With the draft to support both gob and protobuf as serialisation formats
for Packer, along with the SDK changes that propel them, we add a series
of tests that make sure the logic that picks which protocol is solid and
functional.
These tests rely on building several versions of the tester plugin, with
and without protobuf support, to then install them in the tests as
needed to test the logic of Packer using packer build with them, and
templates that require multiple plugins.
When building a plugin, we may want some customisation capabilities
beyond changing the version/pre-release/metadata, and instead run
commands or change files on the filesystem.
To do so, we introduce functions under the BuildCustomisation type,
which have two responsabilities: changing the current state of the
plugin's directory, and cleaning up afterwards.
These customisations are passed as parameters to the BuildSimplePlugin
function, and are called one-by-one, deferring their cleanup after the
build process is finished.
A first implementation of such a customisation is added with this
commit, in order to change the version of a module that the plugin
depends on, which we'll use to change the version of the plugin SDK in
order to test how Packer behaves with different versions of the SDK for
a single plugin.
When testing a packer build or validate, one common use case is to check
which plugins are loaded and used by Packer to run a command.
This is generally done through Grep, but repeating the same pattern can
be redundant, and if the output changes, all those need to be updated.
Therefore, this commit introduces a PluginsUsed gadget, which can be
orchestrated to ensure a plugin is used, or not used, allowing to check
for multiple plugins at once.
As a small reorganisation attempt, this commit moves some file-system
oriented functions from plugin.go into its own file so they are more
logically grouped.
In terms of organisation, we keep functions that interact with plugins
into its own file, therefore the function that build plugin versions
should really be in plugin.go, not in suite.go.
The compiledPlugins map used to be a global variable, which can be
problematic as we move to independent test suites, since those test
suites run in the same process space, the global variable could point to
now deleted plugin versions/paths in separate suites, which would make
tests fail with random errors.
To avoid this, the map is now scoped to the test suite, and a new copy
is created lazily if used by the test suite.
Having only one test suite for the whole of Packer makes it harder to
segregate between test types, and makes for a longer runtime as no tests
run in parallel by default.
This commit splits the packer_test suite into several components in
order to make extension easier.
First we have `lib`: this package embeds the core for running Packer
test suites. This ships facilities to build your own test suite for
Packer core, and exposes convenience methods and structures for building
plugins, packer core, and use it to run a test suite in a temporary
directory.
Then we have two separate test suites: one for plugins, and one for core
itself, the latter of which does not depend on plugins being compiled at
all.
This sets the stage for more specialised test suites in the future, each
of which can run in parallel on different parts of the code.
When a command is run, it is the expectation that no test should make
Packer panic. If it did, something is wrong and Packer should be fixed
so it doesn't panic anymore in that situation.
The way we did the check before was adding a PanicCheck after the
command ran, so we could make sure of that during `Assert`.
However, since we introduced the possibility to have multiple runs,
having this addition as part of the run loop meant that the PanicCheck
would be run as many times as there were runs.
While this worked, this implied that we'd do the same check multiple
times on a single command output, which is not optimal.
Instead, this commit moves the check to within the `Run` function, this
way for each run of the command we do the check once, and then we can
assert the results of the command on what output it produced.
Previously duplicate detection for local variables happened during
`Initialise`, through a call to `checkForDuplicateLocalDefinition`.
This works in a majority of cases, but for commands like `console`, this
was not detected as the return diagnostics for `Initialise` are ignored.
That check can be done as early as during parsing however, as the names
of blocks are not dynamic in the slightest (no interpolation possible),
so we move that detection logic into `Parse`, so that the behaviour is
coherent between all commands.
The logic for evaluating local variables used to rely on their
definition order, with some edge cases. Typically `locals` blocks define
multiple local variables, which don't necessarily appear in the same
order at evaluation as within the template, leading to inconsistent
behaviour, as the order in which those are added to the list of local
variables is non-deterministic.
To avoid this problem, we change how local variables are evaluated, and
we're adopting a workflow similar to datasources, where the local
variables first build a list of direct dependencies. Then when
evaluation happens, we evaluate all the dependencies recursively for
each local variable, which takes care of this issue.
As with Datasources, we add a cap to the recursion: 10. I.e. if the
evaluation of a single variable provokes an infinite recursion, we stop
at that point and return an error to the user, telling them to fix their
template.
The SkipNoAcc function on PackerTestSuite allows to mark a test run as
not to be run every time we run `make test`, but only when PACKER_ACC=1
is set in the environment.
This allows us to skip executing tests that are either long-running, or
that depend on external dependencies (typically Github), which have a
higher potential to fail on a normal run of Packer tests.
When a test fails to exert its assertions on the command-line output, a
test fails, but we don't necessarily can troubleshoot what happened,
especially when this happens in a CI environment.
Therefore, for convenience, we add the faculty for packerCommand.Assert
to automatically dump a command's output (both stdout and stderr) if a
test fails.
Some commands need to have an input in order to work.
For those, we add the capability for the packerCommand struct to have
their stdin defined from a string, which is then fed to the command
being executed.
When a Packer command is created for testing the tool, we generally run
it once, then the command is essentially nooping.
This change allows us to run Packer multiple times with the same
parameters, and make sure all runs conform to a specific list of checks.
This allows us to more reliably test non-deterministic behaviours.
Remotely installing plugins with a pre-release as part of the constraint
is unsupported by Packer, and should error if that happens.
This test makes sure that this gets treated as an error if that's the
case, even before attempting to connect to the source.
'%d' gets output as-is in the temporary workdir we create. This is
unnecessary and could even be problematic in some cases, so we scrub it
from the MkdirTemp call.
Since legacy config files may declare single plugin components, we need
to warn that they're not supported anymore.
This is in process of being PR'd into main, but to ensure the config
works as intended and we do get the error, we add some tests for that.
These changes include a series of test cases for validating packer init
using the force and upgrade flag. Include in this test is a test case
for validating the plugin installation error when init encounters a
plugin whose reported version does not match the version within the
plugin name.
```
--- PASS: Test_PackerCoreSuite (18.66s)
--- PASS: Test_PackerCoreSuite/TestPackerInitForce (4.91s)
--- PASS: Test_PackerCoreSuite/TestPackerInitForce/installs_any_missing_plugins (2.76s)
--- PASS: Test_PackerCoreSuite/TestPackerInitForce/reinstalls_plugins_matching_version_constraints (2.14s)
--- PASS: Test_PackerCoreSuite/TestPackerInitUpgrade (3.70s)
--- PASS: Test_PackerCoreSuite/TestPackerInitUpgrade/upgrades_a_plugin_to_the_latest_matching_version_constraints (2.02s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithMixedVersions (1.96s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithMixedVersions/skips_the_plugin_installation_with_mixed_versions_before_exiting_with_an_error (1.96s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithNonGithubSource (1.22s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithNonGithubSource/try_installing_from_a_non-github_source,_should_fail (0.07s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithNonGithubSource/manually_install_plugin_to_the_expected_source (0.59s)
--- PASS: Test_PackerCoreSuite/TestPackerInitWithNonGithubSource/re-run_packer_init_on_same_template,_should_succeed_silently (0.55s)
```
To make sure we do scrub the metadata in the plugin name when installing
it from a local binary, we add a test that does that installation with
both alternatives: 1.0.0-dev and 1.0.0-dev+metadata, which should result
in only one alternative being installed (the last one that succeeded).
The installation with a metadata part in the version for a plugin had
one test that relied on the plugin directories being populated with
packer plugins install --path.
This could change in the future, while the command should remain
functional, so we explicitely call it in the test instead of through the
function that creates/populates a temp plugin dir.
When building a pipeline to count the number of lines returned by
Packer, it can be a bit cumbersome to have to chain the calls to
MkPipeCheck to do that check, so we add one convenience function for the
simplest case: counting the number of lines on stdout, without any kind
of filtering.
When manually installing a plugin to the plugin directory, we compute a
SHA256SUM file from the plugin binary, and install it alongside it so we
can test the loading process for Packer.
In the introduction of the function, we added a check that if we were
running on Windows, we'd remove the extension of the sumfile's name
before writing it.
This is actually not necessary (and breaks the loading logic) as Packer
looks for the name of the plugin with extension, followed by
_SHA256SUM in order to compare the effective digest of the file to the
one written to this file.
Since this prevents the tests that use this function from succeeding in
a Windows environment, we remove this extra step.
Windows relies on the `TMP` (or alternatives) being set in the
environment in order to be able to create temporary directories and
files.
If this is not set, the `os.TempDir` function defaults on the windows
installation root directory (typically C:\Windows), leading to
permission errors when running Packer in the context of a test, as we're
installing plugins in a temporary directory.
To avoid this problem, we get the current setting from the test's
invocation environment, and forward it to the subcommand we execute for
our tests.
Since on Windows extensions are mandatory in order to have something
executable, we compile Packer with a `.exe` suffix during tests, so we
can use the executable afterwards to run tests with.
As we're introducing a --ignore-prerelease-plugins flag to both the
validate and build subcommands, we need to make sure they work as we
expect it to, so we add a test case for that.
Plugins with metadata information in their file name (i.e.
v1.0.0+metadata) should be ignored by Packer as they could introduce
ambiguity since the metadata is free-form, so we add that test to make
sure Packer behaves coherently.
If a plugin is installed with a non-canonical version in its name (e.g.
01.01.01), Packer rejects it with a message to that effect in stderr, so
we add a test for this use-case.
Installing a plugin manually to a directory is something needed for some
tests, especially those not relying on packer commands to install
plugins as they reject/correct the path/version.
Therefore this function is introduced so we have an easy way to install
a binary as a plugin somewhere on the provided plugin directory.
The ExpectedInstalledName function used to compute the expected name of
a plugin binary for a given version, based on the invoker's environment,
used to cleanup the version string passed in parameter of the function,
which could be problematic.
Besides the logic applied would produce some invalid binary names as the
prerelease plugin would not have a `-` separator, so the resulting
plugin would be ignored, and we couldn't test metadata rejection with
this logic.
This commit therefore changes how the function works: the version string
is still parsed to account for manipulation errors, but the string is
left as-is for the final binary name.
If for some reason we only want to run a test on either stream without
doing some manipulation beforehand, we can run a PipeChecker, however
these would error if no pipe gadget was defined, preventing this
use-case.
Instead of errorring then, this commit just ignores if no pipe is
present, as none is required for the test to run.
Since the command object is created using the TestSuite as argument, we
can keep a reference to the test suite's scoped T() instance for the
command's lifecycle, and reuse it later during `Assert`, instead of
needing to pass it as argument when invoking the function.
When troubleshooting a pipeline for a test, it can be useful to print
the input out without necessarily preventing the pipeline to work.
The Tee gadget is exactly made for this purpose, the input of the Tee is
printed out through `t.Log`, and the input is forwarded to the next step
in the pipeline.
When the PACKER_PLUGIN_PATH envvar is defined in the environment, Packer
uses it as the source of truth for the directories in which to look for
plugins to load.
Previously, we used to support multiple directories separated by the
OS-specific path separator (i.e. : on UNIX, ; on Windows).
Since this changed, and Packer returns an error to the user, we make
sure that this is well-documented and tested through this extra test.
The IntCompare checker converts the input string into an int through
strconv.Atoi, which fails when the string isn't valid base 10. This
definition of "valid" also excludes whitespace, which occurs often
through piping gadgets, but shouldn't be a reason to fail that check, so
we trim the whitespace from the input string.
Lucas Bajolet
dependabot[bot]
Wilken Rivera