Since the command object is created using the TestSuite as argument, we
can keep a reference to the test suite's scoped T() instance for the
command's lifecycle, and reuse it later during `Assert`, instead of
needing to pass it as argument when invoking the function.
When troubleshooting a pipeline for a test, it can be useful to print
the input out without necessarily preventing the pipeline to work.
The Tee gadget is exactly made for this purpose, the input of the Tee is
printed out through `t.Log`, and the input is forwarded to the next step
in the pipeline.
When the PACKER_PLUGIN_PATH envvar is defined in the environment, Packer
uses it as the source of truth for the directories in which to look for
plugins to load.
Previously, we used to support multiple directories separated by the
OS-specific path separator (i.e. : on UNIX, ; on Windows).
Since this changed, and Packer returns an error to the user, we make
sure that this is well-documented and tested through this extra test.
The IntCompare checker converts the input string into an int through
strconv.Atoi, which fails when the string isn't valid base 10. This
definition of "valid" also excludes whitespace, which occurs often
through piping gadgets, but shouldn't be a reason to fail that check, so
we trim the whitespace from the input string.
When running the core acceptance tests we compile a temporary Packer
binary to run the tests with.
However in some cases we may want to run a test with another,
pre-compiled version of Packer.
For this use-case, we introduce an environment variable to change the
executable we're running tests with.
When the Run command was introduced, despite the path to the temporary
packer binary being stored in the command, we were invoking Packer
through "packer", which would resolve whatever is in our PATH that
resolves first.
This was not the intention, so we fix this problem with this commit.
When a test has to run in a working directory that is not the current
one we're running tests from, this may imply creating a temporary
directory, moving files into it, and then running the test from this
directory.
This can be a bit verbose to write all this code, so we abstract the
easy case through the TempWorkdir function, which works similarly to the
MakePluginDir function, by creating a temp dir, populating it with the
requested files, and returning a function to clean it up if needed.
For some tests we may need Packer to run in another directory than the
one we're invoking the tests from, so we add a new function to the
packerCommand structure to change that.
The name and semantics were a bit unclear with how they were previously
named, so this commit changes the name of those functions so it's
clearer they are expecting something, and what they're expecting.
As we've introduced pipelines, we can use those to compose a version of
grep that doesn't have specific logic.
Besides, this refactor allows us to expose grep as a function with
variadic options, so this makes it more concise and clear to assert an
input with Grep.
By default PackerCommands are run with PACKER_LOG=1.
If for any reason we don't want that, we can remove it from the
environment so we only see the user-facing logs.
As a common use case in console-oriented pipelines, we check that a
specific command returned a certain number of lines.
With the combination of LineCount and Compare, we can do exactly this.
When running a pipeline on a command's output, a simple check is making
sure the pipeline returned something empty or not.
This is the goal of those two implementations, basically either the
input is empty as expected, or it errors, and the reverse.
When writing tests, one may need to write a one-off checker for a
packer command that ran, without having to completely implement the
Checker interface.
This commit introduces a generic CustomChecker implementation (i.e. a
function) that can be one-off implemented by developers if their test
doesn't fit the existing gadgets, and the need is not generic/reusable
enough to justify introducing a new gadget for other users.
The ExpectedInstalledName function returns the expected full name of a
plugin binary after it's installed.
This is used for tests that need to copy the binary to some place before
they can run commands and ensure the logic for managing plugins conforms
to the docs/specs.
Since sometimes we want to check for matches, and sometimes we want to
check for a lack of match, we add one more option for the Grep gadget:
inverse.
This essentially replicates `grep -v`, and will succeed only if the
regex provided did NOT match on the requested streams.
This commit adds a few scenarios of plugin installations to the test
suite, in order to document and ensure we behave appropriately when
installing pre-releases/metadata.
When building the temporary plugin directory for a test, we didn't check
that the LoadPluginVersion call succeeded and returned a path, which
may cause errors down the line when attempting to install the plugin.
To avoid this problem, we do the check at that time, and immediately
fail if a plugin isn't found.
MakePluginDir used to only load plugins that were precompiled at the
start of the tests, but now when invoked with any list of plugins, this
will attempt to compile plugins one-by-one, so we don't need to modify
the tests in several places when running tests.
There's still value in compiling the plugins in advance though: as they
run in parallel, they all get compiled at once, so we shave off a few
seconds from the test run.
Calling BuildSimplePlugin for a particular version used to mean that we
had to build it regardless of if it was previously done or not.
This commit changes this behaviour so that it checks first that the
plugin wasn't pre-compiled, and if it was, we immediately return.
When running "Assert" on a packer command, we run a series of checkers
on the command's output/error code, which provoke test failures if they
fail.
Panic checking used to be part of Run, but in the end this would make
more sense to have that as a regular checker if asserting the results of
a packer command, so that's the approach we adopt with this commit.
When running a PackerCommand for acceptance tests, we generally run the
test on a temporary plugin directory, populated by test plugins.
Setting that temporary directory means we need to set the environment
variable, which while it could be easier with a constant for the name
for example, isn't too straightforward.
Therefore for those tests we add a new function for PackerCommand so
that it automatically sets that envvar for the current command.
The NewPluginBuildConfig function was essentially a shortcut to
`version.Must(version.NewSemver(v))', which is superfluous at this
point, we can directly pass the version string to BuildSimplePlugin and
let that function do the creation/check.
Since the MakePluginDir function takes the TestSuite as receiver, we
don't need to additionally pass in a reference to testing.T, since the
test suite already contains one instance, and offers a function to get
it from.
The PluginVersionConfig structure was first introduced when building
the early versions of the test package, but it was an unnecessary
abstraction over go-version.Version.
So we remove that structure definition, and instead we directly use the
version for building those temporary plugins.
The CopyFile function is essentially a go recreation of the `cp'
command, which copies one file from a source path to a destination
directory or file.
This can be used for several tests in the future.
When creating a temporary plugin directory, we had to build a cleanup
function, which could be as simple as `os.Mkdirall` without any kind of
warning that a directory failed to be cleaned-up, or we could do some
more work in order to report the possible issues around this.
That code would quickly be redundant, as there's not a ton of
variability in the code that can be written for this step, so we
abstract it through a pre-defined cancellation function which can be
safely defer invoked.
Creating the Name() function for every gadget we have is superfluous a
bit, as we're essentially parroting the name of the test itself as
implementation for the function.
So instead of requiring every checker implements `Name', we now default
to returning the type name, but if the Name function exists for the
checker, we invoke it and return the value for that function.
This allows us to only define the function where needed, and not
systematically.
Since compiling plugins is quick, but each invocation still takes a bit
of time, we run those compilation jobs in parallel to shave of a couple
seconds from a test run.
Acceptance testing, i.e. running Packer core commands in a controlled
environment and ensuring the behaviour is consistent to what we
expect/document, is not something we have a robust and usable framework
for at the moment.
This commit is a proposal for a base testing framework of the sort, that
is meant to be shipped with packer core, and which will eventually host
most of the tests we currently do in command where we mock an
environment.
The mini plugin implemented here is a minimal Packer plugin that we use
for acceptance testing Packer core.
There's a few components exposed so we can write templates using it, and
make sure Packer interprets it all as it should, and runs/errors as we
expect it.
* docs: add plugin loading spec documentation
The logic for discovering and loading plugins is not well documented on
the current documentation.
This causes issues for users that have to troubleshoot why a particular
plugin cannot be found or installed, so this commit adds a specification
document, detailing what are Packer's expectations when it comes to
discovering plugins.
* Apply suggestions from code review
* Update plugin loading specification navbar
---------
Co-authored-by: Wilken Rivera <[email protected]>
The acceptance tests were failing to be run on CI as the linter reported
some typecheck mismatches.
This could be due to an older version of golangci-lint running on them,
so we bump it to v1.54.0, empirically determined as when the lint
started being adequately reported.
Both linters are not supported anymore and trigger warnings every time
we try to lint our code.
To not see those warnings anymore, we remove them from the
configuration.
Lucas Bajolet
Wilken Rivera
Lucas BajoletandWilken Rivera