With the draft to support both gob and protobuf as serialisation formats
for Packer, along with the SDK changes that propel them, we add a series
of tests that make sure the logic that picks which protocol is solid and
functional.
These tests rely on building several versions of the tester plugin, with
and without protobuf support, to then install them in the tests as
needed to test the logic of Packer using packer build with them, and
templates that require multiple plugins.
When building a plugin, we may want some customisation capabilities
beyond changing the version/pre-release/metadata, and instead run
commands or change files on the filesystem.
To do so, we introduce functions under the BuildCustomisation type,
which have two responsabilities: changing the current state of the
plugin's directory, and cleaning up afterwards.
These customisations are passed as parameters to the BuildSimplePlugin
function, and are called one-by-one, deferring their cleanup after the
build process is finished.
A first implementation of such a customisation is added with this
commit, in order to change the version of a module that the plugin
depends on, which we'll use to change the version of the plugin SDK in
order to test how Packer behaves with different versions of the SDK for
a single plugin.
As we're trying to move away from gob for serialising data over the
wire, this commit adds the capability for Packer to pick dynamically
between gob or protobuf for the serialisation format to communicate with
plugins.
As it stands, if all the plugins discovered are compatible with
protobuf, and we have not forced gob usage, protobuf will be the
serialisation format picked.
If any plugin is not compatible with protobuf, gob will be used for
communicating with all the plugins that will be used over the course of
a command.
When testing a packer build or validate, one common use case is to check
which plugins are loaded and used by Packer to run a command.
This is generally done through Grep, but repeating the same pattern can
be redundant, and if the output changes, all those need to be updated.
Therefore, this commit introduces a PluginsUsed gadget, which can be
orchestrated to ensure a plugin is used, or not used, allowing to check
for multiple plugins at once.
As a small reorganisation attempt, this commit moves some file-system
oriented functions from plugin.go into its own file so they are more
logically grouped.
In terms of organisation, we keep functions that interact with plugins
into its own file, therefore the function that build plugin versions
should really be in plugin.go, not in suite.go.
The compiledPlugins map used to be a global variable, which can be
problematic as we move to independent test suites, since those test
suites run in the same process space, the global variable could point to
now deleted plugin versions/paths in separate suites, which would make
tests fail with random errors.
To avoid this, the map is now scoped to the test suite, and a new copy
is created lazily if used by the test suite.
Having only one test suite for the whole of Packer makes it harder to
segregate between test types, and makes for a longer runtime as no tests
run in parallel by default.
This commit splits the packer_test suite into several components in
order to make extension easier.
First we have `lib`: this package embeds the core for running Packer
test suites. This ships facilities to build your own test suite for
Packer core, and exposes convenience methods and structures for building
plugins, packer core, and use it to run a test suite in a temporary
directory.
Then we have two separate test suites: one for plugins, and one for core
itself, the latter of which does not depend on plugins being compiled at
all.
This sets the stage for more specialised test suites in the future, each
of which can run in parallel on different parts of the code.
In a couple places in the codebase we didn't check the errors from
functions we execute.
In some cases this is harmless (or at least ignorable), but others may
need to log what went wrong, so for all the reported occurrences we
either ignore explicitly or handle the error with a log.
The logOutput function had a bunch of nested checks in the code for the
function, as well as one ineffasign pointed out by golangci-lint.
This commit changes how special cases are processed, and changes a bit
how log redirection to a file is handled.
Prior to this commit, only when PACKER_LOG and PACKER_LOG_FILE are set
does Packer redirect logs to the specified path, which is a bit
superfluous as setting PACKER_LOG_FILE should be enough on its own to
redirect logs and enable verbose logs.
Now, when neither is set, no logOutput is set, if PACKER_LOG_FILE is set
(regardless of PACKER_LOG), we redirect to the specified file, and
finally if PACKER_LOG is set, we redirect to stderr.
Some of the variables we create are flagged by our linters as
ineffective assignments, which makes sense as those are generally fed by
code below, so we don't need to use the declaration/assignation syntax
(:=) but instead can fall back to using var with a type to get the zero
value of the declared entity.
Test files have some calls that don't necessarily get checked since they
have a very low probability to fail, and in a testing environment it's
acceptable to not have bulletproof code.
Therefore, we add a rule to not run that check on any file that ends
with `_test.go` (Go test convention), so the linter isn't as noisy as it
used to be.
Since the enumer implementation we used hadn't been updated for 5+
years, this didn't work with recent linux/go versions, and enumer
crashed while attempting to parse/analyse the source files.
There's another alternative on Github, forked from the one we used,
which seems more maintained now, and does produce the expected files in
Packer.
Before change
```
~> govulncheck ./...
=== Symbol Results ===
Vulnerability #1: GO-2024-2947
Leak of sensitive information to log files in
github.com/hashicorp/go-retryablehttp
More info: https://pkg.go.dev/vuln/GO-2024-2947
Module: github.com/hashicorp/go-retryablehttp
Found in: github.com/hashicorp/[email protected]
Fixed in: github.com/hashicorp/[email protected]
Example traces found:
#1: hcl2template/function/vault.go:30:30: function.init calls template.Vault, which eventually calls retryablehttp.Client.Do
Your code is affected by 1 vulnerability from 1 module.
```
After Change
```
~> govulncheck ./...
No vulnerabilities found.
```
* doc fix for unattended debian/ubuntu installer linkrot
the links for chef-maintained preseeds/autoinstallers for debian/ubuntu both rotted. i'm just getting started with packer, so i hope this edit is accurate.
* Remove webarchive links for legacy JSON
When possible point users to HCL2 templates for getting started examples.
* As per the style guide aim for inclusive language
---------
Co-authored-by: Wilken Rivera <[email protected]>
* Update UpsertBucket to first call GetBucket, this will allow bucket level role based authentication, as CreateBucket uses project level auth
* Fix one incorrect test failure message
* Document use of TMPDIR required for remote plugin installation
* Update website/content/docs/configure.mdx
Co-authored-by: Lucas Bajolet <[email protected]>
---------
Co-authored-by: Lucas Bajolet <[email protected]>
When a command is run, it is the expectation that no test should make
Packer panic. If it did, something is wrong and Packer should be fixed
so it doesn't panic anymore in that situation.
The way we did the check before was adding a PanicCheck after the
command ran, so we could make sure of that during `Assert`.
However, since we introduced the possibility to have multiple runs,
having this addition as part of the run loop meant that the PanicCheck
would be run as many times as there were runs.
While this worked, this implied that we'd do the same check multiple
times on a single command output, which is not optimal.
Instead, this commit moves the check to within the `Run` function, this
way for each run of the command we do the check once, and then we can
assert the results of the command on what output it produced.
Previously duplicate detection for local variables happened during
`Initialise`, through a call to `checkForDuplicateLocalDefinition`.
This works in a majority of cases, but for commands like `console`, this
was not detected as the return diagnostics for `Initialise` are ignored.
That check can be done as early as during parsing however, as the names
of blocks are not dynamic in the slightest (no interpolation possible),
so we move that detection logic into `Parse`, so that the behaviour is
coherent between all commands.
The logic for evaluating local variables used to rely on their
definition order, with some edge cases. Typically `locals` blocks define
multiple local variables, which don't necessarily appear in the same
order at evaluation as within the template, leading to inconsistent
behaviour, as the order in which those are added to the list of local
variables is non-deterministic.
To avoid this problem, we change how local variables are evaluated, and
we're adopting a workflow similar to datasources, where the local
variables first build a list of direct dependencies. Then when
evaluation happens, we evaluate all the dependencies recursively for
each local variable, which takes care of this issue.
As with Datasources, we add a cap to the recursion: 10. I.e. if the
evaluation of a single variable provokes an infinite recursion, we stop
at that point and return an error to the user, telling them to fix their
template.
GetVarsByType is a function that gets a list of Traversals from a hcl
Block.
This approach works when what we are visiting is indeed one, however
when we can get an immediate list of Traversals, but want to filter them
based on their roots, we have to reimplement parts of that function.
Therefore, we split this function in two, GetVarsByType still keeps its
current behaviour, but the filtering step is exposed as another function
now: FilterTraversalsByType, so we can reuse it elsewhere.
The SkipNoAcc function on PackerTestSuite allows to mark a test run as
not to be run every time we run `make test`, but only when PACKER_ACC=1
is set in the environment.
This allows us to skip executing tests that are either long-running, or
that depend on external dependencies (typically Github), which have a
higher potential to fail on a normal run of Packer tests.
When a test fails to exert its assertions on the command-line output, a
test fails, but we don't necessarily can troubleshoot what happened,
especially when this happens in a CI environment.
Therefore, for convenience, we add the faculty for packerCommand.Assert
to automatically dump a command's output (both stdout and stderr) if a
test fails.
Some commands need to have an input in order to work.
For those, we add the capability for the packerCommand struct to have
their stdin defined from a string, which is then fed to the command
being executed.
When a Packer command is created for testing the tool, we generally run
it once, then the command is essentially nooping.
This change allows us to run Packer multiple times with the same
parameters, and make sure all runs conform to a specific list of checks.
This allows us to more reliably test non-deterministic behaviours.
Updates the provisioners documentation for a better reader experience by including a list of the built-in provisioners and a link to the community supported provisioners that are all withing the navigation tree.
Ref: #12898
Signed-off-by: Ryan Johnson <[email protected]>
Adds an example of a string template being used.
The example demonstrates how a template sequence can be used to embed the value of a variable into a string that can be used as script content.
Ref: #12651
Signed-off-by: Ryan Johnson <[email protected]>
Adds documentation for `ssh_keypair_name`, `ssh_agent_auth`, `temporary_key_pair_name`, and `ssh_private_key_file`.
The note is updated noting that not all builders support these options.
Ref: #10722
Signed-off-by: Ryan Johnson <[email protected]>
Lucas Bajolet
Lucas Bajolet
Wilken Rivera
dependabot[bot]
hashicorp-tsccr[bot]andhashicorp-tsccr[bot]
Jeanne Angeles Franco
Devashish
Jamie Finnigan
Wilken Rivera
graevyandWilken Rivera
Jenna Goldstrich
Ryan Johnson
Ryan Johnson