As with the go version to 1.18.9, this fix concerns mitigations to the
GO-2022-1144 vulnerability.
Since we depend on golang.org/net too, we need to update it to a version
that is not vulnerable anymore, and this is starting at version 0.4.0
This PR add [`tart` builder](https://github.com/cirruslabs/packer-plugin-tart) to the list of all the builders.
Tart is a open-source toolset for building, running and managing Linux and macOS virtual machines on Apple Silicon. Part of the toolset is this packer plugin.
The go 1.18.9 version fixes a vulnerability GO-2022-1144, which concerns
the net/http and golang.org/x/net packages.
These are used in the codebase, and therefore automated tools report the
generated binaries as vulnerable to this.
Note that while Packer is indeed vulnerable to this, this is a DoS
attack. This is therefore unlikely to impact Packer severely, especially
as it requires a deliberate attempt to provoke an OOM/excessive GC
cycles.
Nonetheless, since this vulnerability is fixed with go 1.18.9, we bump
the version used to build/test the tools to this version.
```
Usage: packer validate [options] TEMPLATE
Checks the template is valid by parsing the template and also
checking the configuration with the various builders, provisioners, etc.
If it is not valid, the errors will be shown and the command will exit
with a non-zero exit status. If it is valid, it will exit with a zero
exit status.
Options:
-syntax-only Only check syntax. Do not verify config of the template.
-except=foo,bar,baz Validate all builds other than these.
-only=foo,bar,baz Validate only these builds.
-machine-readable Produce machine-readable output.
-var 'key=value' Variable for templates, can be used multiple times.
-var-file=path JSON or HCL2 file containing user variables, can be used multiple times.
-no-warn-undeclared-var Disable warnings for user variable files containing undeclared variables.
-evaluate-datasources Evaluate data sources during validation (HCL2 only, may incur costs); Defaults to false.
```
As a follow-up to tracking the template type for an iteration, we
actively check that it remains consistent between two invocations of
packer, and reject the update if it is not the case.
In order to ensure users keep the same templates between two packer
invocations for a single iteration, we start tracking the type of
template used when building images.
Currently the HCL2 upgrade command generates a valid variable block for all variables
within the JSON variables property. However JSON templates in Packer support variable
interpolation for input variables defined within a variable definition file without
it being declared within the variables property. When upgrading a JSON template to HCL2
the user variable reference gets converted to var.<var_name>. The upgraded template ultimately
fails on validation or build execution with undefined variable errors, with this change the upgrade
command will now create a variable block for all input variables referenced within the build template
to ensure all required variables been defined.
The `packer-image-iteration' datasource is an undocumented, and
unexported datasource, so it cannot be used by clients.
Since this is dead weight, we can remove it safely from the codebase.
In JSON templates, we register the builds to process on HCP at the
initialisation of the Registry.
However, during the HCP breakout, this code was duplicated with a slight
difference for the JSON templates, and the build names were registered
once during initialisation, with the expected full name, and once at the
start of PopulateIteration, with the truncated name.
This caused a bug when processing a template that contains a builder
with both a name and a type, as both would be registered, but only one
would complete, leading to the iteration being unable to complete.
When the options for bypassing/enabling assigned and undeclared
variables were added to Packer, the website documentation for those
commands and options was not updated.
This commit adds some documentation for those options.
When packer validate is invoked, it does not try to evaluate the
datasources before attempting to decide if the template is valid.
In many cases, this works, but sometimes it will fail as the value is
unknown by the validation code.
Since the validation code for all the elements of a Packer template is
left to be implemented by plugins, we cannot rely on checking for
unknown values everywhere, especially since the unknown references are
replaced automatically by a value of the right type for the
configuration expected.
So, in order for such configurations to be validable, we add an extra
option to packer validate, that will let users evaluate the datasources
from a template.
* Update validation options for undeclared variables
In an effort to help users move from JSON to HCL2 templates the support for
variable definitions files are being updated to ignore undeclared
variable warnings on build execution. For legacy JSON templates builds no
warnings are displayed when var-files contain undeclared variables.
Since preferred mode HCL2 templates is to be explicit with variable
declarations - they must be declared to be used - validation for
undeclared variables still warns when running `packer validate`. A new
flag has been added to the validate command that can be used to disable
undeclared variable warnings.
* Update validation test for unused variables
Example Run
```
~> go run . validate -no-warn-undeclared-var -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
The configuration is valid.
~> go run . validate -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl
files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
The configuration is valid.
~> go run . build -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 744 microseconds.
==> Wait completed after 798 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
* Rename Strict field to WarnOnUndeclaredVar
The field name Strict is a bit vague since it is only used for
checking against undeclared variables within a var-file definition.
To mitigate against potential overloading of this field it is
being renamed to be more explicit on its usage.
* command/build: Add warn-on-undeclared-var flag
Now that the default behaviour is to not display warnings for undeclared variables
an optional flag has been added to toggle the old behaviour.
```
~> go run . build -warn-on-undeclared-var -var-file command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 762 microseconds.
==> Wait completed after 799 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
The ancestry inferral code relied on HCP datasource outputs for deciding
what would be the ancestry for an image built with HCP.
This brings a dependency into the hcp package, which is not really
necessary as we only need a few information from those entities, hence
this commit removes the full dependency on the structures, in favour of
more focused data structures where we only cherry pick what is necessary
for this code.
This commit reorganises the code for both the registry/API and the
Orchestrator/Registry.
The main difference with the previous version is how stuff is exposed.
Now we only expose a Registry interface to the outside (previously named
Orchestrator), which has several implementations: null is the default,
and is returned if HCP is not enabled.
The other implementations being HCL/JSON, both private to the hcp
sub-package.
The api (previously `registry') is the set of functionality that
abstracts and calls the HCP API.
It was meant to be merged with the `hcp' package, but because of a
dependency loop with the datasources, both are separated for now.
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
To be able to track HCP builds during a Packer build, we need to
propagate the name used for registering the build to HCP.
This works out-of-the-box for JSON templates, as the build's name is
always the builder's, so there's no confusion on that.
For HCP templates however, there's a possibility for a template to
define a name for the build, which is returned by a CoreBuild's Name()
function, in addition to the source's Type/Name.
The builds are registered to HCP with the source.String() function
however, which does not contain the build's name. This prevents any
build containing a name to work, as their name/source.String does not
match.
The name we registered the build with is stored in a CoreBuild as the
Type attribute, but cannot be safely accessed, as builds are types
within the command as packersdk.Build, which only exposes Name() for
this purpose.
In order to circumvent this problem, and as a way to present a prototype
of solution that will likely have to be discussed before we merge it,
this commit hotfixes the issue.
In order to determine whether or not a build is done or needs
processing, we check 3 things:
1. that its status is not done
2. that its ID is not empty
3. that it has no images linked to it
The was the image check was phrased indicated that we checked for the
reverse, so we reverse the naming to make it less confusing.
When Packer tries to get the latest version of a plugin, it will attempt
to get the latest version from a list that is picked from the list of
releases, fetched directly from Github.
What may happen is that a release is triggered, the assets are not yet
available, including the checksum file, which is uploaded last, at the
end of the release process.
This causes any client that tries to get the latest version of a plugin
to fail during that window, as the code will fail immediately whenever a
checksum file is unavailable.
To circumvent this issue, and fallback to a previous version, we
acknowledge that a checksum was unavailable then trying to get the
latest version, and fallback to a previous one.
The mockPluginGetter would always succeed in returning a file from the
list of suggested releases, which does not let us test what happens when
a release is not yet complete, and a checksum file fails to be fetched
from the source.
This commit changes this behaviour so that we get an error when a
checksum does not exist, mirrorring what happens in real conditions.
In azure templates, the shared_image_gallery was mistakenly considered
an attribute while this is supposed to be a block.
This is due to a heuristic we use for deciding whether a JSON object is
to be translated to an attribute or a block that fell short as the
shared_image_gallery does not contain complex types.
This cannot be fixed trivially for the general case, so we add this
entity to the list of workarounds until we can implement something more
robust.
Wilken Rivera
hc-github-team-packer
lonegunmanb
Wilken Rivera
Lucas Bajolet
Fedor Korotkov
dependabot[bot]
Christophe Jauffret
HashiBot
Ashlee M Boyer
Zachary Shilton
Paulo Dias
Lucas Bajolet