As a follow-up to tracking the template type for an iteration, we
actively check that it remains consistent between two invocations of
packer, and reject the update if it is not the case.
In order to ensure users keep the same templates between two packer
invocations for a single iteration, we start tracking the type of
template used when building images.
Currently the HCL2 upgrade command generates a valid variable block for all variables
within the JSON variables property. However JSON templates in Packer support variable
interpolation for input variables defined within a variable definition file without
it being declared within the variables property. When upgrading a JSON template to HCL2
the user variable reference gets converted to var.<var_name>. The upgraded template ultimately
fails on validation or build execution with undefined variable errors, with this change the upgrade
command will now create a variable block for all input variables referenced within the build template
to ensure all required variables been defined.
The `packer-image-iteration' datasource is an undocumented, and
unexported datasource, so it cannot be used by clients.
Since this is dead weight, we can remove it safely from the codebase.
In JSON templates, we register the builds to process on HCP at the
initialisation of the Registry.
However, during the HCP breakout, this code was duplicated with a slight
difference for the JSON templates, and the build names were registered
once during initialisation, with the expected full name, and once at the
start of PopulateIteration, with the truncated name.
This caused a bug when processing a template that contains a builder
with both a name and a type, as both would be registered, but only one
would complete, leading to the iteration being unable to complete.
When the options for bypassing/enabling assigned and undeclared
variables were added to Packer, the website documentation for those
commands and options was not updated.
This commit adds some documentation for those options.
When packer validate is invoked, it does not try to evaluate the
datasources before attempting to decide if the template is valid.
In many cases, this works, but sometimes it will fail as the value is
unknown by the validation code.
Since the validation code for all the elements of a Packer template is
left to be implemented by plugins, we cannot rely on checking for
unknown values everywhere, especially since the unknown references are
replaced automatically by a value of the right type for the
configuration expected.
So, in order for such configurations to be validable, we add an extra
option to packer validate, that will let users evaluate the datasources
from a template.
* Update validation options for undeclared variables
In an effort to help users move from JSON to HCL2 templates the support for
variable definitions files are being updated to ignore undeclared
variable warnings on build execution. For legacy JSON templates builds no
warnings are displayed when var-files contain undeclared variables.
Since preferred mode HCL2 templates is to be explicit with variable
declarations - they must be declared to be used - validation for
undeclared variables still warns when running `packer validate`. A new
flag has been added to the validate command that can be used to disable
undeclared variable warnings.
* Update validation test for unused variables
Example Run
```
~> go run . validate -no-warn-undeclared-var -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
The configuration is valid.
~> go run . validate -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl
files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
The configuration is valid.
~> go run . build -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 744 microseconds.
==> Wait completed after 798 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
* Rename Strict field to WarnOnUndeclaredVar
The field name Strict is a bit vague since it is only used for
checking against undeclared variables within a var-file definition.
To mitigate against potential overloading of this field it is
being renamed to be more explicit on its usage.
* command/build: Add warn-on-undeclared-var flag
Now that the default behaviour is to not display warnings for undeclared variables
an optional flag has been added to toggle the old behaviour.
```
~> go run . build -warn-on-undeclared-var -var-file command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 762 microseconds.
==> Wait completed after 799 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
The ancestry inferral code relied on HCP datasource outputs for deciding
what would be the ancestry for an image built with HCP.
This brings a dependency into the hcp package, which is not really
necessary as we only need a few information from those entities, hence
this commit removes the full dependency on the structures, in favour of
more focused data structures where we only cherry pick what is necessary
for this code.
This commit reorganises the code for both the registry/API and the
Orchestrator/Registry.
The main difference with the previous version is how stuff is exposed.
Now we only expose a Registry interface to the outside (previously named
Orchestrator), which has several implementations: null is the default,
and is returned if HCP is not enabled.
The other implementations being HCL/JSON, both private to the hcp
sub-package.
The api (previously `registry') is the set of functionality that
abstracts and calls the HCP API.
It was meant to be merged with the `hcp' package, but because of a
dependency loop with the datasources, both are separated for now.
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
To be able to track HCP builds during a Packer build, we need to
propagate the name used for registering the build to HCP.
This works out-of-the-box for JSON templates, as the build's name is
always the builder's, so there's no confusion on that.
For HCP templates however, there's a possibility for a template to
define a name for the build, which is returned by a CoreBuild's Name()
function, in addition to the source's Type/Name.
The builds are registered to HCP with the source.String() function
however, which does not contain the build's name. This prevents any
build containing a name to work, as their name/source.String does not
match.
The name we registered the build with is stored in a CoreBuild as the
Type attribute, but cannot be safely accessed, as builds are types
within the command as packersdk.Build, which only exposes Name() for
this purpose.
In order to circumvent this problem, and as a way to present a prototype
of solution that will likely have to be discussed before we merge it,
this commit hotfixes the issue.
In order to determine whether or not a build is done or needs
processing, we check 3 things:
1. that its status is not done
2. that its ID is not empty
3. that it has no images linked to it
The was the image check was phrased indicated that we checked for the
reverse, so we reverse the naming to make it less confusing.
When Packer tries to get the latest version of a plugin, it will attempt
to get the latest version from a list that is picked from the list of
releases, fetched directly from Github.
What may happen is that a release is triggered, the assets are not yet
available, including the checksum file, which is uploaded last, at the
end of the release process.
This causes any client that tries to get the latest version of a plugin
to fail during that window, as the code will fail immediately whenever a
checksum file is unavailable.
To circumvent this issue, and fallback to a previous version, we
acknowledge that a checksum was unavailable then trying to get the
latest version, and fallback to a previous one.
The mockPluginGetter would always succeed in returning a file from the
list of suggested releases, which does not let us test what happens when
a release is not yet complete, and a checksum file fails to be fetched
from the source.
This commit changes this behaviour so that we get an error when a
checksum does not exist, mirrorring what happens in real conditions.
In azure templates, the shared_image_gallery was mistakenly considered
an attribute while this is supposed to be a block.
This is due to a heuristic we use for deciding whether a JSON object is
to be translated to an attribute or a block that fell short as the
shared_image_gallery does not contain complex types.
This cannot be fixed trivially for the general case, so we add this
entity to the list of workarounds until we can implement something more
robust.
Some literal strings without format-string directives nor arguments were
littered around the hcl2_upgrade command's code, which is pointed out by
the linters.
Since they serve no purpose, we remove them.
In previous versions of hcl2_upgrade, we referenced a list of
passthrough statements from the original JSON template to be replaced by
their equivalent Go-template in HCL2.
This works but requires us to explicitely accept every possible
templated variable in order to not ever encounter <no value> in our
generated HCL2 templates.
This is suboptimal, hence this commit changes approach by pre-visting
the AST from the original Go Template, escaping every sequence that is
not covered by the list of functions we can migrate. Pipes are also
excluded from this function.
* Add documentation for configuring Packer for HCP Packer
This change works to document the two configuration options for setting
up Packer to publish build artifacts to an active HCP Packer registry.
Currently the HashiCorp Cloud Platform documents the process for
configuring Packer via a HCL template, which is the preferred route.
This documentation focuses more on the use of environment variables for
configuring Packer for HCP Packer, which is necessary for legacy JSON
template users looking to publish to HCP Packer.
In light of the new HCP Packer documentation page we've opted for a shorter
version of the HCP Packer introduction section.
* Applying the first round of suggestions to tighten up the documentation.
* Move HCP Packer side bar to top
* Move intro into docs
* Add redirects for intro pages
Co-authored-by: Laura Pacilio <[email protected]>
* website: fix broken links on /docs/templates
* fix: redirected install-plugins link
* fix: debugging link
* fix: secrets manager link in docs
* fix: secrets manager link in source
* fix: amazon ami plugin link in docs
* fix: amazon ami plugin link in source
* fix: extending plugins link
* fix: plugins/builders/amazon links
* fix: various builders links
* fix: various amazon builder links
* fix: redirected terminology link
* fix: custom-provisioners link
* fix: docker-push redirected plugin link
* fix: googlecompute plugin links
* fix: hyperv iso plugin links
* website: update link to hcl upgrade guide
Co-authored-by: Wilken Rivera <[email protected]>
Co-authored-by: Wilken Rivera <[email protected]>
The HCP error messages were sometimes a bit terse in terms of the
information it bubbles up to the user.
This made them useful for people who already knew what to look for, but
newcomers would almost certainly be lost because of the lack of
information in those.
To improve on this situation, we reword most of those error messages in
this commit.
dependabot[bot]
Wilken Rivera
Lucas Bajolet
Christophe Jauffret
HashiBot
Ashlee M Boyer
Zachary Shilton
Paulo Dias
Lucas Bajolet
Jenna Goldstrich
