The fmt command reformats HCL2 templates, provided it can parse the file
and reformat its contents according to the standards set by the HCL
library's formatters.
However, if the file is malformed for some reason, the command will fail
with a parse error, but while the parse error message is shown, the
actual errors in the template(s) are not forwarded, making it hard for
users to understand what went wrong with the contents of the file
they're trying to format.
In order to be more helpful with those errors, we now forward those
parsing errors to the UI.
Since we now support loading pre-releases, we also want Packer to be
able to ignore them by user demand, so we put in place the
infrastructure to modulate this.
When Packer is loaded, we used to perform plugin discovery.
This was done for every call to Packer, including when it is executed as
a plugin, arguably against what the comments document.
Doing this as early in the loading process makes it harder to change
this behaviour, as we'd need to introduce flags aside from the rest, and
handle them manually, which is not optimal.
Therefore, we change this: now when Packer starts executing, it will not
attempt to discover installed plugins anymore, and instead will only try
to load them when a configuration has been parsed, and is being used to
perform actions (typically build/validate).
When Packer orders the plugins by their version, we'd assumed it was
ordered from highest to lowest.
However, this was the case because our implementation of `Less' was
actually returning `>=', which is not what it was supposed to do.
This commit therefore fixes the implementation of `Less' to do what it
is documented to do, and ensures the behaviour is correct through
additional testing.
Changing this requires some changes to the loading process as well
because of the aforementioned assumption with regards to ordering.
Right now we had two paths for discovering installed plugins, i.e.
through plugin-getter's `ListInstallations' function, or through the
`Discover' call, which relied on a glob to list the installations.
This was required since we allowed plugins to be installed in multiple
locations, and with different constraints.
Now that we force a certain convention, we can consolidate the logic
into ListInstallations, and rely on that logic in `Discover' to load a
plugin into the PluginConfig for the current Packer run.
Since we'll only look in the plugin directory, and not from multiple
sources now, for installing/listing plugins, we can simplify the
structures which used to accept multiple directories so they only accept
one.
Since we're removing the alternative plugin installation directories in
favour of only supporting installing them in the PACKER_PLUGIN_PATH
directory, we only return one directory when getting the known plugin
directories.
Not validating the bucket's name during parse leads to configurations
being marked as valid, even if the bucket name is not, which will fail
during a real build afterwards.
To avoid this problem and fail with an error as quickly as possible, we
add a check during parsing, so that it gets reported for validate as
well.
cty's new "refinements" concept allows us to reduce the range of unknown
values from our functions. This initial changeset focuses only on
declaring which functions are guaranteed to return a non-null result,
which is a helpful baseline refinement because it allows "== null" and
"!= null" tests to produce known results even when the given value is
otherwise unknown.
This commit also includes some updates to test results that are now
refined based on cty's own built-in refinement behaviors, just as a
result of us having updated cty in the previous commit.
If a variable is defined and overriden in the packer command-line, but
there's a problem during the evaluation of this override (type error
typically), we show an error message which details the problem.
This message points to a temporary in-memory HCL2 "file" that we use for
parsing and evaluating the expression for the variable, but since it's
virtual, there's no point in using this as the source for the error, as
it will always yield "line 0" and no contents.
So, in order to limit confusion here, we remove the source for this
error message.
When a user invokes packer for a build or validation, the template being
processed needs components to be present for Packer to process it
without error.
If the component cannot be found from the plugins loaded (or from the
components bundled with Packer), Packer errors, and the command fails.
This is expected, but the error message does not suggest anything to fix
the error, potantially leaving users confused at the problem.
This commit suggests either a replacement (in case of a typo), or points
to the web documentation for Packer, specifically the integrations, so
they can look for the plugin they're missing, and install it, so
subsequent invocations of Packer work.
The previous implementation of the GetVarsByType function worked only on
top-level attributes, ignoring the nested blocks in the structure.
This implies that if a datasource depends on another through an
expression within a nested block, we may not execute it first, and then
executing this datasource before its dependent is possible, resulting in
an error in the end.
This commit is an attempt at making this more reliable for HCL configs,
but only works on configs lifted from HCL files for now. We need to make
this more reliable for later iterations.
Datasources use their attribute's expressions to determine whether or
not they depend on another datasource, in order to get the list of
dependencies and execute them before executing a datasource.
This code may be useful later on for figuring out the dependencies for
any block, so we move this code to the utils.go file, and use this for
datasources.
When a datasource fails to be evaluated because a cycle has been
detected, we point out one of the links of the chain now so that users
have a better idea of what to look at.
Since datasources are recursively evaluated depending on their
dependencies, we don't need to pre-execute those that depend on nothing,
as the recursive traversal of the datasources will take care of that for
us.
Not sure why this was defined and returned, but the value was set, but
never used, as such this is not useful to keep in the code, so let's
simplify this now.
Local variables can't have a validation block in their definition, so
this step in not useful and should be removed.
Besides, since the validation was done on the local variables before
evaluation, it did nothing at all, as the PackerConfig.LocalVariables
collection gets populated during evaluation, so this is essentially a
no-op, and can be safely removed.
Since this feature is no longer something we plan to activate later, as
it contradicts with our efforts to remove bundled plugins, and
encouraging users to move to either manually installing plugins, or
managing them through `packer init', we clean-up the code for this
feature.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Since bundled plugins will be removed in an upcoming version of Packer,
this commit adds a new warning message whenever a template uses one such
plugin.
This warning has been implemented on build, validate, console and the
inspect subcommands.
In addition to warning about the upcoming change and potential issue
this will cause, this warning message proposes solutions to the user so
they know what they'll have to do in order not to rely on those bundled
plugins later.
In HCL2, if a template contains a `required_plugins' block, but the
required plugins are not installed, we print a warning.
Prior to this commit, this was reported as an error once per missing
plugin, which was redundant.
This commit changes that by gathering all the missing plugins once, and
printing them as a bullet list, so the message isn't redundant anymore.
Packer's telemetry package captures the field names from a configuration
on legacy JSON templates, but did not contain any code for handling HCL2
template configurations.
This commit adds a routine to extract the field names from a HCL2
configuration once flattened, and adds some glue code to propagate the
configurations to the telemetry structures.
* hcl2template: use && for chaining conditions
For deciding if we need to run an error provisioner block, we checked
using a nested if that we didn't need to avoid running it.
This is unnecessary, and adds a superfluous level of inbrication to the
code, so we chain both conditions with &&.
* hcl2: don't warn on excluded post-processor
When a post-processor is excluded through the -except command-line
argument for packer build, we used to print a warning on HCL2 templates
if this except statement did not match a build.
However, since we can use the except option to avoid running
post-processors, we should account for this as well when printing out a
warning that there were no matches.
This commit adds this counter to the except checks for the
post-processors as well.
This commit irons out one of the pain points of the HCP rework by
introducing a HCPPublisher interface, implemented both by the JSON Core,
and the HCL2 PackerConfig, which keeps a map of the build names used by
Packer to the build names pushed on HCP.
This in turn lets us go back to the old implementation of the GetBuilds
function, which returns a list of (filtered) builds, and eventually an
error if something went wrong while processing.
While looking into #11932 I found that Packer will throw an error when
an unknown source is referenced from within the sources attribute for
a build block. The hcl.Diagnostics error includes a Subject, which
should highlight where in the HCL2 configuration file the unknown source resides.
But when creating the BuildBlock no HCL2Ref data is copied over, thus the
Subject is displayed with with zero value for an hcl.Range type.
This change updates the build creation logic to copy the HCL2Ref data
from the build block at decode time so that it can be properly
referenced downstream.
Closes#11932
Failure due to change in output
```
--- FAIL: TestValidateCommand_ShowLineNumForMissing (0.00s)
--- FAIL: TestValidateCommand_ShowLineNumForMissing/test-fixtures/validate-invalid/missing_build_block.pkr.hcl (0.00s)
validate_test.go:377: Unexpected output: (
"""
Error: Unknown source file.cho
- on line 0:
+ on test-fixtures/validate-invalid/missing_build_block.pkr.hcl line 6:
(source code not available)
... // 4 identical lines
"""
)
validate_test.go:379:
FAIL
FAIL github.com/hashicorp/packer/command 1.002s
```
* Update validation options for undeclared variables
In an effort to help users move from JSON to HCL2 templates the support for
variable definitions files are being updated to ignore undeclared
variable warnings on build execution. For legacy JSON templates builds no
warnings are displayed when var-files contain undeclared variables.
Since preferred mode HCL2 templates is to be explicit with variable
declarations - they must be declared to be used - validation for
undeclared variables still warns when running `packer validate`. A new
flag has been added to the validate command that can be used to disable
undeclared variable warnings.
* Update validation test for unused variables
Example Run
```
~> go run . validate -no-warn-undeclared-var -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
The configuration is valid.
~> go run . validate -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl
files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
The configuration is valid.
~> go run . build -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 744 microseconds.
==> Wait completed after 798 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
* Rename Strict field to WarnOnUndeclaredVar
The field name Strict is a bit vague since it is only used for
checking against undeclared variables within a var-file definition.
To mitigate against potential overloading of this field it is
being renamed to be more explicit on its usage.
* command/build: Add warn-on-undeclared-var flag
Now that the default behaviour is to not display warnings for undeclared variables
an optional flag has been added to toggle the old behaviour.
```
~> go run . build -warn-on-undeclared-var -var-file command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 762 microseconds.
==> Wait completed after 799 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
This commit reorganises the code for both the registry/API and the
Orchestrator/Registry.
The main difference with the previous version is how stuff is exposed.
Now we only expose a Registry interface to the outside (previously named
Orchestrator), which has several implementations: null is the default,
and is returned if HCP is not enabled.
The other implementations being HCL/JSON, both private to the hcp
sub-package.
The api (previously `registry') is the set of functionality that
abstracts and calls the HCP API.
It was meant to be merged with the `hcp' package, but because of a
dependency loop with the datasources, both are separated for now.
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
To be able to track HCP builds during a Packer build, we need to
propagate the name used for registering the build to HCP.
This works out-of-the-box for JSON templates, as the build's name is
always the builder's, so there's no confusion on that.
For HCP templates however, there's a possibility for a template to
define a name for the build, which is returned by a CoreBuild's Name()
function, in addition to the source's Type/Name.
The builds are registered to HCP with the source.String() function
however, which does not contain the build's name. This prevents any
build containing a name to work, as their name/source.String does not
match.
The name we registered the build with is stored in a CoreBuild as the
Type attribute, but cannot be safely accessed, as builds are types
within the command as packersdk.Build, which only exposes Name() for
this purpose.
In order to circumvent this problem, and as a way to present a prototype
of solution that will likely have to be discussed before we merge it,
this commit hotfixes the issue.
The HCP error messages were sometimes a bit terse in terms of the
information it bubbles up to the user.
This made them useful for people who already knew what to look for, but
newcomers would almost certainly be lost because of the lack of
information in those.
To improve on this situation, we reword most of those error messages in
this commit.
When a template describes a build block without a source reference, the
build should be considered invalid as we won't have a CoreBuild produced
as a result of the need to have both.
In current versions of Packer, this will produce an error message
hinting that nothing will happen because of the lack of either build or
source block.
This commit takes the defined block, and points out to it as missing a
source block as being the reason why nothing is happening, making it
clearer what is required for an HCL2 build to be processed.
When a variable is set in a pkrvars file, but isn't defined, an error
message is output, but does not deliver an example of what is expected
by Packer in order to complete a build.
To remedy that, we improve the error message by giving an example of
variable block to include in the build template.
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <[email protected]>
When a `packer build' is run on a config that does not define anything
to be built, the process would run normally, and only state that no
artifacts were produced after builds were run successfully.
This is misleading as we reach the end of the command without having a
proper warning that nothing was even attempted to be built.
This commit adds a warning when that happens, and leaves immediately
after.
Before this commit, some HCP-related logic was embedded in the parser,
making all of it common to all commands.
This complexifies the logic as some parts rely on the template being in
a HCP-valid environment, and the datasources need to be evaluated in
order for some operations to be executed.
To simplify this and avoid those pitfalls, we move this logic to its own
set of functions that will be performed after the parsing is done, on
any valid HCL2 or JSON template.
When an override block is either not an object, or if one of its
contents are not an object, Packer would crash trying to forcefully
cast the cty.Value to a cty.Collection.
To avoid this behaviour, we add extra checks that return hcl.Diagnostics
when they fail.
When we try to validate a build that contains references to
hcp_packer_image, the data is not fetched, and the value is therefore
unknown.
However, during the decoding phase for the build blocks, we attempt to
fetch the ancestry information for the current build, from the
information previously fetched from HCP.
Since we're validating, there's no way this is set, and attempting to
cast without checking causes Packer to crash on the conversion.
To avoid this, we only attempt this conversion if the value is known.
Closes#11870
* Add support for data source lookup
* Update failing test
* Fix typo for HCP_PACKER_BUILD_FINGERPRINT env in test
* Add test Fingerprint where needed; failing tests were not showing
because they are properly set for acctest
* Update to use BuildContext
Lucas Bajolet
Devashish
Wilken Rivera
guoguangwu
hashicorp-copywrite[bot]
Wilken Rivera
Lucas Bajolet
Eng Zer Jun
Sylvia Moss