Single-component plugins are a relic from the past that has been
deprecated from version 1.7.0 and onwards.
Since we're revisiting how plugins are installed/loaded, and the changes
will be incompatible with those, we remove them in preparation of this
work.
The plugins installed command list installed plugins, and prints out all
their paths and versions.
The results are listed in a list of unique versions, in ascending
order.
When listing plugins outside of a specific identifier however, because
of the insertion in the list works, we only consider the version number,
and ignore all subsequent insertions with the same version.
This causes a problem when we have multiple plugins installed with the
same version, as only the first one that is discovered (typically the
one with lexicographical precedence) gets inserted, and the others are
ignored.
To support such a use case, we change the insertion routine to not only
support versions, but also paths when finding a spot to insert the
installation into.
The --force option for packer init and packer plugins install enforces
installation of a plugin, even if it is already locally installed.
This will become useful if for some reason a pre-existing plugin
binary/version is already installed, and we want to overwrite it.
When a user invokes packer for a build or validation, the template being
processed needs components to be present for Packer to process it
without error.
If the component cannot be found from the plugins loaded (or from the
components bundled with Packer), Packer errors, and the command fails.
This is expected, but the error message does not suggest anything to fix
the error, potantially leaving users confused at the problem.
This commit suggests either a replacement (in case of a typo), or points
to the web documentation for Packer, specifically the integrations, so
they can look for the plugin they're missing, and install it, so
subsequent invocations of Packer work.
When invoking `packer plugins install' to install a plugin, or `packer
init', the checksum file would be installed with 0555 permissions.
This led in turn to further attempts at installing the plugin will
succeed, but the checksum file would not be updated, as it was marked
non-writable by the owner of the file, leading potentially to a
situation where the plugin binary and the checksum would be out-of-sync,
but could not be updated unless the user changed it.
To avoid such a problem, we write the checksum file with 0644
permissions, so the owner can read/write, while the other users can only
read it.
Invoking Packer with the CHECKPOINT_DISABLE env. variable the telemetry reporter is left uninitialized in order to disable telemetry reporting.
Any method calls on the nil reporter is expected to check if the reporter is active or in NOOP mode. This change fixes a crash when calling SetBundledUsage()
on a nil CheckpointTelemetry type that occurs when using a bundled plugin with CHECKPOINT_DISABLE=1.
Since this feature is no longer something we plan to activate later, as
it contradicts with our efforts to remove bundled plugins, and
encouraging users to move to either manually installing plugins, or
managing them through `packer init', we clean-up the code for this
feature.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
* Add test case for loading plugin in CWD
* Add test case to validate checksume files are ignored
* Update Discover to include CWD "." in PluginFolders if KnowPluginFolders is unset
When copying a plugin's checksum file (packer-plugin-*_SHA256SUM) installed by `packer plugins install` or `packer init`
into a separate directory the file may be copied with the executable bit turned out. If unchanged after the copy, Packer would
discover the checksum file as a possible plugin match and error when trying to execute describe on the plugin look a like. This change
adds a checksum file test to the plugin matching logic. If the discovered plugin name is a checksum it is excluded from the discovered plugin list.
Starting with Go 1.19 the loading of binaries from the current working directory was
deemed as a possible security problem. Thus the use of exec.Command or exec.LookPath no longer resolves
an executable within the current working directory. This change updates the discover logic to return absolute
paths for any discovered plugin, which is called directly when passed to exec.Command or exec.LookPath. By doing
this Packer is able to load a custom plugin sitting in the current working directory as it did in version prior to v1.9.2.
Since bundled plugins will be removed in an upcoming version of Packer,
this commit adds a new warning message whenever a template uses one such
plugin.
This warning has been implemented on build, validate, console and the
inspect subcommands.
In addition to warning about the upcoming change and potential issue
this will cause, this warning message proposes solutions to the user so
they know what they'll have to do in order not to rely on those bundled
plugins later.
Packer will try to discover installed plugins in all of the directories
defined by packer.KnowPluginFolders. In a previous release logic was
added to scan nested directories in order to load plugins installed by
`packer plugins install`. This change resulted in a nested directory
scan for each folder within the KnownPluginFolders slice.
This change reduces the nested directory scan to only the directories
where plugins would have been installed using `packer plugins install`
The template type for a builder, provisioner, post-processor or
datasource was not tracked with telemetry data.
This commit adds the infrastructure to track this, bumping the schema
for non-crashes to `beta/packer/6' at the same time.
Packer's telemetry package captures the field names from a configuration
on legacy JSON templates, but did not contain any code for handling HCL2
template configurations.
This commit adds a routine to extract the field names from a HCL2
configuration once flattened, and adds some glue code to propagate the
configurations to the telemetry structures.
This change fixes a regression introduced in Packer 1.8.6, where configurations custom builder names
, via the name attribute, would internally interpolate the proper build names, but would display the
name to STDOUT as uninterpolated values. The change updates the creation of a CoreBuild for legacy JSON templates
to use use the interpolated name as the final build type, which gets rendered to the screen.
Test Results Before Fix
```
~> go test ./... -run='TestCoreBuild_buildNameIntepolation' -v
--- FAIL: TestCoreBuild_buildTypeIntepolation (0.01s)
core_test.go:930: build type interpolation failed; expected "test.mybuild-RandomToken", got "test.{{user `build_name`}}"
core_test.go:930: build type interpolation failed; expected "test.build-vardata", got "test.{{user `build_name`}}"
core_test.go:930: build type interpolation failed; expected "test.build-12345", got "test.{{user `build_name`}}"
FAIL
```
Test Results After Fix
```
~> go test ./... -run='TestCoreBuild_buildNameIntepolation' -v
=== RUN TestCoreBuild_buildTypeIntepolation
--- PASS: TestCoreBuild_buildTypeIntepolation (0.01s)
PASS
```
Closes#12281
This commit irons out one of the pain points of the HCP rework by
introducing a HCPPublisher interface, implemented both by the JSON Core,
and the HCL2 PackerConfig, which keeps a map of the build names used by
Packer to the build names pushed on HCP.
This in turn lets us go back to the old implementation of the GetBuilds
function, which returns a list of (filtered) builds, and eventually an
error if something went wrong while processing.
When a build is cancelled by a user, the current implementation of
CoreBuild.Run does not return an error signifying that there was a
problem with the build.
This in turn causes the HCP code to attempt to update the status of the
build to DONE, but fails because there are no artifacts to push.
This commit changes this behaviour by returning the error provoked by
the cancellation to the caller, so it can be taken into account when
setting the status of the build.
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
To be able to track HCP builds during a Packer build, we need to
propagate the name used for registering the build to HCP.
This works out-of-the-box for JSON templates, as the build's name is
always the builder's, so there's no confusion on that.
For HCP templates however, there's a possibility for a template to
define a name for the build, which is returned by a CoreBuild's Name()
function, in addition to the source's Type/Name.
The builds are registered to HCP with the source.String() function
however, which does not contain the build's name. This prevents any
build containing a name to work, as their name/source.String does not
match.
The name we registered the build with is stored in a CoreBuild as the
Type attribute, but cannot be safely accessed, as builds are types
within the command as packersdk.Build, which only exposes Name() for
this purpose.
In order to circumvent this problem, and as a way to present a prototype
of solution that will likely have to be discussed before we merge it,
this commit hotfixes the issue.
When Packer tries to get the latest version of a plugin, it will attempt
to get the latest version from a list that is picked from the list of
releases, fetched directly from Github.
What may happen is that a release is triggered, the assets are not yet
available, including the checksum file, which is uploaded last, at the
end of the release process.
This causes any client that tries to get the latest version of a plugin
to fail during that window, as the code will fail immediately whenever a
checksum file is unavailable.
To circumvent this issue, and fallback to a previous version, we
acknowledge that a checksum was unavailable then trying to get the
latest version, and fallback to a previous one.
The mockPluginGetter would always succeed in returning a file from the
list of suggested releases, which does not let us test what happens when
a release is not yet complete, and a checksum file fails to be fetched
from the source.
This commit changes this behaviour so that we get an error when a
checksum does not exist, mirrorring what happens in real conditions.
JSON templates used only to report the builder's type in HCP builds,
even if the name was specified.
This commit changes this behaviour to include both if they're available,
in a similar fashion as what is done on the HCL2 templates.
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <[email protected]>
Before this commit, some HCP-related logic was embedded in the parser,
making all of it common to all commands.
This complexifies the logic as some parts rely on the template being in
a HCP-valid environment, and the datasources need to be evaluated in
order for some operations to be executed.
To simplify this and avoid those pitfalls, we move this logic to its own
set of functions that will be performed after the parsing is done, on
any valid HCL2 or JSON template.
* registry: add heartbeat call for running builds
When a build is running, we send periodic heartbeats to HCP Packer's
API.
This will be used on the service side to detect if a build is stalled on
the core side because of a crash or any other malfunction that caused
Packer not to send an update on the status of a build.
* registry: only update status on status update
Prior to this commit, we'd send updates to both the labels and the
cloud-provider whenever an update to the status of a build would be
sent.
This would cause a bug in which once a build reached the post-processing
state, and its cloud-provider was set, the status could not be updated
anymore, as the cloud-provider would be set and further updates are
rejected by the platform.
To avoid this problem, we only transmit the status when doing a status
update, and no other fields along with it.
* internal: publicise CompleteBuild function
The markBuildComplete private function used to be called from a final
status update to DONE, through the UpdateBuildStatus function.
The problem being that the (now) CompleteBuild function not only updates
the status of the build, but also all its metadata.
For consistency, we remove the indirection, and explicitely call
CompleteBuild when we want to finish a build.
* internal: block status updates on DONE builds
* Parse plugins from nested installed directories in json/non-required block HCL
* Add Packer Plugins installed test
* Add checksum validation for multi-component plugins installed by the packer plugins subcommand (#11732)
This change copies a bit of the logic in hcl2template/plugin in order to
strengthen the check for multi-component plugins installed via the
`packer plugins install` commands.
* Replace ioutil with os pkg
In go1.16 the ioutil std lib package was deprecated in favor of the os
and io packages. This change just updates all Temp creation methods to
their os pkg equivalents.
Co-authored-by: Wilken Rivera <[email protected]>
When obtaining image metadata for a HCP Packer build, Packer core will
use mapstructure to decode the image metadata and validate that is has
all of the required data set. Before this change Packer core would fail
if there is extra metadata fields in the image metadata that Packer is
not aware.
This change removes the restriction of erroring on unused metadata
fields. Packer should only care about the fields it knows and ignore the
rest. In the future, we will add new fields to the SDK but plugins may
begin sending that data sooner.
* fixes typos in base docs
* fixes typos in example code
* fixes typos in website docs
* fixes link to Terraform provider
* fixes typo in function name
* Update handling of registry artifacts
This change uses the github.com/hashicorp/packer-plgin-sdk/packer/registryimage for querying
Artifact State for HCP Registry Image metadata. To handle the conversion
of the RPC response, mapstructure was introduced to conversion state
data into an registryimage.Image before publishing to a image bucket.
* Update to use registry image from packersdk
* Rename internal registry service pkg
* Update vendored plugins to latest version
* The latest release of Amazon, GoogleCompute, and Azure have support
for publishing images to the HCP Packer registry.
* Add Packer version to main context at variable interpolation
Test Results Before Change
```
=== RUN TestCoreBuild_packerVersion
testing.go:30: err: template: root:1:2: executing "root" at <packer_version>: error calling packer_version: packer_version not available
--- FAIL: TestCoreBuild_packerVersion (0.00s)
```
Test Results After Change
```
=== RUN TestCoreBuild_packerVersion
--- PASS: TestCoreBuild_packerVersion (0.00s)
```
* Add packer_core_version to interpolation context for HCL configs
Test Results Before Change
```
=== RUN TestParse_build/provisioner_with_packer_version_interpolation
common_test.go:109: Parser.getBuilds() unexpected diagnostics. testdata/build/provisioner_packer_version_interpolation.pkr.hcl:8,5-24: Failed preparing provisioner-block
"shell" ""; render 'slice_string': template: root:1:2: executing "root" at <packer_version>: error calling packer_version: packer_version not available in:
{{packer_version}}
--- FAIL: TestParse_build (0.01s)
```
Test Results After Change
```
--- PASS: TestParse_build (0.02s)
--- PASS: TestParse_build/provisioner_with_packer_version_interpolation (0.00s)
```
* Add working registry pkg
* Add custom error for handling the loading of PAR environment variables
* Working Publish to Build, with proper error handling for bucket names
* Update hcp-sdk-go to use branch instead of mod replace directive
* Update Packer build status configuration
* Add support for HCP_PACKER_BUILD_FINGERPRINT env
* Add support for publishing one or more PARtifacts from a single build
* add git shas to this branch
* Add ability to set provider name if available
* Add working RegistryBuilder type
* Add RegistryPostProcessor as wrapper post-processor
* When in PAR mode a empty RegistryPostProcessor is added to the end of
the post-processor list to publish all final image data.
* Add support for updating a build from PAR that is not in a DONE state
* Fix a small issue with creation the initial builds for an empty
iteration.
* Add PAR URL to post-processor display
* Implement hcp_packer_registry block (#11168)
* Update vendored Amazon plugin to v1.0.1-dev
* Fix panic when running a Packer registry build in a clean directory
* Remove the publishing of post-processor metadata from the registry
post-processor.
* Remove metadata add from registry_builder
* Update registry builder to skip a build that was found to be DONE
Co-authored-by: Megan Marsh <[email protected]>
Co-authored-by: Sylvia Moss <[email protected]>
* start using `go:generate packer-sdc struct-markdown`
* Update Makefile
remove @go install ./cmd/struct-markdown
* run go generate for struct-markdown
* use //go:generate packer-sdc mapstructure-to-hcl2
* run go generate for mapstructure-to-hcl2
* remove struct-markdown and mapstructure-to-hcl2
* vendor vendors
Lucas Bajolet
Wilken Rivera
guoguangwu
hashicorp-copywrite[bot]
Wilken Rivera
Eng Zer Jun
Lucas Bajolet
Sylvia Moss
Jenna GoldstrichandWilken Rivera
Kerim Satirli
Ilya Voronin