* hcl2template: pass user variable values to plugins as packer_user_variables (#13686)
HCL2 builds do not send the packer_user_variables config key to the
builder, provisioner and post-processor plugins. Legacy JSON builds
send this key (through CoreBuild.packerConfig()). Without the key, the
plugin SDK keeps interpolate.Context.UserVariables nil. Then the
template function {{ user "name" }} fails with 'error calling user:
test'. This failure occurs in each string that a plugin interpolates at
run time, for example the contents of the vagrant post-processor's
vagrantfile_template.
Add the function PackerConfig.userVariableValues(). This function
converts the input variable values to strings, equivalent to the legacy
user variables. The core sends the map at each plugin handoff point:
builder, provisioner, post-processor, and enforced provisioner.
The map does not contain the sensitive variables. A sensitive value
goes to a plugin only if the template refers to it explicitly. The map
does not contain values that are not primitive (lists, maps, objects).
Legacy user variables were always strings.
Co-authored-by: Claude Fable 5 <[email protected]>
* Dependency upgrade (#13689)
* go.mod version upgrade
* actions upgrade
* plugin-getter: install plugins from non-GitHub HTTP sources (#13691)
Packer can only install plugins from github.com, with
releases.hashicorp.com consulted first for HashiCorp-published plugins.
This has been a long-standing gap for air-gapped and policy-restricted
environments (#11164): the source address parser already accepts any
hostname, but both existing getters reject non-github.com sources at
install time.
Add a remote plugin getter that installs plugins from the host named in
a required_plugins source address. The host serves the directory
structure of releases.hashicorp.com under the source's path: an
index.json listing versions, and per version a SHA256SUMS file, the
zips it lists, and - when the zip names carry no plugin protocol
version - the version's manifest.json. A plugin published on
releases.hashicorp.com is therefore mirrored as a verbatim copy of its
tree, with every checksum file and signature upstream-authored. A
plugin published as GitHub release assets is mirrored by copying each
release's assets into a version directory, renaming their SHA256SUMS
file to the unprefixed convention with content unchanged, and writing
an index.json listing the versions. Both kinds of content can be
served side by side by one host.
Getter selection happens per source address: github.com sources keep
the release and github getters unchanged, while any other host is
served by the remote getter over HTTPS. Sources with three or more
components are supported, up to the existing 16-component limit, so
nested artifact-repository paths and hosts that embed the upstream
origin in their path all resolve.
Version discovery, constraint solving, checksum verification, and the
binary naming rules match the existing getters. The installed filename
is rebuilt from validated checksum-file fields and never taken from the
server's response, checksum entries matching neither known naming shape
are rejected rather than guessed at, and nothing the remote metadata
supplies is used to fetch from another origin or path. index.json
parsing is covered by fixtures captured from the live releases API, so
a format change there fails tests rather than user installs.
Closes#11164
* fix: update link to CONTRIBUTING.md for consistency with main branch
* Merge pull request #13701 from hashicorp/sanya-hashicorp/fix-vuln
Module Dependency update
* build(deps): bump google.golang.org/grpc (#13704)
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.83.1 to 1.83.2.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.83.1...v1.83.2)
---
updated-dependencies:
- dependency-name: google.golang.org/grpc
dependency-version: 1.83.2
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* [COMPLIANCE] Add/Update Copyright Headers (#13699)
* [COMPLIANCE] Add/Update Copyright Headers
* make generate update
* make generate
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: sanya <[email protected]>
Co-authored-by: Sanya <[email protected]>
* Packer release 1.16.1 (#13705)
* Module upgrade (#13707)
---------
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Erik Berg <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
Co-authored-by: Benjamin Holmes <[email protected]>
Co-authored-by: elomito <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
As we're trying to move away from gob for serialising data over the
wire, this commit adds the capability for Packer to pick dynamically
between gob or protobuf for the serialisation format to communicate with
plugins.
As it stands, if all the plugins discovered are compatible with
protobuf, and we have not forced gob usage, protobuf will be the
serialisation format picked.
If any plugin is not compatible with protobuf, gob will be used for
communicating with all the plugins that will be used over the course of
a command.
When Discovering plugins installed through the `Discover` function, we
use the base name of the plugin binary we discovered preliminarly, then
we match its name against a regex to extract the prefix for the plugin's
components.
Extracting the base path used to be done with `path.Base`, which while
working perfectly on UNIX systems, does not on Windows as it uses `\\`
as their path separator.
To circumvent this problem, we use the `filepath` package to extract the
base name of the plugin instead, making the discovery logic work again
on Windows.
Listing installed plugins on Windows requires the extension to be set in
the ListOptions, otherwise they are not discovered.
While working on the discovery code, and consolidating it in a single
location, we've forgotten to pass the argument to ListInstallations, so
that makes it impossible to automatically discover installed components
on Windows.
This commit fixes this issue for the plugins required, and the general
discovery process during build/validate.
When running a packer command on an HCL2 template, depending on whether
or not there are required_plugin blocks defined, Packer may need to
discover and register a plugin's components multiple times.
This is the behaviour ever since those blocks were introduced to Packer,
but given we are doing the operation multiple times, this is suboptimal.
This commit changes the way things works by first doing the restricted
discovery of plugins (as dictated by required_plugins), then proceeding
to the global discovery, with the change that subsequent component
discoveries will not have precedence over those pre-discovered anymore.
This allows us to invert the call order of both discovery phases safely,
and maintains the constraints described in the templates.
Given that calling the describe command on plugins and deserialising the
output as a plugin description is something done in multiple places in
the code, we factorise this operation so we don't need to copy/paste the
code around.
While migrating to a unified approach for loading plugins, the API
major and minor versions were not added to the constraints for
discovering plugins from the environment, leading to Packer potentially
considering plugins that are not compatible with itself.
This should not be possible, but was due to this omission, which we fix
with this commit.
Since we now support loading pre-releases, we also want Packer to be
able to ignore them by user demand, so we put in place the
infrastructure to modulate this.
When Packer orders the plugins by their version, we'd assumed it was
ordered from highest to lowest.
However, this was the case because our implementation of `Less' was
actually returning `>=', which is not what it was supposed to do.
This commit therefore fixes the implementation of `Less' to do what it
is documented to do, and ensures the behaviour is correct through
additional testing.
Changing this requires some changes to the loading process as well
because of the aforementioned assumption with regards to ordering.
Right now we had two paths for discovering installed plugins, i.e.
through plugin-getter's `ListInstallations' function, or through the
`Discover' call, which relied on a glob to list the installations.
This was required since we allowed plugins to be installed in multiple
locations, and with different constraints.
Now that we force a certain convention, we can consolidate the logic
into ListInstallations, and rely on that logic in `Discover' to load a
plugin into the PluginConfig for the current Packer run.
Since the plugin directory should now be unique instead of a list, we
can use it directly from the receiver's structure instead of as a
parameter to the function.
Since we'll only look in the plugin directory, and not from multiple
sources now, for installing/listing plugins, we can simplify the
structures which used to accept multiple directories so they only accept
one.
Since we'll be only accepting plugins installed locally along with a
shasum, and those that adopt the convention we introduced with
required_plugins, we remove support for plugins that only have a
packer-plugin-.* type name.
Since we're removing the alternative plugin installation directories in
favour of only supporting installing them in the PACKER_PLUGIN_PATH
directory, we only return one directory when getting the known plugin
directories.
Single-component plugins are a relic from the past that has been
deprecated from version 1.7.0 and onwards.
Since we're revisiting how plugins are installed/loaded, and the changes
will be incompatible with those, we remove them in preparation of this
work.
Since this feature is no longer something we plan to activate later, as
it contradicts with our efforts to remove bundled plugins, and
encouraging users to move to either manually installing plugins, or
managing them through `packer init', we clean-up the code for this
feature.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
* Add test case for loading plugin in CWD
* Add test case to validate checksume files are ignored
* Update Discover to include CWD "." in PluginFolders if KnowPluginFolders is unset
When copying a plugin's checksum file (packer-plugin-*_SHA256SUM) installed by `packer plugins install` or `packer init`
into a separate directory the file may be copied with the executable bit turned out. If unchanged after the copy, Packer would
discover the checksum file as a possible plugin match and error when trying to execute describe on the plugin look a like. This change
adds a checksum file test to the plugin matching logic. If the discovered plugin name is a checksum it is excluded from the discovered plugin list.
Starting with Go 1.19 the loading of binaries from the current working directory was
deemed as a possible security problem. Thus the use of exec.Command or exec.LookPath no longer resolves
an executable within the current working directory. This change updates the discover logic to return absolute
paths for any discovered plugin, which is called directly when passed to exec.Command or exec.LookPath. By doing
this Packer is able to load a custom plugin sitting in the current working directory as it did in version prior to v1.9.2.
Packer will try to discover installed plugins in all of the directories
defined by packer.KnowPluginFolders. In a previous release logic was
added to scan nested directories in order to load plugins installed by
`packer plugins install`. This change resulted in a nested directory
scan for each folder within the KnownPluginFolders slice.
This change reduces the nested directory scan to only the directories
where plugins would have been installed using `packer plugins install`
* Parse plugins from nested installed directories in json/non-required block HCL
* Add Packer Plugins installed test
* Add checksum validation for multi-component plugins installed by the packer plugins subcommand (#11732)
This change copies a bit of the logic in hcl2template/plugin in order to
strengthen the check for multi-component plugins installed via the
`packer plugins install` commands.
* Replace ioutil with os pkg
In go1.16 the ioutil std lib package was deprecated in favor of the os
and io packages. This change just updates all Temp creation methods to
their os pkg equivalents.
Co-authored-by: Wilken Rivera <[email protected]>
* used components that don't have a required_plugin block will make Packer 'implicitly' require those. These components are manually selected and commented for now.
* add tests
* docs
This changes wraps the plugin client start call with an anonymous function so that Packer starts a
new plugin for each occurrence of a particular plugin block.
Before only one subprocess was being created causing subsquent calls to fail as it was trying start
an already started plugin subprocess.
Before Change
```
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
2021/02/05 16:09:13 On error:
2021/02/05 16:09:13 Waiting on builds to complete...
2021/02/05 16:09:13 Starting build run: null.basic-example
2021/02/05 16:09:13 Running builder:
2021/02/05 16:09:13 [INFO] (telemetry) Starting builder
on examples/basic-example.pkr.hcl line 29:
(source code not available)
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
2021/02/05 16:09:13 packer.test plugin: [INFO] communicator disabled, will not connect
2021/02/05 16:09:13 packer.test plugin: Unable to load communicator config from state to populate provisionHookData
on examples/basic-example.pkr.hcl line 38:
2021/02/05 16:09:13 packer.test plugin: Running the provision hook
(source code not available)
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
on examples/basic-example.pkr.hcl line 42:
(source code not available)
2021/02/05 16:09:13 [INFO] (telemetry) Starting provisioner comment
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
```
After change
```
null.basic-example: output will be in this color.
==> null.basic-example: ____ _
==> null.basic-example: | __ ) ___ __ _ (_) _ __
==> null.basic-example: | _ \ / _ \ / _` | | | | '_ \
==> null.basic-example: | |_) | | __/ | (_| | | | | | | |
==> null.basic-example: |____/ \___| \__, | |_| |_| |_|
==> null.basic-example: |___/
==> null.basic-example:
==> null.basic-example: Running local shell script: /tmp/packer-shell646549657
null.basic-example: This is a shell script
==> null.basic-example: Pausing at breakpoint provisioner.
==> null.basic-example: Press enter to continue.
==> null.basic-example: In the middle of Provisioning run
==> null.basic-example: Running local shell script: /tmp/packer-shell177279484
null.basic-example: This is another shell script
==> null.basic-example: _____ _
==> null.basic-example: | ____| _ __ __| |
==> null.basic-example: | _| | '_ \ / _` |
==> null.basic-example: | |___ | | | | | (_| |
==> null.basic-example: |_____| |_| |_| \__,_|
==> null.basic-example:
Build 'null.basic-example' finished after 1 second 32 milliseconds.
==> Wait completed after 1 second 32 milliseconds
==> Builds finished. The artifacts of successful builds are:
--> null.basic-example: Did not export anything. This is the null builder
Please enter the commit message for your changes. Lines starting
```
This adds the new `required_plugins` block to be nested under the packer block.
Example:
```hcl
packer {
required_plugins {
aws = {
version = ">= 2.7.0"
source = "azr/aws"
}
azure = ">= 2.7.0"
}
}
```
For example on darwin_amd64 Packer will install those under :
* "${PACKER_HOME_DIR}/plugin/github.com/azr/amazon/packer-plugin-amazon_2.7.0_x5.0_darwin_amd64"
* "${PACKER_HOME_DIR}/plugin/github.com/hashicorp/azure/packer-plugin-azure_2.7.0_x5.0_darwin_amd64_x5"
+ docs
+ tests
Lucas Bajolet
Lucas Bajolet
Devashish
Wilken Rivera
Jenna GoldstrichandWilken Rivera
Ilya Voronin
Adrien Delorme
Megan Marsh
Adrien Delorme
Sylvia Moss