As we're trying to move away from gob for serialising data over the
wire, this commit adds the capability for Packer to pick dynamically
between gob or protobuf for the serialisation format to communicate with
plugins.
As it stands, if all the plugins discovered are compatible with
protobuf, and we have not forced gob usage, protobuf will be the
serialisation format picked.
If any plugin is not compatible with protobuf, gob will be used for
communicating with all the plugins that will be used over the course of
a command.
When Discovering plugins installed through the `Discover` function, we
use the base name of the plugin binary we discovered preliminarly, then
we match its name against a regex to extract the prefix for the plugin's
components.
Extracting the base path used to be done with `path.Base`, which while
working perfectly on UNIX systems, does not on Windows as it uses `\\`
as their path separator.
To circumvent this problem, we use the `filepath` package to extract the
base name of the plugin instead, making the discovery logic work again
on Windows.
Listing installed plugins on Windows requires the extension to be set in
the ListOptions, otherwise they are not discovered.
While working on the discovery code, and consolidating it in a single
location, we've forgotten to pass the argument to ListInstallations, so
that makes it impossible to automatically discover installed components
on Windows.
This commit fixes this issue for the plugins required, and the general
discovery process during build/validate.
When running a packer command on an HCL2 template, depending on whether
or not there are required_plugin blocks defined, Packer may need to
discover and register a plugin's components multiple times.
This is the behaviour ever since those blocks were introduced to Packer,
but given we are doing the operation multiple times, this is suboptimal.
This commit changes the way things works by first doing the restricted
discovery of plugins (as dictated by required_plugins), then proceeding
to the global discovery, with the change that subsequent component
discoveries will not have precedence over those pre-discovered anymore.
This allows us to invert the call order of both discovery phases safely,
and maintains the constraints described in the templates.
Given that calling the describe command on plugins and deserialising the
output as a plugin description is something done in multiple places in
the code, we factorise this operation so we don't need to copy/paste the
code around.
While migrating to a unified approach for loading plugins, the API
major and minor versions were not added to the constraints for
discovering plugins from the environment, leading to Packer potentially
considering plugins that are not compatible with itself.
This should not be possible, but was due to this omission, which we fix
with this commit.
Since we now support loading pre-releases, we also want Packer to be
able to ignore them by user demand, so we put in place the
infrastructure to modulate this.
When Packer orders the plugins by their version, we'd assumed it was
ordered from highest to lowest.
However, this was the case because our implementation of `Less' was
actually returning `>=', which is not what it was supposed to do.
This commit therefore fixes the implementation of `Less' to do what it
is documented to do, and ensures the behaviour is correct through
additional testing.
Changing this requires some changes to the loading process as well
because of the aforementioned assumption with regards to ordering.
Right now we had two paths for discovering installed plugins, i.e.
through plugin-getter's `ListInstallations' function, or through the
`Discover' call, which relied on a glob to list the installations.
This was required since we allowed plugins to be installed in multiple
locations, and with different constraints.
Now that we force a certain convention, we can consolidate the logic
into ListInstallations, and rely on that logic in `Discover' to load a
plugin into the PluginConfig for the current Packer run.
Since the plugin directory should now be unique instead of a list, we
can use it directly from the receiver's structure instead of as a
parameter to the function.
Since we'll only look in the plugin directory, and not from multiple
sources now, for installing/listing plugins, we can simplify the
structures which used to accept multiple directories so they only accept
one.
Since we'll be only accepting plugins installed locally along with a
shasum, and those that adopt the convention we introduced with
required_plugins, we remove support for plugins that only have a
packer-plugin-.* type name.
Since we're removing the alternative plugin installation directories in
favour of only supporting installing them in the PACKER_PLUGIN_PATH
directory, we only return one directory when getting the known plugin
directories.
Single-component plugins are a relic from the past that has been
deprecated from version 1.7.0 and onwards.
Since we're revisiting how plugins are installed/loaded, and the changes
will be incompatible with those, we remove them in preparation of this
work.
Since this feature is no longer something we plan to activate later, as
it contradicts with our efforts to remove bundled plugins, and
encouraging users to move to either manually installing plugins, or
managing them through `packer init', we clean-up the code for this
feature.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
* Add test case for loading plugin in CWD
* Add test case to validate checksume files are ignored
* Update Discover to include CWD "." in PluginFolders if KnowPluginFolders is unset
When copying a plugin's checksum file (packer-plugin-*_SHA256SUM) installed by `packer plugins install` or `packer init`
into a separate directory the file may be copied with the executable bit turned out. If unchanged after the copy, Packer would
discover the checksum file as a possible plugin match and error when trying to execute describe on the plugin look a like. This change
adds a checksum file test to the plugin matching logic. If the discovered plugin name is a checksum it is excluded from the discovered plugin list.
Starting with Go 1.19 the loading of binaries from the current working directory was
deemed as a possible security problem. Thus the use of exec.Command or exec.LookPath no longer resolves
an executable within the current working directory. This change updates the discover logic to return absolute
paths for any discovered plugin, which is called directly when passed to exec.Command or exec.LookPath. By doing
this Packer is able to load a custom plugin sitting in the current working directory as it did in version prior to v1.9.2.
Packer will try to discover installed plugins in all of the directories
defined by packer.KnowPluginFolders. In a previous release logic was
added to scan nested directories in order to load plugins installed by
`packer plugins install`. This change resulted in a nested directory
scan for each folder within the KnownPluginFolders slice.
This change reduces the nested directory scan to only the directories
where plugins would have been installed using `packer plugins install`
* Parse plugins from nested installed directories in json/non-required block HCL
* Add Packer Plugins installed test
* Add checksum validation for multi-component plugins installed by the packer plugins subcommand (#11732)
This change copies a bit of the logic in hcl2template/plugin in order to
strengthen the check for multi-component plugins installed via the
`packer plugins install` commands.
* Replace ioutil with os pkg
In go1.16 the ioutil std lib package was deprecated in favor of the os
and io packages. This change just updates all Temp creation methods to
their os pkg equivalents.
Co-authored-by: Wilken Rivera <[email protected]>
* used components that don't have a required_plugin block will make Packer 'implicitly' require those. These components are manually selected and commented for now.
* add tests
* docs
This changes wraps the plugin client start call with an anonymous function so that Packer starts a
new plugin for each occurrence of a particular plugin block.
Before only one subprocess was being created causing subsquent calls to fail as it was trying start
an already started plugin subprocess.
Before Change
```
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
2021/02/05 16:09:13 On error:
2021/02/05 16:09:13 Waiting on builds to complete...
2021/02/05 16:09:13 Starting build run: null.basic-example
2021/02/05 16:09:13 Running builder:
2021/02/05 16:09:13 [INFO] (telemetry) Starting builder
on examples/basic-example.pkr.hcl line 29:
(source code not available)
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
2021/02/05 16:09:13 packer.test plugin: [INFO] communicator disabled, will not connect
2021/02/05 16:09:13 packer.test plugin: Unable to load communicator config from state to populate provisionHookData
on examples/basic-example.pkr.hcl line 38:
2021/02/05 16:09:13 packer.test plugin: Running the provision hook
(source code not available)
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
on examples/basic-example.pkr.hcl line 42:
(source code not available)
2021/02/05 16:09:13 [INFO] (telemetry) Starting provisioner comment
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
```
After change
```
null.basic-example: output will be in this color.
==> null.basic-example: ____ _
==> null.basic-example: | __ ) ___ __ _ (_) _ __
==> null.basic-example: | _ \ / _ \ / _` | | | | '_ \
==> null.basic-example: | |_) | | __/ | (_| | | | | | | |
==> null.basic-example: |____/ \___| \__, | |_| |_| |_|
==> null.basic-example: |___/
==> null.basic-example:
==> null.basic-example: Running local shell script: /tmp/packer-shell646549657
null.basic-example: This is a shell script
==> null.basic-example: Pausing at breakpoint provisioner.
==> null.basic-example: Press enter to continue.
==> null.basic-example: In the middle of Provisioning run
==> null.basic-example: Running local shell script: /tmp/packer-shell177279484
null.basic-example: This is another shell script
==> null.basic-example: _____ _
==> null.basic-example: | ____| _ __ __| |
==> null.basic-example: | _| | '_ \ / _` |
==> null.basic-example: | |___ | | | | | (_| |
==> null.basic-example: |_____| |_| |_| \__,_|
==> null.basic-example:
Build 'null.basic-example' finished after 1 second 32 milliseconds.
==> Wait completed after 1 second 32 milliseconds
==> Builds finished. The artifacts of successful builds are:
--> null.basic-example: Did not export anything. This is the null builder
Please enter the commit message for your changes. Lines starting
```
This adds the new `required_plugins` block to be nested under the packer block.
Example:
```hcl
packer {
required_plugins {
aws = {
version = ">= 2.7.0"
source = "azr/aws"
}
azure = ">= 2.7.0"
}
}
```
For example on darwin_amd64 Packer will install those under :
* "${PACKER_HOME_DIR}/plugin/github.com/azr/amazon/packer-plugin-amazon_2.7.0_x5.0_darwin_amd64"
* "${PACKER_HOME_DIR}/plugin/github.com/hashicorp/azure/packer-plugin-azure_2.7.0_x5.0_darwin_amd64_x5"
+ docs
+ tests
Lucas Bajolet
Lucas Bajolet
Devashish
hashicorp-copywrite[bot]
Wilken Rivera
Jenna GoldstrichandWilken Rivera
Ilya Voronin
Adrien Delorme
Megan Marsh
Adrien Delorme
Sylvia Moss