The source parsing logic was heavily directed towards Github compatible
source URIs, however if we want to support more cases, we need to make
sure we are able to specify those URIs, and to load plugins installed
from those sources.
Right now, since the getters available are only github.com, we will not
support remotely instlling plugins from sources other than github.com,
with the same set of constraints as before. However, we do support now
installing from a local plugin binary to any kind of source, and we
support loading them, including if a template wants this plugin
installed locally with version constraints.
When introduced back in January, the version variable extracted from the
binary name had a typo in its name, and was named `protocolVerionStr'
instead of `protocolVersionStr'.
This commit was already merged into main, so it's too late to fix at
introduction site, but we can fix it today as a separate commit ontop
the stack.
If a plugin is installed manually, its version number could be valid but
non-canonical (ex: 1.2.3 vs 01.002.0003).
Since these two versions refer to the same version, but the looks are
different, this may become ambiguous which version should be loaded.
To avoid such a situation, we reject explicitely non-canonical version
numbers in plugins, but only in path, we're aware that because of
metadata, the version from `describe' may already differ from the file
name.
As with versions, API versions are useful possibly for ordering plugin
installations in order for Packer to choose which plugin to load.
This could be unnecessary as API versions are stable, and only in dev
plugins this could be a problem normally (there shouldn't be two same
releases of a plugin), but this cements API version in ordering plugins
so we avoid surprises later down the line.
As with the version of the plugin, the API version should also match
between the path and the self-reported API version from the describe
command.
This was not checked before, so users could masquerade a plugin's use of
an API version that may be incompatible with Packer.
To avoid this problem, we make sure both versions are the same, so that
they work as expected.
When Packer discovers binary a bunch of checks are performed, which
ultimately end with a checksum match check.
This however should be the very first thing we do, even before
attempting to run `describe' on the plugin binary we're discovering.
So this commit moves this checksum match to the top of the discovery
process for binaries.
When discovering the installed plugins locally, we perform a couple of
checks on the version, namely that it is valid, if it is a prerelease,
it needs to be a dev, and that the self-reported version matches the one
hinted at through the name of the binary.
This was done through regexes, but those were a wee bit simple when
dealing with versions that have metadata. Those binaries would be
completely ignored by Packer, and never loaded, although they are a
valid use case.
The version library we already used supports those however, and
comparisons are more reliable with them.
So, in order to simplify our code, and make it more reliable, we're
exclusively using this library to perform parsing and comparisons of
versions during the discovery phase.
Since we now support loading pre-releases, we also want Packer to be
able to ignore them by user demand, so we put in place the
infrastructure to modulate this.
When a pre-release version of a plugin is locally installed, it may or
may not be loaded depending on the constraints expressed in the template
being executed.
If the template contains constraints for loading the plugin, it would be
ignored, while if that wasn't present, it would be loaded.
This is inconsistent, and deserves to be addressed, which is what this
commit does.
With this change, plugin pre-releases are now loaded, provided the
version reported matches the constraints, independently from its
pre-release suffix `-dev'.
Also, if a release with the same version is also installed alongside the
pre-release version of a plugin, it will have precedence over the
pre-release.
When Packer orders the plugins by their version, we'd assumed it was
ordered from highest to lowest.
However, this was the case because our implementation of `Less' was
actually returning `>=', which is not what it was supposed to do.
This commit therefore fixes the implementation of `Less' to do what it
is documented to do, and ensures the behaviour is correct through
additional testing.
Changing this requires some changes to the loading process as well
because of the aforementioned assumption with regards to ordering.
When a plugin is loaded from Packer, we now check that the version it
reports matches what the name implies. In case there's a mismatch, we
log, and reject the binary.
Right now we had two paths for discovering installed plugins, i.e.
through plugin-getter's `ListInstallations' function, or through the
`Discover' call, which relied on a glob to list the installations.
This was required since we allowed plugins to be installed in multiple
locations, and with different constraints.
Now that we force a certain convention, we can consolidate the logic
into ListInstallations, and rely on that logic in `Discover' to load a
plugin into the PluginConfig for the current Packer run.
The plugin installation list should be sorted according to a name first,
then version second basis.
Right now, we only rely on the glob to add plugin installs to this list,
making the order unreliable since the lexicographical order is not the
order in which we want to see the same plugin ordered (e.g. v1.0.9 >
1.0.10).
To fix this, we implement a logic for sorting a list of installations
that does what's described above with more accuracy.
Since we'll only look in the plugin directory, and not from multiple
sources now, for installing/listing plugins, we can simplify the
structures which used to accept multiple directories so they only accept
one.
The plugins installed command list installed plugins, and prints out all
their paths and versions.
The results are listed in a list of unique versions, in ascending
order.
When listing plugins outside of a specific identifier however, because
of the insertion in the list works, we only consider the version number,
and ignore all subsequent insertions with the same version.
This causes a problem when we have multiple plugins installed with the
same version, as only the first one that is discovered (typically the
one with lexicographical precedence) gets inserted, and the others are
ignored.
To support such a use case, we change the insertion routine to not only
support versions, but also paths when finding a spot to insert the
installation into.
The --force option for packer init and packer plugins install enforces
installation of a plugin, even if it is already locally installed.
This will become useful if for some reason a pre-existing plugin
binary/version is already installed, and we want to overwrite it.
When invoking `packer plugins install' to install a plugin, or `packer
init', the checksum file would be installed with 0555 permissions.
This led in turn to further attempts at installing the plugin will
succeed, but the checksum file would not be updated, as it was marked
non-writable by the owner of the file, leading potentially to a
situation where the plugin binary and the checksum would be out-of-sync,
but could not be updated unless the user changed it.
To avoid such a problem, we write the checksum file with 0644
permissions, so the owner can read/write, while the other users can only
read it.
Since this feature is no longer something we plan to activate later, as
it contradicts with our efforts to remove bundled plugins, and
encouraging users to move to either manually installing plugins, or
managing them through `packer init', we clean-up the code for this
feature.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
When Packer tries to get the latest version of a plugin, it will attempt
to get the latest version from a list that is picked from the list of
releases, fetched directly from Github.
What may happen is that a release is triggered, the assets are not yet
available, including the checksum file, which is uploaded last, at the
end of the release process.
This causes any client that tries to get the latest version of a plugin
to fail during that window, as the code will fail immediately whenever a
checksum file is unavailable.
To circumvent this issue, and fallback to a previous version, we
acknowledge that a checksum was unavailable then trying to get the
latest version, and fallback to a previous one.
* Parse plugins from nested installed directories in json/non-required block HCL
* Add Packer Plugins installed test
* Add checksum validation for multi-component plugins installed by the packer plugins subcommand (#11732)
This change copies a bit of the logic in hcl2template/plugin in order to
strengthen the check for multi-component plugins installed via the
`packer plugins install` commands.
* Replace ioutil with os pkg
In go1.16 the ioutil std lib package was deprecated in favor of the os
and io packages. This change just updates all Temp creation methods to
their os pkg equivalents.
Co-authored-by: Wilken Rivera <[email protected]>
* used components that don't have a required_plugin block will make Packer 'implicitly' require those. These components are manually selected and commented for now.
* add tests
* docs
This change adds a simple hostname validation check to validate that a
plugins source address is github.com before continuing with the Get
call. An issue was encountered when using a hostname different from
github.com, where the getter would continue to pull a plugin from GitHub
even if the hostname was something like "example.com". See log details
below.
Before change
```
2021/02/16 12:49:41 [TRACE] fetching checksums file for the "0.0.2" version of the example.com/hashicorp/docker plugin in
"/home/wilken/.packer.d/plugins/example.com/hashicorp/docker"...
2021/02/16 12:49:41 [DEBUG] github-getter: getting "https://github.com/hashicorp/packer-plugin-docker/releases/download/v0.0.2/packer-plugin-docker_v0.0.2_SHA256SUMS"
2021/02/16 12:49:42 [TRACE] Ignoring remote binary packer-plugin-docker_v0.0.2_x5.0_linux_arm64.zip, wrong system, expected
2021/02/16 12:49:42 [TRACE] About to get: packer-plugin-docker_v0.0.2_x5.0_linux_amd64.zip
2021/02/16 12:49:42 [DEBUG] github-getter: getting "https://github.com/hashicorp/packer-plugin-docker/releases/download/v0.0.2/packer-plugin-docker_v0.0.2_x5.0_linux_amd64.zip"
```
After change
```
2021/02/16 13:36:32 [TRACE] for plugin example.com/hashicorp/docker found 0 matching installation(s)
2021/02/16 13:36:32 [TRACE] getting available versions for the the example.com/hashicorp/docker plugin
2021/02/16 13:36:32 [TRACE] &{%!q(*github.Client=<nil>) "packer-getter-github-1.7.0-dev"} getter could not get release: example.com/hashicorp/docker doesn't appear to be a valid github.com source address; check source and try again.
2021/02/16 13:36:32 [DEBUG] will try to install: []
2021/02/16 13:36:32 [INFO] (telemetry) Finalizing. no release version found for the example.com/hashicorp/docker plugin matching the constraint(s): ">=v0.0.2"
2021/02/16 13:36:32 waiting for all plugin processes to complete...
```
This adds the new `required_plugins` block to be nested under the packer block.
Example:
```hcl
packer {
required_plugins {
aws = {
version = ">= 2.7.0"
source = "azr/aws"
}
azure = ">= 2.7.0"
}
}
```
For example on darwin_amd64 Packer will install those under :
* "${PACKER_HOME_DIR}/plugin/github.com/azr/amazon/packer-plugin-amazon_2.7.0_x5.0_darwin_amd64"
* "${PACKER_HOME_DIR}/plugin/github.com/hashicorp/azure/packer-plugin-azure_2.7.0_x5.0_darwin_amd64_x5"
+ docs
+ tests
Lucas Bajolet
guoguangwu
hashicorp-copywrite[bot]
Wilken Rivera
Jenna GoldstrichandWilken Rivera
Dom Del Nano
Adrien Delorme