When remotely installing a plugin, constraints are used by Packer to
determine which version of a plugin to install.
These constraints can be arbitrarily complex, including operators and
ranges in which to look for valid versions.
However, the versions specified in those constraints should always be
final releases, and not a pre-release since we don't explicitly support
remotely installing pre-releases.
This commit therefore addds checks to make sure these are reported ASAP,
even before the source is contacted to list releases and picking one to
install.
The global `PACKER_CONFIG` config file was already deprecated from
Packer core, but now with 1.11.0 since we remove support for
mono-component plugins, we are also removing the capability for that
config file to declare them.
Instead of silently not using those, Packer will now error with a
message pointing to the web docs on how to manage their plugins with the
updated workflows for Packer 1.11 and above.
Compared to Terraform, Packer was lacking a capability to encode/decode
strings to/from base64-encoded text encoded with another encoding.
This could be problematic in some cases, mainly when working with
Windows, as most of the OS uses UTF-16LE as its standard encoding for
many operations.
Therefore, we take a page from Terraform here, and add those functions
to what Packer supports in an HCL2 context.
When a server returns a code that is not 200, we error in the current
state.
This is not conformant to the HTTP norm, as anything in the 2xx range is
considered a success, so the datasource should not error in this case.
Therefore, this commit fixes the condition in which we report an error,
so that anything in the 2xx range is now considered a success by the
datasource.
When updating the docs in prevision for Packer 1.11.0, we changed the
templates that show how plugins are installed/discovered with commands
like packer init.
While doing so, a template had its component renamed to coolcloud, but
the following prose did not change, making the text inconsistent.
Since there are other mentions of myawesomecloud in the codebase, we
choose to settle on this one for that example too.
When Discovering plugins installed through the `Discover` function, we
use the base name of the plugin binary we discovered preliminarly, then
we match its name against a regex to extract the prefix for the plugin's
components.
Extracting the base path used to be done with `path.Base`, which while
working perfectly on UNIX systems, does not on Windows as it uses `\\`
as their path separator.
To circumvent this problem, we use the `filepath` package to extract the
base name of the plugin instead, making the discovery logic work again
on Windows.
Listing installed plugins on Windows requires the extension to be set in
the ListOptions, otherwise they are not discovered.
While working on the discovery code, and consolidating it in a single
location, we've forgotten to pass the argument to ListInstallations, so
that makes it impossible to automatically discover installed components
on Windows.
This commit fixes this issue for the plugins required, and the general
discovery process during build/validate.
If a plugin is installed in the PACKER_PLUGIN_PATH, and its version
contains metadata, we reject it. This is because metadata is free-form
data, which could then make it possible to have multiple conflicting
versions of a plugin installed, so we don't support it and explicitely
reject plugins like those.
A valid plugin with metadata in its version information should be
installed without its metadata part, so there can only be one variant of
the plugin installed at a specific version.
Since the plugins install subcommand can install both remotely and
locally sourced plugins, we update the documentation for it on the
web-docs to reflect this change.
The ParsePluginSource function can be invoked from either a HCL2 context
(when parsing a required_plugins block), or from the command-line
itself.
While in the first context a hcl.Diagnostics is coherent, in case the
source to parse is a command-line argument, for example when installing
or removing a plugin, the error message cannot have an HCL context,
leading to errors that are incorrectly prefixed by a <nil> string dure
to the lack of a reference to attach the diagnostic to.
Therefore, in order to fix this behaviour, the logic that parses plugin
sources now returns an error, and attaching the error to an HCL subject
is done independently, if needed.
When specifying/installing plugins, a source URI is required for Packer
to be able to locate or install a plugin to the local plugin hierarchy.
The plugin hierarchy is based on the plugin source, where each component
in this hierarchy will become a directory.
In order to avoid sources with too many levels of nesting, causing a lot
of mkdirs, we limit the number of sources to 16 in this commit, this
should be long enough for most of our users.
Since we're changing how packer manages plugin installation with 1.11.0,
we reflect those changes to the website documentation.
Now, we only describe the packer init and packer plugins install
commands, along with the `--path` flag for installing from a local
source.
The explanations of how packer discovers and picks which version of a
plugin to load are also included, along with the list of constraints
that determine whether a plugin can be considered or not to be loadable.
With the changes coming up for 1.11.0, we update the Packer
configuration docs so they describe the one way to load plugins, and the
two environment variables which can be set to change this behaviour.
When running a packer command on an HCL2 template, depending on whether
or not there are required_plugin blocks defined, Packer may need to
discover and register a plugin's components multiple times.
This is the behaviour ever since those blocks were introduced to Packer,
but given we are doing the operation multiple times, this is suboptimal.
This commit changes the way things works by first doing the restricted
discovery of plugins (as dictated by required_plugins), then proceeding
to the global discovery, with the change that subsequent component
discoveries will not have precedence over those pre-discovered anymore.
This allows us to invert the call order of both discovery phases safely,
and maintains the constraints described in the templates.
When a user defines PACKER_PLUGIN_PATH in their environment, we need to
error if their path defines multiple directories separated by `:`.
This used to be supported, but this is removed with 1.11 as we're
simplifying the loading process for plugins, so we opted to fall-back to
only one plugin directory supported.
In the full example for both the hcp-packer-artifact and
hcp-packer-version the hcp-packer-version reference in the example
template was mistakenly spelled as "hcp_packer_version", which won't
work, so we fix that typo here.
```
~> packer init mondoo_req.pkr.hcl
Failed getting the "github.com/mondoohq/cnspec" plugin:
error:
Remote installation of the plugin version 10.8.1-dev is unsupported.
This is likely an upstream issue with the 10.8.1 release, which should be reported.
If you require this specific version of the plugin, download the binary and install it manually.
packer plugins install --path '<plugin_binary>' github.com/mondoohq/cnspec
```
This change is an attempt to remove the need for additional temporary files, along with
calls to stat the temp files, to reduce the number of files being created, opened, and closed.
In addition to this change, the logic for falling back to a previous version if the highest matched version
is a pre-release has been removed. Instead we will assume that any prior versions will exhibit the same issue and
return immediately. A user can install the version manually if they will or they can modify their version constraint
to a properly released version.
When a checksum file for a release is downloaded and iterated upon to
find the compatible binary for a release, we used to log each
non-compatible entry in the logs.
This is noisy as we know there's going to be multiple entries that don't
match the host's os/arch, and there's no good reason to show those, so
we silence them.
When installing a plugin from a remote source, we list the installed
plugins that match the constraints specified, and if the constraint is
already satisfied, we don't do anything.
However, since remote installation is only relevant for releases of a
plugin, we should only look at the installed releases of a plugin, and
not consider pre-releases for that step.
This wasn't the case before this commit, as if a prerelease version of a
commit (ex: 10.8.1-dev), and we try to invoke `packer init` with a
constraint on this version specifically, Packer would locate that
pre-release and assume it was already installed, so would silently
succeed the command and do nothing.
This isn't the expected behaviour as we should install the final release
of that plugin, regardless of any prerelease installation of the plugin.
So this commit fixes that by only listing releases, so we don't report
the plugin being already installed if a prerelease is what's installed.
When packer init is invoked with a --force argument, but no --update, we
clamp the version to install based on the last one locally installed.
Doing this may however cause the constraint to always be false if the
latest available version of a plugin is a pre-release, as none of the
upstream constraints will match that.
Therefore this commit changes how the constraint is derived from the
local list of installations, so that only the last installation that
matches the original constraint will be used, and not a pre-release.
Since we're hardening what Packer is able to load locally when it comes
to plugins, we need also to harden the installation process a bit.
While testing we noticed some remotes had published their plugins with
version mismatches between the tag and the binary.
This was not a problem in the past, as Packer did not care for this,
only the binary name was important, and the plugin could be installed
without problem.
Nowadays however, since Packer enforces the plugin version reported in
the name to be the same as the plugin self-reported version, this makes
it impossible for the installed plugin to load anymore in such an
instance.
Therefore in order to limit confusion, and so users are able to
understand the problem and report it to the plugins with that mismatch,
we instead install the plugin as a dev version, and report it to the
user so they can still use it, but are able to report it to the plugin
developers.
When listing installed plugins, we check that the plugin's reported
version through describe matches what's in the name of the file.
Doing do, we were parsing the same version string twice without
modifying it, which was not necessary, so this commit changes that.
Whenever a Github release exposes an entry for another OS/arch
combination, this gets registered as an error, which in the event no
binary is compatible with the host's OS/arch, gets reported at the end
of the getter process.
While this is sound in theory, in practice we get the list of all the
combinations that don't match the host's, which is not something a
Packer user can act on, and might therefore be more confusing than
helping to solve the issue.
Therefore we opt in this commit to stop registering those cases as real
errors, and only log them as an INFO statement.
Since we named the version from the getter `version', this means we have
a naming conflict inside the loop that attempts to install a versioned
candidate for a plugin, making it impossible to invoke something from
the go-version package.
Since we'll introduce a change that needs the latter capability, we must
either rename the local variable to something else than `version', or we
need to alias the package locally.
This commit implements the latter, opting to call the package goversion.
When installing a remote plugin, and after we've either successfully
installed a binary, or exhausted all the possible sources, we print a
final error message if nothing was reported.
However, given that errs is a pointer to a structure, and if no errors
were produced, the the error list could be nil, leading to the call to
`Len()' to crash Packer.
This is exceedingly rare as in general the code attempts to read
multiple sources from Github, and therefore we almost always get some
error reported, but while changing the function's code, I managed to
make it crash while removing/changing some error statements.
Therefore to avoid future surprises, we first check that `errs' is not
nil before invoking `Len()' on it, as no errors and no plugins installed
mean that something went wrong all the same.
Given that calling the describe command on plugins and deserialising the
output as a plugin description is something done in multiple places in
the code, we factorise this operation so we don't need to copy/paste the
code around.
The zipfile containing the binaries we attempt to install from a remote
source is placed in the temporary directory of the host.
In general it is wiped automatically by the OS, but in some cases
(windows typically), it isn't.
To avoid cluttering the temporary directory, we clean-up after
ourselves, and remove the temporary zip file that we create when
attempting to install a plugin, regardless of it succeeding or not.
Lucas Bajolet
Lucas Bajolet
Wilken Rivera
dependabot[bot]
Estelle PoulinandWilken Rivera