Since the DAG package was lifted from Terraform, its contents are more
than what we need for now, so this commit cleans-up the package to keep
only the currently needed parts of code.
If we need to support more in the future, we can revert this commit, or
pickup the changes again from Terraform.
The implementation of the DAG as extracted from Terraform relied on a
Root vertex being injected into the graph as the last node to visit.
This is used as a sanity check for Terraform, but doesn't apply to our
use-case for now, as we are always executing everything and have no need
for this root node.
Instead, we change how Validate operates so it does not error in case
there is no valid root node for the graph, but enables us calling it to
check for self-referencing edges, and circular dependencies.
The dag package is a port over from Terraform to Packer, changing what
little there was to fit our current dependency ecosystem.
Most of the changes are on the type of diagnostics returned, as
Terraform has its own type for them, while we rely on hcl's Diagnostics.
Other than that, the functionality is essentially equivalent, and the
code was barely touched.
In HCP's metadata package, especially the VCS/git parts, we keep the
current HEAD for a repository, along with the state it is in, in order
to report it to HCP Packer when the build completes.
However, when a build is run on a template from an empty Git repository,
and HCP Packer is enabled, the code would crash when trying to get the
information on the current HEAD, as it doesn't exist.
The git library we use returns an error in such a case, but this was
ignored, leading to a crash when attempting to get the hash to this
reference later on.
This commit fixes the problem by NOT ignoring the error to get the head,
and immediately stop processing the git data as it doesn't yet exist.
* Update UpsertBucket to first call GetBucket, this will allow bucket level role based authentication, as CreateBucket uses project level auth
* Fix one incorrect test failure message
With Packer 1.10.1 we started warning when a build failed to complete
because of a potential incompatibility with the builder being used.
This led to cases in which the build failed for other reasons, and
Packer would still warn of potential incompatibilities, even if the
builder was in effect HCP compatible.
We attempted to fix this issue by introducing a new error type, and
checks when we read the artifacts linked to a build, however this loop
would fail when any one of the artifacts is not compatible with HCP
Packer, leading to false failures.
To avoid this problem, we log incompatibilities to the verbose logger,
and only signal the problem if all the artifacts could not be used to
upload data to HCP Packer, in which case it's almost certain that if the
build succeeded and no artifacts are registered to the build, that all
the components used are not compatible with HCP, and should be reported
as such to users.
When running a build with HCP Packer enabled, Packer attempts to push
the build status to HCP.
If the build fails, we update the status to BUILD_FAILED, and that's the
end of it.
If however the build succeeds, Packer attempts to get the HCP artifact
from the builder, which will only succeed if the builder supports it.
Otherwise, we'll get either nil, or an artifact type that is not
compatible with what is expected for HCP support.
When either of those happens, we warn that the builder may not support
HCP Packer at all, so users are aware of the problem.
However, when the error was introduced, it only looked at the fact that
an error was produced, independently of the type of error. This caused
legitimate errors while building to be reported as potential
incompatibility between the builder and HCP, which was confusing to
users.
This commit changes this by introducing a new error type, only produced
when the artifact either is nil, or failed to be deserialised into a HCP
artifact, which lets us produce the incompatibility warning with more
accuracy.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
When setting a project id via the HCP_PROJECT_ID env the client will try to validate the project
by checking that it has an associated registry. If the project is invalid or not a valid UUID an error
will be displayed to the user
* Add comment to clarify usage of SDK loaded env. variables
HCP supports two types of service principals: Organization-level and project-level.
When a user tries to publish to an active HCP Packer registry using a plsp the client
fails when configuring the client due to a API permission error; namely plsp do not have
the permissions to query an org for a list of projects. Setting the HCP_PROJECT_ID does
not resolve the issue because the call to ListProjects is still executed.
This changes updates the client configuration params to obtain both the HCP Organization and
Project IDs that will be used for connecting to the HCP Packer registry. With this change
if a user provides a project Id via the HCP_PROJECT_ID environment variable no call to ListProjects will
be made. Instead the value will be take as is and used to create the connection. A user connecting with
a project level service principals must provide a valid HCP_PROJECT_ID in order to connect.
With HCP supporting multi-projects now, Packer needs to take it into
account when picking a project from an organisation.
This commit adds two cases:
1. multiple projects are defined, none is supplied through
HCP_PROJECT_ID: in this case we will default to the oldest project
defined for the organisation.
2. we supply HCP_PROJECT_ID: in this case, we pick the project with the
corresponding ID, and use it for publishing metadata.
In packer v1.8.5, the bucket's description was not properly set in the
bucket object we use for HCP, therefore all the buckets created by
Packer did not have their description updated.
Before the change
```
--- FAIL: TestReadFromHCLBuildBlock (0.00s)
--- FAIL: TestReadFromHCLBuildBlock/configure_bucket_using_only_hcp_packer_registry_block (0.00s)
types.bucket_test.go:380: expected the build to to have contents of hcp_packer_registry block but it does not: ®istry.Bucket{
Slug: "hcp_packer_registry-block-test",
- Description: "",
+ Description: "description from hcp_packer_registry block",
Destination: "",
BucketLabels: {"org": "test"},
... // 5 identical fields
}
FAIL
FAIL github.com/hashicorp/packer/internal/hcp/registry 1.072s
FAIL
```
After Change
```
~> go test ./...
? github.com/hashicorp/packer/internal/hcp/api [no test files]
ok github.com/hashicorp/packer/internal/hcp/env (cached)
ok github.com/hashicorp/packer/internal/hcp/registry 1.130s
```
Fingerprints are how we link a packer build to an iteration on HCP.
These are computed automatically from the Git SHA in the current state,
and are unique to the bucket/iteration.
The main problem with this approach is that while sound in theory, it
quickly falls apart when users want to run the same build configuration
twice, but expect a new image to be created.
With the current model, this fails, as the iteration with the current
SHA already exists.
While this is solvable through environment variables, or by committing a
change to the repository, we think this is not clear enough, and causes
an extra step to what should otherwise be a simple process.
Therefore, to lower the barrier of entry into HCP, we change this
behaviour with this commit.
Now, fingerprints are randomly generated ULIDs instead of a git SHA, and
a new one is always generated, unless one is already specified in the
environment.
This makes continuation of an existing iteration a conscious choice
rather than something automatic, and virtually eliminates conflicts such
as the ones described above.
The iteration structure that we use for linking a packer build to an
iteration on HCP defines a `Labels' attribute, which is never set nor
read from at any point.
Since it is unused, we remove it in this commit.
This commit irons out one of the pain points of the HCP rework by
introducing a HCPPublisher interface, implemented both by the JSON Core,
and the HCL2 PackerConfig, which keeps a map of the build names used by
Packer to the build names pushed on HCP.
This in turn lets us go back to the old implementation of the GetBuilds
function, which returns a list of (filtered) builds, and eventually an
error if something went wrong while processing.
Related to the previous behaviour problem, this changes how the status
of a build is set on HCP Packer, as a cancelled build is different from
a failed one, we can set it to the appropriate state on cancellation.
As a follow-up to tracking the template type for an iteration, we
actively check that it remains consistent between two invocations of
packer, and reject the update if it is not the case.
In order to ensure users keep the same templates between two packer
invocations for a single iteration, we start tracking the type of
template used when building images.
In JSON templates, we register the builds to process on HCP at the
initialisation of the Registry.
However, during the HCP breakout, this code was duplicated with a slight
difference for the JSON templates, and the build names were registered
once during initialisation, with the expected full name, and once at the
start of PopulateIteration, with the truncated name.
This caused a bug when processing a template that contains a builder
with both a name and a type, as both would be registered, but only one
would complete, leading to the iteration being unable to complete.
The ancestry inferral code relied on HCP datasource outputs for deciding
what would be the ancestry for an image built with HCP.
This brings a dependency into the hcp package, which is not really
necessary as we only need a few information from those entities, hence
this commit removes the full dependency on the structures, in favour of
more focused data structures where we only cherry pick what is necessary
for this code.
This commit reorganises the code for both the registry/API and the
Orchestrator/Registry.
The main difference with the previous version is how stuff is exposed.
Now we only expose a Registry interface to the outside (previously named
Orchestrator), which has several implementations: null is the default,
and is returned if HCP is not enabled.
The other implementations being HCL/JSON, both private to the hcp
sub-package.
The api (previously `registry') is the set of functionality that
abstracts and calls the HCP API.
It was meant to be merged with the `hcp' package, but because of a
dependency loop with the datasources, both are separated for now.
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
In order to determine whether or not a build is done or needs
processing, we check 3 things:
1. that its status is not done
2. that its ID is not empty
3. that it has no images linked to it
The was the image check was phrased indicated that we checked for the
reverse, so we reverse the naming to make it less confusing.
The HCP error messages were sometimes a bit terse in terms of the
information it bubbles up to the user.
This made them useful for people who already knew what to look for, but
newcomers would almost certainly be lost because of the lack of
information in those.
To improve on this situation, we reword most of those error messages in
this commit.
Previously, we'd get the fingerprint for an iteration when creating the
bucket.
Since we moved the remainder of the logic for getting the iteration to a
separate Initialize method, we also move the logic that reads the
fingerprint from the environment to this function.
Fingerprint initialize was previously occurring during the creation of a
bucket. We want to be able to initialize a bucket and defer the setting
of a fingerprint to a later point.
* Update test to reflect new function signatures for Bucket and Iteration
Loadgin the slug from the environment within the Bucket implied that the
Bucket was successfully created, which may fail outside of a Git
directory when the iteration fingerprint is not set through environment
variables.
To make it possible to print as much information regarding the
environment to the users at once rather than stopping immediately when
this step fails, we move the logic to read the default value from the
environment to the HCP setup code.
The HCP_PACKER_REGISTRY environment variable had its behaviour changed
recently, as prior versions of Packer expected the attribute to be
forcefully set to a value that is neither "off" nor "0" in order to
get HCP integration to work, or that a "hcp_packer_registry" block was
defined in an HCL template. Now, this environment variable defaults to
not explicitely enable, but instead to explicitely disable HCP
integration, and the feature switch fall upon HCP_PACKER_BUCKET_NAME.
As an extra feature, we keep the prior behaviour alive when it is
explicitely defined as a value to enable it. That way we can report
errors if the rest is not defined, rather than silently ignore it.
This function we add to env is the first stone to enable this behaviour.
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <[email protected]>
The current behaviour for HCP integration is based on the presence of
the HCP_PACKER_REGISTRY environment variable, where if it is either not
"0" or "off", the HCP integration is enabled.
This commit changes the behaviour to only use this variable to
explicitely disable HCP, and the HCP_PACKER_BUCKET_NAME variable will
condition if the HCP integration is enabled for a build or not in JSON
templates.
For HCL templates, the integration can also be disabled if the
HCP_PACKER_REGISTRY variable is set to "0" or "off", and will be
considered enabled if the HCP_PACKER_BUCKET_NAME is set, or there is a
"hcp_packer_registry" block in a build.
Lucas Bajolet
Devashish
Jenna Goldstrich
Wilken RiveraandLucas Bajolet
hashicorp-copywrite[bot]
Eng Zer Jun