JSON templates used only to report the builder's type in HCP builds,
even if the name was specified.
This commit changes this behaviour to include both if they're available,
in a similar fashion as what is done on the HCL2 templates.
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <[email protected]>
Before this commit, some HCP-related logic was embedded in the parser,
making all of it common to all commands.
This complexifies the logic as some parts rely on the template being in
a HCP-valid environment, and the datasources need to be evaluated in
order for some operations to be executed.
To simplify this and avoid those pitfalls, we move this logic to its own
set of functions that will be performed after the parsing is done, on
any valid HCL2 or JSON template.
* registry: add heartbeat call for running builds
When a build is running, we send periodic heartbeats to HCP Packer's
API.
This will be used on the service side to detect if a build is stalled on
the core side because of a crash or any other malfunction that caused
Packer not to send an update on the status of a build.
* registry: only update status on status update
Prior to this commit, we'd send updates to both the labels and the
cloud-provider whenever an update to the status of a build would be
sent.
This would cause a bug in which once a build reached the post-processing
state, and its cloud-provider was set, the status could not be updated
anymore, as the cloud-provider would be set and further updates are
rejected by the platform.
To avoid this problem, we only transmit the status when doing a status
update, and no other fields along with it.
* internal: publicise CompleteBuild function
The markBuildComplete private function used to be called from a final
status update to DONE, through the UpdateBuildStatus function.
The problem being that the (now) CompleteBuild function not only updates
the status of the build, but also all its metadata.
For consistency, we remove the indirection, and explicitely call
CompleteBuild when we want to finish a build.
* internal: block status updates on DONE builds
* Parse plugins from nested installed directories in json/non-required block HCL
* Add Packer Plugins installed test
* Add checksum validation for multi-component plugins installed by the packer plugins subcommand (#11732)
This change copies a bit of the logic in hcl2template/plugin in order to
strengthen the check for multi-component plugins installed via the
`packer plugins install` commands.
* Replace ioutil with os pkg
In go1.16 the ioutil std lib package was deprecated in favor of the os
and io packages. This change just updates all Temp creation methods to
their os pkg equivalents.
Co-authored-by: Wilken Rivera <[email protected]>
When obtaining image metadata for a HCP Packer build, Packer core will
use mapstructure to decode the image metadata and validate that is has
all of the required data set. Before this change Packer core would fail
if there is extra metadata fields in the image metadata that Packer is
not aware.
This change removes the restriction of erroring on unused metadata
fields. Packer should only care about the fields it knows and ignore the
rest. In the future, we will add new fields to the SDK but plugins may
begin sending that data sooner.
* fixes typos in base docs
* fixes typos in example code
* fixes typos in website docs
* fixes link to Terraform provider
* fixes typo in function name
* Update handling of registry artifacts
This change uses the github.com/hashicorp/packer-plgin-sdk/packer/registryimage for querying
Artifact State for HCP Registry Image metadata. To handle the conversion
of the RPC response, mapstructure was introduced to conversion state
data into an registryimage.Image before publishing to a image bucket.
* Update to use registry image from packersdk
* Rename internal registry service pkg
* Update vendored plugins to latest version
* The latest release of Amazon, GoogleCompute, and Azure have support
for publishing images to the HCP Packer registry.
* Add Packer version to main context at variable interpolation
Test Results Before Change
```
=== RUN TestCoreBuild_packerVersion
testing.go:30: err: template: root:1:2: executing "root" at <packer_version>: error calling packer_version: packer_version not available
--- FAIL: TestCoreBuild_packerVersion (0.00s)
```
Test Results After Change
```
=== RUN TestCoreBuild_packerVersion
--- PASS: TestCoreBuild_packerVersion (0.00s)
```
* Add packer_core_version to interpolation context for HCL configs
Test Results Before Change
```
=== RUN TestParse_build/provisioner_with_packer_version_interpolation
common_test.go:109: Parser.getBuilds() unexpected diagnostics. testdata/build/provisioner_packer_version_interpolation.pkr.hcl:8,5-24: Failed preparing provisioner-block
"shell" ""; render 'slice_string': template: root:1:2: executing "root" at <packer_version>: error calling packer_version: packer_version not available in:
{{packer_version}}
--- FAIL: TestParse_build (0.01s)
```
Test Results After Change
```
--- PASS: TestParse_build (0.02s)
--- PASS: TestParse_build/provisioner_with_packer_version_interpolation (0.00s)
```
* Add working registry pkg
* Add custom error for handling the loading of PAR environment variables
* Working Publish to Build, with proper error handling for bucket names
* Update hcp-sdk-go to use branch instead of mod replace directive
* Update Packer build status configuration
* Add support for HCP_PACKER_BUILD_FINGERPRINT env
* Add support for publishing one or more PARtifacts from a single build
* add git shas to this branch
* Add ability to set provider name if available
* Add working RegistryBuilder type
* Add RegistryPostProcessor as wrapper post-processor
* When in PAR mode a empty RegistryPostProcessor is added to the end of
the post-processor list to publish all final image data.
* Add support for updating a build from PAR that is not in a DONE state
* Fix a small issue with creation the initial builds for an empty
iteration.
* Add PAR URL to post-processor display
* Implement hcp_packer_registry block (#11168)
* Update vendored Amazon plugin to v1.0.1-dev
* Fix panic when running a Packer registry build in a clean directory
* Remove the publishing of post-processor metadata from the registry
post-processor.
* Remove metadata add from registry_builder
* Update registry builder to skip a build that was found to be DONE
Co-authored-by: Megan Marsh <[email protected]>
Co-authored-by: Sylvia Moss <[email protected]>
* start using `go:generate packer-sdc struct-markdown`
* Update Makefile
remove @go install ./cmd/struct-markdown
* run go generate for struct-markdown
* use //go:generate packer-sdc mapstructure-to-hcl2
* run go generate for mapstructure-to-hcl2
* remove struct-markdown and mapstructure-to-hcl2
* vendor vendors
* used components that don't have a required_plugin block will make Packer 'implicitly' require those. These components are manually selected and commented for now.
* add tests
* docs
This change adds a simple hostname validation check to validate that a
plugins source address is github.com before continuing with the Get
call. An issue was encountered when using a hostname different from
github.com, where the getter would continue to pull a plugin from GitHub
even if the hostname was something like "example.com". See log details
below.
Before change
```
2021/02/16 12:49:41 [TRACE] fetching checksums file for the "0.0.2" version of the example.com/hashicorp/docker plugin in
"/home/wilken/.packer.d/plugins/example.com/hashicorp/docker"...
2021/02/16 12:49:41 [DEBUG] github-getter: getting "https://github.com/hashicorp/packer-plugin-docker/releases/download/v0.0.2/packer-plugin-docker_v0.0.2_SHA256SUMS"
2021/02/16 12:49:42 [TRACE] Ignoring remote binary packer-plugin-docker_v0.0.2_x5.0_linux_arm64.zip, wrong system, expected
2021/02/16 12:49:42 [TRACE] About to get: packer-plugin-docker_v0.0.2_x5.0_linux_amd64.zip
2021/02/16 12:49:42 [DEBUG] github-getter: getting "https://github.com/hashicorp/packer-plugin-docker/releases/download/v0.0.2/packer-plugin-docker_v0.0.2_x5.0_linux_amd64.zip"
```
After change
```
2021/02/16 13:36:32 [TRACE] for plugin example.com/hashicorp/docker found 0 matching installation(s)
2021/02/16 13:36:32 [TRACE] getting available versions for the the example.com/hashicorp/docker plugin
2021/02/16 13:36:32 [TRACE] &{%!q(*github.Client=<nil>) "packer-getter-github-1.7.0-dev"} getter could not get release: example.com/hashicorp/docker doesn't appear to be a valid github.com source address; check source and try again.
2021/02/16 13:36:32 [DEBUG] will try to install: []
2021/02/16 13:36:32 [INFO] (telemetry) Finalizing. no release version found for the example.com/hashicorp/docker plugin matching the constraint(s): ">=v0.0.2"
2021/02/16 13:36:32 waiting for all plugin processes to complete...
```
This changes wraps the plugin client start call with an anonymous function so that Packer starts a
new plugin for each occurrence of a particular plugin block.
Before only one subprocess was being created causing subsquent calls to fail as it was trying start
an already started plugin subprocess.
Before Change
```
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
2021/02/05 16:09:13 On error:
2021/02/05 16:09:13 Waiting on builds to complete...
2021/02/05 16:09:13 Starting build run: null.basic-example
2021/02/05 16:09:13 Running builder:
2021/02/05 16:09:13 [INFO] (telemetry) Starting builder
on examples/basic-example.pkr.hcl line 29:
(source code not available)
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
2021/02/05 16:09:13 packer.test plugin: [INFO] communicator disabled, will not connect
2021/02/05 16:09:13 packer.test plugin: Unable to load communicator config from state to populate provisionHookData
on examples/basic-example.pkr.hcl line 38:
2021/02/05 16:09:13 packer.test plugin: Running the provision hook
(source code not available)
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
???????: failed loading comment: error dial unix /tmp/packer-plugin358226172: connect: no such file or directory
on examples/basic-example.pkr.hcl line 42:
(source code not available)
2021/02/05 16:09:13 [INFO] (telemetry) Starting provisioner comment
dial unix /tmp/packer-plugin358226172: connect: no such file or directory
```
After change
```
null.basic-example: output will be in this color.
==> null.basic-example: ____ _
==> null.basic-example: | __ ) ___ __ _ (_) _ __
==> null.basic-example: | _ \ / _ \ / _` | | | | '_ \
==> null.basic-example: | |_) | | __/ | (_| | | | | | | |
==> null.basic-example: |____/ \___| \__, | |_| |_| |_|
==> null.basic-example: |___/
==> null.basic-example:
==> null.basic-example: Running local shell script: /tmp/packer-shell646549657
null.basic-example: This is a shell script
==> null.basic-example: Pausing at breakpoint provisioner.
==> null.basic-example: Press enter to continue.
==> null.basic-example: In the middle of Provisioning run
==> null.basic-example: Running local shell script: /tmp/packer-shell177279484
null.basic-example: This is another shell script
==> null.basic-example: _____ _
==> null.basic-example: | ____| _ __ __| |
==> null.basic-example: | _| | '_ \ / _` |
==> null.basic-example: | |___ | | | | | (_| |
==> null.basic-example: |_____| |_| |_| \__,_|
==> null.basic-example:
Build 'null.basic-example' finished after 1 second 32 milliseconds.
==> Wait completed after 1 second 32 milliseconds
==> Builds finished. The artifacts of successful builds are:
--> null.basic-example: Did not export anything. This is the null builder
Please enter the commit message for your changes. Lines starting
```
This adds the new `required_plugins` block to be nested under the packer block.
Example:
```hcl
packer {
required_plugins {
aws = {
version = ">= 2.7.0"
source = "azr/aws"
}
azure = ">= 2.7.0"
}
}
```
For example on darwin_amd64 Packer will install those under :
* "${PACKER_HOME_DIR}/plugin/github.com/azr/amazon/packer-plugin-amazon_2.7.0_x5.0_darwin_amd64"
* "${PACKER_HOME_DIR}/plugin/github.com/hashicorp/azure/packer-plugin-azure_2.7.0_x5.0_darwin_amd64_x5"
+ docs
+ tests
* move maps of plugins back in core
* go mod vendor
* more fixes
* fix imports
* Update core_test.go
* fix build
* more fixes
* more fixes
* up vendors after fixing sdk
* Update post_processor_mock.hcl2spec.go
* Leave implementatino of MapOf in the sdk for plugi tests
Other wise use the interface
* go mod tidy
* add MapOfDatasource type too
This add :
* discovery of `packer-plugin-*` binaries from the known folders and ask them to describe themselves
* tests
For testing: in go we create a bash script that in turn calls back to Go. I could not make the tests to work on windows and then would like to postpone testing this for when we know more about the finite layout of this feature. That is mainly: how things are going to work with init, versioning and such.
Lucas Bajolet
Eng Zer Jun
Lucas Bajolet
Sylvia Moss
Jenna GoldstrichandWilken Rivera
Kerim Satirli
Ilya Voronin
Dom Del Nano
Wilken Rivera
Adrien Delorme