Commit Graph
53 Commits
Author SHA1 Message Date
Lucas Bajolet 318529f373 hcl2template: remove value validation for locals
Local variables can't have a validation block in their definition, so
this step in not useful and should be removed.

Besides, since the validation was done on the local variables before
evaluation, it did nothing at all, as the PackerConfig.LocalVariables
collection gets populated during evaluation, so this is essentially a
no-op, and can be safely removed.
2023-09-06 12:07:37 -04:00
Lucas Bajolet eb9e1a4795 packer: remove implicit required plugins
Since this feature is no longer something we plan to activate later, as
it contradicts with our efforts to remove bundled plugins, and
encouraging users to move to either manually installing plugins, or
managing them through `packer init', we clean-up the code for this
feature.
2023-08-17 16:51:49 -04:00
hashicorp-copywrite[bot] 19055df3ec [COMPLIANCE] License changes (#12568)
* Updating the license from MPL to Business Source License

Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.

* Update copyright file headers to BUSL-1.1

---------

Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
2023-08-10 15:53:29 -07:00
Lucas Bajolet a2930bda4f packer: warn on bundled plugins usage
Since bundled plugins will be removed in an upcoming version of Packer,
this commit adds a new warning message whenever a template uses one such
plugin.

This warning has been implemented on build, validate, console and the
inspect subcommands.

In addition to warning about the upcoming change and potential issue
this will cause, this warning message proposes solutions to the user so
they know what they'll have to do in order not to rely on those bundled
plugins later.
2023-07-18 15:36:12 -04:00
hashicorp-copywrite[bot] b7df3ca36f [COMPLIANCE] Add Copyright and License Headers (#12254)
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
2023-03-02 15:37:05 -05:00
Wilken Rivera 57cbe4e203 core: Update validation options for undeclared variables (#12104)
* Update validation options for undeclared variables

In an effort to help users move from JSON to HCL2 templates the support for
variable definitions files are being updated to ignore undeclared
variable warnings on build execution. For legacy JSON templates builds no
warnings are displayed when var-files contain undeclared variables.

Since preferred mode HCL2 templates is to be explicit with variable
declarations - they must be declared to be used - validation for
undeclared variables still warns when running `packer validate`. A new
flag has been added to the validate command that can be used to disable
undeclared variable warnings.

* Update validation test for unused variables

Example Run
```
~>  go run . validate -no-warn-undeclared-var -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
The configuration is valid.

~>  go run . validate -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable

The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl
files,
such as variables.pkr.hcl

variable "unused" {
  type    = string
  default = null

}

The configuration is valid.

~>  go run . build -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
file.chocolate: output will be in this color.

Build 'file.chocolate' finished after 744 microseconds.

==> Wait completed after 798 microseconds

==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```

* Rename Strict field to WarnOnUndeclaredVar

The field name Strict is a bit vague since it is only used for
checking against undeclared variables within a var-file definition.
To mitigate against potential overloading of this field it is
being renamed to be more explicit on its usage.

* command/build: Add warn-on-undeclared-var flag

Now that the default behaviour is to not display warnings for undeclared variables
an optional flag has been added to toggle the old behaviour.

```
~>  go run . build -warn-on-undeclared-var -var-file command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable

The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl files,
such as variables.pkr.hcl

variable "unused" {
  type    = string
  default = null

}

file.chocolate: output will be in this color.

Build 'file.chocolate' finished after 762 microseconds.

==> Wait completed after 799 microseconds

==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
2022-11-14 17:06:45 -05:00
Lucas Bajolet 1cee460d0d hcp: extract all HCP-related code to hcp package
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.

This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
2022-11-14 13:31:35 -05:00
Lucas Bajolet 7cd095062a parser: extract HCP logic out of the HCL parser
Before this commit, some HCP-related logic was embedded in the parser,
making all of it common to all commands.

This complexifies the logic as some parts rely on the template being in
a HCP-valid environment, and the datasources need to be evaluated in
order for some operations to be executed.

To simplify this and avoid those pitfalls, we move this logic to its own
set of functions that will be performed after the parsing is done, on
any valid HCL2 or JSON template.
2022-08-12 11:54:21 -04:00
Adrien Delorme 5d17d7fe65 prevent duplicate local block creation (#11534)
* prevent duplicate local block creation

* remove duplicate locals block bug

* local variables: first get block, then decode it + simplify retry loop

* Update types.packer_config.go

* revert go get of hcl lib
2022-01-28 13:21:53 -05:00
a534c743cf HCP Packer registry enabled Packer client (#11154)
* Add working registry pkg

* Add custom error for handling the loading of PAR environment variables

* Working Publish to Build, with proper error handling for bucket names

* Update hcp-sdk-go to use branch instead of mod replace directive

* Update Packer build status configuration

* Add support for HCP_PACKER_BUILD_FINGERPRINT env

* Add support for publishing one or more PARtifacts from a single build

* add git shas to this branch

* Add ability to set provider name if available

* Add working RegistryBuilder type

* Add RegistryPostProcessor as wrapper post-processor

* When in PAR mode a empty RegistryPostProcessor is added to the end of
  the post-processor list to publish all final image data.

* Add support for updating a build from PAR that is not in a DONE state

* Fix a small issue with creation the initial builds for an empty
  iteration.

* Add PAR URL to post-processor display

* Implement hcp_packer_registry block (#11168)

* Update vendored Amazon plugin to v1.0.1-dev

* Fix panic when running a Packer registry build in a clean directory

* Remove the publishing of post-processor metadata from the registry
  post-processor.

* Remove metadata add from registry_builder

* Update registry builder to skip a build that was found to be DONE

Co-authored-by: Megan Marsh <[email protected]>
Co-authored-by: Sylvia Moss <[email protected]>
2021-08-05 09:25:19 -04:00
Adrien Delorme 03d79a2c39 HCL2 variables: split validation from getting value (#10843)
* HCL2 variables: split validation from getting value, to only

This way we do this only once and log this only once. The errors were being ignored anyways.

* Update types.variables_test.go
2021-03-30 14:58:26 -04:00
Adrien Delorme 77a29fc2f8 Allow to have dynamic blocks in a build block + tests (#10825)
This :
* allows to have a `build.dynamic` block
* add tests
* makes sure to show a correct message when a source was not found
  * display only name of source (instead of a weird map printout) 
  * use a "Did you mean %q" feature where possible 


Because dynamic blocks need all variables to be evaluated and available, I moved parsing of everything that is not a variable to "after" variables are extrapolated. Meaning that dynamic block get expanded in the `init` phase and then only we start interpreting HCL2 content.

After #10819 fix #10657
2021-03-30 15:53:04 +02:00
Adrien Delorme f32b67c3bb Simplify error message when config file can't be "stat'd" (#10763)
* remove confusing message when a stat error happens while trying to list HCL2 files

* leave early if our first GetHCL2Files has errors
2021-03-15 14:07:07 +01:00
Adrien Delorme 72e4dc4cb5 update docs to remove 'magic' required_plugin block usages 2021-02-15 13:58:58 +01:00
Sylvia Moss 3ddb17ad86 Don't allow data sources to be used inside data sources (#10559) 2021-02-04 11:25:44 +01:00
Adrien Delorme ed091163be HCL2 Parse packer.required_plugins block + packer init (#10304)
This adds the new `required_plugins` block to be nested under the packer block.

Example:
```hcl
packer {
  required_plugins {
    aws = {
      version = ">= 2.7.0"
      source = "azr/aws"
    }
    azure = ">= 2.7.0"
  }
}
```

For example on darwin_amd64 Packer will install those under :
* "${PACKER_HOME_DIR}/plugin/github.com/azr/amazon/packer-plugin-amazon_2.7.0_x5.0_darwin_amd64"
* "${PACKER_HOME_DIR}/plugin/github.com/hashicorp/azure/packer-plugin-azure_2.7.0_x5.0_darwin_amd64_x5"

+ docs
+ tests
2021-02-02 18:05:04 +01:00
Megan MarshandAdrien Delorme fbbda0f9d9 Sensitive locals (#10509)
* Allow locals to be delcared as individual blocks, and give them the Sensitive flag

* add docs for new local block

* linting

* add tests

* modified parsing to use schema, check for dupes properly

* update comment

fix wording a liiitle

* add tests for duplicate variables definition in two different files

* remove unnecessary slice initialisation

* fix crash by returning when decode error is hit

* parseLocalVariables: only treat a local vars if its not nil

also return in case of error
return locals in case of error too

* fix duplicate_locals test for windows

Co-authored-by: Adrien Delorme <[email protected]>
2021-01-26 10:21:44 +01:00
Sylvia Moss 291121dd55 (2) Implement datasources (#10440) 2021-01-20 10:37:16 +01:00
Megan Marsh 88b7b0e14a remove sdk 2020-12-17 13:29:25 -08:00
Wilken Rivera 4e58987026 command/fmt: Ensure all variable files ending in .pkrvars.hcl get formatted (#10377)
Before change
```
⇶  packer fmt -check /tmp/unformatted.pkrvars.hcl
Error: Cannot tell whether /tmp/unformatted.pkrvars.hcl contains HCL2 configuration data

⇶  echo $?
1
```

After fix
```
⇶  packer fmt -check /tmp/unformatted.pkrvars.hcl
/tmp/unformatted.pkrvars.hcl

⇶  echo $?
3

⇶  packer fmt -check command/test-fixtures/fmt
command/test-fixtures/fmt/unformatted.pkr.hcl
command/test-fixtures/fmt/unformatted.auto.pkrvars.hcl
command/test-fixtures/fmt/unformatted.pkrvars.hcl

```
2020-12-14 10:29:58 -05:00
Megan Marsh 39ab646236 move plugin and rpc code into sdk; other minor tweaks (#10359) 2020-12-09 12:39:54 +01:00
Megan Marsh a6fdeca099 move logsecretfilter into packer sdk 2020-11-30 14:30:30 -08:00
Megan Marsh bc85854a53 refactor packer version out of hcltemplate code. 2020-11-09 12:29:53 -08:00
Adrien Delorme 4bc16455b4 HCL2: add a packer block with a required_version input setting (#10149)
* add the possibility to set the packer.required_version field; to make sure the template file works with that version of Packer
* add tests
* add documentation on packer.required_version

Example:

packer {
  required_version = ">= 1.2.0, < 2.0.0"
}
2020-10-27 10:03:36 +01:00
Adrien Delorme 4cb218749b remove unecessary check 2020-10-05 15:34:35 +02:00
Adrien Delorme ebb1cb406c ignore walk error 2020-10-02 10:53:17 +02:00
Adrien DelormeandMegan Marsh bb22cfcf34 HCL2: pass sensitive variables to packer.LogSecretFilter
Co-Authored-By: Megan Marsh <[email protected]>
2020-10-02 10:17:07 +02:00
Sylvia Moss 9181a435aa Make max_retries a string to allow variable interpolation (#9673) 2020-08-05 17:41:20 +02:00
Adrien Delorme 44616d3bff refactor initialization out from packer configs + tests (#9627)
The initialization of packer core in JSON also validates that `null` variables were set, except in the case of `packer validate --syntax-only` , but after the refactor to allow to have all commands work with HCL2 and JSON this subtlety was lost.

This refactors the initialisation of the core in order to allow to have `packer validate --syntax-only` not error in case a variable is not set. Since these calls are refactored this works for HCL2 too.

fix #9478
2020-07-24 10:58:03 +02:00
Adrien DelormeandMegan Marsh 125178d943 core: Decode: when in HCL2 decoding mode; reset the whole struct before preparing it. (#9622)
* core: Decode when in HCL2 decoding mode; reset the whole struct before preparing it.
* HCL2: add path variables + docs & tests

Co-authored-by: Megan Marsh <[email protected]>
2020-07-23 09:25:07 +02:00
Adrien Delorme bac9c74447 packer console for HCL2 (#9359) 2020-06-05 17:23:54 +02:00
Adrien Delorme 059a782b0e remove debug log 2020-05-28 17:07:11 +02:00
Adrien Delorme 125ffe36c3 Allow to override some source fields from the build section
Using `hcl.MergeBodies`
2020-05-25 17:09:37 +02:00
Adrien Delorme 526d040534 document Parser.Parse better 2020-04-30 11:51:24 +02:00
Adrien Delorme 5c2b8da63b get builds from PackerConfig instead of parser
to allow just reading the config and to not start anything. This will allow to later on run `validate --syntax-only`.

Note that none of the builder/provisioner/post-processor config will be read but simply ignored. HCL2 still needs the body to be properly formatted and it should detect most syntax errors.
2020-04-29 16:36:40 +02:00
Adrien Delorme 2df21496b3 hcl2template: let PackerConfig actually start a builder/provisioner/post-processor 2020-04-29 16:15:42 +02:00
Adrien Delorme 2af40c762b HCL2: pass on builder type and name (#8956)
* sets `packer_build_name` and `packer_builder_type` variables for builder provisioners and post-processors in HCL2
* allows to use the new `${source.type}` and `${source.name}` variables in HCL2
* fixes #8932 

Note that the common.PackerConfig is used everywhere and was not set for HCL2, this had some implications: 

For #8923 you can see the issue here:

https://github.com/hashicorp/packer/blob/dde74232f251434715cb76664426d9c1e1c91bd8/builder/lxd/config.go#L61-L63

More random examples of where this could cause an issue :

https://github.com/hashicorp/packer/blob/0785c2f6fca9c22bf25528e0176042799dd79df9/provisioner/ansible-local/provisioner.go#L380-L381

https://github.com/hashicorp/packer/blob/b4efd13a4d69a937a22caa5dd84eb98f7fdce5cf/builder/amazon/ebs/builder.go#L232-L236



* [All references to PackerConfig.PackerBuildName](https://sourcegraph.com/github.com/hashicorp/packer@ff6a039d5bb45e34ff761d9c52e8b98972288447/-/blob/common/packer_config.go#L7:2&tab=references)

* [All references to PackerConfig.PackerBuilderType](https://sourcegraph.com/github.com/hashicorp/packer@ff6a039d5bb45e34ff761d9c52e8b98972288447/-/blob/common/packer_config.go#L8:2&tab=references)
2020-04-09 11:14:37 +02:00
Adrien Delorme d068430abf make sure locals are evaluated only once variables are + test this (#8918)
fix #8898
2020-03-19 15:30:34 +01:00
Adrien Delorme ad8dafa3bd HCL: add tests and fixes around var-file and var args (#8914) 2020-03-19 13:57:22 +01:00
Adrien Delorme c8300b620a allow to use hcl files as var files in HCL mode
fix #8781
2020-03-12 15:41:40 +01:00
Adrien Delorme 0ccff0d5b9 all variables must have a value.
A variable's default value can be set to null to force user to set it.
2020-03-09 17:25:56 +01:00
Adrien Delorme 6d8cce501e tweak validation & add tests 2020-03-09 16:16:59 +01:00
Adrien Delorme e851050774 don't force to set unused variables too 2020-03-04 12:32:29 +01:00
Adrien Delorme 59cc246942 Merge remote-tracking branch 'origin/master' into fix_8730 2020-03-04 12:30:01 +01:00
Adrien Delorme 50896d4ddf WIP 2020-03-03 11:15:56 +01:00
Sylvia Moss 591b684f08 Fix HCL2 local variables decoding to allow local usage within another local in the same locals block (#8755) 2020-02-21 12:12:30 +01:00
Adrien Delorme a8fcb2d91a HCL2: add support for dynamic blocks, document for loops and splat expressions (#8720) 2020-02-20 10:51:34 +01:00
Adrien Delorme cebfb1c735 give correct error when variable is unset 2020-02-17 17:15:52 +01:00
Adrien Delorme 193dad46e6 Hcl2 input variables, local variables and functions (#8588)
Mainly redefine or reused what Terraform did.

* allow to used `variables`, `variable` and `local` blocks
* import the following functions and their docs from Terraform: abs, abspath, basename, base64decode, base64encode, bcrypt, can, ceil, chomp, chunklist, cidrhost, cidrnetmask, cidrsubnet, cidrsubnets, coalesce, coalescelist, compact, concat, contains, convert, csvdecode, dirname, distinct, element, file, fileexists, fileset, flatten, floor, format, formatdate, formatlist, indent, index, join, jsondecode, jsonencode, keys, length, log, lookup, lower, max, md5, merge, min, parseint, pathexpand, pow, range, reverse, rsadecrypt, setintersection, setproduct, setunion, sha1, sha256, sha512, signum, slice, sort, split, strrev, substr, timestamp, timeadd, title, trim, trimprefix, trimspace, trimsuffix, try, upper, urlencode, uuidv4, uuidv5, values, yamldecode, yamlencode, zipmap.
2020-02-06 11:49:21 +01:00
Adrien Delorme 0102f5b6fa remove unecessary inexistant block panic + add test
the hcl library will error for us
2020-01-06 16:56:42 +01:00