Cut version 1.3.0

Going to revert this change for now, becuase of potential issues that…

CHANGELOG.md

@@ -1,14 +1,22 @@
-## 1.2.6 (Unreleased)
+## 1.3.0 (September 11, 2018)
 
 ### IMPROVEMENTS:
-
+* azure/arm: Retry cleanup of individual resources on error [GH-6644]
 * builder/alicloud: Support source image coming from marketplace [GH-6588]
+* builder/amazon-chroot: Add new `root_volume_type` option. [GH-6669]
+* builder/amazon-chroot: If you have a PV source AMI, with the Amazon Chroot
+    builder, and the destination AMI is type HVM, you can now enable
+    ena_support, example: [GH-6670]
 * builder/amazon-chroot: New feature `root_volume_tags` to tag the created
     volumes. [GH-6504]
+* builder/amazon: Create a random interim AMI name when encrypt_boot is true so
+    that ami name is not searchable. [GH-6657]
 * builder/azure: Implement clean_image_name template engine. [GH-6558]
 * builder/cloudstack: Add option to use a fixed port via public_port. [GH-6532]
 * builder/digitalocean: Add support for tagging to instances [GH-6546]
 * builder/googlecompute: Add new `min_cpu_platform` feature [GH-6607]
+* builder/googlecompute: Update the list of public image projects that we
+    search, based on GCE documentation. [GH-6648]
 * builder/lxc: Allow unplivileged LXC containers. [GH-6279]
 * builder/oci: Add `metadata` feature to Packer config. [GH-6498]
 * builder/openstack: Add support for getting config from clouds-public.yaml.
@@ -18,34 +26,77 @@
 * builder/openstack: Migrate floating IP usage to Network v2 API from Compute
     API. [GH-6373]
 * builder/openstack: Support Block Storage volumes as boot volume. [GH-6596]
+* builder/oracle-oci: Add support for freeform tagging of OCI images [GH-6338]
 * builder/qemu: add ssh agent support. [GH-6541]
 * builder/qemu: New `use_backing_file` feature [GH-6249]
+* builder/vmware-iso: Add support for disk compaction [GH-6411]
 * builder/vmware-iso: Try to use ISO files uploaded to the datastore when
     building remotely instead of uploading them freshly every time [GH-5165]
 * command/validate: Warn users if config needs fixing. [GH-6423]
+* core: Add a 'split' function to parse template variables. [GH-6357]
+* core: Add a template function allowing users to read keys from consul
+    [GH-6577]
+* core: Add a template function allowing users to read keys from vault
+    [GH-6533]
 * core: Add progress-bar to download step. [GH-5851]
+* core: Create a new root-level Packer template option, "sensitive-variables"
+    which allows users to list which variables they would like to have scrubbed
+    from the Packer logs. [GH-6610]
+* core: Create new config options, "boot_keygroup_interval" and
+    "boot_key_interval" that can be set at the builder-level to supercede
+    PACKER_KEY_INTERVAL for the bootcommand. [GH-6616]
 * core: Deduplicate ui and log lines that stream to terminal [GH-6611]
+* core: Refactor and deduplicate ssh code across builders. This should be a no-
+    op but is a big win for maintainability. [GH-6621] [GH-6613]
+* post-processor/compress: Add support for xz compression [GH-6534]
 * post-processor/vagrant: Support for Docker images. [GH-6494]
+* post-processor/vsphere: Add new `esxi_host` option. [GH-5366]
 * postprocessor/vagrant: Add support for Azure. [GH-6576]
+* provisioner/ansible: Add new "extra var", packer_http_addr. [GH-6501]
 * provisioner/ansible: Enable {{.WinRMPassword}} template engine. [GH-6450]
 * provisioner/shell-local: Create PACKER_HTTP_ADDR environment variable
     [GH-6503]
 
+
 ### BUG FIXES:
 * builder/amazon-ebssurrogate: Clean up volumes at end of build. [GH-6514]
+* builder/amazon: Increase default waiter timeout for AWS
+    WaitUntilImageAvailable command [GH-6601]
+* builder/amazon: Increase the MaxRetries in the Amazon client from the default
+    to 20, to work around users who regularly reach their requestlimit and are
+    being throttled. [GH-6641]
+* builder/amazon: Properly apply environment overrides to our custom-written
+    waiters. [GH-6649]
 * builder/azure: Generated password satisfies Azure password requirements
     [GH-6480]
+* builder/hyper-v: Buider no longer errors if skip_compaction isn't true when
+    skip_export is true, and compaction efficiency is improved [GH-6393]
 * builder/lxc: Correctly pass "config" option to "lxc launch". [GH-6563]
+* builder/lxc: Determine lxc root according to the running user [GH-6543]
+* builder/lxc: Fix file copying for unprivileged LXC containers [GH-6544]
+* builder/oracle-oci: Update OCI sdk, fixing validation bug that occurred when
+    RSA key was encrypted. [GH-6492]
 * builder/vmware-iso: Fix crash caused by invalid datacenter url. [GH-6529]
+* builder/vmware: Maintain original boot order during CreateVMX step for
+    vmware-iso builder [GH-6204]
+* communicator/chroot: Fix quote escaping so that ansible provisioner works
+    properly. [GH-6635]
 * core: Better error handling in downloader when connection error occurs.
     [GH-6557]
 * core: Fix broken pathing checks in checksum files. [GH-6525]
+* provisioner/shell Create new template option allowing users to choose to
+    source env vars from a file rather than declaring them inline. This
+    resolves a bug that occurred when users had complex quoting in their
+    `execute_command`s [GH-6636]
+* provisioner/shell-local: Windows inline scripts now default to being appended
+    with ".cmd", fixing a backwards incompatibility in v1.2.5 [GH-6626]
 * provisioner/windows-restart: Provisioner now works when used in conjuction
     with SSH communicator [GH-6606]
 
 ### BACKWARDS INCOMPATIBILITIES:
 * builder/amazon: "owners" field on source_ami_filter is now required for
     secuirty reasons. [GH-6585]
+* builder/vmware-iso: validation will fail for templates using esxi that have the "disk_type_id" set to something other than "thin" or "" and that do not have "skip_compaction": true also set. Use `packer fix` to fix this. [GH-6411]
 
 ## 1.2.5 (July 16, 2018)
 

Makefile

@@ -11,6 +11,8 @@ GOPATH=$(shell go env GOPATH)
 # gofmt
 UNFORMATTED_FILES=$(shell find . -not -path "./vendor/*" -name "*.go" | xargs gofmt -s -l)
 
+EXECUTABLE_FILES=$(shell find . -type f -perm +111 | egrep -v '^\./(website/vendor|vendor/|\.git|bin/|scripts/|pkg/)' | egrep -v '.*(\.sh|\.bats|\.git)' | egrep -v './provisioner/ansible/test-fixtures/exit1')
+
 # Get the git commit
 GIT_DIRTY=$(shell test -n "`git status --porcelain`" && echo "+CHANGES" || true)
 GIT_COMMIT=$(shell git rev-parse --short HEAD)
@@ -74,6 +76,15 @@ fmt-check: ## Check go code formatting
 		echo "Check passed."; \
 	fi
 
+mode-check: ## Check that only certain files are executable
+	@echo "==> Checking that only certain files are executable..."
+	@if [ ! -z "$(EXECUTABLE_FILES)" ]; then \
+		echo "These files should not be executable or they must be white listed in the Makefile:"; \
+		echo "$(EXECUTABLE_FILES)" | xargs -n1; \
+		exit 1; \
+	else \
+		echo "Check passed."; \
+	fi
 fmt-docs:
 	@find ./website/source/docs -name "*.md" -exec pandoc --wrap auto --columns 79 --atx-headers -s -f "markdown_github+yaml_metadata_block" -t "markdown_github+yaml_metadata_block" {} -o {} \;
 
@@ -89,7 +100,7 @@ generate: deps ## Generate dynamically generated code
 	goimports -w common/bootcommand/boot_command.go
 	gofmt -w command/plugin.go
 
-test: deps fmt-check ## Run unit tests
+test: deps fmt-check mode-check ## Run unit tests
 	@go test $(TEST) $(TESTARGS) -timeout=2m
 	@go tool vet $(VET)  ; if [ $$? -eq 1 ]; then \
 		echo "ERROR: Vet found problems in the code."; \

builder/amazon/chroot/builder.go

@@ -42,6 +42,7 @@ type Config struct {
 	PreMountCommands  []string                   `mapstructure:"pre_mount_commands"`
 	RootDeviceName    string                     `mapstructure:"root_device_name"`
 	RootVolumeSize    int64                      `mapstructure:"root_volume_size"`
+	RootVolumeType    string                     `mapstructure:"root_volume_type"`
 	SourceAmi         string                     `mapstructure:"source_ami"`
 	SourceAmiFilter   awscommon.AmiFilterOptions `mapstructure:"source_ami_filter"`
 	RootVolumeTags    awscommon.TagMap           `mapstructure:"root_volume_tags"`
@@ -223,6 +224,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 				EnableAMISriovNetSupport: b.config.AMISriovNetSupport,
 				EnableAMIENASupport:      b.config.AMIENASupport,
 				AmiFilters:               b.config.SourceAmiFilter,
+				AMIVirtType:              b.config.AMIVirtType,
 			},
 			&StepCheckRootDevice{},
 		)
@@ -232,6 +234,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 		&StepFlock{},
 		&StepPrepareDevice{},
 		&StepCreateVolume{
+			RootVolumeType: b.config.RootVolumeType,
 			RootVolumeSize: b.config.RootVolumeSize,
 			RootVolumeTags: b.config.RootVolumeTags,
 			Ctx:            b.config.ctx,

builder/amazon/chroot/step_create_volume.go

@@ -2,6 +2,7 @@ package chroot
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"log"
 
@@ -21,6 +22,7 @@ import (
 type StepCreateVolume struct {
 	volumeId       string
 	RootVolumeSize int64
+	RootVolumeType string
 	RootVolumeTags awscommon.TagMap
 	Ctx            interpolate.Context
 }
@@ -53,11 +55,21 @@ func (s *StepCreateVolume) Run(ctx context.Context, state multistep.StateBag) mu
 
 	var createVolume *ec2.CreateVolumeInput
 	if config.FromScratch {
+		rootVolumeType := ec2.VolumeTypeGp2
+		if s.RootVolumeType == "io1" {
+			err := errors.New("Cannot use io1 volume when building from scratch")
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		} else if s.RootVolumeType != "" {
+			rootVolumeType = s.RootVolumeType
+		}
 		createVolume = &ec2.CreateVolumeInput{
 			AvailabilityZone: instance.Placement.AvailabilityZone,
 			Size:             aws.Int64(s.RootVolumeSize),
-			VolumeType:       aws.String(ec2.VolumeTypeGp2),
+			VolumeType:       aws.String(rootVolumeType),
 		}
+
 	} else {
 		// Determine the root device snapshot
 		image := state.Get("source_image").(*ec2.Image)
@@ -70,26 +82,13 @@ func (s *StepCreateVolume) Run(ctx context.Context, state multistep.StateBag) mu
 			}
 		}
 
-		if rootDevice == nil {
-			err := fmt.Errorf("Couldn't find root device!")
+		ui.Say("Creating the root volume...")
+		createVolume, err = s.buildCreateVolumeInput(*instance.Placement.AvailabilityZone, rootDevice)
+		if err != nil {
 			state.Put("error", err)
 			ui.Error(err.Error())
 			return multistep.ActionHalt
 		}
-
-		ui.Say("Creating the root volume...")
-		vs := *rootDevice.Ebs.VolumeSize
-		if s.RootVolumeSize > *rootDevice.Ebs.VolumeSize {
-			vs = s.RootVolumeSize
-		}
-
-		createVolume = &ec2.CreateVolumeInput{
-			AvailabilityZone: instance.Placement.AvailabilityZone,
-			Size:             aws.Int64(vs),
-			SnapshotId:       rootDevice.Ebs.SnapshotId,
-			VolumeType:       rootDevice.Ebs.VolumeType,
-			Iops:             rootDevice.Ebs.Iops,
-		}
 	}
 
 	if len(tagSpecs) > 0 {
@@ -137,3 +136,33 @@ func (s *StepCreateVolume) Cleanup(state multistep.StateBag) {
 		ui.Error(fmt.Sprintf("Error deleting EBS volume: %s", err))
 	}
 }
+
+func (s *StepCreateVolume) buildCreateVolumeInput(az string, rootDevice *ec2.BlockDeviceMapping) (*ec2.CreateVolumeInput, error) {
+	if rootDevice == nil {
+		return nil, fmt.Errorf("Couldn't find root device!")
+	}
+	createVolumeInput := &ec2.CreateVolumeInput{
+		AvailabilityZone: aws.String(az),
+		Size:             rootDevice.Ebs.VolumeSize,
+		SnapshotId:       rootDevice.Ebs.SnapshotId,
+		VolumeType:       rootDevice.Ebs.VolumeType,
+		Iops:             rootDevice.Ebs.Iops,
+	}
+	if s.RootVolumeSize > *rootDevice.Ebs.VolumeSize {
+		createVolumeInput.Size = aws.Int64(s.RootVolumeSize)
+	}
+
+	if s.RootVolumeType == "" || s.RootVolumeType == *rootDevice.Ebs.VolumeType {
+		return createVolumeInput, nil
+	}
+
+	if s.RootVolumeType == "io1" {
+		return nil, fmt.Errorf("Root volume type cannot be io1, because existing root volume type was %s", *rootDevice.Ebs.VolumeType)
+	}
+
+	createVolumeInput.VolumeType = aws.String(s.RootVolumeType)
+	// non io1 cannot set iops
+	createVolumeInput.Iops = nil
+
+	return createVolumeInput, nil
+}

builder/amazon/chroot/step_create_volume_test.go

@@ -0,0 +1,73 @@
+package chroot
+
+import (
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func buildTestRootDevice() *ec2.BlockDeviceMapping {
+	return &ec2.BlockDeviceMapping{
+		Ebs: &ec2.EbsBlockDevice{
+			VolumeSize: aws.Int64(10),
+			SnapshotId: aws.String("snap-1234"),
+			VolumeType: aws.String("gp2"),
+		},
+	}
+}
+
+func TestCreateVolume_Default(t *testing.T) {
+	stepCreateVolume := new(StepCreateVolume)
+	_, err := stepCreateVolume.buildCreateVolumeInput("test-az", buildTestRootDevice())
+	assert.NoError(t, err)
+}
+
+func TestCreateVolume_Shrink(t *testing.T) {
+	stepCreateVolume := StepCreateVolume{RootVolumeSize: 1}
+	testRootDevice := buildTestRootDevice()
+	ret, err := stepCreateVolume.buildCreateVolumeInput("test-az", testRootDevice)
+	assert.NoError(t, err)
+	// Ensure that the new value is equal to the size of the old root device
+	assert.Equal(t, *ret.Size, *testRootDevice.Ebs.VolumeSize)
+}
+
+func TestCreateVolume_Expand(t *testing.T) {
+	stepCreateVolume := StepCreateVolume{RootVolumeSize: 25}
+	testRootDevice := buildTestRootDevice()
+	ret, err := stepCreateVolume.buildCreateVolumeInput("test-az", testRootDevice)
+	assert.NoError(t, err)
+	// Ensure that the new value is equal to the size of the value passed in
+	assert.Equal(t, *ret.Size, stepCreateVolume.RootVolumeSize)
+}
+
+func TestCreateVolume_io1_to_io1(t *testing.T) {
+	stepCreateVolume := StepCreateVolume{RootVolumeType: "io1"}
+	testRootDevice := buildTestRootDevice()
+	testRootDevice.Ebs.VolumeType = aws.String("io1")
+	testRootDevice.Ebs.Iops = aws.Int64(1000)
+	ret, err := stepCreateVolume.buildCreateVolumeInput("test-az", testRootDevice)
+	assert.NoError(t, err)
+	assert.Equal(t, *ret.VolumeType, stepCreateVolume.RootVolumeType)
+	assert.Equal(t, *ret.Iops, *testRootDevice.Ebs.Iops)
+}
+
+func TestCreateVolume_io1_to_gp2(t *testing.T) {
+	stepCreateVolume := StepCreateVolume{RootVolumeType: "gp2"}
+	testRootDevice := buildTestRootDevice()
+	testRootDevice.Ebs.VolumeType = aws.String("io1")
+	testRootDevice.Ebs.Iops = aws.Int64(1000)
+
+	ret, err := stepCreateVolume.buildCreateVolumeInput("test-az", testRootDevice)
+	assert.NoError(t, err)
+	assert.Equal(t, *ret.VolumeType, stepCreateVolume.RootVolumeType)
+	assert.Nil(t, ret.Iops)
+}
+
+func TestCreateVolume_gp2_to_io1(t *testing.T) {
+	stepCreateVolume := StepCreateVolume{RootVolumeType: "io1"}
+	testRootDevice := buildTestRootDevice()
+
+	_, err := stepCreateVolume.buildCreateVolumeInput("test-az", testRootDevice)
+	assert.Error(t, err)
+}

builder/amazon/common/access_config.go

@@ -43,6 +43,9 @@ func (c *AccessConfig) Session() (*session.Session, error) {
 		config.WithCredentials(staticCreds)
 	}
 
+	// default is 3, and when it was causing failures for users being throttled
+	config = config.WithMaxRetries(20)
+
 	if c.RawRegion != "" {
 		config = config.WithRegion(c.RawRegion)
 	} else if region := c.metadataRegion(); region != "" {

builder/amazon/common/run_config.go

@@ -34,6 +34,7 @@ func (d *AmiFilterOptions) NoOwner() bool {
 type RunConfig struct {
 	AssociatePublicIpAddress          bool              `mapstructure:"associate_public_ip_address"`
 	AvailabilityZone                  string            `mapstructure:"availability_zone"`
+	BlockDurationMinutes              int64             `mapstructure:"block_duration_minutes"`
 	DisableStopInstance               bool              `mapstructure:"disable_stop_instance"`
 	EbsOptimized                      bool              `mapstructure:"ebs_optimized"`
 	EnableT2Unlimited                 bool              `mapstructure:"enable_t2_unlimited"`
@@ -111,6 +112,11 @@ func (c *RunConfig) Prepare(ctx *interpolate.Context) []error {
 		errs = append(errs, fmt.Errorf("An instance_type must be specified"))
 	}
 
+	if c.BlockDurationMinutes%60 != 0 {
+		errs = append(errs, fmt.Errorf(
+			"block_duration_minutes must be multiple of 60"))
+	}
+
 	if c.SpotPrice == "auto" {
 		if c.SpotPriceAutoProduct == "" {
 			errs = append(errs, fmt.Errorf(

builder/amazon/common/state.go

@@ -169,6 +169,8 @@ func WaitForVolumeToBeAttached(c *ec2.EC2, ctx aws.Context, input *ec2.DescribeV
 			return req, nil
 		},
 	}
+	w.ApplyOptions(opts...)
+
 	return w.WaitWithContext(ctx)
 }
 
@@ -198,6 +200,8 @@ func WaitForVolumeToBeDetached(c *ec2.EC2, ctx aws.Context, input *ec2.DescribeV
 			return req, nil
 		},
 	}
+	w.ApplyOptions(opts...)
+
 	return w.WaitWithContext(ctx)
 }
 
@@ -227,6 +231,8 @@ func WaitForImageToBeImported(c *ec2.EC2, ctx aws.Context, input *ec2.DescribeIm
 			return req, nil
 		},
 	}
+	w.ApplyOptions(opts...)
+
 	return w.WaitWithContext(ctx)
 }
 
@@ -281,11 +287,11 @@ func getOverride(varInfo envInfo) envInfo {
 	return varInfo
 }
 func getEnvOverrides() overridableWaitVars {
-	// Load env vars from environment, and use them to override defaults
+	// Load env vars from environment.
 	envValues := overridableWaitVars{
-		envInfo{"AWS_POLL_DELAY_SECONDS", 2, false},
+		envInfo{"AWS_POLL_DELAY_SECONDS", 0, false},
 		envInfo{"AWS_MAX_ATTEMPTS", 0, false},
-		envInfo{"AWS_TIMEOUT_SECONDS", 300, false},
+		envInfo{"AWS_TIMEOUT_SECONDS", 0, false},
 	}
 
 	envValues.awsMaxAttempts = getOverride(envValues.awsMaxAttempts)

builder/amazon/common/step_get_password.go

@@ -12,6 +12,7 @@ import (
 	"time"
 
 	"github.com/aws/aws-sdk-go/service/ec2"
+	commonhelper "github.com/hashicorp/packer/helper/common"
 	"github.com/hashicorp/packer/helper/communicator"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
@@ -94,13 +95,15 @@ WaitLoop:
 			"Password (since debug is enabled): %s", s.Comm.WinRMPassword))
 	}
 	// store so that we can access this later during provisioning
-	state.Put("winrm_password", s.Comm.WinRMPassword)
+
+	commonhelper.SetSharedState("winrm_password", s.Comm.WinRMPassword, s.BuildName)
 	packer.LogSecretFilter.Set(s.Comm.WinRMPassword)
 
 	return multistep.ActionContinue
 }
 
 func (s *StepGetPassword) Cleanup(multistep.StateBag) {
+	commonhelper.RemoveSharedStateFile("winrm_password", s.BuildName)
 }
 
 func (s *StepGetPassword) waitForPassword(state multistep.StateBag, cancel <-chan struct{}) (string, error) {

builder/amazon/common/step_run_spot_instance.go

@@ -24,6 +24,7 @@ type StepRunSpotInstance struct {
 	AssociatePublicIpAddress          bool
 	AvailabilityZone                  string
 	BlockDevices                      BlockDevices
+	BlockDurationMinutes              int64
 	Debug                             bool
 	Comm                              *communicator.Config
 	EbsOptimized                      bool
@@ -187,8 +188,9 @@ func (s *StepRunSpotInstance) Run(ctx context.Context, state multistep.StateBag)
 	}
 
 	runSpotResp, err := ec2conn.RequestSpotInstances(&ec2.RequestSpotInstancesInput{
-		SpotPrice:           &spotPrice,
-		LaunchSpecification: runOpts,
+		BlockDurationMinutes: &s.BlockDurationMinutes,
+		LaunchSpecification:  runOpts,
+		SpotPrice:            &spotPrice,
 	})
 	if err != nil {
 		err := fmt.Errorf("Error launching source spot instance: %s", err)

builder/amazon/common/step_source_ami_info.go

@@ -21,6 +21,7 @@ type StepSourceAMIInfo struct {
 	SourceAmi                string
 	EnableAMISriovNetSupport bool
 	EnableAMIENASupport      bool
+	AMIVirtType              string
 	AmiFilters               AmiFilterOptions
 }
 
@@ -105,11 +106,13 @@ func (s *StepSourceAMIInfo) Run(_ context.Context, state multistep.StateBag) mul
 
 	// Enhanced Networking can only be enabled on HVM AMIs.
 	// See http://goo.gl/icuXh5
-	if (s.EnableAMIENASupport || s.EnableAMISriovNetSupport) && *image.VirtualizationType != "hvm" {
-		err := fmt.Errorf("Cannot enable enhanced networking, source AMI '%s' is not HVM", s.SourceAmi)
-		state.Put("error", err)
-		ui.Error(err.Error())
-		return multistep.ActionHalt
+	if s.EnableAMIENASupport || s.EnableAMISriovNetSupport {
+		err = s.canEnableEnhancedNetworking(image)
+		if err != nil {
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		}
 	}
 
 	state.Put("source_image", image)
@@ -117,3 +120,16 @@ func (s *StepSourceAMIInfo) Run(_ context.Context, state multistep.StateBag) mul
 }
 
 func (s *StepSourceAMIInfo) Cleanup(multistep.StateBag) {}
+
+func (s *StepSourceAMIInfo) canEnableEnhancedNetworking(image *ec2.Image) error {
+	if s.AMIVirtType == "hvm" {
+		return nil
+	}
+	if s.AMIVirtType != "" {
+		return fmt.Errorf("Cannot enable enhanced networking, AMIVirtType '%s' is not HVM", s.AMIVirtType)
+	}
+	if *image.VirtualizationType != "hvm" {
+		return fmt.Errorf("Cannot enable enhanced networking, source AMI '%s' is not HVM", s.SourceAmi)
+	}
+	return nil
+}

builder/amazon/common/step_source_ami_info_test.go

@@ -0,0 +1,42 @@
+package common
+
+import (
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/ec2"
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func TestStepSourceAmiInfo_PVImage(t *testing.T) {
+	err := new(StepSourceAMIInfo).canEnableEnhancedNetworking(&ec2.Image{
+		VirtualizationType: aws.String("paravirtual"),
+	})
+	assert.Error(t, err)
+}
+
+func TestStepSourceAmiInfo_HVMImage(t *testing.T) {
+	err := new(StepSourceAMIInfo).canEnableEnhancedNetworking(&ec2.Image{
+		VirtualizationType: aws.String("hvm"),
+	})
+	assert.NoError(t, err)
+}
+
+func TestStepSourceAmiInfo_PVImageWithAMIVirtPV(t *testing.T) {
+	stepSourceAMIInfo := StepSourceAMIInfo{
+		AMIVirtType: "paravirtual",
+	}
+	err := stepSourceAMIInfo.canEnableEnhancedNetworking(&ec2.Image{
+		VirtualizationType: aws.String("paravirtual"),
+	})
+	assert.Error(t, err)
+}
+
+func TestStepSourceAmiInfo_PVImageWithAMIVirtHVM(t *testing.T) {
+	stepSourceAMIInfo := StepSourceAMIInfo{
+		AMIVirtType: "hvm",
+	}
+	err := stepSourceAMIInfo.canEnableEnhancedNetworking(&ec2.Image{
+		VirtualizationType: aws.String("paravirtual"),
+	})
+	assert.NoError(t, err)
+}

builder/amazon/ebs/builder.go

@@ -126,6 +126,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			AssociatePublicIpAddress:          b.config.AssociatePublicIpAddress,
 			AvailabilityZone:                  b.config.AvailabilityZone,
 			BlockDevices:                      b.config.BlockDevices,
+			BlockDurationMinutes:              b.config.BlockDurationMinutes,
 			Ctx:                               b.config.ctx,
 			Debug:                             b.config.PackerDebug,
 			EbsOptimized:                      b.config.EbsOptimized,
@@ -178,6 +179,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			EnableAMISriovNetSupport: b.config.AMISriovNetSupport,
 			EnableAMIENASupport:      b.config.AMIENASupport,
 			AmiFilters:               b.config.SourceAmiFilter,
+			AMIVirtType:              b.config.AMIVirtType,
 		},
 		&awscommon.StepKeyPair{
 			Debug:        b.config.PackerDebug,

builder/amazon/ebs/step_create_ami.go

@@ -7,6 +7,7 @@ import (
 
 	"github.com/aws/aws-sdk-go/service/ec2"
 	awscommon "github.com/hashicorp/packer/builder/amazon/common"
+	"github.com/hashicorp/packer/common/random"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
 )
@@ -22,10 +23,17 @@ func (s *stepCreateAMI) Run(ctx context.Context, state multistep.StateBag) multi
 	ui := state.Get("ui").(packer.Ui)
 
 	// Create the image
-	ui.Say(fmt.Sprintf("Creating the AMI: %s", config.AMIName))
+	amiName := config.AMIName
+	if config.AMIEncryptBootVolume {
+		// to avoid having a temporary unencrypted
+		// image named config.AMIName
+		amiName = random.AlphaNum(7)
+	}
+
+	ui.Say(fmt.Sprintf("Creating unencrypted AMI %s from instance %s", amiName, *instance.InstanceId))
 	createOpts := &ec2.CreateImageInput{
 		InstanceId:          instance.InstanceId,
-		Name:                &config.AMIName,
+		Name:                &amiName,
 		BlockDeviceMappings: config.BlockDevices.BuildAMIDevices(),
 	}
 

builder/amazon/ebssurrogate/builder.go

@@ -140,6 +140,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			AssociatePublicIpAddress:          b.config.AssociatePublicIpAddress,
 			AvailabilityZone:                  b.config.AvailabilityZone,
 			BlockDevices:                      b.config.BlockDevices,
+			BlockDurationMinutes:              b.config.BlockDurationMinutes,
 			Ctx:                               b.config.ctx,
 			Debug:                             b.config.PackerDebug,
 			EbsOptimized:                      b.config.EbsOptimized,
@@ -195,6 +196,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			EnableAMISriovNetSupport: b.config.AMISriovNetSupport,
 			EnableAMIENASupport:      b.config.AMIENASupport,
 			AmiFilters:               b.config.SourceAmiFilter,
+			AMIVirtType:              b.config.AMIVirtType,
 		},
 		&awscommon.StepKeyPair{
 			Debug:        b.config.PackerDebug,

builder/amazon/ebsvolume/builder.go

@@ -124,6 +124,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			AssociatePublicIpAddress:          b.config.AssociatePublicIpAddress,
 			AvailabilityZone:                  b.config.AvailabilityZone,
 			BlockDevices:                      b.config.launchBlockDevices,
+			BlockDurationMinutes:              b.config.BlockDurationMinutes,
 			Ctx:                               b.config.ctx,
 			Debug:                             b.config.PackerDebug,
 			EbsOptimized:                      b.config.EbsOptimized,

builder/amazon/instance/builder.go

@@ -210,6 +210,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			AssociatePublicIpAddress: b.config.AssociatePublicIpAddress,
 			AvailabilityZone:         b.config.AvailabilityZone,
 			BlockDevices:             b.config.BlockDevices,
+			BlockDurationMinutes:     b.config.BlockDurationMinutes,
 			Ctx:                      b.config.ctx,
 			Debug:                    b.config.PackerDebug,
 			EbsOptimized:             b.config.EbsOptimized,
@@ -256,6 +257,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			EnableAMISriovNetSupport: b.config.AMISriovNetSupport,
 			EnableAMIENASupport:      b.config.AMIENASupport,
 			AmiFilters:               b.config.SourceAmiFilter,
+			AMIVirtType:              b.config.AMIVirtType,
 		},
 		&awscommon.StepKeyPair{
 			Debug:        b.config.PackerDebug,

builder/azure/arm/config.go

@@ -22,6 +22,7 @@ import (
 	"github.com/hashicorp/packer/builder/azure/common/constants"
 	"github.com/hashicorp/packer/builder/azure/pkcs12"
 	"github.com/hashicorp/packer/common"
+	commonhelper "github.com/hashicorp/packer/helper/common"
 	"github.com/hashicorp/packer/helper/communicator"
 	"github.com/hashicorp/packer/helper/config"
 	"github.com/hashicorp/packer/packer"
@@ -359,7 +360,10 @@ func setRuntimeValues(c *Config) {
 	var tempName = NewTempName()
 
 	c.tmpAdminPassword = tempName.AdminPassword
+	// store so that we can access this later during provisioning
+	commonhelper.SetSharedState("winrm_password", c.tmpAdminPassword, c.PackerConfig.PackerBuildName)
 	packer.LogSecretFilter.Set(c.tmpAdminPassword)
+
 	c.tmpCertificatePassword = tempName.CertificatePassword
 	if c.TempComputeName == "" {
 		c.tmpComputeName = tempName.ComputeName

builder/azure/arm/step_delete_resource_group.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 
 	"github.com/hashicorp/packer/builder/azure/common/constants"
+	retry "github.com/hashicorp/packer/common"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
 )
@@ -93,11 +94,19 @@ func (s *StepDeleteResourceGroup) deleteDeploymentResources(ctx context.Context,
 			resourceType,
 			resourceName))
 
-		err := deleteResource(ctx, s.client,
-			resourceType,
-			resourceName,
-			resourceGroupName)
-		s.reportIfError(err, resourceName)
+		err := retry.Retry(10, 600, 10, func(attempt uint) (bool, error) {
+			err := deleteResource(ctx, s.client,
+				resourceType,
+				resourceName,
+				resourceGroupName)
+			if err != nil {
+				s.reportIfError(err, resourceName)
+				return false, nil
+			}
+
+			return true, nil
+		})
+
 		if err = deploymentOperations.Next(); err != nil {
 			return err
 		}

builder/azure/arm/step_save_winrm_password.go

@@ -3,6 +3,7 @@ package arm
 import (
 	"context"
 
+	commonhelper "github.com/hashicorp/packer/helper/common"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
 )
@@ -14,10 +15,11 @@ type StepSaveWinRMPassword struct {
 
 func (s *StepSaveWinRMPassword) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
 	// store so that we can access this later during provisioning
-	state.Put("winrm_password", s.Password)
+	commonhelper.SetSharedState("winrm_password", s.Password, s.BuildName)
 	packer.LogSecretFilter.Set(s.Password)
 	return multistep.ActionContinue
 }
 
 func (s *StepSaveWinRMPassword) Cleanup(multistep.StateBag) {
+	commonhelper.RemoveSharedStateFile("winrm_password", s.BuildName)
 }

builder/azure/arm/tempname.go

@@ -4,17 +4,7 @@ import (
 	"fmt"
 	"strings"
 
-	"github.com/hashicorp/packer/builder/azure/common"
-)
-
-const (
-	TempNameAlphabet = "0123456789bcdfghjklmnpqrstvwxyz"
-
-	numbers   = "0123456789"
-	lowerCase = "abcdefghijklmnopqrstuvwxyz"
-	upperCase = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
-
-	TempPasswordAlphabet = numbers + lowerCase + upperCase
+	"github.com/hashicorp/packer/common/random"
 )
 
 type TempName struct {
@@ -34,7 +24,7 @@ type TempName struct {
 func NewTempName() *TempName {
 	tempName := &TempName{}
 
-	suffix := common.RandomString(TempNameAlphabet, 10)
+	suffix := random.AlphaNumLower(10)
 	tempName.ComputeName = fmt.Sprintf("pkrvm%s", suffix)
 	tempName.DeploymentName = fmt.Sprintf("pkrdp%s", suffix)
 	tempName.KeyVaultName = fmt.Sprintf("pkrkv%s", suffix)
@@ -46,7 +36,7 @@ func NewTempName() *TempName {
 	tempName.ResourceGroupName = fmt.Sprintf("packer-Resource-Group-%s", suffix)
 
 	tempName.AdminPassword = generatePassword()
-	tempName.CertificatePassword = common.RandomString(TempPasswordAlphabet, 32)
+	tempName.CertificatePassword = random.AlphaNum(32)
 
 	return tempName
 }
@@ -60,16 +50,16 @@ func NewTempName() *TempName {
 func generatePassword() string {
 	var s string
 	for i := 0; i < 100; i++ {
-		s := common.RandomString(TempPasswordAlphabet, 32)
-		if !strings.ContainsAny(s, numbers) {
+		s := random.AlphaNum(32)
+		if !strings.ContainsAny(s, random.PossibleNumbers) {
 			continue
 		}
 
-		if !strings.ContainsAny(s, lowerCase) {
+		if !strings.ContainsAny(s, random.PossibleLowerCase) {
 			continue
 		}
 
-		if !strings.ContainsAny(s, upperCase) {
+		if !strings.ContainsAny(s, random.PossibleUpperCase) {
 			continue
 		}
 

builder/azure/arm/tempname_test.go

@@ -3,6 +3,8 @@ package arm
 import (
 	"strings"
 	"testing"
+
+	"github.com/hashicorp/packer/common/random"
 )
 
 func TestTempNameShouldCreatePrefixedRandomNames(t *testing.T) {
@@ -44,14 +46,14 @@ func TestTempNameShouldCreatePrefixedRandomNames(t *testing.T) {
 func TestTempAdminPassword(t *testing.T) {
 	tempName := NewTempName()
 
-	if !strings.ContainsAny(tempName.AdminPassword, numbers) {
-		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", numbers)
+	if !strings.ContainsAny(tempName.AdminPassword, random.PossibleNumbers) {
+		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", random.PossibleNumbers)
 	}
-	if !strings.ContainsAny(tempName.AdminPassword, lowerCase) {
-		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", lowerCase)
+	if !strings.ContainsAny(tempName.AdminPassword, random.PossibleLowerCase) {
+		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", random.PossibleLowerCase)
 	}
-	if !strings.ContainsAny(tempName.AdminPassword, upperCase) {
-		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", upperCase)
+	if !strings.ContainsAny(tempName.AdminPassword, random.PossibleUpperCase) {
+		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", random.PossibleUpperCase)
 	}
 }
 

builder/azure/common/randomstring.go

@@ -1,45 +0,0 @@
-package common
-
-import (
-	"math/rand"
-	"os"
-	"time"
-)
-
-var pwSymbols = []string{
-	"abcdefghijklmnopqrstuvwxyz",
-	"ABCDEFGHIJKLMNOPQRSTUVWXYZ",
-	"0123456789",
-}
-
-var rnd = rand.New(rand.NewSource(time.Now().UnixNano() + int64(os.Getpid())))
-
-func RandomString(chooseFrom string, length int) (randomString string) {
-	cflen := len(chooseFrom)
-	for i := 0; i < length; i++ {
-		randomString += string(chooseFrom[rnd.Intn(cflen)])
-	}
-	return
-}
-
-func RandomPassword() (password string) {
-	pwlen := 15
-	batchsize := pwlen / len(pwSymbols)
-	pw := make([]byte, 0, pwlen)
-	// choose character set
-	for c := 0; len(pw) < pwlen; c++ {
-		s := RandomString(pwSymbols[c%len(pwSymbols)], rnd.Intn(batchsize-1)+1)
-		pw = append(pw, []byte(s)...)
-	}
-	// truncate
-	pw = pw[:pwlen]
-
-	// permute
-	for c := 0; c < pwlen-1; c++ {
-		i := rnd.Intn(pwlen-c) + c
-		x := pw[c]
-		pw[c] = pw[i]
-		pw[i] = x
-	}
-	return string(pw)
-}

builder/azure/common/randomstring_test.go

@@ -1,15 +0,0 @@
-package common
-
-import (
-	"testing"
-)
-
-func TestRandomPassword_generates_15char_passwords(t *testing.T) {
-	for i := 0; i < 100; i++ {
-		pw := RandomPassword()
-		t.Logf("pw: %v", pw)
-		if len(pw) != 15 {
-			t.Fatalf("len(pw)!=15, but %v: %v (%v)", len(pw), pw, i)
-		}
-	}
-}

builder/docker/communicator_test.go

@@ -68,8 +68,8 @@ func TestUploadDownload(t *testing.T) {
 	hooks[packer.HookProvision] = []packer.Hook{
 		&packer.ProvisionHook{
 			Provisioners: []*packer.HookedProvisioner{
-				{upload, nil, ""},
-				{download, nil, ""},
+				{Provisioner: upload, Config: nil, TypeName: ""},
+				{Provisioner: download, Config: nil, TypeName: ""},
 			},
 		},
 	}
@@ -157,9 +157,9 @@ func TestLargeDownload(t *testing.T) {
 	hooks[packer.HookProvision] = []packer.Hook{
 		&packer.ProvisionHook{
 			Provisioners: []*packer.HookedProvisioner{
-				{shell, nil, ""},
-				{downloadCupcake, nil, ""},
-				{downloadBigcake, nil, ""},
+				{Provisioner: shell, Config: nil, TypeName: ""},
+				{Provisioner: downloadCupcake, Config: nil, TypeName: ""},
+				{Provisioner: downloadBigcake, Config: nil, TypeName: ""},
 			},
 		},
 	}
@@ -266,10 +266,10 @@ func TestFixUploadOwner(t *testing.T) {
 	hooks[packer.HookProvision] = []packer.Hook{
 		&packer.ProvisionHook{
 			Provisioners: []*packer.HookedProvisioner{
-				{fileProvisioner, nil, ""},
-				{dirProvisioner, nil, ""},
-				{shellProvisioner, nil, ""},
-				{verifyProvisioner, nil, ""},
+				{Provisioner: fileProvisioner, Config: nil, TypeName: ""},
+				{Provisioner: dirProvisioner, Config: nil, TypeName: ""},
+				{Provisioner: shellProvisioner, Config: nil, TypeName: ""},
+				{Provisioner: verifyProvisioner, Config: nil, TypeName: ""},
 			},
 		},
 	}

builder/googlecompute/driver_gce.go

@@ -168,7 +168,27 @@ func (d *driverGCE) DeleteDisk(zone, name string) (<-chan error, error) {
 }
 
 func (d *driverGCE) GetImage(name string, fromFamily bool) (*Image, error) {
-	projects := []string{d.projectId, "centos-cloud", "coreos-cloud", "cos-cloud", "debian-cloud", "google-containers", "opensuse-cloud", "rhel-cloud", "suse-cloud", "ubuntu-os-cloud", "windows-cloud", "gce-nvme", "windows-sql-cloud", "rhel-sap-cloud"}
+	projects := []string{
+		d.projectId,
+		// Public projects, drawn from
+		// https://cloud.google.com/compute/docs/images
+		"centos-cloud",
+		"cos-cloud",
+		"coreos-cloud",
+		"debian-cloud",
+		"rhel-cloud",
+		"rhel-sap-cloud",
+		"suse-cloud",
+		"suse-sap-cloud",
+		"ubuntu-os-cloud",
+		"windows-cloud",
+		"windows-sql-cloud",
+		"gce-uefi-images",
+		"gce-nvme",
+		// misc
+		"google-containers",
+		"opensuse-cloud",
+	}
 	var errs error
 	for _, project := range projects {
 		image, err := d.GetImageFromProject(project, name, fromFamily)

builder/googlecompute/step_create_windows_password.go

@@ -13,6 +13,7 @@ import (
 	"os"
 	"time"
 
+	commonhelper "github.com/hashicorp/packer/helper/common"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
 )
@@ -113,6 +114,7 @@ func (s *StepCreateWindowsPassword) Run(_ context.Context, state multistep.State
 	}
 
 	state.Put("winrm_password", data.password)
+	commonhelper.SetSharedState("winrm_password", data.password, c.PackerConfig.PackerBuildName)
 	packer.LogSecretFilter.Set(data.password)
 
 	return multistep.ActionContinue

builder/oracle/oci/step_get_default_credentials.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"log"
 
+	commonhelper "github.com/hashicorp/packer/helper/common"
 	"github.com/hashicorp/packer/helper/communicator"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
@@ -51,9 +52,8 @@ func (s *stepGetDefaultCredentials) Run(ctx context.Context, state multistep.Sta
 	}
 
 	// store so that we can access this later during provisioning
-	state.Put("winrm_password", s.Comm.WinRMPassword)
+	commonhelper.SetSharedState("winrm_password", s.Comm.WinRMPassword, s.BuildName)
 	packer.LogSecretFilter.Set(s.Comm.WinRMPassword)
-
 	return multistep.ActionContinue
 }
 

builder/vmware/iso/builder.go

@@ -139,6 +139,14 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 		b.config.DiskAdapterType = "lsilogic"
 	}
 
+	if !b.config.SkipCompaction {
+		if b.config.RemoteType == "esx5" {
+			if b.config.DiskTypeId == "" {
+				b.config.SkipCompaction = true
+			}
+		}
+	}
+
 	if b.config.DiskTypeId == "" {
 		// Default is growable virtual disk split in 2GB files.
 		b.config.DiskTypeId = "1"
@@ -148,6 +156,13 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 		}
 	}
 
+	if b.config.RemoteType == "esx5" {
+		if b.config.DiskTypeId != "thin" && !b.config.SkipCompaction {
+			errs = packer.MultiErrorAppend(
+				errs, fmt.Errorf("skip_compaction must be 'true' for disk_type_id: %s", b.config.DiskTypeId))
+		}
+	}
+
 	if b.config.GuestOSType == "" {
 		b.config.GuestOSType = "other"
 	}

builder/vmware/iso/driver_esx5.go

@@ -46,7 +46,8 @@ func (d *ESX5Driver) Clone(dst, src string, linked bool) error {
 }
 
 func (d *ESX5Driver) CompactDisk(diskPathLocal string) error {
-	return nil
+	diskPath := d.datastorePath(diskPathLocal)
+	return d.sh("vmkfstools", "--punchzero", diskPath)
 }
 
 func (d *ESX5Driver) CreateDisk(diskPathLocal string, size string, adapter_type string, typeId string) error {

command/build.go

@@ -294,16 +294,16 @@ Usage: packer build [options] TEMPLATE
 
 Options:
 
-  -color=false               Disable color output (on by default)
-  -debug                     Debug mode enabled for builds
-  -except=foo,bar,baz        Build all builds other than these
-  -only=foo,bar,baz          Build only the specified builds
-  -force                     Force a build to continue if artifacts exist, deletes existing artifacts
-  -machine-readable          Machine-readable output
+  -color=false                  Disable color output (on by default)
+  -debug                        Debug mode enabled for builds
+  -except=foo,bar,baz           Build all builds other than these
+  -only=foo,bar,baz             Build only the specified builds
+  -force                        Force a build to continue if artifacts exist, deletes existing artifacts
+  -machine-readable             Machine-readable output
   -on-error=[cleanup|abort|ask] If the build fails do: clean up (default), abort, or ask
-  -parallel=false            Disable parallelization (on by default)
-  -var 'key=value'           Variable for templates, can be used multiple times.
-  -var-file=path             JSON file containing user variables.
+  -parallel=false               Disable parallelization (on by default)
+  -var 'key=value'              Variable for templates, can be used multiple times.
+  -var-file=path                JSON file containing user variables.
 `
 
 	return strings.TrimSpace(helpText)

command/fix.go

@@ -157,6 +157,8 @@ Fixes that are run:
                              Hyper-V ISO builder templates
   hyperv-vmxc-typo           Corrects a typo in the "clone_from_vmxc_path"
                              setting. Replaces with "clone_from_vmcx_path".
+  vmware-compaction          Adds "skip_compaction = true" to "vmware-iso"
+                             builders with incompatible disk_type_id
 
 Options:
 

command/fix_test.go

@@ -2,18 +2,39 @@ package command
 
 import (
 	"path/filepath"
+	"strings"
 	"testing"
+
+	"github.com/hashicorp/packer/packer"
+	"github.com/stretchr/testify/assert"
 )
 
 func TestFix(t *testing.T) {
+	s := &strings.Builder{}
+	ui := &packer.BasicUi{
+		Writer: s,
+	}
 	c := &FixCommand{
 		Meta: testMeta(t),
 	}
 
+	c.Ui = ui
+
 	args := []string{filepath.Join(testFixture("fix"), "template.json")}
 	if code := c.Run(args); code != 0 {
 		fatalCommand(t, c.Meta)
 	}
+	expected := `{
+  "builders": [
+    {
+      "type": "dummy"
+    }
+  ],
+  "push": {
+    "name": "foo/bar"
+  }
+}`
+	assert.Equal(t, expected, strings.TrimSpace(s.String()))
 }
 
 func TestFix_invalidTemplate(t *testing.T) {

common/config.go

@@ -9,6 +9,8 @@ import (
 	"runtime"
 	"strings"
 	"time"
+
+	"github.com/hashicorp/packer/packer"
 )
 
 // PackerKeyEnv is used to specify the key interval (delay) between keystrokes
@@ -41,7 +43,7 @@ func SupportedProtocol(u *url.URL) bool {
 
 	// build a dummy NewDownloadClient since this is the only place that valid
 	// protocols are actually exposed.
-	cli := NewDownloadClient(&DownloadConfig{}, nil)
+	cli := NewDownloadClient(&DownloadConfig{}, new(packer.NoopUi))
 
 	// Iterate through each downloader to see if a protocol was found.
 	ok := false
@@ -173,7 +175,7 @@ func FileExistsLocally(original string) bool {
 
 	// First create a dummy downloader so we can figure out which
 	// protocol to use.
-	cli := NewDownloadClient(&DownloadConfig{}, nil)
+	cli := NewDownloadClient(&DownloadConfig{}, new(packer.NoopUi))
 	d, ok := cli.config.DownloaderMap[u.Scheme]
 	if !ok {
 		return false

common/download.go

@@ -10,19 +10,17 @@ import (
 	"errors"
 	"fmt"
 	"hash"
+	"io"
 	"log"
+	"net/http"
 	"net/url"
 	"os"
 	"path"
+	"path/filepath"
 	"runtime"
 	"strings"
-)
 
-// imports related to each Downloader implementation
-import (
-	"io"
-	"net/http"
-	"path/filepath"
+	"github.com/hashicorp/packer/packer"
 )
 
 // DownloadConfig is the configuration given to instantiate a new
@@ -58,8 +56,7 @@ type DownloadConfig struct {
 
 // A DownloadClient helps download, verify checksums, etc.
 type DownloadClient struct {
-	config     *DownloadConfig
-	downloader Downloader
+	config *DownloadConfig
 }
 
 // HashForType returns the Hash implementation for the given string
@@ -81,33 +78,24 @@ func HashForType(t string) hash.Hash {
 
 // NewDownloadClient returns a new DownloadClient for the given
 // configuration.
-func NewDownloadClient(c *DownloadConfig, bar ProgressBar) *DownloadClient {
-	const mtu = 1500 /* ethernet */ - 20 /* ipv4 */ - 20 /* tcp */
-
-	// If bar is nil, then use a dummy progress bar that doesn't do anything
-	if bar == nil {
-		bar = GetDummyProgressBar()
-	}
-
+func NewDownloadClient(c *DownloadConfig, ui packer.Ui) *DownloadClient {
 	// Create downloader map if it hasn't been specified already.
 	if c.DownloaderMap == nil {
 		c.DownloaderMap = map[string]Downloader{
-			"file":  &FileDownloader{progress: bar, bufferSize: nil},
-			"http":  &HTTPDownloader{progress: bar, userAgent: c.UserAgent},
-			"https": &HTTPDownloader{progress: bar, userAgent: c.UserAgent},
-			"smb":   &SMBDownloader{progress: bar, bufferSize: nil},
+			"file":  &FileDownloader{Ui: ui, bufferSize: nil},
+			"http":  &HTTPDownloader{Ui: ui, userAgent: c.UserAgent},
+			"https": &HTTPDownloader{Ui: ui, userAgent: c.UserAgent},
+			"smb":   &SMBDownloader{Ui: ui, bufferSize: nil},
 		}
 	}
 	return &DownloadClient{config: c}
 }
 
-// A downloader implements the ability to transfer a file, and cancel or resume
-//	it.
+// Downloader defines what capabilities a downloader should have.
 type Downloader interface {
 	Resume()
 	Cancel()
-	Progress() uint64
-	Total() uint64
+	ProgressBar() packer.ProgressBar
 }
 
 // A LocalDownloader is responsible for converting a uri to a local path
@@ -150,17 +138,17 @@ func (d *DownloadClient) Get() (string, error) {
 	var finalPath string
 
 	var ok bool
-	d.downloader, ok = d.config.DownloaderMap[u.Scheme]
+	downloader, ok := d.config.DownloaderMap[u.Scheme]
 	if !ok {
 		return "", fmt.Errorf("No downloader for scheme: %s", u.Scheme)
 	}
 
-	remote, ok := d.downloader.(RemoteDownloader)
+	remote, ok := downloader.(RemoteDownloader)
 	if !ok {
-		return "", fmt.Errorf("Unable to treat uri scheme %s as a Downloader. : %T", u.Scheme, d.downloader)
+		return "", fmt.Errorf("Unable to treat uri scheme %s as a Downloader. : %T", u.Scheme, downloader)
 	}
 
-	local, ok := d.downloader.(LocalDownloader)
+	local, ok := downloader.(LocalDownloader)
 	if !ok && !d.config.CopyFile {
 		d.config.CopyFile = true
 	}
@@ -233,11 +221,9 @@ func (d *DownloadClient) VerifyChecksum(path string) (bool, error) {
 // HTTPDownloader is an implementation of Downloader that downloads
 // files over HTTP.
 type HTTPDownloader struct {
-	current   uint64
-	total     uint64
 	userAgent string
 
-	progress ProgressBar
+	Ui packer.Ui
 }
 
 func (d *HTTPDownloader) Cancel() {
@@ -256,8 +242,7 @@ func (d *HTTPDownloader) Download(dst *os.File, src *url.URL) error {
 		return err
 	}
 
-	// Reset our progress
-	d.current = 0
+	var current int64
 
 	// Make the request. We first make a HEAD request so we can check
 	// if the server supports range queries. If the server/URL doesn't
@@ -301,7 +286,7 @@ func (d *HTTPDownloader) Download(dst *os.File, src *url.URL) error {
 					if _, err = dst.Seek(0, os.SEEK_END); err == nil {
 						req.Header.Set("Range", fmt.Sprintf("bytes=%d-", fi.Size()))
 
-						d.current = uint64(fi.Size())
+						current = fi.Size()
 					}
 				}
 			}
@@ -328,23 +313,22 @@ func (d *HTTPDownloader) Download(dst *os.File, src *url.URL) error {
 		return fmt.Errorf("HTTP error: %s", err.Error())
 	}
 
-	d.total = d.current + uint64(resp.ContentLength)
+	total := current + resp.ContentLength
 
-	bar := d.progress
-	bar.SetTotal64(int64(d.total))
-	progressBar := bar.Start()
-	progressBar.Set64(int64(d.current))
+	bar := d.ProgressBar()
+	bar.Start(total)
+	defer bar.Finish()
+	bar.Add(current)
+
+	body := bar.NewProxyReader(resp.Body)
 
 	var buffer [4096]byte
 	for {
-		n, err := resp.Body.Read(buffer[:])
+		n, err := body.Read(buffer[:])
 		if err != nil && err != io.EOF {
 			return err
 		}
 
-		d.current += uint64(n)
-		progressBar.Set64(int64(d.current))
-
 		if _, werr := dst.Write(buffer[:n]); werr != nil {
 			return werr
 		}
@@ -353,36 +337,16 @@ func (d *HTTPDownloader) Download(dst *os.File, src *url.URL) error {
 			break
 		}
 	}
-	progressBar.Finish()
 	return nil
 }
 
-func (d *HTTPDownloader) Progress() uint64 {
-	return d.current
-}
-
-func (d *HTTPDownloader) Total() uint64 {
-	return d.total
-}
-
 // FileDownloader is an implementation of Downloader that downloads
 // files using the regular filesystem.
 type FileDownloader struct {
 	bufferSize *uint
 
-	active  bool
-	current uint64
-	total   uint64
-
-	progress ProgressBar
-}
-
-func (d *FileDownloader) Progress() uint64 {
-	return d.current
-}
-
-func (d *FileDownloader) Total() uint64 {
-	return d.total
+	active bool
+	Ui     packer.Ui
 }
 
 func (d *FileDownloader) Cancel() {
@@ -449,7 +413,6 @@ func (d *FileDownloader) Download(dst *os.File, src *url.URL) error {
 	}
 
 	/* download the file using the operating system's facilities */
-	d.current = 0
 	d.active = true
 
 	f, err := os.Open(realpath)
@@ -463,44 +426,37 @@ func (d *FileDownloader) Download(dst *os.File, src *url.URL) error {
 	if err != nil {
 		return err
 	}
-	d.total = uint64(fi.Size())
 
-	bar := d.progress
-	bar.SetTotal64(int64(d.total))
-	progressBar := bar.Start()
-	progressBar.Set64(int64(d.current))
+	bar := d.ProgressBar()
+
+	bar.Start(fi.Size())
+	defer bar.Finish()
+	fProxy := bar.NewProxyReader(f)
 
 	// no bufferSize specified, so copy synchronously.
 	if d.bufferSize == nil {
-		var n int64
-		n, err = io.Copy(dst, f)
+		_, err = io.Copy(dst, fProxy)
 		d.active = false
 
-		d.current += uint64(n)
-		progressBar.Set64(int64(d.current))
-
 		// use a goro in case someone else wants to enable cancel/resume
 	} else {
 		errch := make(chan error)
 		go func(d *FileDownloader, r io.Reader, w io.Writer, e chan error) {
 			for d.active {
-				n, err := io.CopyN(w, r, int64(*d.bufferSize))
+				_, err := io.CopyN(w, r, int64(*d.bufferSize))
 				if err != nil {
 					break
 				}
 
-				d.current += uint64(n)
-				progressBar.Set64(int64(d.current))
 			}
 			d.active = false
 			e <- err
-		}(d, f, dst, errch)
+		}(d, fProxy, dst, errch)
 
 		// ...and we spin until it's done
 		err = <-errch
 	}
-	progressBar.Finish()
-	f.Close()
+
 	return err
 }
 
@@ -509,19 +465,8 @@ func (d *FileDownloader) Download(dst *os.File, src *url.URL) error {
 type SMBDownloader struct {
 	bufferSize *uint
 
-	active  bool
-	current uint64
-	total   uint64
-
-	progress ProgressBar
-}
-
-func (d *SMBDownloader) Progress() uint64 {
-	return d.current
-}
-
-func (d *SMBDownloader) Total() uint64 {
-	return d.total
+	active bool
+	Ui     packer.Ui
 }
 
 func (d *SMBDownloader) Cancel() {
@@ -570,7 +515,6 @@ func (d *SMBDownloader) Download(dst *os.File, src *url.URL) error {
 	}
 
 	/* Open up the "\\"-prefixed path using the Windows filesystem */
-	d.current = 0
 	d.active = true
 
 	f, err := os.Open(realpath)
@@ -584,43 +528,39 @@ func (d *SMBDownloader) Download(dst *os.File, src *url.URL) error {
 	if err != nil {
 		return err
 	}
-	d.total = uint64(fi.Size())
 
-	bar := d.progress
-	bar.SetTotal64(int64(d.total))
-	progressBar := bar.Start()
-	progressBar.Set64(int64(d.current))
+	bar := d.ProgressBar()
+
+	bar.Start(fi.Size())
+	defer bar.Finish()
+	fProxy := bar.NewProxyReader(f)
 
 	// no bufferSize specified, so copy synchronously.
 	if d.bufferSize == nil {
-		var n int64
-		n, err = io.Copy(dst, f)
+		_, err = io.Copy(dst, fProxy)
 		d.active = false
 
-		d.current += uint64(n)
-		progressBar.Set64(int64(d.current))
-
 		// use a goro in case someone else wants to enable cancel/resume
 	} else {
 		errch := make(chan error)
 		go func(d *SMBDownloader, r io.Reader, w io.Writer, e chan error) {
 			for d.active {
-				n, err := io.CopyN(w, r, int64(*d.bufferSize))
+				_, err := io.CopyN(w, r, int64(*d.bufferSize))
 				if err != nil {
 					break
 				}
 
-				d.current += uint64(n)
-				progressBar.Set64(int64(d.current))
 			}
 			d.active = false
 			e <- err
-		}(d, f, dst, errch)
+		}(d, fProxy, dst, errch)
 
 		// ...and as usual we spin until it's done
 		err = <-errch
 	}
-	progressBar.Finish()
-	f.Close()
 	return err
 }
+
+func (d *HTTPDownloader) ProgressBar() packer.ProgressBar { return d.Ui.ProgressBar() }
+func (d *FileDownloader) ProgressBar() packer.ProgressBar { return d.Ui.ProgressBar() }
+func (d *SMBDownloader) ProgressBar() packer.ProgressBar  { return d.Ui.ProgressBar() }

common/download_test.go

@@ -12,6 +12,8 @@ import (
 	"runtime"
 	"strings"
 	"testing"
+
+	"github.com/hashicorp/packer/packer"
 )
 
 func TestDownloadClientVerifyChecksum(t *testing.T) {
@@ -36,7 +38,7 @@ func TestDownloadClientVerifyChecksum(t *testing.T) {
 		Checksum: checksum,
 	}
 
-	d := NewDownloadClient(config, nil)
+	d := NewDownloadClient(config, new(packer.NoopUi))
 	result, err := d.VerifyChecksum(tf.Name())
 	if err != nil {
 		t.Fatalf("Verify err: %s", err)
@@ -59,7 +61,7 @@ func TestDownloadClient_basic(t *testing.T) {
 		Url:        ts.URL + "/basic.txt",
 		TargetPath: tf.Name(),
 		CopyFile:   true,
-	}, nil)
+	}, new(packer.NoopUi))
 
 	path, err := client.Get()
 	if err != nil {
@@ -95,7 +97,7 @@ func TestDownloadClient_checksumBad(t *testing.T) {
 		Hash:       HashForType("md5"),
 		Checksum:   checksum,
 		CopyFile:   true,
-	}, nil)
+	}, new(packer.NoopUi))
 
 	if _, err := client.Get(); err == nil {
 		t.Fatal("should error")
@@ -121,7 +123,7 @@ func TestDownloadClient_checksumGood(t *testing.T) {
 		Hash:       HashForType("md5"),
 		Checksum:   checksum,
 		CopyFile:   true,
-	}, nil)
+	}, new(packer.NoopUi))
 
 	path, err := client.Get()
 	if err != nil {
@@ -153,7 +155,7 @@ func TestDownloadClient_checksumNoDownload(t *testing.T) {
 		Hash:       HashForType("md5"),
 		Checksum:   checksum,
 		CopyFile:   true,
-	}, nil)
+	}, new(packer.NoopUi))
 	path, err := client.Get()
 	if err != nil {
 		t.Fatalf("err: %s", err)
@@ -183,7 +185,7 @@ func TestDownloadClient_notFound(t *testing.T) {
 	client := NewDownloadClient(&DownloadConfig{
 		Url:        ts.URL + "/not-found.txt",
 		TargetPath: tf.Name(),
-	}, nil)
+	}, new(packer.NoopUi))
 
 	if _, err := client.Get(); err == nil {
 		t.Fatal("should error")
@@ -211,7 +213,7 @@ func TestDownloadClient_resume(t *testing.T) {
 		Url:        ts.URL,
 		TargetPath: tf.Name(),
 		CopyFile:   true,
-	}, nil)
+	}, new(packer.NoopUi))
 
 	path, err := client.Get()
 	if err != nil {
@@ -273,7 +275,7 @@ func TestDownloadClient_usesDefaultUserAgent(t *testing.T) {
 		CopyFile:   true,
 	}
 
-	client := NewDownloadClient(config, nil)
+	client := NewDownloadClient(config, new(packer.NoopUi))
 	_, err = client.Get()
 	if err != nil {
 		t.Fatal(err)
@@ -306,7 +308,7 @@ func TestDownloadClient_setsUserAgent(t *testing.T) {
 		CopyFile:   true,
 	}
 
-	client := NewDownloadClient(config, nil)
+	client := NewDownloadClient(config, new(packer.NoopUi))
 	_, err = client.Get()
 	if err != nil {
 		t.Fatal(err)
@@ -405,7 +407,7 @@ func TestDownloadFileUrl(t *testing.T) {
 		CopyFile: false,
 	}
 
-	client := NewDownloadClient(config, nil)
+	client := NewDownloadClient(config, new(packer.NoopUi))
 
 	// Verify that we fail to match the checksum
 	_, err = client.Get()
@@ -436,7 +438,7 @@ func SimulateFileUriDownload(t *testing.T, uri string) (string, error) {
 	}
 
 	// go go go
-	client := NewDownloadClient(config, nil)
+	client := NewDownloadClient(config, new(packer.NoopUi))
 	path, err := client.Get()
 
 	// ignore any non-important checksum errors if it's not a unc path

common/progress.go

@@ -1,145 +0,0 @@
-package common
-
-import (
-	"fmt"
-	"github.com/cheggaaa/pb"
-	"github.com/hashicorp/packer/helper/multistep"
-	"github.com/hashicorp/packer/packer"
-	"github.com/hashicorp/packer/packer/rpc"
-	"log"
-	"reflect"
-	"time"
-)
-
-// This is the arrow from packer/ui.go -> TargetedUI.prefixLines
-const targetedUIArrowText = "==>"
-
-// The ProgressBar interface is used for abstracting cheggaaa's progress-
-// bar, or any other progress bar. If a UI does not support a progress-
-// bar, then it must return a null progress bar.
-const (
-	DefaultProgressBarWidth = 80
-)
-
-type ProgressBar = *pb.ProgressBar
-
-// Figure out the terminal dimensions and use it to calculate the available rendering space
-func calculateProgressBarWidth(length int) int {
-	// If the UI's width is signed, then this is an interface that doesn't really benefit from a progress bar
-	if length < 0 {
-		log.Println("Refusing to render progress-bar for unsupported UI.")
-		return length
-	}
-
-	// Figure out the terminal width if possible
-	width, _, err := GetTerminalDimensions()
-	if err != nil {
-		newerr := fmt.Errorf("Unable to determine terminal dimensions: %v", err)
-		log.Printf("Using default width (%d) for progress-bar due to error: %s", DefaultProgressBarWidth, newerr)
-		return DefaultProgressBarWidth
-	}
-
-	// If the terminal width is smaller than the requested length, then complain
-	if width < length {
-		newerr := fmt.Errorf("Terminal width (%d) is smaller than UI message width (%d).", width, length)
-		log.Printf("Using default width (%d) for progress-bar due to error: %s", DefaultProgressBarWidth, newerr)
-		return DefaultProgressBarWidth
-	}
-
-	// Otherwise subtract the minimum length and return it
-	return width - length
-}
-
-// Get a progress bar with the default appearance
-func GetDefaultProgressBar() ProgressBar {
-	bar := pb.New64(0)
-	bar.ShowPercent = true
-	bar.ShowCounters = true
-	bar.ShowSpeed = false
-	bar.ShowBar = true
-	bar.ShowTimeLeft = false
-	bar.ShowFinalTime = false
-	bar.SetUnits(pb.U_BYTES)
-	bar.Format("[=>-]")
-	bar.SetRefreshRate(5 * time.Second)
-	return bar
-}
-
-// Return a dummy progress bar that doesn't do anything
-func GetDummyProgressBar() ProgressBar {
-	bar := pb.New64(0)
-	bar.ManualUpdate = true
-	return bar
-}
-
-// Given a packer.Ui, calculate the number of characters that a packer.Ui will
-// prefix a message with. Then we can use this to calculate the progress bar's width.
-func calculateUiPrefixLength(ui packer.Ui) int {
-	var recursiveCalculateUiPrefixLength func(packer.Ui, int) int
-
-	// Define a recursive closure that traverses through all the known packer.Ui types
-	// and aggregates the length of the message prefix from each particular type
-	recursiveCalculateUiPrefixLength = func(ui packer.Ui, agg int) int {
-		switch ui.(type) {
-
-		case *packer.ColoredUi:
-			// packer.ColoredUi is simply a wrapper around .Ui
-			u := ui.(*packer.ColoredUi)
-			return recursiveCalculateUiPrefixLength(u.Ui, agg)
-
-		case *packer.TargetedUI:
-			// A TargetedUI adds the .Target and an arrow by default
-			u := ui.(*packer.TargetedUI)
-			res := fmt.Sprintf("%s %s: ", targetedUIArrowText, u.Target)
-			return recursiveCalculateUiPrefixLength(u.Ui, agg+len(res))
-
-		case *packer.BasicUi:
-			// The standard BasicUi appends only a newline
-			return agg + len("\n")
-
-		// packer.rpc.Ui returns 0 here to trigger the hack described later
-		case *rpc.Ui:
-			return 0
-
-		case *packer.MachineReadableUi:
-			// MachineReadableUi doesn't emit anything...like at all
-			return 0
-		}
-
-		log.Printf("Calculating the message prefix length for packer.Ui type (%T) is not implemented. Using the current aggregated length of %d.", ui, agg)
-		return agg
-	}
-	return recursiveCalculateUiPrefixLength(ui, 0)
-}
-
-func GetPackerConfigFromStateBag(state multistep.StateBag) *PackerConfig {
-	config := state.Get("config")
-	rConfig := reflect.Indirect(reflect.ValueOf(config))
-	iPackerConfig := rConfig.FieldByName("PackerConfig").Interface()
-	packerConfig := iPackerConfig.(PackerConfig)
-	return &packerConfig
-}
-
-func GetProgressBar(ui packer.Ui, config *PackerConfig) ProgressBar {
-	// Figure out the prefix length by quering the UI
-	uiPrefixLength := calculateUiPrefixLength(ui)
-
-	// hack to deal with packer.rpc.Ui courtesy of @Swampdragons
-	if _, ok := ui.(*rpc.Ui); uiPrefixLength == 0 && config != nil && ok {
-		res := fmt.Sprintf("%s %s: \n", targetedUIArrowText, config.PackerBuildName)
-		uiPrefixLength = len(res)
-	}
-
-	// Now we can use the prefix length to calculate the progress bar width
-	width := calculateProgressBarWidth(uiPrefixLength)
-
-	log.Printf("ProgressBar: Using progress bar width: %d\n", width)
-
-	// Get a default progress bar and set some output defaults
-	bar := GetDefaultProgressBar()
-	bar.SetWidth(width)
-	bar.Callback = func(message string) {
-		ui.Message(message)
-	}
-	return bar
-}

common/progress_test.go

@@ -1,142 +0,0 @@
-package common
-
-import (
-	"github.com/hashicorp/packer/packer"
-	"testing"
-)
-
-// test packer.Ui implementation to verify that progress bar is being written
-type testProgressBarUi struct {
-	messageCalled  bool
-	messageMessage string
-}
-
-func (u *testProgressBarUi) Say(string)                {}
-func (u *testProgressBarUi) Error(string)              {}
-func (u *testProgressBarUi) Machine(string, ...string) {}
-
-func (u *testProgressBarUi) Ask(string) (string, error) {
-	return "", nil
-}
-func (u *testProgressBarUi) Message(message string) {
-	u.messageCalled = true
-	u.messageMessage = message
-}
-
-// ..and now let's begin our actual tests
-func TestCalculateUiPrefixLength_Unknown(t *testing.T) {
-	ui := &testProgressBarUi{}
-
-	expected := 0
-	if res := calculateUiPrefixLength(ui); res != expected {
-		t.Fatalf("calculateUiPrefixLength should have returned a length of %d", expected)
-	}
-}
-
-func TestCalculateUiPrefixLength_BasicUi(t *testing.T) {
-	ui := &packer.BasicUi{}
-
-	expected := 1
-	if res := calculateUiPrefixLength(ui); res != expected {
-		t.Fatalf("calculateUiPrefixLength should have returned a length of %d", expected)
-	}
-}
-
-func TestCalculateUiPrefixLength_TargetedUI(t *testing.T) {
-	ui := &packer.TargetedUI{}
-	ui.Target = "TestTarget"
-	arrowText := "==>"
-
-	expected := len(arrowText + " " + ui.Target + ": ")
-	if res := calculateUiPrefixLength(ui); res != expected {
-		t.Fatalf("calculateUiPrefixLength should have returned a length of %d", expected)
-	}
-}
-
-func TestCalculateUiPrefixLength_TargetedUIWrappingBasicUi(t *testing.T) {
-	ui := &packer.TargetedUI{}
-	ui.Target = "TestTarget"
-	ui.Ui = &packer.BasicUi{}
-	arrowText := "==>"
-
-	expected := len(arrowText + " " + ui.Target + ": " + "\n")
-	if res := calculateUiPrefixLength(ui); res != expected {
-		t.Fatalf("calculateUiPrefixLength should have returned a length of %d", expected)
-	}
-}
-
-func TestCalculateUiPrefixLength_TargetedUIWrappingMachineUi(t *testing.T) {
-	ui := &packer.TargetedUI{}
-	ui.Target = "TestTarget"
-	ui.Ui = &packer.MachineReadableUi{}
-
-	expected := 0
-	if res := calculateUiPrefixLength(ui); res != expected {
-		t.Fatalf("calculateUiPrefixLength should have returned a length of %d", expected)
-	}
-}
-func TestDefaultProgressBar(t *testing.T) {
-	var callbackCalled bool
-
-	// Initialize the default progress bar
-	bar := GetDefaultProgressBar()
-	bar.Callback = func(state string) {
-		callbackCalled = true
-		t.Logf("TestDefaultProgressBar emitted %#v", state)
-	}
-	bar.SetTotal64(1)
-
-	// Set it off
-	progressBar := bar.Start()
-	progressBar.Set64(1)
-
-	// Check to see that the callback was hit
-	if !callbackCalled {
-		t.Fatalf("TestDefaultProgressBar.Callback should be called")
-	}
-}
-
-func TestDummyProgressBar(t *testing.T) {
-	var callbackCalled bool
-
-	// Initialize the dummy progress bar
-	bar := GetDummyProgressBar()
-	bar.Callback = func(state string) {
-		callbackCalled = true
-		t.Logf("TestDummyProgressBar emitted %#v", state)
-	}
-	bar.SetTotal64(1)
-
-	// Now we can go
-	progressBar := bar.Start()
-	progressBar.Set64(1)
-
-	// Check to see that the callback was hit
-	if callbackCalled {
-		t.Fatalf("TestDummyProgressBar.Callback should not be called")
-	}
-}
-
-func TestUiProgressBar(t *testing.T) {
-
-	ui := &testProgressBarUi{}
-
-	// Initialize the Ui progress bar
-	bar := GetProgressBar(ui, nil)
-	bar.SetTotal64(1)
-
-	// Ensure that callback has been set to something
-	if bar.Callback == nil {
-		t.Fatalf("TestUiProgressBar.Callback should be initialized")
-	}
-
-	// Now we can go
-	progressBar := bar.Start()
-	progressBar.Set64(1)
-
-	// Check to see that the callback was hit
-	if !ui.messageCalled {
-		t.Fatalf("TestUiProgressBar.messageCalled should be called")
-	}
-	t.Logf("TestUiProgressBar emitted %#v", ui.messageMessage)
-}

common/random/string.go

@@ -0,0 +1,33 @@
+package random
+
+import (
+	"math/rand"
+	"os"
+	"time"
+)
+
+var (
+	PossibleNumbers   = "0123456789"
+	PossibleLowerCase = "abcdefghijklmnopqrstuvwxyz"
+	PossibleUpperCase = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+
+	PossibleAlphaNum      = PossibleNumbers + PossibleLowerCase + PossibleUpperCase
+	PossibleAlphaNumLower = PossibleNumbers + PossibleLowerCase
+	PossibleAlphaNumUpper = PossibleNumbers + PossibleUpperCase
+)
+
+var rnd = rand.New(rand.NewSource(time.Now().UnixNano() + int64(os.Getpid())))
+
+func Numbers(length int) string       { return String(PossibleNumbers, length) }
+func AlphaNum(length int) string      { return String(PossibleAlphaNum, length) }
+func AlphaNumLower(length int) string { return String(PossibleAlphaNumLower, length) }
+func AlphaNumUpper(length int) string { return String(PossibleAlphaNumUpper, length) }
+
+func String(chooseFrom string, length int) (randomString string) {
+	cflen := len(chooseFrom)
+	bytes := make([]byte, length)
+	for i := range bytes {
+		bytes[i] = chooseFrom[rnd.Intn(cflen)]
+	}
+	return string(bytes)
+}

common/shell-local/config.go

@@ -53,9 +53,6 @@ type Config struct {
 	UseLinuxPathing bool `mapstructure:"use_linux_pathing"`
 
 	Ctx interpolate.Context
-
-	// internal use only; for the provisioner.
-	WinRMPassword string
 }
 
 func Decode(config *Config, raws ...interface{}) error {

common/shell-local/run.go

@@ -10,6 +10,7 @@ import (
 	"strings"
 
 	"github.com/hashicorp/packer/common"
+	commonhelper "github.com/hashicorp/packer/helper/common"
 	"github.com/hashicorp/packer/packer"
 	"github.com/hashicorp/packer/template/interpolate"
 )
@@ -69,7 +70,6 @@ func Run(ui packer.Ui, config *Config) (bool, error) {
 		// buffers and for reading the final exit status.
 		flattenedCmd := strings.Join(interpolatedCmds, " ")
 		cmd := &packer.RemoteCmd{Command: flattenedCmd}
-		packer.LogSecretFilter.Set(config.WinRMPassword)
 		log.Printf("[INFO] (shell-local): starting local command: %s", flattenedCmd)
 		if err := cmd.StartWithUi(comm, ui); err != nil {
 			return false, fmt.Errorf(
@@ -105,7 +105,7 @@ func createInlineScriptFile(config *Config) (string, error) {
 
 	// generate context so you can interpolate the command
 	config.Ctx.Data = &EnvVarsTemplate{
-		WinRMPassword: config.WinRMPassword,
+		WinRMPassword: getWinRMPassword(config.PackerBuildName),
 	}
 
 	for _, command := range config.Inline {
@@ -139,7 +139,7 @@ func createInterpolatedCommands(config *Config, script string, flattenedEnvVars
 		Vars:          flattenedEnvVars,
 		Script:        script,
 		Command:       script,
-		WinRMPassword: config.WinRMPassword,
+		WinRMPassword: getWinRMPassword(config.PackerBuildName),
 	}
 
 	interpolatedCmds := make([]string, len(config.ExecuteCommand))
@@ -169,7 +169,7 @@ func createFlattenedEnvVars(config *Config) (string, error) {
 
 	// interpolate environment variables
 	config.Ctx.Data = &EnvVarsTemplate{
-		WinRMPassword: config.WinRMPassword,
+		WinRMPassword: getWinRMPassword(config.PackerBuildName),
 	}
 	// Split vars into key/value components
 	for _, envVar := range config.Vars {
@@ -196,3 +196,9 @@ func createFlattenedEnvVars(config *Config) (string, error) {
 	}
 	return flattened, nil
 }
+
+func getWinRMPassword(buildName string) string {
+	winRMPass, _ := commonhelper.RetrieveSharedState("winrm_password", buildName)
+	packer.LogSecretFilter.Set(winRMPass)
+	return winRMPass
+}

common/step_download.go

@@ -63,9 +63,6 @@ func (s *StepDownload) Run(_ context.Context, state multistep.StateBag) multiste
 
 	ui.Say(fmt.Sprintf("Retrieving %s", s.Description))
 
-	// Get a progress bar from the ui so we can hand it off to the download client
-	bar := GetProgressBar(ui, GetPackerConfigFromStateBag(state))
-
 	// First try to use any already downloaded file
 	// If it fails, proceed to regular download logic
 
@@ -99,7 +96,7 @@ func (s *StepDownload) Run(_ context.Context, state multistep.StateBag) multiste
 		}
 		downloadConfigs[i] = config
 
-		if match, _ := NewDownloadClient(config, bar).VerifyChecksum(config.TargetPath); match {
+		if match, _ := NewDownloadClient(config, ui).VerifyChecksum(config.TargetPath); match {
 			ui.Message(fmt.Sprintf("Found already downloaded, initial checksum matched, no download needed: %s", url))
 			finalPath = config.TargetPath
 			break
@@ -143,11 +140,8 @@ func (s *StepDownload) download(config *DownloadConfig, state multistep.StateBag
 	var path string
 	ui := state.Get("ui").(packer.Ui)
 
-	// Get a progress bar and hand it off to the download client
-	bar := GetProgressBar(ui, GetPackerConfigFromStateBag(state))
-
-	// Create download client with config and progress bar
-	download := NewDownloadClient(config, bar)
+	// Create download client with config
+	download := NewDownloadClient(config, ui)
 
 	downloadCompleteCh := make(chan error, 1)
 	go func() {
@@ -159,7 +153,6 @@ func (s *StepDownload) download(config *DownloadConfig, state multistep.StateBag
 	for {
 		select {
 		case err := <-downloadCompleteCh:
-			bar.Finish()
 
 			if err != nil {
 				return "", err, true
@@ -174,7 +167,6 @@ func (s *StepDownload) download(config *DownloadConfig, state multistep.StateBag
 
 		case <-time.After(1 * time.Second):
 			if _, ok := state.GetOk(multistep.StateCancelled); ok {
-				bar.Finish()
 				ui.Say("Interrupt received. Cancelling download...")
 				return "", nil, false
 			}

common/step_provision.go

@@ -22,10 +22,6 @@ type StepProvision struct {
 	Comm packer.Communicator
 }
 
-type ProvisionHookData struct {
-	WinRMPassword string
-}
-
 func (s *StepProvision) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
 	comm := s.Comm
 	if comm == nil {
@@ -37,21 +33,12 @@ func (s *StepProvision) Run(_ context.Context, state multistep.StateBag) multist
 	hook := state.Get("hook").(packer.Hook)
 	ui := state.Get("ui").(packer.Ui)
 
-	// Save data we need to give to provisioners
-	WinRMPassword, ok := state.GetOk("winrm_password")
-	if !ok {
-		WinRMPassword = ""
-	} else {
-		WinRMPassword = WinRMPassword.(string)
-	}
-	phd := ProvisionHookData{WinRMPassword.(string)}
-
 	// Run the provisioner in a goroutine so we can continually check
 	// for cancellations...
 	log.Println("Running the provision hook")
 	errCh := make(chan error, 1)
 	go func() {
-		errCh <- hook.Run(packer.HookProvision, ui, comm, phd)
+		errCh <- hook.Run(packer.HookProvision, ui, comm, nil)
 	}()
 
 	for {

fix/fixer.go

@@ -38,6 +38,7 @@ func init() {
 		"powershell-escapes":         new(FixerPowerShellEscapes),
 		"hyperv-deprecations":        new(FixerHypervDeprecations),
 		"hyperv-vmxc-typo":           new(FixerHypervVmxcTypo),
+		"vmware-compaction":          new(FixerVMwareCompaction),
 	}
 
 	FixerOrder = []string{
@@ -57,7 +58,6 @@ func init() {
 		"amazon-private-ip",
 		"docker-email",
 		"powershell-escapes",
-		"hyperv-deprecations",
-		"hyperv-vmxc-typo",
+		"vmware-compaction",
 	}
 }

fix/fixer_docker_email.go

@@ -5,10 +5,14 @@ import "github.com/mitchellh/mapstructure"
 type FixerDockerEmail struct{}
 
 func (FixerDockerEmail) Fix(input map[string]interface{}) (map[string]interface{}, error) {
+	if input["post-processors"] == nil {
+		return input, nil
+	}
+
 	// Our template type we'll use for this fixer only
 	type template struct {
-		Builders       []map[string]interface{}
-		PostProcessors []map[string]interface{} `mapstructure:"post-processors"`
+		Builders []map[string]interface{}
+		PP       `mapstructure:",squash"`
 	}
 
 	// Decode the input into our structure, if we can
@@ -27,7 +31,9 @@ func (FixerDockerEmail) Fix(input map[string]interface{}) (map[string]interface{
 	}
 
 	// Go through each post-processor and delete `docker_login` if present
-	for _, pp := range tpl.PostProcessors {
+	pps := tpl.ppList()
+
+	for _, pp := range pps {
 		_, ok := pp["login_email"]
 		if !ok {
 			continue

fix/fixer_pp_manifest_filename.go

@@ -8,10 +8,13 @@ import (
 type FixerManifestFilename struct{}
 
 func (FixerManifestFilename) Fix(input map[string]interface{}) (map[string]interface{}, error) {
+	if input["post-processors"] == nil {
+		return input, nil
+	}
 
 	// Our template type we'll use for this fixer only
 	type template struct {
-		PostProcessors []interface{} `mapstructure:"post-processors"`
+		PP `mapstructure:",squash"`
 	}
 
 	// Decode the input into our structure, if we can
@@ -21,20 +24,7 @@ func (FixerManifestFilename) Fix(input map[string]interface{}) (map[string]inter
 	}
 
 	// Go through each post-processor and get out all the complex configs
-	pps := make([]map[string]interface{}, 0, len(tpl.PostProcessors))
-	for _, rawPP := range tpl.PostProcessors {
-		switch pp := rawPP.(type) {
-		case string:
-		case map[string]interface{}:
-			pps = append(pps, pp)
-		case []interface{}:
-			for _, innerRawPP := range pp {
-				if innerPP, ok := innerRawPP.(map[string]interface{}); ok {
-					pps = append(pps, innerPP)
-				}
-			}
-		}
-	}
+	pps := tpl.ppList()
 
 	for _, pp := range pps {
 		ppTypeRaw, ok := pp["type"]

fix/fixer_pp_manifest_filename_test.go

@@ -1,8 +1,9 @@
 package fix
 
 import (
-	"reflect"
 	"testing"
+
+	"github.com/stretchr/testify/assert"
 )
 
 func TestFixerManifestPPFilename_Impl(t *testing.T) {
@@ -43,11 +44,7 @@ func TestFixerManifestPPFilename_Fix(t *testing.T) {
 	}
 
 	output, err := f.Fix(input)
-	if err != nil {
-		t.Fatalf("err: %s", err)
-	}
+	assert.NoError(t, err)
 
-	if !reflect.DeepEqual(output, expected) {
-		t.Fatalf("unexpected: %#v\nexpected: %#v\n", output, expected)
-	}
+	assert.Equal(t, expected, output)
 }

fix/fixer_pp_vagrant_override.go

@@ -1,8 +1,6 @@
 package fix
 
-import (
-	"github.com/mitchellh/mapstructure"
-)
+import "github.com/mitchellh/mapstructure"
 
 // FixerVagrantPPOverride is a Fixer that replaces the provider-specific
 // overrides for the Vagrant post-processor with the new style introduced
@@ -10,9 +8,13 @@ import (
 type FixerVagrantPPOverride struct{}
 
 func (FixerVagrantPPOverride) Fix(input map[string]interface{}) (map[string]interface{}, error) {
+	if input["post-processors"] == nil {
+		return input, nil
+	}
+
 	// Our template type we'll use for this fixer only
 	type template struct {
-		PostProcessors []interface{} `mapstructure:"post-processors"`
+		PP `mapstructure:",squash"`
 	}
 
 	// Decode the input into our structure, if we can
@@ -21,21 +23,7 @@ func (FixerVagrantPPOverride) Fix(input map[string]interface{}) (map[string]inte
 		return nil, err
 	}
 
-	// Go through each post-processor and get out all the complex configs
-	pps := make([]map[string]interface{}, 0, len(tpl.PostProcessors))
-	for _, rawPP := range tpl.PostProcessors {
-		switch pp := rawPP.(type) {
-		case string:
-		case map[string]interface{}:
-			pps = append(pps, pp)
-		case []interface{}:
-			for _, innerRawPP := range pp {
-				if innerPP, ok := innerRawPP.(map[string]interface{}); ok {
-					pps = append(pps, innerPP)
-				}
-			}
-		}
-	}
+	pps := tpl.ppList()
 
 	// Go through each post-processor and make the fix if necessary
 	possible := []string{"aws", "digitalocean", "virtualbox", "vmware"}

fix/fixer_pp_vagrant_override_test.go

@@ -1,8 +1,9 @@
 package fix
 
 import (
-	"reflect"
 	"testing"
+
+	"github.com/stretchr/testify/assert"
 )
 
 func TestFixerVagrantPPOverride_Impl(t *testing.T) {
@@ -69,11 +70,7 @@ func TestFixerVagrantPPOverride_Fix(t *testing.T) {
 	}
 
 	output, err := f.Fix(input)
-	if err != nil {
-		t.Fatalf("err: %s", err)
-	}
+	assert.NoError(t, err)
 
-	if !reflect.DeepEqual(output, expected) {
-		t.Fatalf("unexpected: %#v\nexpected: %#v\n", output, expected)
-	}
+	assert.Equal(t, expected, output)
 }

fix/fixer_vmware_compaction.go

@@ -0,0 +1,91 @@
+package fix
+
+import (
+	"github.com/mitchellh/mapstructure"
+)
+
+// FixerVMwareCompaction adds "skip_compaction = true" to "vmware-iso" builders with incompatible disk_type_id
+type FixerVMwareCompaction struct{}
+
+func (FixerVMwareCompaction) Fix(input map[string]interface{}) (map[string]interface{}, error) {
+	// The type we'll decode into; we only care about builders
+	type template struct {
+		Builders []map[string]interface{}
+	}
+
+	// Decode the input into our structure, if we can
+	var tpl template
+	if err := mapstructure.Decode(input, &tpl); err != nil {
+		return nil, err
+	}
+
+	for _, builder := range tpl.Builders {
+		builderTypeRaw, ok := builder["type"]
+		if !ok {
+			continue
+		}
+
+		builderType, ok := builderTypeRaw.(string)
+		if !ok {
+			continue
+		}
+
+		if builderType != "vmware-iso" {
+			continue
+		}
+
+		builderRemoteTypeRaw, ok := builder["remote_type"]
+		if !ok {
+			continue
+		}
+
+		builderRemoteType, ok := builderRemoteTypeRaw.(string)
+		if !ok {
+			continue
+		}
+
+		if builderRemoteType != "esx5" {
+			continue
+		}
+
+		builderDiskTypeIdRaw, ok := builder["disk_type_id"]
+		// set to default when this fixer was added due to incompatibility of defaults
+		if !ok {
+			builderDiskTypeId := "zeroedthick"
+			builder["disk_type_id"] = builderDiskTypeId
+		}
+
+		if ok {
+			builderDiskTypeId, ok := builderDiskTypeIdRaw.(string)
+			if !ok {
+				continue
+			}
+			if builderDiskTypeId == "thin" {
+				continue
+			}
+		}
+
+		builderSkipCompactionRaw, ok := builder["skip_compaction"]
+		// already verified this is not creating a "thin" disk, will need to skip_compaction
+		if ok {
+			builderSkipCompaction, ok := builderSkipCompactionRaw.(bool)
+			if !ok {
+				continue
+			}
+			if !builderSkipCompaction {
+				builder["skip_compaction"] = !builderSkipCompaction
+			}
+			continue
+		}
+
+		builderSkipCompaction := true
+		builder["skip_compaction"] = builderSkipCompaction
+	}
+
+	input["builders"] = tpl.Builders
+	return input, nil
+}
+
+func (FixerVMwareCompaction) Synopsis() string {
+	return `Adds "skip_compaction = true" to "vmware-iso" builders with incompatible disk_type_id`
+}

fix/fixer_vmware_compaction_test.go

@@ -0,0 +1,112 @@
+package fix
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestFixerVMwareCompaction_impl(t *testing.T) {
+	var _ Fixer = new(FixerVMwareCompaction)
+}
+
+func TestFixerVMwareCompaction_Fix(t *testing.T) {
+	cases := []struct {
+		Input    map[string]interface{}
+		Expected map[string]interface{}
+	}{
+		{
+			Input: map[string]interface{}{
+				"type": "virtualbox-iso",
+			},
+
+			Expected: map[string]interface{}{
+				"type": "virtualbox-iso",
+			},
+		},
+		{
+			Input: map[string]interface{}{
+				"type": "vmware-iso",
+			},
+
+			Expected: map[string]interface{}{
+				"type": "vmware-iso",
+			},
+		},
+		{
+			Input: map[string]interface{}{
+				"type":        "vmware-iso",
+				"remote_type": "esx5",
+			},
+
+			Expected: map[string]interface{}{
+				"type":            "vmware-iso",
+				"remote_type":     "esx5",
+				"disk_type_id":    "zeroedthick",
+				"skip_compaction": true,
+			},
+		},
+		{
+			Input: map[string]interface{}{
+				"type":         "vmware-iso",
+				"remote_type":  "esx5",
+				"disk_type_id": "zeroedthick",
+			},
+
+			Expected: map[string]interface{}{
+				"type":            "vmware-iso",
+				"remote_type":     "esx5",
+				"disk_type_id":    "zeroedthick",
+				"skip_compaction": true,
+			},
+		},
+		{
+			Input: map[string]interface{}{
+				"type":            "vmware-iso",
+				"remote_type":     "esx5",
+				"disk_type_id":    "zeroedthick",
+				"skip_compaction": false,
+			},
+
+			Expected: map[string]interface{}{
+				"type":            "vmware-iso",
+				"remote_type":     "esx5",
+				"disk_type_id":    "zeroedthick",
+				"skip_compaction": true,
+			},
+		},
+		{
+			Input: map[string]interface{}{
+				"type":         "vmware-iso",
+				"remote_type":  "esx5",
+				"disk_type_id": "thin",
+			},
+
+			Expected: map[string]interface{}{
+				"type":         "vmware-iso",
+				"remote_type":  "esx5",
+				"disk_type_id": "thin",
+			},
+		},
+	}
+
+	for _, tc := range cases {
+		var f FixerVMwareCompaction
+
+		input := map[string]interface{}{
+			"builders": []map[string]interface{}{tc.Input},
+		}
+
+		expected := map[string]interface{}{
+			"builders": []map[string]interface{}{tc.Expected},
+		}
+
+		output, err := f.Fix(input)
+		if err != nil {
+			t.Fatalf("err: %s", err)
+		}
+
+		if !reflect.DeepEqual(output, expected) {
+			t.Fatalf("unexpected: %#v\nexpected: %#v\n", output, expected)
+		}
+	}
+}

fix/helpers.go

@@ -0,0 +1,25 @@
+package fix
+
+// PP is a convenient way to interact with the post-processors within a fixer
+type PP struct {
+	PostProcessors []interface{} `mapstructure:"post-processors"`
+}
+
+// postProcessors converts the variable structure of the template to a list
+func (pp *PP) ppList() []map[string]interface{} {
+	pps := make([]map[string]interface{}, 0, len(pp.PostProcessors))
+	for _, rawPP := range pp.PostProcessors {
+		switch pp := rawPP.(type) {
+		case string:
+		case map[string]interface{}:
+			pps = append(pps, pp)
+		case []interface{}:
+			for _, innerRawPP := range pp {
+				if innerPP, ok := innerRawPP.(map[string]interface{}); ok {
+					pps = append(pps, innerPP)
+				}
+			}
+		}
+	}
+	return pps
+}

packer/progressbar.go

@@ -0,0 +1,147 @@
+package packer
+
+import (
+	"fmt"
+	"io"
+	"sync"
+	"sync/atomic"
+
+	"github.com/cheggaaa/pb"
+)
+
+// ProgressBar allows to graphically display
+// a self refreshing progress bar.
+type ProgressBar interface {
+	Start(total int64)
+	Add(current int64)
+	NewProxyReader(r io.Reader) (proxy io.Reader)
+	Finish()
+}
+
+// StackableProgressBar is a progress bar that
+// allows to track multiple downloads at once.
+// Every call to Start increments a counter that
+// will display the number of current loadings.
+// Every call to Start will add total to an internal
+// total that is the total displayed.
+// First call to Start will start a goroutine
+// that is waiting for every download to be finished.
+// Last call to Finish triggers a cleanup.
+// When all active downloads are finished
+// StackableProgressBar will clean itself to a default
+// state.
+type StackableProgressBar struct {
+	mtx sync.Mutex // locks in Start & Finish
+	BasicProgressBar
+	items int32
+	total int64
+
+	started bool
+}
+
+var _ ProgressBar = new(StackableProgressBar)
+
+func (spb *StackableProgressBar) start() {
+	spb.BasicProgressBar.ProgressBar = pb.New(0)
+	spb.BasicProgressBar.ProgressBar.SetUnits(pb.U_BYTES)
+
+	spb.BasicProgressBar.ProgressBar.Start()
+	spb.started = true
+}
+
+func (spb *StackableProgressBar) Start(total int64) {
+	spb.mtx.Lock()
+
+	spb.total += total
+	spb.items++
+
+	if !spb.started {
+		spb.start()
+	}
+	spb.SetTotal64(spb.total)
+	spb.prefix()
+	spb.mtx.Unlock()
+}
+
+func (spb *StackableProgressBar) prefix() {
+	spb.BasicProgressBar.ProgressBar.Prefix(fmt.Sprintf("%d items: ", atomic.LoadInt32(&spb.items)))
+}
+
+func (spb *StackableProgressBar) Finish() {
+	spb.mtx.Lock()
+
+	spb.items--
+	if spb.items == 0 {
+		// slef cleanup
+		spb.BasicProgressBar.ProgressBar.Finish()
+		spb.BasicProgressBar.ProgressBar = nil
+		spb.started = false
+		spb.total = 0
+		return
+	}
+	spb.prefix()
+	spb.mtx.Unlock()
+}
+
+// BasicProgressBar is packer's basic progress bar.
+// Current implementation will always try to keep
+// itself at the bottom of a terminal.
+type BasicProgressBar struct {
+	*pb.ProgressBar
+}
+
+var _ ProgressBar = new(BasicProgressBar)
+
+func (bpb *BasicProgressBar) Start(total int64) {
+	bpb.SetTotal64(total)
+	bpb.ProgressBar.Start()
+}
+
+func (bpb *BasicProgressBar) Add(current int64) {
+	bpb.ProgressBar.Add64(current)
+}
+func (bpb *BasicProgressBar) NewProxyReader(r io.Reader) io.Reader {
+	return &ProxyReader{
+		Reader:      r,
+		ProgressBar: bpb,
+	}
+}
+func (bpb *BasicProgressBar) NewProxyReadCloser(r io.ReadCloser) io.ReadCloser {
+	return &ProxyReader{
+		Reader:      r,
+		ProgressBar: bpb,
+	}
+}
+
+// NoopProgressBar is a silent progress bar.
+type NoopProgressBar struct {
+}
+
+var _ ProgressBar = new(NoopProgressBar)
+
+func (npb *NoopProgressBar) Start(int64)                                      {}
+func (npb *NoopProgressBar) Add(int64)                                        {}
+func (npb *NoopProgressBar) Finish()                                          {}
+func (npb *NoopProgressBar) NewProxyReader(r io.Reader) io.Reader             { return r }
+func (npb *NoopProgressBar) NewProxyReadCloser(r io.ReadCloser) io.ReadCloser { return r }
+
+// ProxyReader implements io.ReadCloser but sends
+// count of read bytes to a progress bar
+type ProxyReader struct {
+	io.Reader
+	ProgressBar
+}
+
+func (r *ProxyReader) Read(p []byte) (n int, err error) {
+	n, err = r.Reader.Read(p)
+	r.ProgressBar.Add(int64(n))
+	return
+}
+
+// Close the reader if it implements io.Closer
+func (r *ProxyReader) Close() (err error) {
+	if closer, ok := r.Reader.(io.Closer); ok {
+		return closer.Close()
+	}
+	return
+}

packer/provisioner.go

@@ -44,10 +44,6 @@ type ProvisionHook struct {
 	runningProvisioner Provisioner
 }
 
-type ProvisionHookData struct {
-	WinRMPassword string
-}
-
 // Runs the provisioners in order.
 func (h *ProvisionHook) Run(name string, ui Ui, comm Communicator, data interface{}) error {
 	// Shortcut
@@ -75,15 +71,8 @@ func (h *ProvisionHook) Run(name string, ui Ui, comm Communicator, data interfac
 		h.lock.Unlock()
 
 		ts := CheckpointReporter.AddSpan(p.TypeName, "provisioner", p.Config)
-		// re-run prepare with builder-generated config variables (e.g. WinRMPassword)
-		// Hack Alert. #SorryNotSorry.
-		err := p.Provisioner.Prepare(data)
-		if err != nil {
-			log.Printf("Error performing secondary Prepare: %s", err)
-		}
 
-		// Finally, provision.
-		err = p.Provisioner.Provision(ui, comm)
+		err := p.Provisioner.Provision(ui, comm)
 
 		ts.End(err)
 		if err != nil {

packer/provisioner_mock.go

@@ -14,10 +14,8 @@ type MockProvisioner struct {
 }
 
 func (t *MockProvisioner) Prepare(configs ...interface{}) error {
-	if !t.PrepCalled {
-		t.PrepCalled = true
-		t.PrepConfigs = configs
-	}
+	t.PrepCalled = true
+	t.PrepConfigs = configs
 	return nil
 }
 

packer/rpc/server.go

@@ -114,7 +114,8 @@ func (s *Server) RegisterProvisioner(p packer.Provisioner) {
 
 func (s *Server) RegisterUi(ui packer.Ui) {
 	s.server.RegisterName(DefaultUiEndpoint, &UiServer{
-		ui: ui,
+		ui:       ui,
+		register: s.server.RegisterName,
 	})
 }
 

packer/rpc/ui.go

@@ -1,6 +1,7 @@
 package rpc
 
 import (
+	"io"
 	"log"
 	"net/rpc"
 
@@ -14,10 +15,13 @@ type Ui struct {
 	endpoint string
 }
 
+var _ packer.Ui = new(Ui)
+
 // UiServer wraps a packer.Ui implementation and makes it exportable
 // as part of a Golang RPC server.
 type UiServer struct {
-	ui packer.Ui
+	ui       packer.Ui
+	register func(name string, rcvr interface{}) error
 }
 
 // The arguments sent to Ui.Machine
@@ -60,6 +64,31 @@ func (u *Ui) Say(message string) {
 	}
 }
 
+func (u *Ui) ProgressBar() packer.ProgressBar {
+	if err := u.client.Call("Ui.ProgressBar", new(interface{}), new(interface{})); err != nil {
+		log.Printf("Error in Ui RPC call: %s", err)
+	}
+	return u // Ui is also a progress bar !!
+}
+
+var _ packer.ProgressBar = new(Ui)
+
+func (pb *Ui) Start(total int64) {
+	pb.client.Call("Ui.Start", total, new(interface{}))
+}
+
+func (pb *Ui) Add(current int64) {
+	pb.client.Call("Ui.Add", current, new(interface{}))
+}
+
+func (pb *Ui) Finish() {
+	pb.client.Call("Ui.Finish", nil, new(interface{}))
+}
+
+func (pb *Ui) NewProxyReader(r io.Reader) io.Reader {
+	return &packer.ProxyReader{Reader: r, ProgressBar: pb}
+}
+
 func (u *UiServer) Ask(query string, reply *string) (err error) {
 	*reply, err = u.ui.Ask(query)
 	return
@@ -91,3 +120,26 @@ func (u *UiServer) Say(message *string, reply *interface{}) error {
 	*reply = nil
 	return nil
 }
+
+func (u *UiServer) ProgressBar(_ *string, reply *interface{}) error {
+	// No-op for now, this function might be
+	// used in the future if we want to use
+	// different progress bars with identifiers.
+	u.ui.ProgressBar()
+	return nil
+}
+
+func (pb *UiServer) Finish(_ string, _ *interface{}) error {
+	pb.ui.ProgressBar().Finish()
+	return nil
+}
+
+func (pb *UiServer) Start(total int64, _ *interface{}) error {
+	pb.ui.ProgressBar().Start(total)
+	return nil
+}
+
+func (pb *UiServer) Add(current int64, _ *interface{}) error {
+	pb.ui.ProgressBar().Add(current)
+	return nil
+}

packer/rpc/ui_test.go

@@ -1,8 +1,11 @@
 package rpc
 
 import (
+	"io"
 	"reflect"
 	"testing"
+
+	"github.com/hashicorp/packer/packer"
 )
 
 type testUi struct {
@@ -17,6 +20,12 @@ type testUi struct {
 	messageMessage string
 	sayCalled      bool
 	sayMessage     string
+
+	progressBarCalled               bool
+	progressBarStartCalled          bool
+	progressBarAddCalled            bool
+	progressBarFinishCalled         bool
+	progressBarNewProxyReaderCalled bool
 }
 
 func (u *testUi) Ask(query string) (string, error) {
@@ -46,6 +55,28 @@ func (u *testUi) Say(message string) {
 	u.sayMessage = message
 }
 
+func (u *testUi) ProgressBar() packer.ProgressBar {
+	u.progressBarCalled = true
+	return u
+}
+
+func (u *testUi) Start(int64) {
+	u.progressBarStartCalled = true
+}
+
+func (u *testUi) Add(int64) {
+	u.progressBarAddCalled = true
+}
+
+func (u *testUi) Finish() {
+	u.progressBarFinishCalled = true
+}
+
+func (u *testUi) NewProxyReader(r io.Reader) io.Reader {
+	u.progressBarNewProxyReaderCalled = true
+	return r
+}
+
 func TestUiRPC(t *testing.T) {
 	// Create the UI to test
 	ui := new(testUi)
@@ -88,6 +119,26 @@ func TestUiRPC(t *testing.T) {
 		t.Fatalf("bad: %#v", ui.errorMessage)
 	}
 
+	bar := uiClient.ProgressBar()
+	if ui.progressBarCalled != true {
+		t.Errorf("ProgressBar not called.")
+	}
+
+	bar.Start(100)
+	if ui.progressBarStartCalled != true {
+		t.Errorf("progressBar.Start not called.")
+	}
+
+	bar.Add(1)
+	if ui.progressBarAddCalled != true {
+		t.Errorf("progressBar.Add not called.")
+	}
+
+	bar.Finish()
+	if ui.progressBarFinishCalled != true {
+		t.Errorf("progressBar.Finish not called.")
+	}
+
 	uiClient.Machine("foo", "bar", "baz")
 	if !ui.machineCalled {
 		t.Fatal("machine should be called")

packer/ui.go

@@ -37,8 +37,20 @@ type Ui interface {
 	Message(string)
 	Error(string)
 	Machine(string, ...string)
+	ProgressBar() ProgressBar
 }
 
+type NoopUi struct{}
+
+var _ Ui = new(NoopUi)
+
+func (*NoopUi) Ask(string) (string, error) { return "", errors.New("this is a noop ui") }
+func (*NoopUi) Say(string)                 { return }
+func (*NoopUi) Message(string)             { return }
+func (*NoopUi) Error(string)               { return }
+func (*NoopUi) Machine(string, ...string)  { return }
+func (*NoopUi) ProgressBar() ProgressBar   { return new(NoopProgressBar) }
+
 // ColoredUi is a UI that is colored using terminal colors.
 type ColoredUi struct {
 	Color      UiColor
@@ -46,6 +58,8 @@ type ColoredUi struct {
 	Ui         Ui
 }
 
+var _ Ui = new(ColoredUi)
+
 // TargetedUI is a UI that wraps another UI implementation and modifies
 // the output to indicate a specific target. Specifically, all Say output
 // is prefixed with the target name. Message output is not prefixed but
@@ -56,6 +70,8 @@ type TargetedUI struct {
 	Ui     Ui
 }
 
+var _ Ui = new(TargetedUI)
+
 // The BasicUI is a UI that reads and writes from a standard Go reader
 // and writer. It is safe to be called from multiple goroutines. Machine
 // readable output is simply logged for this UI.
@@ -66,6 +82,13 @@ type BasicUi struct {
 	l           sync.Mutex
 	interrupted bool
 	scanner     *bufio.Scanner
+	StackableProgressBar
+}
+
+var _ Ui = new(BasicUi)
+
+func (bu *BasicUi) ProgressBar() ProgressBar {
+	return &bu.StackableProgressBar
 }
 
 // MachineReadableUi is a UI that only outputs machine-readable output
@@ -74,6 +97,8 @@ type MachineReadableUi struct {
 	Writer io.Writer
 }
 
+var _ Ui = new(MachineReadableUi)
+
 func (u *ColoredUi) Ask(query string) (string, error) {
 	return u.Ui.Ask(u.colorize(query, u.Color, true))
 }
@@ -100,6 +125,10 @@ func (u *ColoredUi) Machine(t string, args ...string) {
 	u.Ui.Machine(t, args...)
 }
 
+func (u *ColoredUi) ProgressBar() ProgressBar {
+	return u.Ui.ProgressBar() //TODO(adrien): color me
+}
+
 func (u *ColoredUi) colorize(message string, color UiColor, bold bool) string {
 	if !u.supportsColors() {
 		return message
@@ -153,6 +182,10 @@ func (u *TargetedUI) Machine(t string, args ...string) {
 	u.Ui.Machine(fmt.Sprintf("%s,%s", u.Target, t), args...)
 }
 
+func (u *TargetedUI) ProgressBar() ProgressBar {
+	return u.Ui.ProgressBar()
+}
+
 func (u *TargetedUI) prefixLines(arrow bool, message string) string {
 	arrowText := "==>"
 	if !arrow {
@@ -305,3 +338,7 @@ func (u *MachineReadableUi) Machine(category string, args ...string) {
 		}
 	}
 }
+
+func (u *MachineReadableUi) ProgressBar() ProgressBar {
+	return new(NoopProgressBar)
+}

post-processor/vsphere/post-processor.go

@@ -3,9 +3,13 @@ package vsphere
 import (
 	"bytes"
 	"fmt"
+	"io"
 	"log"
 	"net/url"
+	"os"
 	"os/exec"
+	"regexp"
+	"runtime"
 	"strings"
 
 	"github.com/hashicorp/packer/common"
@@ -19,6 +23,18 @@ var builtins = map[string]string{
 	"mitchellh.vmware-esx": "vmware",
 }
 
+var ovftool string = "ovftool"
+
+var (
+	// Regular expression to validate RFC1035 hostnames from full fqdn or simple hostname.
+	// For example "packer-esxi1". Requires proper DNS setup and/or correct DNS search domain setting.
+	hostnameRegex = regexp.MustCompile(`^[[:alnum:]][[:alnum:]\-]{0,61}[[:alnum:]]|[[:alpha:]]$`)
+
+	// Simple regular expression to validate IPv4 values.
+	// For example "192.168.1.1".
+	ipv4Regex = regexp.MustCompile(`^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$`)
+)
+
 type Config struct {
 	common.PackerConfig `mapstructure:",squash"`
 
@@ -27,6 +43,7 @@ type Config struct {
 	Datastore    string   `mapstructure:"datastore"`
 	DiskMode     string   `mapstructure:"disk_mode"`
 	Host         string   `mapstructure:"host"`
+	ESXiHost     string   `mapstructure:"esxi_host"`
 	Insecure     bool     `mapstructure:"insecure"`
 	Options      []string `mapstructure:"options"`
 	Overwrite    bool     `mapstructure:"overwrite"`
@@ -64,7 +81,11 @@ func (p *PostProcessor) Configure(raws ...interface{}) error {
 	// Accumulate any errors
 	errs := new(packer.MultiError)
 
-	if _, err := exec.LookPath("ovftool"); err != nil {
+	if runtime.GOOS == "windows" {
+		ovftool = "ovftool.exe"
+	}
+
+	if _, err := exec.LookPath(ovftool); err != nil {
 		errs = packer.MultiErrorAppend(
 			errs, fmt.Errorf("ovftool not found: %s", err))
 	}
@@ -122,6 +143,14 @@ func (p *PostProcessor) PostProcess(ui packer.Ui, artifact packer.Artifact) (pac
 		ovftool_uri += "/Resources/" + p.config.ResourcePool
 	}
 
+	if p.config.ESXiHost != "" {
+		if ipv4Regex.MatchString(p.config.ESXiHost) {
+			ovftool_uri += "/?ip=" + p.config.ESXiHost
+		} else if hostnameRegex.MatchString(p.config.ESXiHost) {
+			ovftool_uri += "/?dns=" + p.config.ESXiHost
+		}
+	}
+
 	args, err := p.BuildArgs(source, ovftool_uri)
 	if err != nil {
 		ui.Message(fmt.Sprintf("Failed: %s\n", err))
@@ -129,16 +158,26 @@ func (p *PostProcessor) PostProcess(ui packer.Ui, artifact packer.Artifact) (pac
 
 	ui.Message(fmt.Sprintf("Uploading %s to vSphere", source))
 
-	log.Printf("Starting ovftool with parameters: %s", p.filterLog(strings.Join(args, " ")))
+	log.Printf("Starting ovftool with parameters: %s",
+		strings.Replace(
+			strings.Join(args, " "),
+			password,
+			"<password>",
+			-1))
+
+	var errWriter io.Writer
+	var errOut bytes.Buffer
+	cmd := exec.Command(ovftool, args...)
+	errWriter = io.MultiWriter(os.Stderr, &errOut)
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = errWriter
 
-	var out bytes.Buffer
-	cmd := exec.Command("ovftool", args...)
-	cmd.Stdout = &out
 	if err := cmd.Run(); err != nil {
-		return nil, false, fmt.Errorf("Failed: %s\n%s\n", err, p.filterLog(out.String()))
+		err := fmt.Errorf("Error uploading virtual machine: %s\n%s\n", err, p.filterLog(errOut.String()))
+		return nil, false, err
 	}
 
-	ui.Message(p.filterLog(out.String()))
+	ui.Message(p.filterLog(errOut.String()))
 
 	artifact = NewArtifact(p.config.Datastore, p.config.VMFolder, p.config.VMName, artifact.Files())
 

provisioner/ansible-local/provisioner_test.go

@@ -8,11 +8,12 @@ import (
 	"testing"
 
 	"fmt"
+	"os/exec"
+
 	"github.com/hashicorp/packer/builder/docker"
 	"github.com/hashicorp/packer/packer"
 	"github.com/hashicorp/packer/provisioner/file"
 	"github.com/hashicorp/packer/template"
-	"os/exec"
 )
 
 func TestProvisioner_Impl(t *testing.T) {
@@ -132,7 +133,7 @@ func TestProvisionerProvision_PlaybookFiles(t *testing.T) {
 	}
 
 	comm := &communicatorMock{}
-	if err := p.Provision(&uiStub{}, comm); err != nil {
+	if err := p.Provision(new(packer.NoopUi), comm); err != nil {
 		t.Fatalf("err: %s", err)
 	}
 
@@ -166,7 +167,7 @@ func TestProvisionerProvision_PlaybookFilesWithPlaybookDir(t *testing.T) {
 	}
 
 	comm := &communicatorMock{}
-	if err := p.Provision(&uiStub{}, comm); err != nil {
+	if err := p.Provision(new(packer.NoopUi), comm); err != nil {
 		t.Fatalf("err: %s", err)
 	}
 
@@ -367,8 +368,8 @@ func testProvisionerProvisionDockerWithPlaybookFiles(t *testing.T, templateStrin
 	hooks[packer.HookProvision] = []packer.Hook{
 		&packer.ProvisionHook{
 			Provisioners: []*packer.HookedProvisioner{
-				{ansible, nil, ""},
-				{download, nil, ""},
+				{Provisioner: ansible, Config: nil, TypeName: ""},
+				{Provisioner: download, Config: nil, TypeName: ""},
 			},
 		},
 	}

provisioner/ansible-local/ui_stub.go

@@ -1,15 +0,0 @@
-package ansiblelocal
-
-type uiStub struct{}
-
-func (su *uiStub) Ask(string) (string, error) {
-	return "", nil
-}
-
-func (su *uiStub) Error(string) {}
-
-func (su *uiStub) Machine(string, ...string) {}
-
-func (su *uiStub) Message(string) {}
-
-func (su *uiStub) Say(msg string) {}

provisioner/ansible/adapter_test.go

@@ -24,7 +24,7 @@ func TestAdapter_Serve(t *testing.T) {
 
 	config := &ssh.ServerConfig{}
 
-	ui := new(ui)
+	ui := new(packer.NoopUi)
 
 	sut := newAdapter(done, &l, config, "", newUi(ui), communicator{})
 	go func() {
@@ -93,36 +93,6 @@ func (a addr) String() string {
 	return "test"
 }
 
-type ui int
-
-func (u *ui) Ask(s string) (string, error) {
-	*u++
-	return s, nil
-}
-
-func (u *ui) Say(s string) {
-	*u++
-	log.Println(s)
-}
-
-func (u *ui) Message(s string) {
-	*u++
-	log.Println(s)
-}
-
-func (u *ui) Error(s string) {
-	*u++
-	log.Println(s)
-}
-
-func (u *ui) Machine(s1 string, s2 ...string) {
-	*u++
-	log.Println(s1)
-	for _, s := range s2 {
-		log.Println(s)
-	}
-}
-
 type communicator struct{}
 
 func (c communicator) Start(*packer.RemoteCmd) error {

provisioner/ansible/provisioner.go

@@ -26,6 +26,7 @@ import (
 	"golang.org/x/crypto/ssh"
 
 	"github.com/hashicorp/packer/common"
+	commonhelper "github.com/hashicorp/packer/helper/common"
 	"github.com/hashicorp/packer/helper/config"
 	"github.com/hashicorp/packer/packer"
 	"github.com/hashicorp/packer/template/interpolate"
@@ -57,8 +58,6 @@ type Config struct {
 	UseSFTP              bool     `mapstructure:"use_sftp"`
 	InventoryDirectory   string   `mapstructure:"inventory_directory"`
 	InventoryFile        string   `mapstructure:"inventory_file"`
-	// for internal use only; unsettable by user.
-	winrmpassword string
 }
 
 type Provisioner struct {
@@ -74,35 +73,14 @@ type PassthroughTemplate struct {
 }
 
 func (p *Provisioner) Prepare(raws ...interface{}) error {
-	// This is a bit of a hack. For provisioners that need access to
-	// auto-generated WinRMPasswords, the mechanism of keeping provisioner data
-	// and build data totally segregated breaks down. We get around this by
-	// having the builder stash the WinRMPassword in the state bag, then
-	// grabbing it out of the statebag inside of StepProvision.  Then, when
-	// the time comes to provision for real, we run the prepare step one more
-	// time, now with WinRMPassword defined in the raws, and can store the
-	// password on the provisioner config without overwriting the rest of the
-	// work we've already done in the first prepare run.
-	if len(raws) == 1 {
-		for k, v := range raws[0].(map[interface{}]interface{}) {
-			if k.(string) == "WinRMPassword" {
-				p.config.winrmpassword = v.(string)
-
-				// Even if WinRMPassword is not gonna be used, we've stored the
-				// key and pointed it to an empty string. That means we'll
-				// always reach this on our second-run of Prepare()
-				return nil
-			}
-		}
-	}
-
 	p.done = make(chan struct{})
 
-	// don't interpolate winrmpassword yet; if we've made it to this line, then
-	// we have not obtained the winrm password yet.
+	// Create passthrough for winrm password so we can fill it in once we know
+	// it
 	p.config.ctx.Data = &PassthroughTemplate{
 		WinRMPassword: `{{.WinRMPassword}}`,
 	}
+
 	err := config.Decode(&p.config, &config.DecodeOpts{
 		Interpolate:        true,
 		InterpolateContext: &p.config.ctx,
@@ -222,9 +200,8 @@ func (p *Provisioner) getVersion() error {
 func (p *Provisioner) Provision(ui packer.Ui, comm packer.Communicator) error {
 	ui.Say("Provisioning with Ansible...")
 	// Interpolate env vars to check for .WinRMPassword
-	packer.LogSecretFilter.Set(p.config.winrmpassword)
 	p.config.ctx.Data = &PassthroughTemplate{
-		WinRMPassword: p.config.winrmpassword,
+		WinRMPassword: getWinRMPassword(p.config.PackerBuildName),
 	}
 	for i, envVar := range p.config.AnsibleEnvVars {
 		envVar, err := interpolate.Render(envVar, &p.config.ctx)
@@ -443,7 +420,12 @@ func (p *Provisioner) executeAnsible(ui packer.Ui, comm packer.Communicator, pri
 
 	// remove winrm password from command, if it's been added
 	flattenedCmd := strings.Join(cmd.Args, " ")
-	ui.Say(fmt.Sprintf("Executing Ansible: %s", flattenedCmd))
+	sanitized := flattenedCmd
+	if len(getWinRMPassword(p.config.PackerBuildName)) > 0 {
+		sanitized = strings.Replace(sanitized,
+			getWinRMPassword(p.config.PackerBuildName), "*****", -1)
+	}
+	ui.Say(fmt.Sprintf("Executing Ansible: %s", sanitized))
 
 	if err := cmd.Start(); err != nil {
 		return err
@@ -571,6 +553,12 @@ func newSigner(privKeyFile string) (*signer, error) {
 	return signer, nil
 }
 
+func getWinRMPassword(buildName string) string {
+	winRMPass, _ := commonhelper.RetrieveSharedState("winrm_password", buildName)
+	packer.LogSecretFilter.Set(winRMPass)
+	return winRMPass
+}
+
 // Ui provides concurrency-safe access to packer.Ui.
 type Ui struct {
 	sem chan int
@@ -612,3 +600,7 @@ func (ui *Ui) Machine(t string, args ...string) {
 	ui.ui.Machine(t, args...)
 	<-ui.sem
 }
+
+func (ui *Ui) ProgressBar() packer.ProgressBar {
+	return new(packer.NoopProgressBar)
+}

provisioner/ansible/provisioner_test.go

@@ -16,8 +16,8 @@ import (
 
 // Be sure to remove the Ansible stub file in each test with:
 //   defer os.Remove(config["command"].(string))
-func testConfig(t *testing.T) map[interface{}]interface{} {
-	m := make((map[interface{}]interface{}))
+func testConfig(t *testing.T) map[string]interface{} {
+	m := make(map[string]interface{})
 	wd, err := os.Getwd()
 	if err != nil {
 		t.Fatalf("err: %s", err)

provisioner/file/provisioner.go

@@ -127,12 +127,12 @@ func (p *Provisioner) ProvisionDownload(ui packer.Ui, comm packer.Communicator)
 		defer f.Close()
 
 		// Get a default progress bar
-		pb := common.GetProgressBar(ui, &p.config.PackerConfig)
-		bar := pb.Start()
-		defer bar.Finish()
+		pb := packer.NoopProgressBar{}
+		pb.Start(0) // TODO: find size ? Remove ?
+		defer pb.Finish()
 
 		// Create MultiWriter for the current progress
-		pf := io.MultiWriter(f, bar)
+		pf := io.MultiWriter(f)
 
 		// Download the file
 		if err = comm.Download(src, pf); err != nil {
@@ -176,8 +176,8 @@ func (p *Provisioner) ProvisionUpload(ui packer.Ui, comm packer.Communicator) er
 		}
 
 		// Get a default progress bar
-		pb := common.GetProgressBar(ui, &p.config.PackerConfig)
-		bar := pb.Start()
+		bar := ui.ProgressBar()
+		bar.Start(info.Size())
 		defer bar.Finish()
 
 		// Create ProxyReader for the current progress

provisioner/file/provisioner_test.go

@@ -1,6 +1,7 @@
 package file
 
 import (
+	"bytes"
 	"io/ioutil"
 	"os"
 	"path/filepath"
@@ -99,27 +100,6 @@ func TestProvisionerPrepare_EmptyDestination(t *testing.T) {
 	}
 }
 
-type stubUi struct {
-	sayMessages string
-}
-
-func (su *stubUi) Ask(string) (string, error) {
-	return "", nil
-}
-
-func (su *stubUi) Error(string) {
-}
-
-func (su *stubUi) Machine(string, ...string) {
-}
-
-func (su *stubUi) Message(string) {
-}
-
-func (su *stubUi) Say(msg string) {
-	su.sayMessages += msg
-}
-
 func TestProvisionerProvision_SendsFile(t *testing.T) {
 	var p Provisioner
 	tf, err := ioutil.TempFile("", "packer")
@@ -141,18 +121,21 @@ func TestProvisionerProvision_SendsFile(t *testing.T) {
 		t.Fatalf("err: %s", err)
 	}
 
-	ui := &stubUi{}
+	b := bytes.NewBuffer(nil)
+	ui := &packer.BasicUi{
+		Writer: b,
+	}
 	comm := &packer.MockCommunicator{}
 	err = p.Provision(ui, comm)
 	if err != nil {
 		t.Fatalf("should successfully provision: %s", err)
 	}
 
-	if !strings.Contains(ui.sayMessages, tf.Name()) {
+	if !strings.Contains(b.String(), tf.Name()) {
 		t.Fatalf("should print source filename")
 	}
 
-	if !strings.Contains(ui.sayMessages, "something") {
+	if !strings.Contains(b.String(), "something") {
 		t.Fatalf("should print destination filename")
 	}
 
@@ -197,18 +180,21 @@ func TestProvisionDownloadMkdirAll(t *testing.T) {
 		if err := p.Prepare(config); err != nil {
 			t.Fatalf("err: %s", err)
 		}
-		ui := &stubUi{}
+		b := bytes.NewBuffer(nil)
+		ui := &packer.BasicUi{
+			Writer: b,
+		}
 		comm := &packer.MockCommunicator{}
 		err = p.ProvisionDownload(ui, comm)
 		if err != nil {
 			t.Fatalf("should successfully provision: %s", err)
 		}
 
-		if !strings.Contains(ui.sayMessages, tf.Name()) {
+		if !strings.Contains(b.String(), tf.Name()) {
 			t.Fatalf("should print source filename")
 		}
 
-		if !strings.Contains(ui.sayMessages, "something") {
+		if !strings.Contains(b.String(), "something") {
 			t.Fatalf("should print destination filename")
 		}
 

provisioner/powershell/provisioner.go

@@ -17,6 +17,7 @@ import (
 
 	"github.com/hashicorp/packer/common"
 	"github.com/hashicorp/packer/common/uuid"
+	commonhelper "github.com/hashicorp/packer/helper/common"
 	"github.com/hashicorp/packer/helper/config"
 	"github.com/hashicorp/packer/packer"
 	"github.com/hashicorp/packer/template/interpolate"
@@ -97,9 +98,6 @@ type Config struct {
 	// Changes will not be effective until the system is rebooted."
 	ValidExitCodes []int `mapstructure:"valid_exit_codes"`
 
-	// internal variable, to be written to at provisioner run time.
-	winrmpassword string
-
 	ctx interpolate.Context
 }
 
@@ -119,28 +117,8 @@ type EnvVarsTemplate struct {
 }
 
 func (p *Provisioner) Prepare(raws ...interface{}) error {
-	// This is a bit of a hack. For provisioners that need access to
-	// auto-generated WinRMPasswords, the mechanism of keeping provisioner data
-	// and build data totally segregated breaks down. We get around this by
-	// having the builder stash the WinRMPassword in the state bag, then
-	// grabbing it out of the statebag inside of StepProvision.  Then, when
-	// the time comes to provision for real, we run the prepare step one more
-	// time, now with WinRMPassword defined in the raws, and can store the
-	// password on the provisioner config without overwriting the rest of the
-	// work we've already done in the first prepare run.
-	if len(raws) == 1 {
-		for k, v := range raws[0].(map[interface{}]interface{}) {
-			if k.(string) == "WinRMPassword" {
-				p.config.winrmpassword = v.(string)
-				// Even if WinRMPassword is not gonna be used, we've stored the
-				// key and pointed it to an empty string. That means we'll
-				// always reach this on our second-run of Prepare()
-				return nil
-			}
-		}
-	}
-
-	// Only get here if it's the first time the Provisioner's Prepare is run
+	// Create passthrough for winrm password so we can fill it in once we know
+	// it
 	p.config.ctx.Data = &EnvVarsTemplate{
 		WinRMPassword: `{{.WinRMPassword}}`,
 	}
@@ -157,7 +135,7 @@ func (p *Provisioner) Prepare(raws ...interface{}) error {
 	}, raws...)
 
 	if err != nil {
-		return fmt.Errorf("Error decoding powershell provisioner template: %s", err)
+		return err
 	}
 
 	if p.config.EnvVarFormat == "" {
@@ -283,7 +261,7 @@ func extractScript(p *Provisioner) (string, error) {
 func (p *Provisioner) Provision(ui packer.Ui, comm packer.Communicator) error {
 	ui.Say(fmt.Sprintf("Provisioning with Powershell..."))
 	p.communicator = comm
-	packer.LogSecretFilter.Set(p.config.winrmpassword)
+
 	scripts := make([]string, len(p.config.Scripts))
 	copy(scripts, p.config.Scripts)
 
@@ -414,7 +392,7 @@ func (p *Provisioner) createFlattenedEnvVars(elevated bool) (flattened string) {
 
 	// interpolate environment variables
 	p.config.ctx.Data = &EnvVarsTemplate{
-		WinRMPassword: p.config.winrmpassword,
+		WinRMPassword: getWinRMPassword(p.config.PackerBuildName),
 	}
 	// Split vars into key/value components
 	for _, envVar := range p.config.Vars {
@@ -489,7 +467,7 @@ func (p *Provisioner) createCommandTextNonPrivileged() (command string, err erro
 	p.config.ctx.Data = &ExecuteCommandTemplate{
 		Path:          p.config.RemotePath,
 		Vars:          p.config.RemoteEnvVarPath,
-		WinRMPassword: p.config.winrmpassword,
+		WinRMPassword: getWinRMPassword(p.config.PackerBuildName),
 	}
 	command, err = interpolate.Render(p.config.ExecuteCommand, &p.config.ctx)
 
@@ -501,6 +479,12 @@ func (p *Provisioner) createCommandTextNonPrivileged() (command string, err erro
 	return command, nil
 }
 
+func getWinRMPassword(buildName string) string {
+	winRMPass, _ := commonhelper.RetrieveSharedState("winrm_password", buildName)
+	packer.LogSecretFilter.Set(winRMPass)
+	return winRMPass
+}
+
 func (p *Provisioner) createCommandTextPrivileged() (command string, err error) {
 	// Prepare everything needed to enable the required env vars within the
 	// remote environment
@@ -512,7 +496,7 @@ func (p *Provisioner) createCommandTextPrivileged() (command string, err error)
 	p.config.ctx.Data = &ExecuteCommandTemplate{
 		Path:          p.config.RemotePath,
 		Vars:          p.config.RemoteEnvVarPath,
-		WinRMPassword: p.config.winrmpassword,
+		WinRMPassword: getWinRMPassword(p.config.PackerBuildName),
 	}
 	command, err = interpolate.Render(p.config.ElevatedExecuteCommand, &p.config.ctx)
 	if err != nil {
@@ -571,7 +555,7 @@ func (p *Provisioner) generateElevatedRunner(command string) (uploadedPath strin
 	}
 	// Replace ElevatedPassword for winrm users who used this feature
 	p.config.ctx.Data = &EnvVarsTemplate{
-		WinRMPassword: p.config.winrmpassword,
+		WinRMPassword: getWinRMPassword(p.config.PackerBuildName),
 	}
 
 	p.config.ElevatedPassword, _ = interpolate.Render(p.config.ElevatedPassword, &p.config.ctx)

provisioner/powershell/provisioner_test.go

@@ -14,8 +14,8 @@ import (
 	"github.com/hashicorp/packer/packer"
 )
 
-func testConfig() map[interface{}]interface{} {
-	return map[interface{}]interface{}{
+func testConfig() map[string]interface{} {
+	return map[string]interface{}{
 		"inline": []interface{}{"foo", "bar"},
 	}
 }

provisioner/shell-local/provisioner.go

@@ -10,27 +10,6 @@ type Provisioner struct {
 }
 
 func (p *Provisioner) Prepare(raws ...interface{}) error {
-	// This is a bit of a hack. For provisioners that need access to
-	// auto-generated WinRMPasswords, the mechanism of keeping provisioner data
-	// and build data totally segregated breaks down. We get around this by
-	// having the builder stash the WinRMPassword in the state bag, then
-	// grabbing it out of the statebag inside of StepProvision.  Then, when
-	// the time comes to provision for real, we run the prepare step one more
-	// time, now with WinRMPassword defined in the raws, and can store the
-	// password on the provisioner config without overwriting the rest of the
-	// work we've already done in the first prepare run.
-	if len(raws) == 1 {
-		for k, v := range raws[0].(map[interface{}]interface{}) {
-			if k.(string) == "WinRMPassword" {
-				p.config.WinRMPassword = v.(string)
-				// Even if WinRMPassword is not gonna be used, we've stored the
-				// key and pointed it to an empty string. That means we'll
-				// always reach this on our second-run of Prepare()
-				return nil
-			}
-		}
-	}
-
 	err := sl.Decode(&p.config, raws...)
 	if err != nil {
 		return err

provisioner/shell-local/provisioner_test.go

@@ -48,8 +48,8 @@ func TestConfigPrepare(t *testing.T) {
 	}
 }
 
-func testConfig(t *testing.T) map[interface{}]interface{} {
-	return map[interface{}]interface{}{
+func testConfig(t *testing.T) map[string]interface{} {
+	return map[string]interface{}{
 		"command": "echo foo",
 	}
 }

version/version.go

@@ -9,12 +9,12 @@ import (
 var GitCommit string
 
 // The main version number that is being run at the moment.
-const Version = "1.2.6"
+const Version = "1.3.0"
 
 // A pre-release marker for the version. If this is "" (empty string)
 // then it means that it is a final release. Otherwise, this is a pre-release
 // such as "dev" (in development), "beta", "rc1", etc.
-const VersionPrerelease = "dev"
+const VersionPrerelease = ""
 
 func FormattedVersion() string {
 	var versionString bytes.Buffer

website/config.rb

@@ -2,7 +2,7 @@ set :base_url, "https://www.packer.io/"
 
 activate :hashicorp do |h|
   h.name         = "packer"
-  h.version      = "1.2.5"
+  h.version      = "1.3.0"
   h.github_slug  = "hashicorp/packer"
   h.website_root = "website"
 end

website/source/docs/builders/amazon-chroot.html.md

@@ -254,6 +254,11 @@ each category, the available configuration keys are alphabetized.
     of the `source_ami` unless `from_scratch` is `true`, in which case
     this field must be defined.
 
+-   `root_volume_type` (string) - The type of EBS volume for the chroot environment
+     and resulting AMI. The default value is the type of the `source_ami`, unless
+    `from_scratch` is true, in which case the default value is `gp2`. You can only
+     specify `io1` if building based on top of a `source_ami` which is also `io1`.
+
 -   `root_volume_tags` (object of key/value strings) - Tags to apply to the volumes
     that are *launched*. This is a
     [template engine](/docs/templates/engine.html),

website/source/docs/builders/amazon-ebs.html.md

@@ -142,6 +142,11 @@ builder.
 -   `availability_zone` (string) - Destination availability zone to launch
     instance in. Leave this empty to allow Amazon to auto-assign.
 
+-   `block_duration_minutes` (int64) - Requires `spot_price` to
+    be set. The required duration for the Spot Instances (also known as Spot blocks).
+    This value must be a multiple of 60 (60, 120, 180, 240, 300, or 360).
+    You can't specify an Availability Zone group or a launch group if you specify a duration.
+
 -   `custom_endpoint_ec2` (string) - This option is useful if you use a cloud
     provider whose API is compatible with aws EC2. Specify another endpoint
     like this `https://ec2.custom.endpoint.com`.

website/source/docs/builders/amazon-ebssurrogate.html.md

@@ -135,6 +135,11 @@ builder.
 -   `availability_zone` (string) - Destination availability zone to launch
     instance in. Leave this empty to allow Amazon to auto-assign.
 
+-   `block_duration_minutes` (int64) - Requires `spot_price` to
+    be set. The required duration for the Spot Instances (also known as Spot blocks).
+    This value must be a multiple of 60 (60, 120, 180, 240, 300, or 360).
+    You can't specify an Availability Zone group or a launch group if you specify a duration.
+
 -   `custom_endpoint_ec2` (string) - This option is useful if you use a cloud
     provider whose API is compatible with aws EC2. Specify another endpoint
     like this `https://ec2.custom.endpoint.com`.

website/source/docs/builders/amazon-ebsvolume.html.md

@@ -107,6 +107,11 @@ builder.
 -   `availability_zone` (string) - Destination availability zone to launch
     instance in. Leave this empty to allow Amazon to auto-assign.
 
+-   `block_duration_minutes` (int64) - Requires `spot_price` to
+    be set. The required duration for the Spot Instances (also known as Spot blocks).
+    This value must be a multiple of 60 (60, 120, 180, 240, 300, or 360).
+    You can't specify an Availability Zone group or a launch group if you specify a duration.
+
 -   `custom_endpoint_ec2` (string) - This option is useful if you use a cloud
     provider whose API is compatible with aws EC2. Specify another endpoint
     like this `https://ec2.custom.endpoint.com`.

website/source/docs/builders/amazon-instance.html.md

@@ -164,6 +164,11 @@ builder.
 -   `availability_zone` (string) - Destination availability zone to launch
     instance in. Leave this empty to allow Amazon to auto-assign.
 
+-   `block_duration_minutes` (int64) - Requires `spot_price` to
+    be set. The required duration for the Spot Instances (also known as Spot blocks).
+    This value must be a multiple of 60 (60, 120, 180, 240, 300, or 360).
+    You can't specify an Availability Zone group or a launch group if you specify a duration.
+
 -   `bundle_destination` (string) - The directory on the running instance where
     the bundled AMI will be saved prior to uploading. By default this is `/tmp`.
     This directory must exist and be writable.

website/source/docs/builders/vmware-iso.html.md.erb

@@ -146,8 +146,9 @@ builder.
     The default is `1`.
 
     For ESXi, this defaults to `zeroedthick`. The available options for ESXi
-    are: `zeroedthick`, `eagerzeroedthick`, `thin`, `rdm:dev`, `rdmp:dev`,
-    `2gbsparse`.
+    are: `zeroedthick`, `eagerzeroedthick`, `thin`. `rdm:dev`, `rdmp:dev`,
+    `2gbsparse` are not supported. Due to default disk compaction, when using
+    `zeroedthick` or `eagerzeroedthick` set `skip_compaction` to `true`.
 
     For more information, please consult the [Virtual Disk Manager User's
     Guide](https://www.vmware.com/pdf/VirtualDiskManager.pdf) for desktop
@@ -326,10 +327,12 @@ builder.
     `5m` or five minutes.
 
 -   `skip_compaction` (boolean) - VMware-created disks are defragmented and
-    compacted at the end of the build process using `vmware-vdiskmanager`. In
-    certain rare cases, this might actually end up making the resulting disks
-    slightly larger. If you find this to be the case, you can disable compaction
-    using this configuration value. Defaults to `false`.
+    compacted at the end of the build process using `vmware-vdiskmanager` or
+    `vmkfstools` in ESXi. In certain rare cases, this might actually end up
+    making the resulting disks slightly larger. If you find this to be the case,
+    you can disable compaction using this configuration value. Defaults to
+    `false`. Default to `true` for ESXi when `disk_type_id` is not explicitly
+    defined and `false` otherwise.
 
 -   `skip_export` (boolean) - Defaults to `false`. When enabled, Packer will
     not export the VM. Useful if the build output is not the resultant
@@ -341,10 +344,9 @@ builder.
     section below for more info.
 
 -   `keep_registered` (boolean) - Set this to `true` if you would like to keep
-    the VM registered with the remote ESXi server. This is convenient if you
-    use packer to provision VMs on ESXi and don't want to use ovftool to
-    deploy the resulting artifact (VMX or OVA or whatever you used as `format`).
-    Defaults to `false`.
+    the VM registered with the remote ESXi server. If you do not need to export
+    the vm, then also set `skip_export: true` in order to avoid an unnecessary
+    step of using ovftool to export the vm. Defaults to `false`.
 
 -   `ovftool_options` (array of strings) - Extra options to pass to ovftool
     during export. Each item in the array is a new argument. The options

website/source/docs/other/debugging.html.md

@@ -33,6 +33,14 @@ provided when `PACKER_LOG=1` environment variable is set prior to a build,
 and you can connect to the local machine using the userid and password defined
 in the kickstart or preseed associated with initializing the local VM.
 
+It should be noted that one of the options `-on-error` is to `retry`, the retry
+of the step in question has limitations:
+
+ * the template packer is building is **not** reloaded from file.
+ * the resources specified from builders **are** reloaded from file.
+
+Check the specfics on your builder to confirm their behavior.
+
 ### Windows
 
 As of Packer 0.8.1 the default WinRM communicator will emit the password for a

website/source/docs/post-processors/vsphere.html.md

@@ -43,6 +43,11 @@ Required:
 
 Optional:
 
+-   `esxi_host` (string) - Target vSphere host. Used to assign specific esx host
+    to upload the resulting VM to, when a vCenter Server is used as `host`. Can be
+    either a hostname (e.g. "packer-esxi1", requires proper DNS setup and/or correct
+    DNS search domain setting) or an ipv4 address.
+
 -   `disk_mode` (string) - Target disk format. See `ovftool` manual for
     available options. By default, "thick" will be used.
 

website/source/docs/templates/communicator.html.md

@@ -71,7 +71,7 @@ The SSH communicator has the following options:
 -   `ssh_bastion_password` (string) - The password to use to authenticate
     with the bastion host.
 
--   `ssh_bastion_port` (number) - The port of the bastion host. Defaults to `1`.
+-   `ssh_bastion_port` (number) - The port of the bastion host. Defaults to `22`.
 
 -   `ssh_bastion_private_key_file` (string) - A private key file to use
     to authenticate with the bastion host.