#packer (2019-08)
Discuss Packer for building AMIs and Docker Images
Archive: https://archive.sweetops.com/packer/
2019-08-01
Hello here, i have a question about packer :
From what I understand about packer is that it creates a temporary AMI from the EC2 that packer has booted and then this temporary AMI is copied … except that currently I only work in one AWS region and so I do not need it to create a copy in the same region as the temporary AMI. Is there a way to not copy the MAI and use the 1st AMI? Because the copy operation takes a lot of time at AWS.
@julien M. I don’t think it creates a temp AMI. It starts, provisions, and stops an instance before creating an AMI of the stopped instance.
2019-08-02
yep exeactly, that’s the process : start ec2 -> execute some task -> stop ec2 -> create temp ami -> copy this ami to “final” AMI
tou can see it in my log :
==> amazon-ebs: Stopping the source instance...
amazon-ebs: Stopping instance
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating AMI JlUwHnk from instance i-0a32e679185b6d6e0
amazon-ebs: AMI: ami-09ebc6af6f029a3a5
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Copying/Encrypting AMI (ami-09ebc6af6f029a3a5) to other regions...
amazon-ebs: Copying to: eu-central-1
amazon-ebs: Waiting for all copies to complete...
==> amazon-ebs: Modifying attributes on AMI (ami-0b1906f336702fff5)...
amazon-ebs: Modifying: description
amazon-ebs: Modifying: users
==> amazon-ebs: Modifying attributes on snapshot (snap-066093425d32275fd)...
==> amazon-ebs: Adding tags to AMI (ami-0b1906f336702fff5)...
==> amazon-ebs: Tagging snapshot: snap-066093425d32275fd
==> amazon-ebs: Creating AMI tags
amazon-ebs: Adding tag: "Created": "1564649441"
amazon-ebs: Adding tag: "Project": "lunchr-banking"
amazon-ebs: Adding tag: "Team": "banking"
amazon-ebs: Adding tag: "Name": "packer.basic"
==> amazon-ebs: Creating snapshot tags
==> amazon-ebs: Deregistering the AMI and deleting unencrypted temporary AMIs and snapshots
==> amazon-ebs: Deregistered AMI id: ami-09ebc6af6f029a3a5
==> amazon-ebs: Deleted snapshot: snap-028ddd3a7cb4ca49c
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
Build 'amazon-ebs' finished.
`
the “temp” AMI log :
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating AMI JlUwHnk from instance i-0a32e679185b6d6e0
amazon-ebs: AMI: ami-09ebc6af6f029a3a5
==> amazon-ebs: Waiting for AMI to become ready..
`
and the copy of this “temp” AMI to “final” AMI :
==> amazon-ebs: Copying/Encrypting AMI (ami-09ebc6af6f029a3a5) to other regions...
amazon-ebs: Copying to: eu-central-1
amazon-ebs: Waiting for all copies to complete...
==> amazon-ebs: Modifying attributes on AMI (ami-0b1906f336702fff5)...
amazon-ebs: Modifying: description
amazon-ebs: Modifying: users
==> amazon-ebs: Modifying attributes on snapshot (snap-066093425d32275fd)...
==> amazon-ebs: Adding tags to AMI (ami-0b1906f336702fff5)...
==> amazon-ebs: Tagging snapshot: snap-066093425d32275fd
but i don’t understand this step … why a copy of the first ami …
especially that the AMI I need is already in the right region
2019-08-05
@julien M. mind posting the packer.json we may be able to help diagnose.
Its definitely something with your config.
==> APL - AWS AMI Builder: Provisioning with shell script: /var/folders/tz/wdr45bjs0rgd12w8qv3qn3h00000gn/T/packer-shell983168744
==> APL - AWS AMI Builder: Stopping the source instance...
APL - AWS AMI Builder: Stopping instance
==> APL - AWS AMI Builder: Waiting for the instance to stop...
==> APL - AWS AMI Builder: Creating AMI xxx-Latest-AMZN-05-Aug-19 09_45_53 from instance i-xxx
APL - AWS AMI Builder: AMI: ami-xxx
==> APL - AWS AMI Builder: Waiting for AMI to become ready...
==> APL - AWS AMI Builder: Modifying attributes on AMI (ami-xxx)...
APL - AWS AMI Builder: Modifying: description
==> APL - AWS AMI Builder: Modifying attributes on snapshot (snap-xxx)...
==> APL - AWS AMI Builder: Adding tags to AMI (ami-xxx)...
==> APL - AWS AMI Builder: Tagging snapshot: snap-xxx
==> APL - AWS AMI Builder: Creating AMI tags
APL - AWS AMI Builder: Adding tag: "Name": "xxx-Latest-AMZN-05-Aug-19 09_45_53"
==> APL - AWS AMI Builder: Creating snapshot tags
APL - AWS AMI Builder: Adding tag: "Name": "xxx-Latest-AMZN-05-Aug-19 09_45_53"
==> APL - AWS AMI Builder: Terminating the source AWS instance...
==> APL - AWS AMI Builder: Cleaning up any extra volumes...
==> APL - AWS AMI Builder: No volumes to clean up, skipping
==> APL - AWS AMI Builder: Deleting temporary security group...
==> APL - AWS AMI Builder: Deleting temporary keypair...
Build 'APL - AWS AMI Builder' finished.
2019-08-06
hello @davidvasandani, see below my packer.json :
{
"variables": {
"aws_region": "{{env `PACKER_REGION`}}",
"aws_profile": "{{env `PACKER_PROFILE`}}",
"subnet_id": "{{env `PACKER_SUBNET_ID`}}",
"source_ami_id": "{{env `PACKER_SOURCE_AMI_ID`}}",
"allowed_users_to_launch": "{{env `PACKER_ALLOWED_USER`}}",
"ami_name": "xxx-{{timestamp}}",
"creator": "{{env `USER`}}",
"instance_type": "t3.large",
"encrypted": "false",
"kms_key_id": "",
"datadog_api_key": "{{env `DD_API_KEY`}}",
"environment": "{{env `ENVIRONMENT`}}"
},
"builders": [
{
"type": "amazon-ebs",
"profile": "{{user `aws_profile`}}",
"region": "{{user `aws_region`}}",
"associate_public_ip_address": "true",
"ami_users": "{{ user `allowed_users_to_launch`}}",
"source_ami": "{{user `source_ami_id`}}",
"instance_type": "{{user `instance_type`}}",
"ami_name": "{{user `ami_name`}}",
"ami_description": "xxx",
"encrypt_boot": false,
"kms_key_id": "{{user `kms_key_id`}}",
"ssh_username": "app",
"ssh_private_key_file": "custom-files/ami.key",
"subnet_id": "{{user `subnet_id`}}",
"tags": {
"Created": "{{timestamp}}",
"Project": "xxx",
"Team": "xxx",
"Name": "packer.basic"
}
}
],
"provisioners": [
{
"type": "file",
"source": "../../../xxx.tgz",
"destination": "/opt/app/xxxx.tgz"
},
{
"type": "file",
"source": "./custom-files/datadog.yaml",
"destination": "/tmp/datadog.yaml"
},
{
"type": "file",
"source": "./custom-files/xxx.service",
"destination": "/tmp/xxx.service"
},
{
"type": "file",
"source": "./custom-files/xxx.logrotate",
"destination": "/tmp/xxxx.logrotate"
},
{
"type": "file",
"source": "./custom-files/datadog-ruby.yml",
"destination": "/tmp/ruby-conf.yml"
},
{
"type": "shell",
"environment_vars": [
"DD_API_KEY={{user `datadog_api_key`}}",
"ENV={{user `environment`}}"
],
"script": "ami-app-bootstrap.sh",
"skip_clean": "true",
"pause_before": "10s",
"timeout": "10s"
}
]
}
2019-08-14
hi here, any idea about my problem ?
@julien M. sorry dropped off there.
{
"variables": {
"vpc": "{{env `BUILD_VPC_ID`}}",
"subnet": "{{env `BUILD_SUBNET_ID`}}",
"aws_region": "{{env `AWS_REGION`}}",
"ami_name": "Latest-AMZN-{{isotime \"02-Jan-06 03_04_05\"}}"
},
"builders": [{
"name": "AWS AMI Builder",
"type": "amazon-ebs",
"region": "{{user `aws_region`}}",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "amzn2-ami-ecs-hvm-2.0.*-x86_64-ebs",
"root-device-type": "ebs"
},
"owners": ["137112412989", "591542846629", "801119661308",
"102837901569", "013907871322", "206029621532",
"286198878708", "443319210888"
],
"most_recent": true
},
"instance_type": "t2.micro",
"ssh_username": "ec2-user",
"ami_name": "{{user `ami_name` | clean_ami_name}}",
"tags": {
"Name": "{{user `ami_name`}}"
},
"run_tags": {
"Name": "{{user `ami_name`}}"
},
"run_volume_tags": {
"Name": "{{user `ami_name`}}"
},
"snapshot_tags": {
"Name": "{{user `ami_name`}}"
},
"ami_description": "Amazon Linux",
"associate_public_ip_address": "true",
"vpc_id": "{{user `vpc`}}",
"subnet_id": "{{user `subnet`}}"
}],
"provisioners": [{
"type": "file",
"source": "/Users/davidvasandani/.ssh/vasandani.me_rsa.pub",
"destination": "/tmp/id_rsa.pub"
},
{
"type": "shell",
"execute_command": "echo '' | sudo -S su - root -c '{{ .Path }}'",
"script": "scripts/python.sh"
},
{
"type": "ansible-local",
"playbook_file": "ansible/playbook.yaml",
"role_paths": [
"ansible/roles/common"
],
"playbook_dir": "ansible",
"galaxy_file": "ansible/requirements.yaml"
},
{
"type": "shell",
"inline": [
"rm .ssh/authorized_keys ; sudo rm /root/.ssh/authorized_keys"
]
}
]
}
this is my config
@julien M. the difference between our configs is yours is missing "vpc_id"
Can you add that to the builders section.
2019-08-16
@julien M. any luck?
2019-08-17
hello @davidvasandani so, i have test with “vpc” var but it’s not the solution : i always have a temp AMI and after a copy of this AMI
so i have test with your template and i this case i don’t have a copy
so i use your template and i have made some modifications for my environment
i think my problem come from AMI-name or Tags …. when i wil some time for testing that
2019-08-20
Thanks for reporting back @julien M. !
2019-08-26
Hey everyone! I am looking for a simpler way to use packer in our circleCI workflow. I am currently using the machine execution and having to install everything on top (which is a lot of steps). Has anyone got a more elegant solution?
@Bruce can you give additional details? Whats the current issue you’re facing and what is your ideal state?
Thanks @davidvasandani. I am currently using circleCI to bake the image using machine execution (VM) which then in each step (job) I install packer, set environment variables for AWS creds, pull secrets in from SSM and then use these to build. Its a lot of steps as I found I can not group them in one as the Env to pass through. I was hoping there was an easier way etc building a docker container that can encapsulate what I need. But I haven’t tried that yet.
@Bruce Sorry, you’re using Packer to build a docker image? Highly highly recommend you switch to using a Dockerfile.
It will work much better in CircleCI.
and for your general sanity.
Thanks @davidvasandani I managed to get this working creating a Dockerfile to do the work with circleCI. All kicked off with a script. It was a lot simpler.