#packer (2019-08)

packer

Discuss Packer for building AMIs and Docker Images Archive: https://archive.sweetops.com/packer/

2019-08-28

2019-08-26

Bruce avatar
Bruce

Hey everyone! I am looking for a simpler way to use packer in our circleCI workflow. I am currently using the machine execution and having to install everything on top (which is a lot of steps). Has anyone got a more elegant solution?

davidvasandani avatar
davidvasandani

@Bruce can you give additional details? Whats the current issue you’re facing and what is your ideal state?

Bruce avatar
Bruce

Thanks @davidvasandani. I am currently using circleCI to bake the image using machine execution (VM) which then in each step (job) I install packer, set environment variables for AWS creds, pull secrets in from SSM and then use these to build. Its a lot of steps as I found I can not group them in one as the Env to pass through. I was hoping there was an easier way etc building a docker container that can encapsulate what I need. But I haven’t tried that yet.

davidvasandani avatar
davidvasandani

@Bruce Sorry, you’re using Packer to build a docker image? Highly highly recommend you switch to using a Dockerfile.

davidvasandani avatar
davidvasandani

It will work much better in CircleCI.

davidvasandani avatar
davidvasandani

and for your general sanity.

Bruce avatar
Bruce

Thanks @davidvasandani I managed to get this working creating a Dockerfile to do the work with circleCI. All kicked off with a script. It was a lot simpler.

:100:1

2019-08-20

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks for reporting back @julien M. !

2019-08-17

julien M. avatar
julien M.

hello @davidvasandani so, i have test with “vpc” var but it’s not the solution : i always have a temp AMI and after a copy of this AMI

julien M. avatar
julien M.

so i have test with your template and i this case i don’t have a copy

julien M. avatar
julien M.

so i use your template and i have made some modifications for my environment

julien M. avatar
julien M.

i think my problem come from AMI-name or Tags …. when i wil some time for testing that

julien M. avatar
julien M.

thank you for your time, with you my build job time decrease to 8min !!!

2
:100:2

2019-08-16

davidvasandani avatar
davidvasandani

@julien M. any luck?

2019-08-14

julien M. avatar
julien M.

hi here, any idea about my problem ?

davidvasandani avatar
davidvasandani

@julien M. sorry dropped off there.

davidvasandani avatar
davidvasandani
{
  "variables": {
    "vpc": "{{env `BUILD_VPC_ID`}}",
    "subnet": "{{env `BUILD_SUBNET_ID`}}",
    "aws_region": "{{env `AWS_REGION`}}",
    "ami_name": "Latest-AMZN-{{isotime \"02-Jan-06 03_04_05\"}}"
  },
  "builders": [{
    "name": "AWS AMI Builder",
    "type": "amazon-ebs",
    "region": "{{user `aws_region`}}",
    "source_ami_filter": {
      "filters": {
        "virtualization-type": "hvm",
        "name": "amzn2-ami-ecs-hvm-2.0.*-x86_64-ebs",
        "root-device-type": "ebs"
      },
      "owners": ["137112412989", "591542846629", "801119661308",
        "102837901569", "013907871322", "206029621532",
        "286198878708", "443319210888"
      ],
      "most_recent": true
    },
    "instance_type": "t2.micro",
    "ssh_username": "ec2-user",
    "ami_name": "{{user `ami_name` | clean_ami_name}}",
    "tags": {
      "Name": "{{user `ami_name`}}"
    },
    "run_tags": {
      "Name": "{{user `ami_name`}}"
    },
    "run_volume_tags": {
      "Name": "{{user `ami_name`}}"
    },
    "snapshot_tags": {
      "Name": "{{user `ami_name`}}"
    },
    "ami_description": "Amazon Linux",
    "associate_public_ip_address": "true",
    "vpc_id": "{{user `vpc`}}",
    "subnet_id": "{{user `subnet`}}"
  }],
  "provisioners": [{
      "type": "file",
      "source": "/Users/davidvasandani/.ssh/vasandani.me_rsa.pub",
      "destination": "/tmp/id_rsa.pub"
    },
    {
      "type": "shell",
      "execute_command": "echo '' | sudo -S su - root -c '{{ .Path }}'",
      "script": "scripts/python.sh"
    },
    {
      "type": "ansible-local",
      "playbook_file": "ansible/playbook.yaml",
      "role_paths": [
        "ansible/roles/common"
      ],
      "playbook_dir": "ansible",
      "galaxy_file": "ansible/requirements.yaml"
    },
    {
      "type": "shell",
      "inline": [
        "rm .ssh/authorized_keys ; sudo rm /root/.ssh/authorized_keys"
      ]
    }
  ]
}
davidvasandani avatar
davidvasandani

this is my config

davidvasandani avatar
davidvasandani

@julien M. the difference between our configs is yours is missing "vpc_id"

davidvasandani avatar
davidvasandani

Can you add that to the builders section.

julien M. avatar
julien M.

oh great !!! i test this quickly

:100:1

2019-08-06

julien M. avatar
julien M.

hello @davidvasandani, see below my packer.json :

julien M. avatar
julien M.
{
  "variables": {
    "aws_region": "{{env `PACKER_REGION`}}",
    "aws_profile": "{{env `PACKER_PROFILE`}}",
    "subnet_id": "{{env `PACKER_SUBNET_ID`}}",
    "source_ami_id": "{{env `PACKER_SOURCE_AMI_ID`}}",
    "allowed_users_to_launch": "{{env `PACKER_ALLOWED_USER`}}",
    "ami_name": "xxx-{{timestamp}}",
    "creator": "{{env `USER`}}",
    "instance_type": "t3.large",
    "encrypted": "false",
    "kms_key_id": "",
    "datadog_api_key": "{{env `DD_API_KEY`}}",
    "environment": "{{env `ENVIRONMENT`}}"
  },

  "builders": [
    {
      "type": "amazon-ebs",
      "profile": "{{user `aws_profile`}}",
      "region": "{{user `aws_region`}}",
      "associate_public_ip_address": "true",
      "ami_users": "{{ user `allowed_users_to_launch`}}",
      "source_ami": "{{user `source_ami_id`}}",
      "instance_type": "{{user `instance_type`}}",
      "ami_name": "{{user `ami_name`}}",
      "ami_description": "xxx",
      "encrypt_boot": false,
      "kms_key_id": "{{user `kms_key_id`}}",
      "ssh_username": "app",
      "ssh_private_key_file": "custom-files/ami.key",
      "subnet_id": "{{user `subnet_id`}}",
      "tags": {
        "Created": "{{timestamp}}",
        "Project": "xxx",
        "Team": "xxx",
        "Name": "packer.basic"
      }
    }
  ],

  "provisioners": [
    {
      "type": "file",
      "source": "../../../xxx.tgz",
      "destination": "/opt/app/xxxx.tgz"
    },
    {
      "type": "file",
      "source": "./custom-files/datadog.yaml",
      "destination": "/tmp/datadog.yaml"
    },
    {
      "type": "file",
      "source": "./custom-files/xxx.service",
      "destination": "/tmp/xxx.service"
    },
    {
      "type": "file",
      "source": "./custom-files/xxx.logrotate",
      "destination": "/tmp/xxxx.logrotate"
    },
    {
      "type": "file",
      "source": "./custom-files/datadog-ruby.yml",
      "destination": "/tmp/ruby-conf.yml"
    },
    {
      "type": "shell",
      "environment_vars": [
        "DD_API_KEY={{user `datadog_api_key`}}",
        "ENV={{user `environment`}}"
        ],
      "script": "ami-app-bootstrap.sh",
      "skip_clean": "true",
      "pause_before": "10s",
      "timeout": "10s"
    }
  ]
}

2019-08-05

davidvasandani avatar
davidvasandani

@julien M. mind posting the packer.json we may be able to help diagnose.

davidvasandani avatar
davidvasandani

Its definitely something with your config.

==> APL - AWS AMI Builder: Provisioning with shell script: /var/folders/tz/wdr45bjs0rgd12w8qv3qn3h00000gn/T/packer-shell983168744
==> APL - AWS AMI Builder: Stopping the source instance...
    APL - AWS AMI Builder: Stopping instance
==> APL - AWS AMI Builder: Waiting for the instance to stop...
==> APL - AWS AMI Builder: Creating AMI xxx-Latest-AMZN-05-Aug-19 09_45_53 from instance i-xxx
    APL - AWS AMI Builder: AMI: ami-xxx
==> APL - AWS AMI Builder: Waiting for AMI to become ready...
==> APL - AWS AMI Builder: Modifying attributes on AMI (ami-xxx)...
    APL - AWS AMI Builder: Modifying: description
==> APL - AWS AMI Builder: Modifying attributes on snapshot (snap-xxx)...
==> APL - AWS AMI Builder: Adding tags to AMI (ami-xxx)...
==> APL - AWS AMI Builder: Tagging snapshot: snap-xxx
==> APL - AWS AMI Builder: Creating AMI tags
    APL - AWS AMI Builder: Adding tag: "Name": "xxx-Latest-AMZN-05-Aug-19 09_45_53"
==> APL - AWS AMI Builder: Creating snapshot tags
    APL - AWS AMI Builder: Adding tag: "Name": "xxx-Latest-AMZN-05-Aug-19 09_45_53"
==> APL - AWS AMI Builder: Terminating the source AWS instance...
==> APL - AWS AMI Builder: Cleaning up any extra volumes...
==> APL - AWS AMI Builder: No volumes to clean up, skipping
==> APL - AWS AMI Builder: Deleting temporary security group...
==> APL - AWS AMI Builder: Deleting temporary keypair...
Build 'APL - AWS AMI Builder' finished.

2019-08-02

julien M. avatar
julien M.

yep exeactly, that’s the process : start ec2 -> execute some task -> stop ec2 -> create temp ami -> copy this ami to “final” AMI

julien M. avatar
julien M.

tou can see it in my log :

julien M. avatar
julien M.
==> amazon-ebs: Stopping the source instance...
    amazon-ebs: Stopping instance
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating AMI JlUwHnk from instance i-0a32e679185b6d6e0
    amazon-ebs: AMI: ami-09ebc6af6f029a3a5
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Copying/Encrypting AMI (ami-09ebc6af6f029a3a5) to other regions...
    amazon-ebs: Copying to: eu-central-1
    amazon-ebs: Waiting for all copies to complete...
==> amazon-ebs: Modifying attributes on AMI (ami-0b1906f336702fff5)...
    amazon-ebs: Modifying: description
    amazon-ebs: Modifying: users
==> amazon-ebs: Modifying attributes on snapshot (snap-066093425d32275fd)...
==> amazon-ebs: Adding tags to AMI (ami-0b1906f336702fff5)...
==> amazon-ebs: Tagging snapshot: snap-066093425d32275fd
==> amazon-ebs: Creating AMI tags
    amazon-ebs: Adding tag: "Created": "1564649441"
    amazon-ebs: Adding tag: "Project": "lunchr-banking"
    amazon-ebs: Adding tag: "Team": "banking"
    amazon-ebs: Adding tag: "Name": "packer.basic"
==> amazon-ebs: Creating snapshot tags
==> amazon-ebs: Deregistering the AMI and deleting unencrypted temporary AMIs and snapshots
==> amazon-ebs: Deregistered AMI id: ami-09ebc6af6f029a3a5
==> amazon-ebs: Deleted snapshot: snap-028ddd3a7cb4ca49c
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
Build 'amazon-ebs' finished.

`

julien M. avatar
julien M.

the “temp” AMI log :

==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating AMI JlUwHnk from instance i-0a32e679185b6d6e0
    amazon-ebs: AMI: ami-09ebc6af6f029a3a5
==> amazon-ebs: Waiting for AMI to become ready.. 

`

julien M. avatar
julien M.

and the copy of this “temp” AMI to “final” AMI :

==> amazon-ebs: Copying/Encrypting AMI (ami-09ebc6af6f029a3a5) to other regions...
    amazon-ebs: Copying to: eu-central-1
    amazon-ebs: Waiting for all copies to complete...
==> amazon-ebs: Modifying attributes on AMI (ami-0b1906f336702fff5)...
    amazon-ebs: Modifying: description
    amazon-ebs: Modifying: users
==> amazon-ebs: Modifying attributes on snapshot (snap-066093425d32275fd)...
==> amazon-ebs: Adding tags to AMI (ami-0b1906f336702fff5)...
==> amazon-ebs: Tagging snapshot: snap-066093425d32275fd
julien M. avatar
julien M.

but i don’t understand this step … why a copy of the first ami …

julien M. avatar
julien M.

especially that the AMI I need is already in the right region

2019-08-01

julien M. avatar
julien M.

Hello here, i have a question about packer :

From what I understand about packer is that it creates a temporary AMI from the EC2 that packer has booted and then this temporary AMI is copied … except that currently I only work in one AWS region and so I do not need it to create a copy in the same region as the temporary AMI. Is there a way to not copy the MAI and use the 1st AMI? Because the copy operation takes a lot of time at AWS.

davidvasandani avatar
davidvasandani

@julien M. I don’t think it creates a temp AMI. It starts, provisions, and stops an instance before creating an AMI of the stopped instance.

    keyboard_arrow_up