#codefresh (2019-04)

codefresh

Archive: https://archive.sweetops.com/codefresh/

2019-04-30

Michael Kolb avatar
Michael Kolb

Does Codefresh have API that accepts commands to start an existing pipeline? I have an integration test that will take hours and would like the option to start another pipeline once the integration tests complete.

dustinvb avatar
dustinvb
codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

2019-04-25

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
05:40:00 PM

Docker incident: Docker Hub elevated errors Apr 25, 17:33 UTC Investigating - Docker has reported an incident affecting Docker Hub Web and Docker Hub Automated Builds components. It shouldn’t affect normal pulling and pushing operations. We will keep monitoring this.

More information here: https://status.docker.com/

Docker incident: Docker Hub elevated errors

Codefresh’s Status Page - Docker incident: Docker Hub elevated errors.

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
06:29:59 PM

Docker incident: Docker Hub elevated errors Apr 25, 18:10 UTC Resolved - Docker has reported this incident as Resolved. “[Resolved] The issue has been resolved. The DB is backup again.”

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5cc1ed78790d0e1ca1c8fcd4 Apr 25, 17:33 UTC Investigating - Docker has reported an incident affecting Docker Hub Web and Docker Hub Automated Builds components. It shouldn’t affect normal pulling and pushing operations. We will keep monitoring this.

More…

Docker System Status

Our system status page is a real-time view of the performance and uptime of Docker products and services.

2019-04-24

oscarsullivan_old avatar
oscarsullivan_old

Hi.. anyone got experience using submodules and codefresh?

oscarsullivan_old avatar
oscarsullivan_old
version: '1.0'
stages:
 - build
 - push
 - prepare
 - deploy

steps:
  get_git_token:
    title: Reading Github token
    image: codefresh/cli
    commands:
      - cf_export GITHUB_TOKEN_EXPORT=$(codefresh get context github --decrypt -o yaml | yq -y .spec.data.auth.password)
  updateSubmodules:
    image: codefresh/cfstep-gitsubmodules
    environment:
      - GITHUB_TOKEN=${{GITHUB_TOKEN_EXPORT}}
      - CF_SUBMODULE_SYNC=true
      - CF_SUBMODULE_UPDATE_RECURSIVE=false
  debug:
    title: Debug Submodules
    image: codefresh/cli
    commands:
      - codefresh get contexts
      - ls -lah /codefresh/volume/my_app/models
      - ls -lah /codefresh/volume/my_app/lib/library
  BuildingDockerImage:
    title: Building Docker Image
    stage: "build"
    type: build
    image_name: ${{IMAGE_NAME}}
    working_directory: ./
    tag: '${{CF_BRANCH_TAG_NORMALIZED}}'
    dockerfile: pm2.Dockerfile
    build_arguments:
      - APP_ENV=.env-${{CF_PULL_REQUEST_TARGET}}
      - PORT=${{PORT}}
oscarsullivan_old avatar
oscarsullivan_old

So it gets my git token, updates the submodules in /codefresh/volume/my_app

oscarsullivan_old avatar
oscarsullivan_old

then does an ls pf those dirs to show me it has cloned (it has)

oscarsullivan_old avatar
oscarsullivan_old

and then in the docker build step I have: COPY models/* models/

oscarsullivan_old avatar
oscarsullivan_old

but the models/ dir is empty in the container..

oscarsullivan_old avatar
oscarsullivan_old

So I can only assume that /codefresh/volume/my_app is not the path that my_app is being cloned into during init and therefore not the context of Docker when building

dustinvb avatar
dustinvb

I would recommend trying to set the working_directory: ${{main_clone}} for the step to ensure you’re in the clone repository directory.

2019-04-23

Michael Kolb avatar
Michael Kolb

the docker hub release

Michael Kolb avatar
Michael Kolb

i found the command that i need kubectl config use-context <<cluster name>>

Michael Kolb avatar
Michael Kolb

the container had all the tools installed. Unfortunately, there was not documentation on the docker hub site.

2019-04-22

Michael Kolb avatar
Michael Kolb

Has anyone been using cf_export stating the name of an existing environment variable? I have tried to use it, but the variable output is getting altered. FreeStye Bash Code

 - FEATURE=$(echo ${{CF_BRANCH}} | cut -c 9- )
      - echo $FEATURE
      - cf_export $FEATURE

Output

TI-595                                                                                                                                                  
Exporting TI-595=-595

I am seeing the cf_export command parse the static environment variable value vs. storing the defined value with the assignment cf_export $FEATURE.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Michael Kolb here’s a pipeline step example to setup ENV vars

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  env:
    title: Setup Environment
    stage: Init
    fail_fast: true
    image: ${{build_image}}
    working_directory: &cwd ${{main_clone}}/${{PROJECT}}
    commands:
      - cf_export BUILD_HARNESS_VERSION=0.18.0

      # Github Commenter
      - cf_export GITHUB_OWNER=${{CF_REPO_OWNER}}
      - cf_export GITHUB_REPO=${{CF_REPO_NAME}}
      - cf_export GITHUB_COMMENT_TYPE=pr
      - cf_export GITHUB_PR_ISSUE_NUMBER=${{CF_PULL_REQUEST_NUMBER}}
      - cf_export GITHUB_COMMENT_FORMAT_FILE=${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}/codefresh/terraform/comment.txt.gotmpl
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it exports Codefresh env vars and also those defined in Codefresh UI (Global config section and ENV vars section in each pipeline)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you mean the example does not use the ${{...}} syntax?

Michael Kolb avatar
Michael Kolb

So, in the documentation it says that you can state an existing variable state the name of an existing environment variable (like EXISTING_VAR) https://codefresh.io/docs/docs/codefresh-yaml/variables/#using-cf_export-command

Variables

Codefresh is a Docker-native CI/CD platform. Instantly build , test and deploy Docker images.

Michael Kolb avatar
Michael Kolb

However, this hasn’t stored that variable assignment

Michael Kolb avatar
Michael Kolb

and the displayed use case does not show assignment in the code

Michael Kolb avatar
Michael Kolb
ersion: '1.0'
steps:
  freestyle-step-1:
    description: Freestyle step..
    title: Free styling
    image: alpine:latest
    commands:
      - cf_export VAR1=VALUE1 VAR2=VALUE2 EXISTING_VAR

  freestyle-step-2:
    description: Freestyle step..
    title: Free styling 2
    image: ${{VAR1}}
    commands:
      - echo $VAR2
      - curl http://$EXISTING_VAR/index.php
Michael Kolb avatar
Michael Kolb

is the documentation referencing a pipeline variable defined in the pipeline vs. in the pipeline yaml?

dustinvb avatar
dustinvb

Yes pipeline variables are in the pipeline SPEC file. If you need variables for the pipeline to be set during pipeline execution then you’d use cf_export to have them pop out to pipeline.

dustinvb avatar
dustinvb

The existing variable section is mentioning step-2 in the YAML above.

dustinvb avatar
dustinvb

You do not need to specify a variable name before the cf_export

Michael Kolb avatar
Michael Kolb

thanks, the documentation page could use more examples of cf_export variable assignment like the example that you provided - cf_export GITHUB_OWNER=${{CF_REPO_OWNER}}

dustinvb avatar
dustinvb

@Kostis (Codefresh) Can you add this to your backlog to add in some more use cases to this?

dustinvb avatar
dustinvb

For example command export?

dustinvb avatar
dustinvb
codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

Kostis (Codefresh) avatar
Kostis (Codefresh)

@dustinvb will do

Michael Kolb avatar
Michael Kolb

they display assignment as static references

Michael Kolb avatar
Michael Kolb

and the reference EXISTING_VAR is misleading if this is pulling from the pipeline variable definition vs. an existing variable

Michael Kolb avatar
Michael Kolb

documentation clarification would help, since I have had to run several builds trying to figure out how the cf_export process handles environment variables

Michael Kolb avatar
Michael Kolb

is there any documentation on the codefresh/kube-helm image that provides connection to Kubernetes & Tiller?

dustinvb avatar
dustinvb
codefresh-contrib/images

useful docker images. Contribute to codefresh-contrib/images development by creating an account on GitHub.

codefresh-contrib/cfstep-helm

Docker image for Codefresh Helm step. Contribute to codefresh-contrib/cfstep-helm development by creating an account on GitHub.

2019-04-19

mgrube avatar
mgrube

How are you guys running stuff like integration tests and system tests? CF can’t directly access internal services that are deployed, so are you guys using some kind of proxy container in the cluster to run these kinds of tests?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
codefresh-io/venona

Codefresh runtime-environment agent. Contribute to codefresh-io/venona development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This allows you to run it all in your k8s cluster

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ve used this for things like connecting to artifactory or using consul

dustinvb avatar
dustinvb

Yup this runner allows you to talk to the internal pods of the Kubernetes cluster using the container ips if needed.

mgrube avatar
mgrube

this will just show up as a different run time environment on your pipeline?

dustinvb avatar
dustinvb

Add the runner to the cluster and then select the runner for the pipeline and everything happens on your cluster.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

related to this, I have a question

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we are about to make our kops cluster private. I know we can use venona, but how do the k8s integrations work then?

dustinvb avatar
dustinvb

Yes

dustinvb avatar
dustinvb
Codefresh behind the firewall

How to run Codefresh pipelines in your own secure infrastructure

mgrube avatar
mgrube

Is there a way to split up what runs on CF runtime env and what runs on our infra? I assume that venona still reports everything up to CF UI, right?

dustinvb avatar
dustinvb

Yes all comes back to UI

dustinvb avatar
dustinvb

Each pipeline has a runtime setting. You can use our CLI to call another pipeline from your original as a step in the original and have that child pipeline run on your infra.

1
mgrube avatar
mgrube

I think this should cover our needs, thanks!

2019-04-18

dustinvb avatar
dustinvb

Is there a how-to on helmfile in Codefresh like an example codefresh.yml with an explanation on the steps?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Not that well documented

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The “ctl” command can be ignored. It just shows how to call Helmfile

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In this example, we support both blue/green and rolling

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, this example shows how we use the monochart

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’d be happy to jump on a call and walk you through it

2019-04-15

dustinvb avatar
dustinvb

@Erik Osterman (Cloud Posse) Do you happen to have an example of running https://github.com/cloudposse/github-commenter as a step in Codefresh pipeline using a Docker image? I’ve seen in other project snippets of codefresh.yml formatted examples but nothing in this project.

cloudposse/github-commenter

Command line utility for creating GitHub comments on Commits, Pull Request Reviews or Issues - cloudposse/github-commenter

dustinvb avatar
dustinvb

Came up with this really quickly.

  GitHubCommenter:
    title: Add GitHub Comment
    image: cloudposse/github-commenter:latest
    environment:
      - GITHUB_TOKEN=${{GITHUB_TOKEN}} #Must be created see link for info below.
      - GITHUB_OWNER=${{CF_REPO_OWNER}}
      - GITHUB_REPO=${{CF_REPO_NAME}}
      - GITHUB_COMMENT_TYPE=pr
      - GITHUB_PR_ISSUE_NUMBER=${{CF_PULL_REQUEST_NUMBER}}
      - GITHUB_COMMENT="" #Your custom comment goes here
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Create new S3 Bucket by osterman · Pull Request #75 · cloudposse/testing.cloudposse.co

what Demo of adding a new user bucket why GitOps rocks! =)

dustinvb avatar
dustinvb

Thanks!

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

2019-04-11

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
11:35:20 AM

Bitbucket access issue Apr 11, 11:15 UTC Investigating - We are currently investigating a problem affecting bitbucket integrations

Bitbucket access issue

Codefresh’s Status Page - Bitbucket access issue.

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
01:24:51 PM

Build services disruption Apr 11, 13:11 UTC Investigating - We are currently investigating this issue

Build services disruption

Codefresh’s Status Page - Build services disruption.

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
03:00:05 PM

Bitbucket access issue Apr 11, 14:43 UTC Resolved - The incident has been resolved Apr 11, 11:15 UTC Investigating - We are currently investigating a problem affecting bitbucket integrations

Bitbucket access issue

Codefresh’s Status Page - Bitbucket access issue.

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
03:00:06 PM

Build services disruption Apr 11, 14:43 UTC Resolved - The incident has been resolved Apr 11, 13:11 UTC Investigating - We are currently investigating this issue

Build services disruption

Codefresh’s Status Page - Build services disruption.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
GitOps with Terraform on Codefresh (Webinar)

Infrastructure as code, pipelines as code, and now we even have code as code! =P In this talk, we show you how we build and deploy applications with Terraform using GitOps with Codefresh. Cloud Posse is a power user of Terraform and have written over 140 Terraform modules. We’ll share how we handl

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here are the slides from the webinar today.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Video will be posted as soon as it’s available.

oscarsullivan_old avatar
oscarsullivan_old

Thank you

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Create new S3 Bucket by osterman · Pull Request #75 · cloudposse/testing.cloudposse.co

what Demo of adding a new user bucket why GitOps rocks! =)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
07:07:23 PM

Helm repositories issue Apr 11, 19:04 UTC Resolved - A fix has been deployed to production and confirmed to have fixed the issue. Helm repositories are fully operational. Apr 11, 18:49 UTC Monitoring - The issue has been identified, and the corresponding fix was applied. Helm repositories are accessible again, We’re closely monitoring the platform to ensure everything is working as expected. Apr 11, 18:47 UTC Identified - Currently Helm repositories provided by Codefresh are having issues. This issue is under…

Helm repositories issue

Codefresh’s Status Page - Helm repositories issue.

2019-04-10

dustinvb avatar
dustinvb

Awesome! I made this little example app a little more deployable… Still working out the kinks. https://github.com/codefresh-contrib/example-voting-app

codefresh-contrib/example-voting-app

Docker’s Example Voting App. Contribute to codefresh-contrib/example-voting-app development by creating an account on GitHub.

dustinvb avatar
dustinvb

Requires a few IPs right now and only works with Cloud LB not sure how to make this more portable using Istio or NGINX as both would require a DNS integration using the LB as is while requiring 2 IPs per running Helm Release is a bit much but this is also meant to demo a few things then toss after playing.

2019-04-09

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’ve added support for DocumentDB (MongoDB) to our Codefresh Enterprise terraform module: https://github.com/cloudposse/terraform-aws-codefresh-backing-services

cloudposse/terraform-aws-codefresh-backing-services

Terraform module to provision AWS backing services necessary to run Codefresh Enterprise - cloudposse/terraform-aws-codefresh-backing-services

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This makes everything that is absolutely essential to running Codefresh Enterprise (onprem) a fully managed service by AWS.

2019-04-05

oscarsullivan_old avatar
oscarsullivan_old
Status: Image is up to date for cloudposse/build-harness:0.18.0                                                            
make: *** No rule to make target 'codefresh/notify/slack/deploy'.  Stop.                                                   
[SYSTEM] Error: Failed to run freestyle step: Send notification to Slack channel; caused by NonZeroExitCodeError: Container
 for step title: Send notification to Slack channel, step type: freestyle, operation: Freestyle step failed with exit code:
 2    
  SendSlackDeployNotification:
    title: Send notification to Slack channel
    stage: "deploy"
    image: cloudposse/build-harness:${{BUILD_HARNESS_VERSION}}
    commands:
      - make codefresh/notify/slack/deploy

Anyone aware of why this would be happening? LGTM according to build-harness readme and example here: https://github.com/cloudposse/example-app/blob/29f91a718522e4a702d77d172a41ed1f779d42fe/codefresh/pull-request.yaml#L107

cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

oscarsullivan_old avatar
oscarsullivan_old
oscarsullivan_old avatar
oscarsullivan_old

likewise make init fails

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Need more details.

oscarsullivan_old avatar
oscarsullivan_old

Not too sure what else to share other than BUILD_HARNESS_VERSION == 0.18.0

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov

oscarsullivan_old avatar
oscarsullivan_old

So have tried the Codefresh commands of:

    ...
    commands:
      - make init
oscarsullivan_old avatar
oscarsullivan_old

and as above

Igor Rodionov avatar
Igor Rodionov
cloudposse/example-app

Example application for CI/CD demonstrations of Codefresh - cloudposse/example-app

Igor Rodionov avatar
Igor Rodionov

?

Igor Rodionov avatar
Igor Rodionov

@oscarsullivan_old ^

Igor Rodionov avatar
Igor Rodionov

check this env vars

Igor Rodionov avatar
Igor Rodionov
cloudposse/build-harness

Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more - cloudposse/build-harness

Igor Rodionov avatar
Igor Rodionov
cloudposse/build-harness

Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more - cloudposse/build-harness

Igor Rodionov avatar
Igor Rodionov

To debug it would be useful to see env vars and codefresh.yaml

casey avatar
casey

@oscarsullivan_old I just setup a pipeline using the codefresh/notify/slack/deploy/webapp target using BUILD_HARNESS_VERSION=0.18.0

casey avatar
casey

I would check your environment variables and make sure they are set correctly, I was getting similar errors, and it was because of env variables not being set

oscarsullivan_old avatar
oscarsullivan_old

Thanks, I’ll double check both these points (version and undeclared variables)

oscarsullivan_old avatar
oscarsullivan_old

it is most likely the declaration of env vars

oscarsullivan_old avatar
oscarsullivan_old

I did no special setup to use the build-harness tmeplates

casey avatar
casey

does anyone have suggestions for the strategy I should use for deploying to production from codefresh? I feel unsafe giving the ClusterRole codefresh-role permission to deploy to the whole cluster

casey avatar
casey

Or is that the only way possible?

oscarsullivan_old avatar
oscarsullivan_old

I think I use this

oscarsullivan_old avatar
oscarsullivan_old
module "codefresh_user" {
  source    = "git::<https://github.com/cloudposse/terraform-aws-iam-system-user.git?ref=tags/0.4.1>"
  namespace = "${var.namespace}"
  stage     = "${var.stage}"
  name      = "codefresh"
}

resource "aws_iam_user_policy_attachment" "default" {
  user       = "${module.codefresh_user.user_name}"
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser"
}
oscarsullivan_old avatar
oscarsullivan_old

but not using K8s rn

2019-04-04

oscarsullivan_old avatar
oscarsullivan_old

what format do you use for your Secrets Manager in #aws to connect to the Codefresh private repo? Can anyone please share an example

2019-04-03

oscarsullivan_old avatar
oscarsullivan_old

Oh wow

oscarsullivan_old avatar
oscarsullivan_old

yeh you’re right

oscarsullivan_old avatar
oscarsullivan_old

that’s perfect actually. thanks

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
09:39:51 AM

Webhooks issue Apr 3, 09:27 UTC Resolved - The incident been resolved Apr 3, 09:24 UTC Investigating - We are currently investigating an issue where git webhooks don’t trigger pipelines. Running builds manually works as expected

Webhooks issue

Codefresh’s Status Page - Webhooks issue.

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
12:45:20 PM

Codefresh availability issue Apr 3, 12:29 UTC Investigating - We are currently investigating a problem affecting Codefresh availability

Codefresh availability issue

Codefresh’s Status Page - Codefresh availability issue.

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
01:00:17 PM

Codefresh availability issue Apr 3, 12:46 UTC Monitoring - Codefresh site is up, we’re still monitoring the system Apr 3, 12:29 UTC Investigating - We are currently investigating a problem affecting Codefresh availability

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
01:44:53 PM

Codefresh availability issue Apr 3, 13:34 UTC Update - Google just updated us on an ongoing incident, we’re monitoring the issue with google team Apr 3, 12:46 UTC Monitoring - Codefresh site is up, we’re still monitoring the system Apr 3, 12:29 UTC Investigating - We are currently investigating a problem affecting Codefresh availability

Codefresh Status - Incident History avatar
Codefresh Status - Incident History
02:34:48 PM

Codefresh availability issue Apr 3, 14:23 UTC Resolved - This incident has been resolved. Apr 3, 14:22 UTC Update - We are continuing to monitor for any further issues. Apr 3, 13:34 UTC Update - Google just updated us on an ongoing incident, we’re monitoring the issue with google team Apr 3, 12:46 UTC Monitoring - Codefresh site is up, we’re still monitoring the system Apr 3, 12:29 UTC Investigating - We are currently investigating a problem affecting Codefresh availability

Codefresh availability issue

Codefresh’s Status Page - Codefresh availability issue.

2019-04-02

oscarsullivan_old avatar
oscarsullivan_old

How would you go about checking if .env is a directory or file inside a docker image inside a pipeline?

oscarsullivan_old avatar
oscarsullivan_old

You could have a freestyle step running a bash IF -f or -d; you could have some sort of condition in Dockerfile maybe?

oscarsullivan_old avatar
oscarsullivan_old

And also to add geodesic to CI, good to just use https://github.com/cloudposse/prod.cloudposse.co/blob/master/codefresh.yml ?

cloudposse/prod.cloudposse.co

Example Terraform/Kubernetes Reference Infrastructure for Cloud Posse Production Organization in AWS - cloudposse/prod.cloudposse.co

oscarsullivan_old avatar
oscarsullivan_old

Seems to be only pushing to codefresh registry which works for me. Unsure why it defines an old build harness?

oscarsullivan_old avatar
oscarsullivan_old

Erik, where’s an example of you executing a command in a geodesic module from a freestyle step in codefresh?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
CI/CD Demonstration of Terraform (GitOps) by osterman · Pull Request #22 · cloudposse/testing.cloudposse.co

what Demonstrate how to do CI/CD of Terraform with Codefresh why Larger goal is to apply this to all reference architectures and terraform modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We should move that “use_codefresh” direnv function into geodesic

oscarsullivan_old avatar
oscarsullivan_old

thanks

oscarsullivan_old avatar
oscarsullivan_old

I get this funny bash script actually

oscarsullivan_old avatar
oscarsullivan_old
  DeployingDockerImage:
    title: Deploying Docker Image with Ansible
    image: [r.cfcr.io/user/acme/${{MODULE}}.acme.co.uk:master](http://r\.cfcr\.io/user/acme/\$\{\{MODULE\}\}\.acme\.co\.uk:master)
    command:
      - ansible --version 
oscarsullivan_old avatar
oscarsullivan_old
02:30:13 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@oscarsullivan_old that’s the default behavior of geodesic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s why you can run

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

docker run myco/myinfra | bash

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to install geodesic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to avoid that, do this:

oscarsullivan_old avatar
oscarsullivan_old

what did I miss there Erik?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
CI/CD Demonstration of Terraform (GitOps) by osterman · Pull Request #22 · cloudposse/testing.cloudposse.co

what Demonstrate how to do CI/CD of Terraform with Codefresh why Larger goal is to apply this to all reference architectures and terraform modules

oscarsullivan_old avatar
oscarsullivan_old

because command not cmd?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cmd:
      - "-l"
      - "-c"
      - "./tests/run.sh"
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i am not sure about command vs cmd

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Freestyle

Run commands inside a Docker container

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

looks like cmd is canonical

oscarsullivan_old avatar
oscarsullivan_old

ah I went from this

CollectAllMyDeps:
  title: Install dependencies
  image: python:3.6.4-alpine3.6
  commands:
    - pip install .
oscarsullivan_old avatar
oscarsullivan_old

Ok will try to make it closer looking to what you’ve got

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh, so yea, cmd is the arg passe to the entrypoint

oscarsullivan_old avatar
oscarsullivan_old

could be the difference

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

vs commands is run after the entrypoint

oscarsullivan_old avatar
oscarsullivan_old

ahhh

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can try this:

oscarsullivan_old avatar
oscarsullivan_old

that makes sense

oscarsullivan_old avatar
oscarsullivan_old
step_name:
  title: Step Title
  description: Step description
  image: image/id
  working_directory: ${{step_id}}
  commands: 
    - bash-command1
    - bash-command2
  cmd:
    - arg1
    - arg2
oscarsullivan_old avatar
oscarsullivan_old

although that confuses me again

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cmd: ["-l", "-c", "true"]
commands:
      - "my command"
oscarsullivan_old avatar
oscarsullivan_old
cmd: ["--version"]
commands:
      - "ansible"
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

no

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm… maybe

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oscarsullivan_old avatar
oscarsullivan_old

oscarsullivan_old avatar
oscarsullivan_old

seems dull I can’t just have commands: ansible --version

oscarsullivan_old avatar
oscarsullivan_old

and have to split it up

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

nono

oscarsullivan_old avatar
oscarsullivan_old

I’ll give these new combos a go

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in that example, you are passing --version to the entrypoint which is bash

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so you should get the version of bash back

oscarsullivan_old avatar
oscarsullivan_old

ooh

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you could alternatively change the entrypoint

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

study up on ENTRYPOINT vs CMD in docker

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then see what we’re doing in the Dockerfile for geodesic

oscarsullivan_old avatar
oscarsullivan_old


study […]
Thanks

oscarsullivan_old avatar
oscarsullivan_old

Must admit I don’t really get the differences

oscarsullivan_old avatar
oscarsullivan_old

Seen both in action but no further

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, the subtle nuances are often misunderstood by Dockerfile authors

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and you see them misused and abused

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I liken it to the system call

int execve(const char *filename, char *const argv[],
                  char *const envp[]);
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ENTRYPOINT ~ filename

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

CMD ~ argv

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(this is my mental model, not the docker explanation)

oscarsullivan_old avatar
oscarsullivan_old

lol Erik it was because I had command not commands

oscarsullivan_old avatar
oscarsullivan_old
  DeployingDockerImage:
    title: Deploying Docker Image with Ansible
    image: [r.cfcr.io/xxx/xxx/${{MODULE}}.xxxx.co.uk:master](http://r\.cfcr\.io/xxx/xxx/\$\{\{MODULE\}\}\.xxxx\.co\.uk:master)
    commands:
      - ansible --version 
      # ansible-playbook pod.yml -i inventory/${{inventory}}.yml

Output:

Status: Downloaded newer image for [r.cfcr.io/xxx/xxx/sandbox.xxx.co.uk:master](http://r\.cfcr\.io/xxx/xxx/sandbox\.xxx\.co\.uk:master)                               
ansible 2.7.9                                                                                                              
  config file = None                                                                                                       
  configured module search path = [u'/conf/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']               
  ansible python module location = /usr/lib/python2.7/site-packages/ansible                                                
  executable location = /usr/bin/ansible                                                                                   
  python version = 2.7.15 (default, Jan 24 2019, 16:32:39) [GCC 8.2.0]                                                     
Reading environment variable exporting file contents.                                                                      
Successfully ran freestyle step: Deploying Docker Image with Ansible                              
oscarsullivan_old avatar
oscarsullivan_old

Now I gotta get my private Role + Playbook accessible

oscarsullivan_old avatar
oscarsullivan_old

Thinking either have them in conf, have some kind of wget situation and publicise them (not ideal), try and do an import like with terraform modules?????

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Our strategy for this is to write the key to chamber

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

However, I think for your purposes, it might be sufficient to write the key to a codefresh secret

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can easily add an SSH key to the agent using an environment variable

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
source <(ssh-agent -s)
ssh-add - <<<${ANSIBLE_SSH_PRIVATE_KEY}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

oscarsullivan_old avatar
oscarsullivan_old

Or my geodesic module has a git clone.. but then where do I store the deploy key hmmmm

oscarsullivan_old avatar
oscarsullivan_old

Have come up with this solution

oscarsullivan_old avatar
oscarsullivan_old
04:21:05 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You could also clone as a step in the codefresh pipeline

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that would be accessible from inside of geodesic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the benefit of doing it that way is that you can leverage the git integrations already available in codefresh without putting a secret (e.g. /conf/ssh/deployment-repo-key into git)

2019-04-01

oscarsullivan_old avatar
oscarsullivan_old

So it turned out I was using ‘global variables’ and build variables wrong

oscarsullivan_old avatar
oscarsullivan_old
oscarsullivan_old avatar
oscarsullivan_old

I thought this would mean when I run the build pipeline my Dockerfile’s ARG PORT would get a value of 3060

oscarsullivan_old avatar
oscarsullivan_old

Turns out it doesn’t

oscarsullivan_old avatar
oscarsullivan_old
Build an Image with Build Arguments

Codefresh is a Docker-native CI/CD platform. Instantly build , test and deploy Docker images.

oscarsullivan_old avatar
oscarsullivan_old

and that’s fine.. happy to define it in a codefresh.yml file

oscarsullivan_old avatar
oscarsullivan_old

but then I realised how do I do multi-branch / stage CI… surely I’ll need multiple codefresh.yml because one example argument is APP_ENV

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Did you try to use interpolations with:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
build_arguments:
    - key=value
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
build_arguments:
    - APP_ENV=${{APP_ENV}}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, what we do sometimes is create a reusable pipeline like build.yaml that we call from something like some-app.yaml; the some-app.yaml can trigger the build pipeline and pass settings to it.

oscarsullivan_old avatar
oscarsullivan_old

Ahhh I did not try interpolating. I went from not declaring args in .Yaml to declaring hardcoded and removing the vars from the console

oscarsullivan_old avatar
oscarsullivan_old

I was going to have a pipeline for each stage inside the codefresh UI which runs off the same codefresh.yml and simply has the build args in the UI

oscarsullivan_old avatar
oscarsullivan_old

Does any of this sound like I’ve done / misunderstood it incorrectly, as having multiple codefresh.yml doesn’t feel DRY.. especially to only change a few build args. Thanks

oscarsullivan_old avatar
oscarsullivan_old

Ah.. unless it should be ENV instead of ARG idk

mgrube avatar
mgrube

how many variables are you setting at build time? I usually just use environment variables that are set at run time instead of build time. I add the environment variables through the pod definition. This allows my Dockerfile to be pretty generic

oscarsullivan_old avatar
oscarsullivan_old

So far 2. Vant see it breaching 5

mgrube avatar
mgrube

I see your screenshot that your are setting aws secrets and the port for your container. I don’t think the port matters on your Docker container, so I am assuming its more about your aws secrets. I never was a fan of keeping keys lying around, and I liked having everything controlled through IAM. May I suggest taking a look at https://github.com/uswitch/kiam ? That’s what we use to make pods have specific IAM permissions, and it works great.

uswitch/kiam

Integrate AWS IAM with Kubernetes. Contribute to uswitch/kiam development by creating an account on GitHub.

oscarsullivan_old avatar
oscarsullivan_old

Thanks Mark. I’ve actually removed it now. It was when I was trying put ECS

oscarsullivan_old avatar
oscarsullivan_old

The two args I have are: application port and application env file.

oscarsullivan_old avatar
oscarsullivan_old

Because I then do: COPY ${APP_ENV} .env

    keyboard_arrow_up