#general (2020-01)
General conversations related to DevOps/Automation
General Discussions
2020-01-01
also, for the reference arch. link above (https://github.com/cloudposse/reference-architectures) - assuming i’m starting from a blank slate, what version of terraform should i go with? 11 or 12?
We have not yet updated it to 0.11
:(
Got it… So I’m guessing that means 0.10 is the way to go?
oh, i misspoke
we have not updated it yet to 0.12
you can use 0.11!
(was on my phone)
Perfect thx!
what’s easiest for registering the “apex” domain? will the provisioning process do it, should i do it in the root account’s rt 53, or get it from my regular registrar?
Use route53. It’s magic
From I made the switch I’ve had no reason to use anyone else. It’s even better than cloudflare
And it integrates with everything AWS
Yea, we register all domains in the “root” (aka master) AWS account
then delegate zones from there to sub accounts
cloudposse looks great, looking forward to giving it a try! Been thinking it should exist but didn’t come across it until now (currently using bitnami’s kube-prod-runtime but that only gets maintenance releases recently).
I have one question regarding codefresh (non-enterprise) version, does it give a whitelist ip/range for deploying to kubernetes or do you have to leave the master world-readable (0.0.0.0/0)?
nvm… I see it now on the GKE pro $300/mo plan it mentions static ip, so I guess it’s only either that or enterprise.
Hrm… I feel like someone else has whitelisted codefresh IPs. I thought they offer distinct set of IPs. @dustinvb in #codefresh might know.
but running the enterprise mode with venona is the most secure route
thanks, it’s a nice feature to prod people to higher paid versions I guess… would be nice if they offered similar on AWS (e.g. spend more than $300/mo on developer plan, get static ip) but doesn’t appear to be the case.
I had one other thought on the helm functionality in codefresh, is the source of truth for what’s deployed only in the codefresh ui when using kanban boards, or is it committed to git like in gitops (as in jenkins-x and flux)?
Cool - I hadn’t seen that
Sorry was out on break for holidays.
We have 2 distinct options available SaaS and Hybrid. With our SaaS offering we execute your pipeline on Codefresh resources and you can whitelist our NAT gateways in your firewall to interact with your Kubernetes API endpoints. With our Hybrid offering we execution your pipeline on your Kubernetes cluster behind-the-firewall. Meaning the cluster does not need to have any externally facing IPs. We launch a pod into your cluster which can interact with internal IPs to orchestrate Kubernetes deployments. The agent is a polling agent so will require egress out to g.codefresh.io and docker.io to pick up jobs and pull docker images used by the agent.
https://codefresh.io/docs/docs/enterprise/behind-the-firewall/
Let me know if you need those IPs for the NATs and I’ll DM them to you.
the NAT IPs would be great, thanks!
thanks @dustinvb!
Hey everyone, give a warm welcome to our newest members!
- @Eamon Keane
Good to have you here =)
2020-01-02
Hi @Erik Osterman (Cloud Posse) - not sure if this is the right channel for this, but I was wondering if you take community contributions for new modules? I have an NLB module based off of https://github.com/cloudposse/terraform-aws-alb, if you would be interested in taking it over
Sure, I think we can do that.
Can you work with @Andriy Knysh (Cloud Posse) to import it?
(#terraform is probably the better channel for this..)
Thanks! Will do. It’s not quite done, but I can let you know when it is.
thanks @Joe Hosteny, we’ll work with you on that
2020-01-03
Hey everyone, give a warm welcome to our newest members!
- @carlos catari
Good to have you here =)
12020-01-04
Hey everyone, give a warm welcome to our newest members!
- @Phuc
- @HS
- @Figo Huang
Good to have you here =)
1
12020-01-05
hey guys, i know this is random ask, but has anyone ever had an appgateway fail and not allow you to save anything to it on Azure? Sorry at my last wits with this thing and just trying to figure it out anyway possible
nop but it seems a Azure Support issue
Ya, not much to go on.. but maybe try #azure
@geertn ?
I’ve had some weird issues with Application Gateways but not this.
2020-01-06
Hey everyone, give a warm welcome to our newest members!
- @scottcressi
Good to have you here =)
:) great to be here
I’m gonna probably be developing an 8 hour training session on setting up an app on AWS ECS with some of yalls repos. Be happy to share my experience along the way and presentation/blog/training material
@OGProgrammer that sounds awesome!
2020-01-07
Is anyone else here stuck in the hell that is Azure AKS?
I’ve found that the faster and more streamlined I automate workloads in this platform the more I’m punished
@Zachary Loeber Is there anything of Azure what is good ? I’d like to understand.
I’m not sure there is anything good about Azure. If you’re a large organization which uses Windows and .Net and dread the thought of training your workforce to use AWS or GCP then you default to Azure despite its limitations
Perhaps it is just a massive case of the grass being greener on the other side for me as I’m working with it all day long but I just constantly hit the barriers of Azure’s services. Don’t get me wrong, I’d go o365 any day for a business email migration and AD hybrid or full online migration though.
Nothing is more frustrating than having services that suddenly don’t act the same way they did a week ago either (AKS I’m talking about you!)….
I’m doing 100% linux workloads on Azure Kubernetes Services (with some other IaaS services speckled in) so I’m possibly an aberration though..
no I left AKS in the dust 18 months ago thank god. From the kubernetes release notes I see they’re the only cloud provider still patching their storage tire fire in each release.
Hey everyone, give a warm welcome to our newest members!
- @Aleksey Silak
- @Aaron Chu
- @Chris Maxoutis
- @Matheus Hunsche
Good to have you here =)
12020-01-08
Hey everyone, give a warm welcome to our newest members!
- @sype
- @Josh Lynn
Good to have you here =)
2020-01-09
Hey everyone, give a warm welcome to our newest members!
- @Charlie Le
- @Jubel Han
- @Matt McLane
- @Tom Taubkin
- @Sankara Reddy
Good to have you here =)
Howdy all. What is the relationship between Cloudposse the company and this slack workspace? I saw an invite to join this workspace through a cloudposse github repo
Welcome @David! SweetOps is a community run by Cloud Posse (cloudposse.com). It was started as a place for our users to collaborate and ask questions related to our large collection of open source projects on GitHub (github.com/cloudposse), but also talk shop and get feedback on anything DevOps related.
1
Beautiful, thanks for the clarification!
I discovered not too long ago that almost every terraform module I’ve written has a better, open source version from you all (or terraform-aws-modules, which maybe is cloudposse a bit as well?).
I’ve just gotten started with a few PRs to get my feet wet with contributing on https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/pulls. Thanks for all the awesome modules!
great! @Andriy Knysh (Cloud Posse) will review as soon as he has a chance. we’re a bit swamped with PRs right now!
thanks for pinging us though - we’ll definitely get to it. and do nudge us if we drop the ball!
oh no problem! I love that with terraform modules its super easy to point my module source at a forked repo if I need specific functionality, and then back to the main repo if/when that PR gets merged.
it’s a great contribution model they’ve developed
we’re not directly affiliated with #terraform-aws-modules (a project co-opted in part by @antonbabenko, another prolific contributor). Many users of those modules also use ours.
1
12020-01-10
Hey everyone, give a warm welcome to our newest members!
- @Bernhard Lenz
- @David
- @Nedumaran Rajagopal
- @Rob Rose
Good to have you here =)
2020-01-11
@dustinvb
Are they chewable? I only do chewable?
haha
Hey everyone, give a warm welcome to our newest members!
- @Philip L Bankier
- @David Hubbell
- @Shawn Petersen
Good to have you here =)
2020-01-13
Hey everyone, give a warm welcome to our newest members!
- @Daniel Kempthorne
- @Francisco Rodriguez
- @Wen Lim
Good to have you here =)
22020-01-14
Hi guys, i’m was invited here by @sype we are currently working on a AWS foundation projet for a French customer. Hope I’ll find great discussion here.
1
welcome @Alexandre
Hey everyone, give a warm welcome to our newest members!
- @Flo Sloot
- @James Huffman
- @Andrea Benfatto
- @Nghia Nguyen
- @Alexandre
- @muhaha
Good to have you here =)
1
Anyone here using consul for devops pipelines?
Can you elaborate? I do use Consul for my ECS services
well the kv store at least
@Corey Gale might be
maybe @tamsky
2020-01-15
Hi Guys!
Please I’m having a serious problem that is driving me crazy, I will appreciate everyone’s help
My Laptop is connected to an OPENVPN hosted on AWS, also I need to connect to a client’s company VPN to work from, the problem I’m facing is, Once I connect to the Client’s Office VPN, I loose Internet access…Please can anybody help with this?
Ya, basically what @maarten suggests… sounds like their VPN is jacking the default route. If you can control that on the client side, that would be the way to go. If have trouble with that, then @Zachary Loeber suggestion will be the the quick win.
Hey everyone, give a warm welcome to our newest members!
- @vgdub
- @Michael Coffey
- @Paul Nicholson
- @Ravi Bhure
Good to have you here =)
3
Hi! I plan to start a blog, but I’d like to make it open source and store everything in git repo. The most important for me are posts written in markdown. Anyone has experience with this? What would you recommend, any specific tool for generating static pages?
I’d recommend trying Org-Mode. imho, Github’s flavor of Markdown is pretty meh.
Even github’s ReStructuredText would be a step up from markdown.
Here’s a list of some tools that do the blog publishing steps for org-mode…
Here’s a post on one option that turned up in my feed recently… https://matthewsetter.com/why-antora-is-the-leading-technical-writing-platform/
Thanks!
a couple more recent-ish posts on the same subject..
• https://blog.frankel.ch/my-blogging-stack-publishing-process/
• https://www.definit.co.uk/2020/01/using-gitlab-pipelines-to-deploy-hugo-sites-to-aws/
I use hugo with render.com for my test site and github pages for my live site (behind cloudflare) and azure devops for the pipeline just for the heck of it
2020-01-16
Hey everyone, give a warm welcome to our newest members!
- @claudemir de Almeida Rosa
- @Ben Mathews
- @jujugrrr
- @SoboL
- @Uzair
- @jvaibhav123
- @Alexandr Katulskiy
Good to have you here =)
2020-01-17
thank you @Bot
Hey everyone, give a warm welcome to our newest members!
- @James Rawlings
- @Sean Johnson
- @kemario.lindo
- @Joseph Omojowo
- @carthewd
Good to have you here =)
12020-01-18
Hey everyone, give a warm welcome to our newest members!
- @Bill D
- @caretak3r
Good to have you here =)
looking for reference materials or guides around packaging up terraform code (used to release something) into a docker container, like cloudposse does with geodesic. we don’t need the exact geodesic image, so im playing around with a custom docker image i made
2020-01-19
@roth.andy does something like we do too
I haven’t packaged terraform code in a docker container before. Terraform code goes in a git repo. The container has the tools that executes the code. You pass in creds to pull down the code and execute it
2020-01-20
Hey everyone, give a warm welcome to our newest members!
- @ITO Wataru
- @oliver
- @grv
Good to have you here =)
2020-01-21
How do contributors work for the cloudposse repos? If I’ve had a few PRs merged into a repo, would it be okay to ask to be a contributor, or is that meant only for cloudposse employees?
@David feel free to add yourself as a contributor if that’s the case
remember to update the README.yaml and rebuild
(and thanks for your contributions!)
Thanks guys, good to be part of this slack channel. Not an expert in terraform, here to learn from y’all since I am moving most of our aws stuff to newest tf and exploring new stuff like workspaces etc. Ofcourse, using cloudposse modules
welcome @grv and thanks!
@grv you came to the right place!
make sure to join us for office hours then too: https://cloudposse.com/office-hours (free)
and checkout our past sessions: https://podcast.cloudposse.com
Sweet thanks
Hey everyone, give a warm welcome to our newest members!
- @marcinw
- @burnzy
Good to have you here =)
2020-01-22
Hey everyone, give a warm welcome to our newest members!
- @Oliver Slater - Piksel
- @Blake
Good to have you here =)
@Erik Osterman (Cloud Posse) I’m pulling them in. I got @Chase Ward here from Calm
He’s a developer.. devops dude.. but recently he’s been hot in data, data pipelines, data engineering. Would Looooooooooove to see some discussions popup around devops + data engineering… Airflow at scale? (shit, I dunno all the hot tools nowadays)
Greetings homies!! Excited to be in a community of like-minded people!
We have some experience with Airflow on Kubenetes, but don’t think we can say “at scale” yet. Would definitely like to hear from others doing more with that.
1
maybe we should have a #bigdata channel
2020-01-23
airflow 2.0 (due in around 1 or 2 quarters) and airflow kNative executor should allow much greater scale workflows than currently. Airflow k8s executor and pod operator gets awkward over around 1,000 tasks/dag and 1,000 tasks/hour in my experience.
@Eamon Keane, Are you running airflow deployments in a pipeline to deploy DAG updates to kubernetes or are you cheating and just letting devs skip version control and work right out of shared storage for their DAGs?
(just curious)
And I wonder if spark 3 will ever be released (along the lines of long awaited releases…)
@Zachary Loeber we only use k8s pod operator and bake dags in image so every dag change produces dockerfile and helm chart. Dev can fully test locally independent of airflow…keeps things sane but obviously misses out on built-in airflow operators.
Happen to be running any spark jobs via airflow as well? I do a helm based deployment that re-kicks off their DAG creation (my immature favorite airflow insider term is fillin’ up the dagbag!) using pre-created shared storage and other pipeline witchcraft. They are using the kubernetes scheduler to kick off spark operator jobs (which are just a fancy bash-operator using spark-submit from all I can tell) that use some pre-deployed secrets to allow for backend storage connections to the spark-history server. It seems bonkers to me considering I do the same kind of deployments for another team without airflow at all using native kubernetes cronjobs and some python….
don’t use spark, no. We’re using airflow as a fancy k8s cronjob orchestrator. It is quite useful to have the UI for visibility and the execution variables like next_ds available for input to the dags and easy retries/backfills and dependency management. Everyone starts off thinking they can do it themselves and regrets it… that’s how airflow was started in the first place!
do you run it in its own cluster and deploy to other clusters then?
we deploy it in the same cluster as it launches pods (one airflow deployment per cluster and per environment)… it’s used for ETL with data regs, so one cluster would be awkward.
I suspect a better k8s workflow engine could be built with tekton, but that’s probably a year or two away.
Hey everyone, give a warm welcome to our newest members!
- @Chase Ward
- @Hussein Khazaal
- @ballew
- @Robert
Good to have you here =)
2020-01-24
Hey everyone, give a warm welcome to our newest members!
- @Bradford Toney
- @Bananahands
- @sathish krishnan
Good to have you here =)
2020-01-25
Hey everyone, give a warm welcome to our newest members!
- @Mohammad
Good to have you here =)
2020-01-26
Hey everyone, give a warm welcome to our newest members!
- @Oliver Fasterling
- @Omri Siri
- @Nitzan Yemal
Good to have you here =)
2020-01-27
Hello, does anyone knows how to do this kind of slide deck : https://hashicorp.github.io/field-workshops-terraform/slides/aws/terraform-oss/index.html#1 ?
here’s the backing repo, probably some clues there, https://github.com/hashicorp/field-workshops-terraform
Hey everyone, give a warm welcome to our newest members!
- @Lenucksi
Good to have you here =)
@Alexandre Yes, well something similar at least. Here is a hugo based reveal.js themed slide deck I did for a generic devops 101 thing I did for work a bit ago: https://github.com/zloeber/deck.loeber.live
The rendered site helps I suppose: https://deck.loeber.live/#/home
fancy!
2020-01-28
It was a lot of effort to avoid having to use powerpoint ‘eh?
Still waiting on a Powerpoint provider for Terraform
It’s probably not impossible - https://docs.microsoft.com/en-us/office/dev/add-ins/reference/overview/powerpoint-add-ins-reference-overview
and of course our all time favorite https://github.com/ndmckinley/terraform-provider-dominos
@Erik Osterman (Cloud Posse) actually didn’t know that the link you posted was a serious foundation for some cool projects until i clicked it (i should have seen the mastercard logo)
I know a company on this slack team using that rest provider.
it’s the ultimate escape hatch
if something provides a REST API, you can terraform it!
Hey everyone, give a warm welcome to our newest members!
- @Jim Park
- @Jillian Rowe
- @Pradeep Kumar Ashokan
- @Pú
- @Yunier
- @Yonatan Koren
Good to have you here =)
2020-01-29
Hey everyone, give a warm welcome to our newest members!
- @Igor Bronovskyi
- @Graeme Gillies
- @Miracle Max
- @gui
- @creature
- @leonyi
Good to have you here =)
Hey
Thanks for the warm welcome! Super happy to hang out with like minded individuals.
Glad you stopped by!
2020-01-30
Hey everyone, give a warm welcome to our newest members!
- @Ryan Bonner
- @dhondapatinaveenreddy
- @leo.hu
- @Jeff Levin
- @bradym
- @Lennart Wilke
- @chase4131
- @Chris OBrien
Good to have you here =)
Looking for simple condition to set in EC2 instance template with ALB, that will hold the 2nd (HA instance being deployed) so I can get the first up with some manual config running then remove condition to lauch the instanceB
why not use beanstalk for that?
manual CF provision in AWS china, batch file we normally use not working something with the handshake from global to china
aha, that sound like no fun
not a cluster, just 1 instance in HA config in template, I’m in hurry and too green to fig it out, lol and it’s late so It dawned on me to throw it out on sweetops
Yeah, no fun when you have to input 50 parameters manually before the fed saml login times out in 30min and have to log back in
Just want to hold on the 2nd instance being created while I do the manual config on the first instance for this new ec2 stack created
distributed lock to stop the 2nd instance coming up (cloud init) until lock released/timeout ?
2020-01-31
Hey everyone, give a warm welcome to our newest members!
- @Sri
- @Raymond Liu
- @Alex Tamoykin
- @Julian Gindi
- @JJ Ferman
- @Sebastian Stadil
Good to have you here =)