#general (2024-04)

General conversations related to DevOps/Automation

General Discussions

2024-04-01

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Yili Kerim
  • @alla2

Good to have you here =)

wave2

2024-04-02

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Jay
  • @paparao.prema75
  • @Ritika Kumar

Good to have you here =)

wave4

2024-04-03

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Jessie Puls
  • @Gobi Dasu
  • @umuhammad97
  • @Wendell
  • @jbontech
  • @djaoas

Good to have you here =)

wave2

2024-04-04

SweetOps avatar
SweetOps
07:00:04 PM

Hey everyone, give a warm welcome to our newest members!

  • @Andrey Klyukin
  • @Dale
  • @mikaelec

Good to have you here =)

wave2

2024-04-05

Dhamodharan avatar
Dhamodharan

#general Not sure I can post my query here. Still looking for a solution.

I am trying to migrate a database from AWS RDS to GCP, the size is around 120GB. I thought it would be fast if I create a dump file and copy it to GCP and restore it. But while doing that it’s taking almost 7hours time. Is there any other approach that I can anticipate this very fast? Any suggestions?.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Have you eliminated CPU/IO bottlenecks as the cause?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

How are you transferring the dump?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you transferring and loading it at the same time? e.g. piping into the database as your transferring, or simply transferring the dump itself is taking 7 hours.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

To speed up the transfer of the dump, see about parallelizing the transfer. One way to do that without much extra tooling is to just use the split command to chunk it on lines, then use rsync to download files in parallel. On the other end you can reassemble the files.

Dhamodharan avatar
Dhamodharan

Hello @Erik Osterman (Cloud Posse) thanks for the update.. Sorry i was travelling, so couldnt look at your reply. Transferring file took an hour only, dump restore is taking 7hours time.. I could upgrade machine type to achieve high network bandwith which reduced 3hours. But still its pretty high for the critical application migration. Do you still recommend to use the split and rsync will help here??

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok, so looks like “pay to play” will have the highest ROI. You can resize the database instance after migration.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Just by upgrading the machine type you achieved greater than a 50% reduction in time.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The other thing you can do is disable indexes on load, and add those afterwards.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Since the transfer time is an hour, it depends on how much time you have to prepare for the lift & shift. You’ll continue to get modest gains by splitting the dump and transferring it in parallel and reassembling on the other side.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would first try increasing the size of the instance even more, just to prove a point. There’s a tradeoff between the cost of people time and machine time, and opportunity cost. The longer it takes to find the solution, the longer the overall time to complete the migration will take (not just the time to ETL the data). So if you receive push back, find out what the true concerns are.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The overall rule-of-thumb advice we give our customers for migrations is the standard (good, fast, cheap). Pick 2.

SweetOps avatar
SweetOps
07:00:05 PM

Hey everyone, give a warm welcome to our newest members!

  • @Adel Hashem
  • @Vamshi Mylaram
  • @tfji
  • @Ben Rosand
  • @Ben
  • @Samuel Frimpong
  • @Justin

Good to have you here =)

2024-04-06

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Justine Devasia
  • @career.bash016

Good to have you here =)

wave2

2024-04-07

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Deivids Egle

Good to have you here =)

wave2

2024-04-08

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Halo Hsu

Good to have you here =)

wave2

2024-04-09

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Msk
  • @NXT1

Good to have you here =)

wave1

2024-04-10

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Bolt17

Good to have you here =)

wave2
Dhamodharan avatar
Dhamodharan

Hello #general, I am setting a CloudRun service and want to pass some environment variables. I could configure it in the CloudRun console under variables and secrets and pass a static value. But my requirement is i want read the variable from the secretsmanager. So the secretname should be passed in the variable section but the value should be retried from secretsmanager. I couldn’t find any references to read the variable from secretsmanager. Can someone help me if have suggestions..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I suggest using #gcp so you reach that audience.

2024-04-11

SweetOps avatar
SweetOps
07:00:07 PM

Hey everyone, give a warm welcome to our newest members!

  • @linear_asks
  • @amnesia-allow0f
  • @miko
  • @Atmosly Platform engineering
  • @Tony
  • @Jason G
  • @Simon Heather

Good to have you here =)

wave4
miko avatar

Sup guys!

wave2

2024-04-12

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @MYacineK
  • @Aditya PR
  • @Alex Berber
  • @Aditya Pandhikanda Ramesh

Good to have you here =)

wave2

2024-04-13

tamish60 avatar
tamish60

Hello guys

wave1
SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Julio Perez
  • @dropinboy
  • @Ben
  • @tamish60
  • @ChrisP
  • @Carl Anders Düvel

Good to have you here =)

wave5

2024-04-14

nrs-status avatar
nrs-status

gotta test run 100+ variations of a deployment (terraform infra + ansible provisioning). what would you guys use to automate this?

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @nick
  • @nrs-status

Good to have you here =)

wave3

2024-04-15

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @microgag
  • @Maks
  • @Stephan Helas
  • @Shelley

Good to have you here =)

wave3

2024-04-16

SweetOps avatar
SweetOps
07:00:09 PM

Hey everyone, give a warm welcome to our newest members!

  • @Tom Bee
  • @Amos Gewirtz
  • @Andrew Chemis
  • @Eddie N.

Good to have you here =)

wave2
3

2024-04-17

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Krishna
  • @Aniket
  • @Saksham
  • @Mark Brooks
  • @Jmarz
  • @Michael Klatskin
  • @Zack

Good to have you here =)

wave2
sweetops2

2024-04-18

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Dinko Mitic
  • @Matt
  • @davidjaffie
  • @William
  • @Maxime

Good to have you here =)

1
wave3

2024-04-19

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @emitor

Good to have you here =)

wave1
sweetops1

2024-04-20

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Lloyd Chang

Good to have you here =)

wave1
sweetops1

2024-04-21

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @David C

Good to have you here =)

wave1

2024-04-22

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Bernie
  • @Marvin de Bruin
  • @Steffan Norberhuis
  • @Ashwin Reddy
  • @Stanley Tso
  • @tony384

Good to have you here =)

1
wave3
tony384 avatar
tony384

wave Hello, team!

1
RB avatar

Unsure where to post this. In the newsletter, i cannot see it clearly on mobile because the formatting of the image shows up on the left of the text giving the text room enough for about one word per line

RB avatar

It makes it challenging to read

RB avatar

I’m using the stock android gmail client

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I agree.. Need to change platform i think.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks for speaking up!

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I made some tweaks, let me know if Monday’s newsletter is better.

1
RB avatar

Looks much better

2024-04-23

SweetOps avatar
SweetOps
07:00:01 PM

Hey everyone, give a warm welcome to our newest members!

  • @Prasanth
  • @Leonie JM

Good to have you here =)

wave2

2024-04-24

Prasanth avatar
Prasanth

wave Hello, team!

SweetOps avatar
SweetOps
07:00:33 PM

Hey everyone, give a warm welcome to our newest members!

  • @rahulbhatu1991
  • @Maurysnake
  • @Coby

Good to have you here =)

2
wave2

2024-04-25

OScar avatar

Been a little while, how is everyone holding up?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Wow, it sure has been!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Haven’t seen you around on #office-hours

OScar avatar

Hey @Erik Osterman (Cloud Posse)! I’ve been a little preoccupied at AWS, working on an upcoming AWS Native Service!

OScar avatar

I still wish I had time to contribute, and join the office hours!

SweetOps avatar
SweetOps
07:00:01 PM

Hey everyone, give a warm welcome to our newest members!

  • @King
  • @natali.fpaura
  • @Patrick Berrigan
  • @Jason
  • @Joe Lauzze

Good to have you here =)

golang1
wave2
Joe Lauzze avatar
Joe Lauzze

Hello

wave1

2024-04-26

SweetOps avatar
SweetOps
07:00:04 PM

Hey everyone, give a warm welcome to our newest members!

  • @James
  • @Vikash

Good to have you here =)

wave3
wave1
2

2024-04-27

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Dhruv Tiwari
  • @James Berger - 3Play Media

Good to have you here =)

wave3

2024-04-28

SweetOps avatar
SweetOps
07:00:04 PM

Hey everyone, give a warm welcome to our newest members!

  • @Emerson H
  • @Max

Good to have you here =)

2
wave1

2024-04-29

SweetOps avatar
SweetOps
07:00:04 PM

Hey everyone, give a warm welcome to our newest members!

  • @Noam Guy
  • @Ronie

Good to have you here =)

wave3

2024-04-30

SweetOps avatar
SweetOps
07:00:04 PM

Hey everyone, give a warm welcome to our newest members!

  • @andrea
  • @rjshk013

Good to have you here =)

wave1
    keyboard_arrow_up