#general (2024-04)

General conversations related to DevOps/Automation

General Discussions

2024-04-01

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Yili Kerim
  • @alla2

Good to have you here =)

wave1

2024-04-02

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Jay
  • @paparao.prema75
  • @Ritika Kumar

Good to have you here =)

wave3

2024-04-03

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Jessie Puls
  • @Gobi Dasu
  • @umuhammad97
  • @Wendell
  • @jbontech
  • @djaoas

Good to have you here =)

wave1

2024-04-04

SweetOps avatar
SweetOps
07:00:04 PM

Hey everyone, give a warm welcome to our newest members!

  • @Andrey Klyukin
  • @Dale
  • @mikaelec

Good to have you here =)

wave1

2024-04-05

Dhamodharan avatar
Dhamodharan

#general Not sure I can post my query here. Still looking for a solution.

I am trying to migrate a database from AWS RDS to GCP, the size is around 120GB. I thought it would be fast if I create a dump file and copy it to GCP and restore it. But while doing that it’s taking almost 7hours time. Is there any other approach that I can anticipate this very fast? Any suggestions?.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Have you eliminated CPU/IO bottlenecks as the cause?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

How are you transferring the dump?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you transferring and loading it at the same time? e.g. piping into the database as your transferring, or simply transferring the dump itself is taking 7 hours.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

To speed up the transfer of the dump, see about parallelizing the transfer. One way to do that without much extra tooling is to just use the split command to chunk it on lines, then use rsync to download files in parallel. On the other end you can reassemble the files.

Dhamodharan avatar
Dhamodharan

Hello @Erik Osterman (Cloud Posse) thanks for the update.. Sorry i was travelling, so couldnt look at your reply. Transferring file took an hour only, dump restore is taking 7hours time.. I could upgrade machine type to achieve high network bandwith which reduced 3hours. But still its pretty high for the critical application migration. Do you still recommend to use the split and rsync will help here??

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok, so looks like “pay to play” will have the highest ROI. You can resize the database instance after migration.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Just by upgrading the machine type you achieved greater than a 50% reduction in time.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The other thing you can do is disable indexes on load, and add those afterwards.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Since the transfer time is an hour, it depends on how much time you have to prepare for the lift & shift. You’ll continue to get modest gains by splitting the dump and transferring it in parallel and reassembling on the other side.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I would first try increasing the size of the instance even more, just to prove a point. There’s a tradeoff between the cost of people time and machine time, and opportunity cost. The longer it takes to find the solution, the longer the overall time to complete the migration will take (not just the time to ETL the data). So if you receive push back, find out what the true concerns are.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The overall rule-of-thumb advice we give our customers for migrations is the standard (good, fast, cheap). Pick 2.

SweetOps avatar
SweetOps
07:00:05 PM

Hey everyone, give a warm welcome to our newest members!

  • @Adel Hashem
  • @Vamshi Mylaram
  • @tfji
  • @Ben Rosand
  • @Ben
  • @Samuel Frimpong
  • @Justin

Good to have you here =)

2024-04-06

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Justine Devasia
  • @career.bash016

Good to have you here =)

wave1

2024-04-07

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Deivids Egle

Good to have you here =)

wave1

2024-04-08

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @Halo Hsu

Good to have you here =)

wave1

2024-04-09

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Msk
  • @NXT1

Good to have you here =)

wave1

2024-04-10

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Bolt17

Good to have you here =)

wave1
Dhamodharan avatar
Dhamodharan

Hello #general, I am setting a CloudRun service and want to pass some environment variables. I could configure it in the CloudRun console under variables and secrets and pass a static value. But my requirement is i want read the variable from the secretsmanager. So the secretname should be passed in the variable section but the value should be retried from secretsmanager. I couldn’t find any references to read the variable from secretsmanager. Can someone help me if have suggestions..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I suggest using #gcp so you reach that audience.

2024-04-11

SweetOps avatar
SweetOps
07:00:07 PM

Hey everyone, give a warm welcome to our newest members!

  • @linear_asks
  • @amnesia-allow0f
  • @miko
  • @Atmosly Platform engineering
  • @Tony
  • @Jason G
  • @Simon Heather

Good to have you here =)

wave3
miko avatar

Sup guys!

wave2

2024-04-12

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @MYacineK
  • @Aditya PR
  • @Alex Berber
  • @Aditya Pandhikanda Ramesh

Good to have you here =)

wave1

2024-04-13

tamish60 avatar
tamish60

Hello guys

wave1
SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @Julio Perez
  • @dropinboy
  • @Ben
  • @tamish60
  • @ChrisP
  • @Carl Anders Düvel

Good to have you here =)

wave4

2024-04-14

nrs-status avatar
nrs-status

gotta test run 100+ variations of a deployment (terraform infra + ansible provisioning). what would you guys use to automate this?

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @nick
  • @nrs-status

Good to have you here =)

wave1

2024-04-15

SweetOps avatar
SweetOps
07:00:02 PM

Hey everyone, give a warm welcome to our newest members!

  • @microgag
  • @Maks
  • @Stephan Helas
  • @Shelley

Good to have you here =)

wave1
    keyboard_arrow_up