#random (2019-09)

Non-work banter and water cooler conversation

A place for non-work-related flimflam, faffing, hodge-podge or jibber-jabber you’d prefer to keep out of more focused work-related channels. Archive: https://archive.sweetops.com/random/

2019-09-30

Manuel Urbano avatar
Manuel Urbano
09:33:03 AM

@Manuel Urbano has joined the channel

Walter Heck avatar
Walter Heck
11:55:46 AM

@Walter Heck has joined the channel

Ognen Mitev avatar
Ognen Mitev
12:38:20 PM

@Ognen Mitev has joined the channel

Dipesh Patel avatar
Dipesh Patel
05:04:59 PM

@Dipesh Patel has joined the channel

2019-09-29

daniel avatar
daniel
12:48:56 PM

@daniel has joined the channel

coreycarvalho avatar
coreycarvalho
02:29:17 PM

@coreycarvalho has joined the channel

2019-09-28

Nick S avatar
Nick S
06:30:40 PM

@Nick S has joined the channel

2019-09-27

roth.andy avatar
roth.andy

Just in case even 1 person in the 1,411 people here don’t know it: Python 2 end of life is January 1.

Tell your friends. Tell your coworkers at the water cooler. Tell your mailman. Tell your customer you’ve had for 10 years who doesn’t want to change anything. Tell your dog .

Friends don’t let friends use python 2.

https://www.python.org/doc/sunset-python-2/

https://pythonclock.org/

Sunsetting Python 2

The official home of the Python Programming Language

Python 2.7 Countdown

Python 2.7 will retire in 2020. Learn more and see the countdown here.

:--1:8
1
doge2
1
Maciek Strömich avatar
Maciek Strömich
How is Python 2 supported in RHEL after 2020? - Red Hat Customer Portal

How is Python 2 supported in RHEL after 2020? How long will Python 2.7 be supported within Red Hat products? As per https://pythonclock.org the Python project is retiring development on Python 2 Jan 1st, 2020. What happens after this date? Python2 is available in an Application stream in RHEL8. Can you let me know what that means for the Python 2 lifecycle within RHEL8? Will Python 2 be available for the entire life of RHEL 8? Will Red Hat continue to provide bugfix/security errata to Python 2 after the EOL of the language?

Robert avatar
Robert
08:10:43 PM
Robert avatar
Robert

@Maciek Strömich ^^

Robert avatar
Robert

But interesting.. I didn’t know that RHEL would support Python 2 until mid 2024.

Maciek Strömich avatar
Maciek Strömich

@Robert considering the speed of doing things in large enterprises/government institutions (some still are forced to use IE6/IE7 ffs ) it’s was kind of obvious that this would happen especially that RHcustomers pay for the support

Doug Farrell avatar
Doug Farrell
06:01:22 PM

@Doug Farrell has joined the channel

Rajesh Babu Gangula avatar
Rajesh Babu Gangula
07:18:57 PM

@Rajesh Babu Gangula has joined the channel

Sebastian Cavedale avatar
Sebastian Cavedale
08:53:38 PM

@Sebastian Cavedale has joined the channel

jacob avatar
jacob
10:23:19 PM

@jacob has joined the channel

2019-09-21

jetstreamin avatar
jetstreamin

anyone ever try this out?

jetstreamin avatar
jetstreamin
Eclipse Vert.x

Eclipse Vert.x is a tool-kit for building reactive applications on the JVM.

jetstreamin avatar
jetstreamin

I like what they are preaching over there

jetstreamin avatar
jetstreamin

the docs are amazing, https://vertx.io/docs/

Vert.x Documentation

Eclipse Vert.x is a tool-kit for building reactive applications on the JVM.

2019-09-17

casey avatar
casey

Anyone have experience/know of a framework (i’m thinking of something like an API gateway) that allows you to combine responses from 2 different microservices and return to client as a unified response?

An example is having a Post microservice, and a analytics microservice. -The post service returns post information for each post (text, author, title, description … etc) -Analytics service has a different DB and hold analytics related info for each post, e.g. (views, likes, .. etc)

I’m looking for a way, to have a single request from a client to then request information from analytics and post service, join the responses and return to client as aggregate response

I have done some research and found KrakenD but have never heard of it before.

PiotrP avatar
PiotrP
Is it possible to consolidate multiple responses and send one response in NGINX

I have Nginx/openresty and some other services running on one VM. Basically VM accepts requests on Openresty and then openresty forwards requests to appropriate service. e.g. below requests getting

mrhen avatar
mrhen

Essentially an api / endpoint that hits two different endpoints and maps the results?

casey avatar
casey

yeah

mrhen avatar
mrhen

Smells more like a data mapping problem than something a whole framework would do

casey avatar
casey

what are common approaches to solving that problem?

mrhen avatar
mrhen

I’ve used a few different libraries for mapping data

mrhen avatar
mrhen

but haven’t been super happy with any of them

mrhen avatar
mrhen

but it isn’t too hard to roll your own

casey avatar
casey

the simplest approach is to call analytics service from post service, but it would be slower

mrhen avatar
mrhen

Well, that’s kind of an interesting question. Is this “middle” layer part of one of the other microservices or is it its own service?

mrhen avatar
mrhen

The other question worth asking is why not have the client send two API calls?

casey avatar
casey

^^ true

mrhen avatar
mrhen

what language or API framework do the microservices currently use?

casey avatar
casey

what do you mean middle layer?

mrhen avatar
mrhen

client -> middle -> Post, Analytics

mrhen avatar
mrhen

the client has to hit some api

casey avatar
casey

okay gotcha

mrhen avatar
mrhen

It sounds like that API should hit the other two APIs

casey avatar
casey

the frameworks we currently have are java/spring-boot and python/django

mrhen avatar
mrhen

so, Django Rest Framework has pretty good mapping support

mrhen avatar
mrhen

it can get pretty complicated though

mrhen avatar
mrhen

but I’ve set up “remote api” serializers using it

mrhen avatar
mrhen

so client -> API 1 -> API 2

mrhen avatar
mrhen

we used it to migrate the client over to API 2 with no disruption

mrhen avatar
mrhen

but you could use it to do what you are talking about

mrhen avatar
mrhen

it’s not out of the box, though

casey avatar
casey

so in your example above, all requests go through API 1 ?

mrhen avatar
mrhen

yeah

mrhen avatar
mrhen

your example would add a second call

mrhen avatar
mrhen

client -> API 1 -> API 2 + API 3

mrhen avatar
mrhen

where API 1 is calling both 2 and 3

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(and ideally async)

mrhen avatar
mrhen

(well, it’s Python… so… I don’t think it gets that for free)

:--1:1
casey avatar
casey

and then join everything together in API 1?

mrhen avatar
mrhen

yeah

mrhen avatar
mrhen

that’s a way to do it

mrhen avatar
mrhen

but it worked well enough

mrhen avatar
mrhen

can you see that link?

mrhen avatar
mrhen

that’s the code we used for the the base remote serializer

mrhen avatar
mrhen

that works with Django Rest Framework

casey avatar
casey

yeah i see that, thank you

mrhen avatar
mrhen

but if you aren’t used to DRF

mrhen avatar
mrhen

it might not be obvious how to get it all hooked up

mrhen avatar
mrhen

but you can see the AuthClient bit at the top

mrhen avatar
mrhen

that is API 1’s interaction class for talking to API 2

casey avatar
casey

ah I see

casey avatar
casey

does managing api changes become difficult?

mrhen avatar
mrhen

yes

mrhen avatar
mrhen

you are explicitly coupling API 1 to API 2 and 3

mrhen avatar
mrhen

and the the client gets coupled to API 1

mrhen avatar
mrhen

so going back to asking the question, why not two calls?

casey avatar
casey

i dont know

mrhen avatar
mrhen

if the client already knows about API 2 and 3

casey avatar
casey

i didnt think of that

mrhen avatar
mrhen

then it probably isn’t worth it to try and combine the calls

mrhen avatar
mrhen

unless you are making thousands of them a minute

mrhen avatar
mrhen

here’s how you’d end up using the RemoteSerializer: https://gist.github.com/MrHen/6e96f2aa2729446c15ed4ef4189a087b#file-serializers-py

casey avatar
casey

okay so maybe some background. We are somewhat in the phase of migrating to microservices. So authentication is handled by one of those microservices. In the case of 2 requests would we have to have a centralized authentication?

mrhen avatar
mrhen

ah, yeah, okay

mrhen avatar
mrhen

so you’ll probably want to add the auth calls

mrhen avatar
mrhen

directly to the previously existing api

mrhen avatar
mrhen

and then once hte auth microservice is up and running

mrhen avatar
mrhen

switch the client to hit the auth service

mrhen avatar
mrhen

then remove the call between the existing api and the auth service

mrhen avatar
mrhen

but if you need to do backend permissions / validation

mrhen avatar
mrhen

you might always have a call between the existing api and the auth service

mrhen avatar
mrhen

it really depends on the details

mrhen avatar
mrhen

but I would say that the end shape should probably be the client making two calls

mrhen avatar
mrhen

but that’s just my opinion

mrhen avatar
mrhen

we recently made the exact same change in our platform

mrhen avatar
mrhen

the RemoteSerializer from above was used to start migrating calls over to the new api

casey avatar
casey

okay so youre saying, (I’m assuming there are 2 APIs: API1 and API2) if API1 handles authentication and API2 does not, then call the auth endpoints from API2 -> API1 when receiving request on API2

casey avatar
casey

?

casey avatar
casey

and if authentication is separated into its own service then API1 would call auth service (API3) and API2 would also call auth service(API3)

mrhen avatar
mrhen

yeah, something like that

mrhen avatar
mrhen

the details vary case by case

mrhen avatar
mrhen

but hopefully that helps a bit

mrhen avatar
mrhen

I don’t know that you are going to find a dedicated backend framework that will hook all this up

mrhen avatar
mrhen

I do know that Django has some remote auth functionality

casey avatar
casey

Well i found KrakenD which seems to aggregate responses from multiple services

casey avatar
casey

and using a config file for rules

casey avatar
casey

have you ever heard of that?

mrhen avatar
mrhen

I haven’t

casey avatar
casey

and also yes, this is very helpful. Thanks

mrhen avatar
mrhen

oh, actually, I have seen some stuff like this

mrhen avatar
mrhen

usually it’s integrating with external apis

mrhen avatar
mrhen

KrakenD seems kind of interesting

mrhen avatar
mrhen

I’m not sure I’d want to bake in that dependency though

casey avatar
casey

yeah it seems to do exactly what I want, but i’m worried it could stop being developed

mrhen avatar
mrhen

yeah, tough call

casey avatar
casey

like this is the main functionality i really like about it

casey avatar
casey
mrhen avatar
mrhen

yeah, looks kind of interesting

mrhen avatar
mrhen

I think people tend to be a little too eager to combine api calls, though

mrhen avatar
mrhen

I think making multiple calls can be totally fine

mrhen avatar
mrhen

it depends on the situation

mrhen avatar
mrhen

the best argument for combining calls is related to things like VPCs or protected / hidden endpoints

mrhen avatar
mrhen

the best argument against combining calls is that a change to API A will require a change to the client and your KrakenD config

casey avatar
casey

here is the exact situation, which maybe would make sense for a combined call or not. Let me know what you think.

  • request to API1 get most recent posts (so you dont know postIds when making initial call from client)
  • client has to wait for response (post Ids) from API1 in order to make the call to API2
casey avatar
casey

I do like the 2 requests approach though…. I just dont want it to be too slow

mrhen avatar
mrhen

it won’t be slower than combining the two calls

mrhen avatar
mrhen

since both calls still have to happen

mrhen avatar
mrhen

do you send a single request to API 2 with the post ids?

mrhen avatar
mrhen

or are you looping over each post id and calling API 2 for each id?

casey avatar
casey

yeah single request

casey avatar
casey

but when combining you could do them asynchrounously

casey avatar
casey

i guess not a huge gain

mrhen avatar
mrhen

you still need the post ids for the second call, right?

casey avatar
casey

yeah you needs post ids for second call

mrhen avatar
mrhen

so you still have to wait for the post ids to “resolve”

mrhen avatar
mrhen

which is going to be the longest part of the call

casey avatar
casey

ahh yeah youre right

mrhen avatar
mrhen

I’d start by trying two calls from the client

mrhen avatar
mrhen

and measure how long it takes

mrhen avatar
mrhen

since it would be the simplest approach

casey avatar
casey

Awesome, yeah i’m going to start with that.

mrhen avatar
mrhen

and if you run into trouble, you’ve already got a couple good options as backup

1
casey avatar
casey

thanks for your help

mrhen avatar
mrhen

no problem!

mrhen avatar
mrhen

good luck!

casey avatar
casey

thanks!!

2019-09-10

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hilarious

Alex Siegman avatar
Alex Siegman

Man, if I had an office I’d totally build daytripper~ lol

2019-09-09

Nikola Velkovski avatar
Nikola Velkovski
dekuNukem/daytripper

A Multifunctional Laser Tripwire. Contribute to dekuNukem/daytripper development by creating an account on GitHub.

:--1:1
1

2019-09-08

johntellsall avatar
johntellsall

Los Angeles: hi all! I hope you will join us this Wednesday at the TestableLA meetup. This time we’re talking about testing Machine Learning, and testing DevOps! https://www.meetup.com/testable/events/wmkmdqyzmbpb/

The Monthly Meetup: Testing Machine Learning and DevOps attachment image

Wed, Sep 11, 2019, 6:30 PM: TestableLA is all about testing - how to write better tests, and how to make your code easier to test. We welcome talks on a variety of technologies and programming languag

2019-09-05

loren avatar
loren

anyone use mergify? looks pretty slick… https://mergify.io/

oscar avatar
oscar

No but have used individual things together to create a similar workflow

Maciek Strömich avatar
Maciek Strömich

so… you still need to have tests, you still need to expose coverage, you still need to provide codestyle

Maciek Strömich avatar
Maciek Strömich

so why not to utilize github protected branching with checks

Maciek Strömich avatar
Maciek Strömich

run all those things in your CI and ask PR to be mergeable only after at least some number of people reviewed the change and all the created tests are passing

oscar avatar
oscar

^ basically what I had happening. There’s then this github tool you can use that checks if all Statuses have passed and if so merge

oscar avatar
oscar

Seems like mergify is convenient but you sacrifice ability to perhaps use other tools?

Maciek Strömich avatar
Maciek Strömich

for me it’s a question of do i really want another 3rd party service to access my sourcecode

Maciek Strömich avatar
Maciek Strömich

github.com ok, my ci of choice ok, but github apps doing security scanning, some merge helper tools it’s questionable

loren avatar
loren

we’ve been using dependabot to create prs to test new versions of dependencies. it gets to be a lot of prs. the CI does run all such tests on all such PRs and marks the commit with the status, but then we still have to go in and mark it approved and merge it. just looking for a way to automate that workflow for specific conditions

1
loren avatar
loren

we mostly just have public projects, so not especially concerned about read access (they already have it). the write access required is a concern though, would be great if github had more granular repo permissions

Maciek Strömich avatar
Maciek Strömich

a scheduled job in you ci which pulls the list of open prs and goes through them to check whether or not it was approved and merge the ones which have tests passed and been approved seems like few liner in python

Maciek Strömich avatar
Maciek Strömich

especially that you can set github in a way that it prevents merging if a pr was not approved by a certain number of people

loren avatar
loren

Yeah sure, just nice to build on something used by a wider community. Decisions around config names and structures get paralyzing, and there are a lot of unknown unknowns for any one person. Having input from lots of users and use cases is huge

1
Maciek Strömich avatar
Maciek Strömich

in some projects we use whitesource bolt for github to keep an eye on vulns in our dependencies, and the way they do writes to our projects is by creating PRs that are reviewed by us

2019-09-03

oscar avatar
oscar
Cloud Conformity, Continuous Assurance

Cloud Conformity highlights violations of AWS and Azure best practices, delivering over 500 different checks across all key areas — security, reliabili…

:--1:3

2019-09-02

Maciek Strömich avatar
Maciek Strömich

https://www.youtube.com/watch?v=uDi3zqb3gAA quite old but still worth a watch about how a missing IAM key led to circleCI’s biggest security incident caused by 3rd party provider

:--1:1
    keyboard_arrow_up