#random (2019-09)

Non-work banter and water cooler conversation

A place for non-work-related flimflam, faffing, hodge-podge or jibber-jabber you’d prefer to keep out of more focused work-related channels.

Archive: https://archive.sweetops.com/random/

2019-09-02

Maciek Strömich avatar
Maciek Strömich

https://www.youtube.com/watch?v=uDi3zqb3gAA quite old but still worth a watch about how a missing IAM key led to circleCI’s biggest security incident caused by 3rd party provider

1

2019-09-03

oscar avatar
Cloud Conformity, Continuous Assurance

Cloud Conformity highlights violations of AWS and Azure best practices, delivering over 500 different checks across all key areas — security, reliabili…

3

2019-09-05

loren avatar

anyone use mergify? looks pretty slick… https://mergify.io/

oscar avatar

No but have used individual things together to create a similar workflow

Maciek Strömich avatar
Maciek Strömich

so… you still need to have tests, you still need to expose coverage, you still need to provide codestyle

Maciek Strömich avatar
Maciek Strömich

so why not to utilize github protected branching with checks

Maciek Strömich avatar
Maciek Strömich

run all those things in your CI and ask PR to be mergeable only after at least some number of people reviewed the change and all the created tests are passing

oscar avatar

^ basically what I had happening. There’s then this github tool you can use that checks if all Statuses have passed and if so merge

oscar avatar

Seems like mergify is convenient but you sacrifice ability to perhaps use other tools?

Maciek Strömich avatar
Maciek Strömich

for me it’s a question of do i really want another 3rd party service to access my sourcecode

Maciek Strömich avatar
Maciek Strömich

github.com ok, my ci of choice ok, but github apps doing security scanning, some merge helper tools it’s questionable

loren avatar

we’ve been using dependabot to create prs to test new versions of dependencies. it gets to be a lot of prs. the CI does run all such tests on all such PRs and marks the commit with the status, but then we still have to go in and mark it approved and merge it. just looking for a way to automate that workflow for specific conditions

1
loren avatar

we mostly just have public projects, so not especially concerned about read access (they already have it). the write access required is a concern though, would be great if github had more granular repo permissions

Maciek Strömich avatar
Maciek Strömich

a scheduled job in you ci which pulls the list of open prs and goes through them to check whether or not it was approved and merge the ones which have tests passed and been approved seems like few liner in python

Maciek Strömich avatar
Maciek Strömich

especially that you can set github in a way that it prevents merging if a pr was not approved by a certain number of people

loren avatar

Yeah sure, just nice to build on something used by a wider community. Decisions around config names and structures get paralyzing, and there are a lot of unknown unknowns for any one person. Having input from lots of users and use cases is huge

1
Maciek Strömich avatar
Maciek Strömich

in some projects we use whitesource bolt for github to keep an eye on vulns in our dependencies, and the way they do writes to our projects is by creating PRs that are reviewed by us

2019-09-08

johntellsall avatar
johntellsall

Los Angeles: hi all! I hope you will join us this Wednesday at the TestableLA meetup. This time we’re talking about testing Machine Learning, and testing DevOps! https://www.meetup.com/testable/events/wmkmdqyzmbpb/

The Monthly Meetup: Testing Machine Learning and DevOpsattachment image

Wed, Sep 11, 2019, 6:30 PM: TestableLA is all about testing - how to write better tests, and how to make your code easier to test. We welcome talks on a variety of technologies and programming languag

2019-09-09

Nikola Velkovski avatar
Nikola Velkovski
dekuNukem/daytripper

A Multifunctional Laser Tripwire. Contribute to dekuNukem/daytripper development by creating an account on GitHub.

1
1

2019-09-10

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hilarious

Alex Siegman avatar
Alex Siegman

Man, if I had an office I’d totally build daytripper~ lol

2019-09-17

casey avatar

Anyone have experience/know of a framework (i’m thinking of something like an API gateway) that allows you to combine responses from 2 different microservices and return to client as a unified response?

An example is having a Post microservice, and a analytics microservice. -The post service returns post information for each post (text, author, title, description … etc) -Analytics service has a different DB and hold analytics related info for each post, e.g. (views, likes, .. etc)

I’m looking for a way, to have a single request from a client to then request information from analytics and post service, join the responses and return to client as aggregate response

I have done some research and found KrakenD but have never heard of it before.

PiotrP avatar
Is it possible to consolidate multiple responses and send one response in NGINX

I have Nginx/openresty and some other services running on one VM. Basically VM accepts requests on Openresty and then openresty forwards requests to appropriate service. e.g. below requests getting

mrhen avatar

Essentially an api / endpoint that hits two different endpoints and maps the results?

casey avatar

yeah

mrhen avatar

Smells more like a data mapping problem than something a whole framework would do

casey avatar

what are common approaches to solving that problem?

mrhen avatar

I’ve used a few different libraries for mapping data

mrhen avatar

but haven’t been super happy with any of them

mrhen avatar

but it isn’t too hard to roll your own

casey avatar

the simplest approach is to call analytics service from post service, but it would be slower

mrhen avatar

Well, that’s kind of an interesting question. Is this “middle” layer part of one of the other microservices or is it its own service?

mrhen avatar

The other question worth asking is why not have the client send two API calls?

casey avatar

^^ true

mrhen avatar

what language or API framework do the microservices currently use?

casey avatar

what do you mean middle layer?

mrhen avatar

client -> middle -> Post, Analytics

mrhen avatar

the client has to hit some api

casey avatar

okay gotcha

mrhen avatar

It sounds like that API should hit the other two APIs

casey avatar

the frameworks we currently have are java/spring-boot and python/django

mrhen avatar

so, Django Rest Framework has pretty good mapping support

mrhen avatar

it can get pretty complicated though

mrhen avatar

but I’ve set up “remote api” serializers using it

mrhen avatar

so client -> API 1 -> API 2

mrhen avatar

we used it to migrate the client over to API 2 with no disruption

mrhen avatar

but you could use it to do what you are talking about

mrhen avatar

it’s not out of the box, though

casey avatar

so in your example above, all requests go through API 1 ?

mrhen avatar

yeah

mrhen avatar

your example would add a second call

mrhen avatar

client -> API 1 -> API 2 + API 3

mrhen avatar

where API 1 is calling both 2 and 3

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(and ideally async)

mrhen avatar

(well, it’s Python… so… I don’t think it gets that for free)

1
casey avatar

and then join everything together in API 1?

mrhen avatar

yeah

mrhen avatar

that’s a way to do it

mrhen avatar

but it worked well enough

mrhen avatar

can you see that link?

mrhen avatar

that’s the code we used for the the base remote serializer

mrhen avatar

that works with Django Rest Framework

casey avatar

yeah i see that, thank you

mrhen avatar

but if you aren’t used to DRF

mrhen avatar

it might not be obvious how to get it all hooked up

mrhen avatar

but you can see the AuthClient bit at the top

mrhen avatar

that is API 1’s interaction class for talking to API 2

casey avatar

ah I see

casey avatar

does managing api changes become difficult?

mrhen avatar

yes

mrhen avatar

you are explicitly coupling API 1 to API 2 and 3

mrhen avatar

and the the client gets coupled to API 1

mrhen avatar

so going back to asking the question, why not two calls?

casey avatar

i dont know

mrhen avatar

if the client already knows about API 2 and 3

casey avatar

i didnt think of that

mrhen avatar

then it probably isn’t worth it to try and combine the calls

mrhen avatar

unless you are making thousands of them a minute

mrhen avatar

here’s how you’d end up using the RemoteSerializer: https://gist.github.com/MrHen/6e96f2aa2729446c15ed4ef4189a087b#file-serializers-py

casey avatar

okay so maybe some background. We are somewhat in the phase of migrating to microservices. So authentication is handled by one of those microservices. In the case of 2 requests would we have to have a centralized authentication?

mrhen avatar

ah, yeah, okay

mrhen avatar

so you’ll probably want to add the auth calls

mrhen avatar

directly to the previously existing api

mrhen avatar

and then once hte auth microservice is up and running

mrhen avatar

switch the client to hit the auth service

mrhen avatar

then remove the call between the existing api and the auth service

mrhen avatar

but if you need to do backend permissions / validation

mrhen avatar

you might always have a call between the existing api and the auth service

mrhen avatar

it really depends on the details

mrhen avatar

but I would say that the end shape should probably be the client making two calls

mrhen avatar

but that’s just my opinion

mrhen avatar

we recently made the exact same change in our platform

mrhen avatar

the RemoteSerializer from above was used to start migrating calls over to the new api

casey avatar

okay so youre saying, (I’m assuming there are 2 APIs: API1 and API2) if API1 handles authentication and API2 does not, then call the auth endpoints from API2 -> API1 when receiving request on API2

casey avatar

and if authentication is separated into its own service then API1 would call auth service (API3) and API2 would also call auth service(API3)

mrhen avatar

yeah, something like that

mrhen avatar

the details vary case by case

mrhen avatar

but hopefully that helps a bit

mrhen avatar

I don’t know that you are going to find a dedicated backend framework that will hook all this up

mrhen avatar

I do know that Django has some remote auth functionality

casey avatar

Well i found KrakenD which seems to aggregate responses from multiple services

casey avatar

and using a config file for rules

casey avatar

have you ever heard of that?

mrhen avatar

I haven’t

casey avatar

and also yes, this is very helpful. Thanks

mrhen avatar

oh, actually, I have seen some stuff like this

mrhen avatar

usually it’s integrating with external apis

mrhen avatar

KrakenD seems kind of interesting

mrhen avatar

I’m not sure I’d want to bake in that dependency though

casey avatar

yeah it seems to do exactly what I want, but i’m worried it could stop being developed

mrhen avatar

yeah, tough call

casey avatar

like this is the main functionality i really like about it

mrhen avatar

yeah, looks kind of interesting

mrhen avatar

I think people tend to be a little too eager to combine api calls, though

mrhen avatar

I think making multiple calls can be totally fine

mrhen avatar

it depends on the situation

mrhen avatar

the best argument for combining calls is related to things like VPCs or protected / hidden endpoints

mrhen avatar

the best argument against combining calls is that a change to API A will require a change to the client and your KrakenD config

casey avatar

here is the exact situation, which maybe would make sense for a combined call or not. Let me know what you think.

  • request to API1 get most recent posts (so you dont know postIds when making initial call from client)
  • client has to wait for response (post Ids) from API1 in order to make the call to API2
casey avatar

I do like the 2 requests approach though…. I just dont want it to be too slow

mrhen avatar

it won’t be slower than combining the two calls

mrhen avatar

since both calls still have to happen

mrhen avatar

do you send a single request to API 2 with the post ids?

mrhen avatar

or are you looping over each post id and calling API 2 for each id?

casey avatar

yeah single request

casey avatar

but when combining you could do them asynchrounously

casey avatar

i guess not a huge gain

mrhen avatar

you still need the post ids for the second call, right?

casey avatar

yeah you needs post ids for second call

mrhen avatar

so you still have to wait for the post ids to “resolve”

mrhen avatar

which is going to be the longest part of the call

casey avatar

ahh yeah youre right

mrhen avatar

I’d start by trying two calls from the client

mrhen avatar

and measure how long it takes

mrhen avatar

since it would be the simplest approach

casey avatar

Awesome, yeah i’m going to start with that.

mrhen avatar

and if you run into trouble, you’ve already got a couple good options as backup

1
casey avatar

thanks for your help

mrhen avatar

no problem!

mrhen avatar

good luck!

casey avatar

thanks!!

2019-09-21

jetstreamin avatar
jetstreamin

anyone ever try this out?

jetstreamin avatar
jetstreamin
Eclipse Vert.x

Eclipse Vert.x is a tool-kit for building reactive applications on the JVM.

jetstreamin avatar
jetstreamin

I like what they are preaching over there

jetstreamin avatar
jetstreamin

the docs are amazing, https://vertx.io/docs/

Vert.x Documentation

Eclipse Vert.x is a tool-kit for building reactive applications on the JVM.

2019-09-27

roth.andy avatar
roth.andy

Just in case even 1 person in the 1,411 people here don’t know it: Python 2 end of life is January 1.

Tell your friends. Tell your coworkers at the water cooler. Tell your mailman. Tell your customer you’ve had for 10 years who doesn’t want to change anything. Tell your dog .

Friends don’t let friends use python 2.

https://www.python.org/doc/sunset-python-2/

https://pythonclock.org/

Sunsetting Python 2

The official home of the Python Programming Language

Python 2.7 Countdown

Python 2.7 will retire in 2020. Learn more and see the countdown here.

8
1
doge2
1
Maciek Strömich avatar
Maciek Strömich
How is Python 2 supported in RHEL after 2020? - Red Hat Customer Portal

How is Python 2 supported in RHEL after 2020? How long will Python 2.7 be supported within Red Hat products? As per https://pythonclock.org the Python project is retiring development on Python 2 Jan 1st, 2020. What happens after this date? Python2 is available in an Application stream in RHEL8. Can you let me know what that means for the Python 2 lifecycle within RHEL8? Will Python 2 be available for the entire life of RHEL 8? Will Red Hat continue to provide bugfix/security errata to Python 2 after the EOL of the language?

Robert avatar
Robert
08:10:43 PM
Robert avatar

@Maciek Strömich ^^

Robert avatar

But interesting.. I didn’t know that RHEL would support Python 2 until mid 2024.

Maciek Strömich avatar
Maciek Strömich

@Robert considering the speed of doing things in large enterprises/government institutions (some still are forced to use IE6/IE7 ffs ) it’s was kind of obvious that this would happen especially that RHcustomers pay for the support

Doug Farrell avatar
Doug Farrell
06:01:22 PM

@Doug Farrell has joined the channel

Rajesh Babu Gangula avatar
Rajesh Babu Gangula
07:18:57 PM

@Rajesh Babu Gangula has joined the channel

Sebastian Cavedale avatar
Sebastian Cavedale
08:53:38 PM

@Sebastian Cavedale has joined the channel

jacob avatar
jacob
10:23:19 PM

@jacob has joined the channel

2019-09-28

Nick S avatar
Nick S
06:30:40 PM

@Nick S has joined the channel

2019-09-29

daniel avatar
daniel
12:48:56 PM

@daniel has joined the channel

coreycarvalho avatar
coreycarvalho
02:29:17 PM

@coreycarvalho has joined the channel

2019-09-30

Manuel Urbano avatar
Manuel Urbano
09:33:03 AM

@Manuel Urbano has joined the channel

Walter Heck avatar
Walter Heck
11:55:46 AM

@Walter Heck has joined the channel

Ognen Mitev avatar
Ognen Mitev
12:38:20 PM

@Ognen Mitev has joined the channel

Dipesh Patel avatar
Dipesh Patel
05:04:59 PM

@Dipesh Patel has joined the channel

    keyboard_arrow_up