#random (2019-09)

https://www.youtube.com/watch?v=uDi3zqb3gAA quite old but still worth a watch about how a missing IAM key led to circleCI’s biggest security incident caused by 3rd party provider

https://www.cloudconformity.com/

anyone use mergify? looks pretty slick… https://mergify.io/

No but have used individual things together to create a similar workflow

so… you still need to have tests, you still need to expose coverage, you still need to provide codestyle

so why not to utilize github protected branching with checks

run all those things in your CI and ask PR to be mergeable only after at least some number of people reviewed the change and all the created tests are passing

^ basically what I had happening. There’s then this github tool you can use that checks if all Statuses have passed and if so merge

Seems like mergify is convenient but you sacrifice ability to perhaps use other tools?

for me it’s a question of do i really want another 3rd party service to access my sourcecode

github.com ok, my ci of choice ok, but github apps doing security scanning, some merge helper tools it’s questionable

we’ve been using dependabot to create prs to test new versions of dependencies. it gets to be a lot of prs. the CI does run all such tests on all such PRs and marks the commit with the status, but then we still have to go in and mark it approved and merge it. just looking for a way to automate that workflow for specific conditions

we mostly just have public projects, so not especially concerned about read access (they already have it). the write access required is a concern though, would be great if github had more granular repo permissions

a scheduled job in you ci which pulls the list of open prs and goes through them to check whether or not it was approved and merge the ones which have tests passed and been approved seems like few liner in python

especially that you can set github in a way that it prevents merging if a pr was not approved by a certain number of people

Yeah sure, just nice to build on something used by a wider community. Decisions around config names and structures get paralyzing, and there are a lot of unknown unknowns for any one person. Having input from lots of users and use cases is huge

in some projects we use whitesource bolt for github to keep an eye on vulns in our dependencies, and the way they do writes to our projects is by creating PRs that are reviewed by us

Los Angeles: hi all! I hope you will join us this Wednesday at the TestableLA meetup. This time we’re talking about testing Machine Learning, and testing DevOps! https://www.meetup.com/testable/events/wmkmdqyzmbpb/

https://github.com/dekuNukem/daytripper

Hilarious

https://twitter.com/dexhorthy/status/856639005462417409

Man, if I had an office I’d totally build daytripper~ lol

Anyone have experience/know of a framework (i’m thinking of something like an API gateway) that allows you to combine responses from 2 different microservices and return to client as a unified response?

An example is having a Post microservice, and a analytics microservice. -The post service returns post information for each post (text, author, title, description … etc) -Analytics service has a different DB and hold analytics related info for each post, e.g. (views, likes, .. etc)

I’m looking for a way, to have a single request from a client to then request information from analytics and post service, join the responses and return to client as aggregate response

I have done some research and found KrakenD but have never heard of it before.

Essentially an api / endpoint that hits two different endpoints and maps the results?

yeah

Smells more like a data mapping problem than something a whole framework would do

what are common approaches to solving that problem?

I’ve used a few different libraries for mapping data

but haven’t been super happy with any of them

but it isn’t too hard to roll your own

the simplest approach is to call analytics service from post service, but it would be slower

Well, that’s kind of an interesting question. Is this “middle” layer part of one of the other microservices or is it its own service?

The other question worth asking is why not have the client send two API calls?

^^ true

what language or API framework do the microservices currently use?

what do you mean middle layer?

client -> middle -> Post, Analytics

the client has to hit some api

okay gotcha

It sounds like that API should hit the other two APIs

the frameworks we currently have are java/spring-boot and python/django

so, Django Rest Framework has pretty good mapping support

it can get pretty complicated though

but I’ve set up “remote api” serializers using it

so client -> API 1 -> API 2

we used it to migrate the client over to API 2 with no disruption

but you could use it to do what you are talking about

it’s not out of the box, though

so in your example above, all requests go through API 1 ?

yeah

your example would add a second call

client -> API 1 -> API 2 + API 3

where API 1 is calling both 2 and 3

and then join everything together in API 1?

yeah

that’s a way to do it

but it worked well enough

https://gist.github.com/MrHen/6e96f2aa2729446c15ed4ef4189a087b

can you see that link?

that’s the code we used for the the base remote serializer

that works with Django Rest Framework

yeah i see that, thank you

but if you aren’t used to DRF

it might not be obvious how to get it all hooked up

but you can see the AuthClient bit at the top

that is API 1’s interaction class for talking to API 2

ah I see

does managing api changes become difficult?

yes

you are explicitly coupling API 1 to API 2 and 3

and the the client gets coupled to API 1

so going back to asking the question, why not two calls?

i dont know

if the client already knows about API 2 and 3

i didnt think of that

then it probably isn’t worth it to try and combine the calls

unless you are making thousands of them a minute

here’s how you’d end up using the RemoteSerializer: https://gist.github.com/MrHen/6e96f2aa2729446c15ed4ef4189a087b#file-serializers-py

okay so maybe some background. We are somewhat in the phase of migrating to microservices. So authentication is handled by one of those microservices. In the case of 2 requests would we have to have a centralized authentication?

ah, yeah, okay

so you’ll probably want to add the auth calls

directly to the previously existing api

and then once hte auth microservice is up and running

switch the client to hit the auth service

then remove the call between the existing api and the auth service

but if you need to do backend permissions / validation

you might always have a call between the existing api and the auth service

it really depends on the details

but I would say that the end shape should probably be the client making two calls

but that’s just my opinion

we recently made the exact same change in our platform

the RemoteSerializer from above was used to start migrating calls over to the new api

okay so youre saying, (I’m assuming there are 2 APIs: API1 and API2) if API1 handles authentication and API2 does not, then call the auth endpoints from API2 -> API1 when receiving request on API2

?

and if authentication is separated into its own service then API1 would call auth service (API3) and API2 would also call auth service(API3)

yeah, something like that

the details vary case by case

but hopefully that helps a bit

I don’t know that you are going to find a dedicated backend framework that will hook all this up

I do know that Django has some remote auth functionality

Well i found KrakenD which seems to aggregate responses from multiple services

https://docs.djangoproject.com/en/2.2/howto/auth-remote-user/#authentication-using-remote-user

and using a config file for rules

have you ever heard of that?

I haven’t

and also yes, this is very helpful. Thanks

oh, actually, I have seen some stuff like this

usually it’s integrating with external apis

KrakenD seems kind of interesting

I’m not sure I’d want to bake in that dependency though

yeah it seems to do exactly what I want, but i’m worried it could stop being developed

yeah, tough call

like this is the main functionality i really like about it

yeah, looks kind of interesting

I think people tend to be a little too eager to combine api calls, though

I think making multiple calls can be totally fine

it depends on the situation

the best argument for combining calls is related to things like VPCs or protected / hidden endpoints

the best argument against combining calls is that a change to API A will require a change to the client and your KrakenD config

here is the exact situation, which maybe would make sense for a combined call or not. Let me know what you think.

• request to API1 get most recent posts (so you dont know postIds when making initial call from client)
• client has to wait for response (post Ids) from API1 in order to make the call to API2

I do like the 2 requests approach though…. I just dont want it to be too slow

it won’t be slower than combining the two calls

since both calls still have to happen

do you send a single request to API 2 with the post ids?

or are you looping over each post id and calling API 2 for each id?

yeah single request

but when combining you could do them asynchrounously

i guess not a huge gain

you still need the post ids for the second call, right?

yeah you needs post ids for second call

so you still have to wait for the post ids to “resolve”

which is going to be the longest part of the call

ahh yeah youre right

I’d start by trying two calls from the client

and measure how long it takes

since it would be the simplest approach

Awesome, yeah i’m going to start with that.

and if you run into trouble, you’ve already got a couple good options as backup

thanks for your help

no problem!

good luck!

thanks!!

anyone ever try this out?

https://vertx.io/

I like what they are preaching over there

the docs are amazing, https://vertx.io/docs/

Just in case even 1 person in the 1,411 people here don’t know it: Python 2 end of life is January 1.

Tell your friends. Tell your coworkers at the water cooler. Tell your mailman. Tell your customer you’ve had for 10 years who doesn’t want to change anything. Tell your dog .

Friends don’t let friends use python 2.

https://www.python.org/doc/sunset-python-2/

https://pythonclock.org/

@Doug Farrell has joined the channel

@Rajesh Babu Gangula has joined the channel

@Sebastian Cavedale has joined the channel

@jacob has joined the channel

@Nick S has joined the channel

@daniel has joined the channel

@coreycarvalho has joined the channel

@Manuel Urbano has joined the channel

@Walter Heck has joined the channel

@Ognen Mitev has joined the channel

@Dipesh Patel has joined the channel