1 of 44

Developer Experience

How BillEase balances Speed, Quality, and Usability of Python Microservices

Alec Gerona, VP of Engineering, BillEase

2 of 44

So what's in this for you?

3 of 44

Developer Experience (DX)

What is this?

4 of 44

repository

python version

pre-commit

testing

upkeep

5 of 44

repository

Every microservice repository in BillEase comes from our boilerplate. This is what the developer first see and what they first play with. Here you'll see how the team has decided how to logically partition our code. ��Voila, this is what the root-view of our microservice built on FastAPI looks like. ��<ol>�<li>app/ - everything that FastAPI needs to function - config injection from env variables, routes, security</li>�<li>db/ - sql files, here at BillEase we use a query builder instead of ORMs like SQLAlchemy. Long story short - we got burned by Django ORM and wanted more flexibility and control over our queries.</li>�<li>lib/ - external SDKs, third-party APIs, all these interfaces go here.</li>�<li>requirements/ - we have three "requirements.txt" files. Base, test, and dev. They all just inherit from each other. Base is installed by our docker image. Test is installed by our test runner, which installs base by extension. Dev is installed locally by devs which installs base and test.</li>�<li>tests/ - all our test files. We'll go into this a bit more later.</li>�<li>util/ - helper functions we don't know where else to put.</li>�</ol>

6 of 44

python version

7 of 44

pre-commit

8 of 44

flake8

linting

9 of 44

isort

10 of 44

"Any color you like"

black

styling

One of the most contentious module we ever installed. ��I hope everyone's clear with the difference between styling and linting? You get styling wrong and your code reviewer gets mad at you, you get linting wrong and you bring production down.��Anyone here using a styler in their code? Maybe Prettier for Javascript? ��It's pretty much like that for Black, just much more polarizing. For example, if you spent any time working with Python you'd know that strings are often done with single quotes. Not so with Black. How about line length? That's right. With black it's 88. 10% over 80. Black is notoriously known for having these decisions set and unconfigurable as well. ��I remember the arguments Jasper, our panel representative over there, and I had for hours discussing this. With him constantly sending me use cases of Black formatting some code "weirdly" and me just saying "it'll be consistently weird, then" until finally I wore him down and now we're using "any color we like". ��Jokes aside, the main benefit of Black is to have the smallest git diff possible so reviews won't be longer than it needs to be.

11 of 44

mypy

typehints

This one's interesting. One of Python's lesser known features is that it actually supports type-hinting. It was actually supported as early as Python 3.5. ��While I've spoken with a lot of developers about their usage of type hints, with some who actually have heard of it, fewer who actually used it, most who haven't. And of those who actually use it, they are not even enforced. It's treated more as a nice-to-have during code reviews. ��Here at BillEase, we believe this should be enforced. During our experience testing and utilizing FastAPI as our Python framework of choice, and if you guys didn't know, FastAPI (which uses Pydantic internally) and Python typehints are two peas in a pod, we came to the conclusion that doing this in practice essentially acts as a self-code review. The only thing remaining is to enforce it. BillEase is also a contributor of Pydantic so please star them as well.��Mypy is simply a package that cross checks your type hints usage along with PEP standards. We highly recommend you to check it out. This is not sponsored by the way.

12 of 44

testing

Let's talk testing. Though this talk is focused on Developer Experience, testing - automated testing forms a significant role in our development so let's talk about it for a minute. ��Raise of hands, how many here does automated testing for their own code? Unit testing? Integration testing? End to end? ��Now, keep your hands up for those who use code coverage to augment their testing?��For those who do, have you ever reached 100%? Does it even matter? ��All hands down. ��Lastly, raise your hands if you're happy with the state of developer-written tests in your company?��There's loads of articles around that espouses different approaches. A lot may say that unit tests are king - we should 100% all code that we can and mock those that we can't. Others say that end to end tests are the way since that gives you the most confidence in your tests. So which is it? ��At BillEase we knew that whichever way we go, we have a couple of must-haves to even have a chance of making developers happy to do testing.

13 of 44

So what does testing as a first-class citizen look like?

14 of 44

Need to see how current apis work and see sample responses?

Look at a test.

15 of 44

Need to add a new endpoint and need to see if it's working?

Write a test.

16 of 44

Need to see if the current build even runs?

Run a test.

17 of 44

tox

18 of 44

pytest

19 of 44

20 of 44

21 of 44

22 of 44

23 of 44

24 of 44

One more setup we need to do, is to freeze time. Have you ever done this for your tests? Very neat if you need to test time-sensitive features such as this one. In this case we'd like to know if our expected response still holds true a week later. This means our consumer actually has some behind-the-scenes logic that relies on time. ��To make this test robust, we are freezing time on execution since we don't really care what timestamp it should have.��For the actual execution of the test, we see line 79 initiating class <code>SQSConsumer()</code>. This is an in-house class we have that starts the consumption process by running a thread. This is why we have line 82 to make sure this thread closes after consuming. Fun fact, do you know why it's called <code>join()</code>?��We also have a simple patching here. This is also a plug for our open source library for interacting with RabbitMQ - PyRMQ. Please check it out and star it. Obviously we're not publishing to an actual RMQ queue so we just mock. ��We then time travel to one week later and call an api to confirm our expectations.

25 of 44

As for the last part of our test, we'll do our assertions. ��We first do a select on the Message table. Presumably our consumer must have inserted an entry here. ��Then we assert our responses.��This is where <code>syrupy</code> comes in next. Not sure where it got the name but it's a well-made zero-dependency library for snapshots. Instead of hardcoding assertions which works for small responses or really insular codes, we can just snapshot the entire response and assert that. Notice that we pop the timestamp out of the snapshot since this is dynamic. ��For assertion overall, you can see that we snapshot twice, once in line 95 being the Message the consumer saved into the db, and line 96 being the response we got from the check API. These snapshots are saved in a relative directory to your tests. ��Finally, the RMQ publish event is asserted and we're done with this test. ��Simple, right? So to recap, we were able to setup sorta-real AWS resources and test our actual code's usage of them. Now we know that if our snapshot has changed, something is wrong with the code.

26 of 44

coverage

27 of 44

28 of 44

"Just put a realistic coverage percentage"

29 of 44

"How should we approach coverage to maximize testing?"

30 of 44

31 of 44

32 of 44

upkeep

33 of 44

CodeRabbit

34 of 44

Renovate

35 of 44

36 of 44

osv.dev

Open Source Vulnerabilities

37 of 44

38 of 44

39 of 44

40 of 44

41 of 44

Semantic Release

42 of 44

Changelog

Release

43 of 44

Summary

repository
python version
pre-commit
testing
Upkeep

44 of 44

Need slides?

alecgerona.io