Transcript

Over the last 20 or 25 videos, we have learned a lot of concepts. And in all of that, we have never written a test. That was intentional. And we'll see why in this video. Because writing a bad test, writing a test in hurry, writing an incomplete test is more harmful than not having tests at all. Because they give you a false sense of confidence about your code, about your application. And that is why I did not want to keep a small part in testing for each one of those videos where the concepts themselves were of importance and not the test. So in this video we are going to cover everything there is about testing and we'll justify it as much as we can.

So let's define what exactly do we mean by a test. A test is a second program that runs your program and it makes claims about what all things happened. And there are a lot of reasons to write tests. We have this whole area this whole style of development called test-driven development where you write the tests first and then write the code so that you have your goal in mind what you want to achieve with your program. So there are different different motivations of writing test either as a guidance or documentation but primarily the most useful reason the most important reason for writing tests is to catch regressions.

What do I mean by regressions? Regression basically means that you wrote some code, you wrote some logic in your codebase a few months back and as you keep adding more and more features to your codebase, you want to ensure that the previous features still function exactly as they used to when you first time tested them manually or first time use them. And every single thing that we are going to cover in this video, every type of test, every metric, every tool that is just a variation of this definition that uh we just saw and testing itself is a huge field. Uh there are people whose whole job is about testing.

There are job titles dedicated to testing and different kinds of testing. So to scope it a little bit of what we mean in this video about what kind of testing we are going to cover what a backend engineer writes and runs themselves which is about 80% of the testing that happens in a normal team. We are not going to cover the corporate style manual QA process and other tools that are available like test management tools, mobile testing or browser automation for front ends and all these areas of testing. The application that we are going to test in this video is the same task board that we saw in the last video.

So let's see the code before we write any tests on top of it. So this is our codebase. It has around 11 Go files and that is pretty much the whole server. The cmd/board is the binary, the executable binary and main is the only place that knows how to put everything together. Internal/board is the domain for different different features like tasks, columns etc. Internal/hub this holds uh the rooms since we are talking about boards and task management. And every subscriber in a room has its own counter. Then we have internal/bus. This is an interface with two methods and it has two different kinds of implementations.

First one keeps everything in process. That is the one that we are going to use for testing and that is one of the uh core use cases of interfaces so that you can have the same same API same interface but two different implementations behind it. The second one the real one is going to use radius for our actual application. Then we have internal store. This also has the similar structure. We have an interface with three methods and there is posgress behind them. And we also have internal HTTP API. This holds the whole route table so that you can look at one file and know what all API endpoints that we are exposing all the URLs.

And one thing uh to notice here is there are no tests in this codebase yet. That's what we are going to write. And for those who do not have any familiarity with the Go programming language, that is fine. One of the claims of this playlist is you do not need to know any specific programming language to watch these videos. As long as you are a programmer and you code in any other language, that is fine. We are using a particular language in this so that you have a concrete understanding of what exactly the code does. We will of course keep most part of this video as theoretical as conceptual instead of focusing too much on code.

But there has to be a few parts which needs to show some code so that you can map the same understanding to a different programming language instead of just having a theoretical understanding because tests in theory is not a lot. It's it's not very complex right the code the thinking tests in code that is the core part of it. So let's start by understanding tests. First thing that I have already mentioned a test is a program. It is a different program than your existing application or your back end. But it is a program in itself. So a test is a piece or a snippet of code that calls your existing code and it looks at what it gives some kind of inputs which we decide beforehand and it looks at the output and if it does not match what we were expecting then it complains it throws errors it it crashes it panics whatever terminology that you are familiar with in a very primary sense even Even if you don't have any kind of testing tools, any testing libraries or frameworks or the whole technicalities of the whole test ecosystem in a very fundamental sense if you wrote a normallooking program and does not matter whatever programming language let's say the same programming language that you uh wrote your back end in and you just checked two things you called some part of your actual back end through a different function.

Let's call this function your test and you just wrote an if condition that if output is not equals to 20 or something then you throw error or you return error or you throw exception whatever you're familiar with right that is a test even without going into all the complexities of testing ecosystem but over the time people have realized that that is not enough considering how complex our back end has become, how complex our applications has become in real world. Handwriting your conditions, handwriting your test runners, handwriting your mocking, all these different concepts every time you want to test your program that is not very scalable and that is the logic behind any other library.

Why do people write libraries in the first place? so that they solve the problem once and they let everyone else solve their problems using the same library instead of reinventing the wheel every time. Okay. So what a programming language gives you if you're using a programming language like Go then it comes with its own testing um module its own testing features and every programming language has uh some kind of inbuilt testing mechanism but we use libraries and frameworks on top of that to make our experience more developer friendly and more uh intuitive. So what a programming language or a framework gives you for testing is a test runner.

What is a testr runner? It is a program that finds all your test files. So I am talking in in the context of go programming language. So in a go testr runner, it finds all your test files across the whole codebase and it runs them and in the end it gives you a very uh beautiful looking report of how many tests passed and how many of them failed. So this is the first test in this codebase. What it does, it creates a board and it moves a task from one column to another and it checks that the task ended up where we expected it to end up or where we asked it to end up. And here I have made a mistake on purpose so that we can watch it fail.

And this is what it gives us the test binary. It prints the file the line number and what it wanted versus what it got. Uh the task it got was in doing and the test wanted it done. Now we fix the expectation and let's run it again and it passes instantly. Now let's look at what is happening behind it. I run it with dash work which tells the tool to keep the folder it builds in instead of deleting it after it runs and it prints me the path. And if you look inside it there is a file in there called underscore test main.go and we did not write it. the tool wrote it. And if you read it, it says package main and it has a list called tests with every test function it found in our package.

And at the bottom, it has a funk main which is the main executable for this testing program which calls testing.main start and then the os.exit for exiting the program. Now we run it once more with dash x and what that does is it prints every command that the tool runs and it reads the last few lines. So it links a binary called board.est and then it runs board.est. So go test writes a main function, compiles it into a program and executes the program. So far what we learned a test is a program and what is a runner? a runner, no matter what programming language, it writes the main function, the main executable where the uh execution starts for any kind of binary for you instead of you having to write it and then it collects all the results.

That's pretty much the whole idea behind using all these frameworks and test runners. Okay, moving on to the next concept which is the part inside your test that actually does the complaining. That is what we call as an assertion. An assertion is a check that either stops or fails the whole test when the condition is not true. In most languages to use all these assertions, we use a library and the libraries they give you uh these very useful functions and sentences like assert equals, assert not equals, assert number and all this stuff. And but in a programming language like Go there is no separate assert function in the standard library you just write an if statement and when the condition is wrong you just call t dot error f and you just write the message yourself and uh the language works it out.

One thing to remember here when you are writing tests is in all your assertions in a language like Go you directly write the message. So it works out without you thinking a lot. But in other programming languages always make sure that you are writing a very meaningful failure message so that whoever is looking at the failing test they know exactly what to check what exactly failed and enough information so that they can debug the whole thing. I have already mentioned this but regression is something that you will hear a lot when we are talking about tests and regression is basically a bug that you had already fixed in your past but it comes again it comes back that's what we mean by regression so uh it it can happen because somebody in your team they changed a function or they changed an interface or they changed a particular a logical behavior and something happened, right?

Which is very normal when you are working in a team. But the best part about writing tests is you catch the regressions instantly because typically what we have is in our CI system in our continuous integration system which is what we mean when we say if you're using something like GitHub actions or something there usually we have a CI workflow for running our tests. So every time we raise a new PR before that PR is merged into your main branch or develop branch that whole test suite runs and if any of the tests fail then we know instantly that this new piece of code that we are trying to merge it is causing regressions.

It is going to cause bugs that we have already fixed and because of that you can instantly fix it instead of finding out from your users complaining about a feature failing. And that behavior is the most useful thing about a test suite which is you can move rapidly. You can add features very quickly without worrying about your existing features breaking because you already have a very deterministic test suite which tests every single condition, every single bugs that had happened in the past. So you are enforcing that with the new piece of code those bugs are still staying fixed instead of coming back.

So now that we know what a single test is, let's explore what are the different kinds of tests that are available to us. Starting with unit test. What do we call as unit test? A unit test is a test of one unit of your code in isolation. It does not care about the surrounding code where it runs. Now the question is what can we consider as a unit? Is it a function or is it a strct or is it just a package or it is a whole service? How do you know what can we consider as a unit? And the answer to that is there is no universal definition for that. There is no universal answer. It is just something that you have to learn to recognize over time when you write more and more tests.

So if you take a function that does something some kind of execution and that function calls two or three other things to do its job. In that case you have two choices. You can either while testing this main function which calls other functions, you can either let the function call the actual other functions or you can hand in mocks while running or testing this function. And in both the cases, both of them can be considered as unit tests. So let's see what that looks like in the task board that we just saw. Here there is a same behavior and we are testing it in four different ways. In the first one, the we give the board a mock bus instead of the real bus logic which is just a small object which records what it was asked to publish and that's it.

It does nothing else. It does not actually perform any kind of bus logic. We'll get to what are the different kinds of mocks or standins that are available to us. The second one here it uses the real bus, the actual bus and the actual hub. So when we do a move in the board, this actually goes to a subscriber. Now the third one, it starts an actual HTTP server on an actual port and it sends a real post request and it also reads the actual response. Now if you look at the numbers, all are 0 0. And if you look at the fourth one, we are running the same check but this time against a real posgress database which takes 10 and a half seconds.

And most of it is because the container the posgress container is booting up. So the the thing that I want to point out here is the most expensive time-taking part of a test is not how much of logic that you are trying to cover because we just saw that one of the tests went through the actual entire HTTP stack and for that also we got zero. But the one that actually cost us some latency was the Postgress one. Why? Because it left our existing process. Right? That is the point to remember. It's not how much you are covering in a unit test or a test. It is about whether you are leaving your existing process and going into an external territory or not.

Okay. Now let's cover two more. Integration test. An integration test is a test that checks two or more parts that are working together. And most of the cases when we write an integration test, the first part is of course our application, our back end, our logic. But the other part or parts, they are usually external, which means uh something that we did not write something like a database. And then an end to- end test that is a test that goes through the whole system in the same way that an actual user will go an actual user of our platform all the activities an actual user will do. This test will have all those properties.

What does that mean? uh the integration test and the end to end test they both sound similar but the difference is an end toend test will have to cover every single path which means that what is the first thing that a user need to do to start using our platform which is authentication. So the first part of your end to end test will have to include authentication and then it will take that token that session whatever it recorded for an actual user login it will use the same session then to test other parts let's say the user navigates to the dashboard page then it calls this API then it creates a task all these things so it will try to map it will try to follow a user through their journey into the platform that is the end to end test but in integration testing We will be using our internal logic so that we don't actually have to do a login.

We can just create a task with because the authentication middleware is something that we have enforced for our users. We can of course bypass it. So integration testing does not have to respect every single path a user will take in our platform. Now if you have read anything about testing or you have read any articles then you must have seen this famous test pyramid where the wide base that is for the unit tests then comes a few integration tests and then on the top a cap for the end to end tests and what that picture actually means is it is saying that the bottom ones the base ones which has a longer width they are extremely cheap and they extremely fast.

The ones on top, they keep getting more and more expensive and more and more slow. And when we say expensive, it does not always have to mean money or paying a bill. It can be expensive in the sense of uh spending your CPU resources, spending your memory, spending your time. So this whole pyramid thing, it is about cost. The higher you go on the pyramid, the more expensive and slow your tests are. And the pyramid has been a very good resource for a lot of years when people talk about tests and how much to test and which parts of test and a lot of arguments around testing itself. There are a lot of resources you'll find on the pyramid concept.

But instead of going the theoretical way of thinking about test, I want to discuss a more practical way something that I use. So instead of asking how much of the system a particular test covers, you should ask how much a test is allowed to touch in your system. Okay? And I'll I'll explain what that means. A small test, it runs in one process. It does not touch your network. It does not touch your database. And you do not have to do any artificial sleeping. And it also does not touch your file system. a medium level test that may talk to a few things that is on the same machine. A database that is running in your local container or a radius instance that is in another container in your local only.

Similarly, everything is in the same machine but it still cannot reach the internet. It does not go outside the machine. But a large test, the final upper level test that can do anything. Okay, that's that's a better mental model to think about tests in my opinion. And the reason is nobody can claim nobody can check whether a test is actually a unit test or not. Why? Because there is no universal definition of a unit test and nobody agrees with anyone when it comes to discussions about testing. Everyone has their own opinions of what exactly is a unit test. But the machine the the machine where you are doing the development and testing itself that can check whether your test actually opened a socket or not.

Because of that it becomes a rule which you can enforce instead of a vague argument or a vague opinion. There is a famous highly discussed term that we use whenever we are talking about testing is a flaky test. What a flaky test means is sometimes the test passes, sometimes the test fails. It is not deterministic and it depends on the environment that it is running. But if you think about tests in this model, in the model where we are thinking inside the machine, inside the process or outside the machine, then the whole flakiness of any test that can completely eliminate it. Why? Because if it is inside the machine then we can push the determinism of the test to its peak because it does not talk to the internet itself.

That's where most of the flakiness lies because there are external services that servers have their own clocks and all these complexities. So the first three unit test, integration test and end to- end test they answer the question that how much of the system does this test actually touch. So this whole thing is about scope. Then we have different varieties of them to answer different different other questions. For example, we have regression testing. Those are the tests that we use to catch the bugs to catch the errors that have already happened and we don't want them to happen again. Okay, they are specifically for catching a particular list of bugs.

Then functional testing that is what uh I mentioned in the first of the video when I say test-driven development where you write the specification of the application specification of your feature first and based on that specification which is the test itself then you write your application or your pro your program and because of that after you're done with your application or your feature or your logic you already have a readym made uh test or readymade verification layer which will tell you whether you are done or not. Okay, that's what we mean by functional testing. Then we have performance testing which mostly tests how fast your program is and not just the speed, it also tests how much memory that you are consuming, how much CPU that you are consuming and all those uh separate things which is concerned about performance.

Then there is security testing which is to expose whether your program tries to bypass any security layer like any secure container layer or your file systems or uh the cookies or something which is related to privacy or security. That's what you are trying to find with security testing whether your program is secure or not. Whether it respects privacy or not. Okay. So those are a variation of different tests for different different scenarios but they are not a different kind of test than a unit or integration or end to end test. They are just different variations. So technically a regression test can be a unit test and similarly uh a security test which checks whether you are breaching any kind of privacy or not that can be an end toend test also.

Similarly uh functional testing and integration testing they sound or seem similar because functional testing is just a variation of integration testing. So all these different kinds of testing that we uh have a list for they are just all variations of these three different kinds of test unit integration and end to end which we have already scoped to our mental model which is about the scope either it is in process or in machine or it touches internet or not okay that's pretty much all the terminologies that you need to know to move ahead okay I just wanted to cover these terminologies ies because this is what most people still use in the industry and you need to know what those things mean but personally I don't really think in terms of unit integration or end to end in my mental model it is only about the things that the tests are touching the things that they are uh interacting with okay the mental model is the scope okay and that is how we are going to uh approach everything for the rest of the you.

Now, we'll see why most people don't write tests in the first place. Look at the task board. When somebody moves a task, the handler asks the board to apply that change and then it publishes an event on the event bus. So, every other connected browser finds out about that particular task moving. We covered this in the uh previous video in the realtime systems how events propagate from the client to the server through the event bus to all the other clients. So in a real back end that particular handler it also has to write to the database posgress and then it might also have to call a payment provider um to check their credits or something like that and then it might also have to send an email which is again an external service.

So now the question is how do you test a functionality of a particular workflow when you have to interact with your database interact with your payment provider and uh send a actual email to your users and all these external things that you need to do all these external connections that you have to establish. How do you test that? And the answer to that is you give that particular function something which looks and acts like the external service would. And that mock that standin that is what we call as a test double. And the name test double it comes from movies and films because they use stunt doubles in movies.

So that the the stunt man the stun double they can perform risky action stunts in place of the actual actor but when we watch the movie for us it looks like the actor is performing the stunts. Now the test double is a is a name of the category. There are five different types here for the test double and they do different kinds of things. Five different things and pretty much all of us call all those five different things a mock because that's the most intuitive word for everything related to testables. The first one is called a dummy. What is a dummy? A dummy is a value that you pass into a function because the function's signature asks for it but it is never actually used.

For example, let's say your your constructor for your service in your back end. It needs a logger which comes from your main file from your main function or somewhere. But in this case while testing you do not really need logging. We do not really care about logging. So what you do you just hand a dummy function or a function which does nothing just returns or prints or something that like that it does not really have all the logging related features and you just pass it in the constructor and you say that okay this is your logger now let's move on to the next thing that is what we mean as a dummy the second one is called a stub what does a stub do basically gives you the same response does not matter how many times you ask and what you task.

For example, let's say you asked for user ID 7 and it gives you back an object which has the users details which resembles whatever uh your database has stored similar kinds of fields, right? And we use stubs so that if we have a dependency then it passes instead of blocking on getting an actual user. Okay, the stub basically means getting a pre-written answer every time. does not matter how many times you ask and what we ask. The third is called a spy which is the same as a stub. It also answers the same thing every time. But on top of that, it also records what all changes that you made to it.

So let's say you ask for user ID 7, it gives you user ID 7. But if you say change the user's name from X to Y, then it will also record that you tried to change the name from X to Y. Fourth is called a mock. A mock usually has some conditions pre-written for it. It has a few expectations set beforehand. For example, if you have already configured beforehand for the mock that a particular save it should be called exactly once with this argument and during the testing if that does not happen then it complains or it fails or it errors. That's what we mean by mock. And the fifth one is a fake. A fake is um basically an implementation which is not suited for production.

It only exists for testing and it it matches the behavior. It matches the inputs and outputs of the actual thing the actual implementation. For example, let's say you are trying to test your service and in particular a method in the service which creates a user. So in that place if you're testing something which depends on this service method what you do you pass a fake for that service method which instead of saving the user in your actual postgrid database what it will do it will create a map or a set or whatever data storage in memory in the program and it will save the user there. So from the perspective of the function that is actually calling the fake the user still got saved it can still ask for the user back but that is not the actual database it was something else a test implementation a dummy implementation.

So let's take one test of the move handler and there are two ways to write the check at the end. The first way is to ask the board what state it is right now which is like asking is the task already in the done column or did the sequence number go up by one. So that is called state verification. The second way to do the check is ask the double what exactly did it see? Was the publish function called once with an event whose type is task. That is called interaction verification and also sometimes called as behavior verification. Now here now if the code changes but the behavior does not. For example, if somebody decides that the handler should publish one combined event instead of two from the outside nothing really changed.

Every browser still sees the same board. If we look at the state verification that does not really notice this change because the state the end state is still the same but here the interaction test the behavior verification that fails you. Why? Because the number of calls changed instead of two events instead of two event publish now it is one and then and that is why it starts failing. And this kind of thing has and this also has a name which is called as change detector test. So instead of testing the final state or the result it checks the actual behavior of the program the step-by-step behavior not the end state of what it achieved and Google has a very interesting write up on this which you can take a look.

So this is the thumb rule for this section. only assert only check what the system is right now not how it went there or how it was achieved only do interaction verification or change detection when that is absolutely necessary which most of the times it is not some of the use cases that you can um have for an actual use for interaction testing is let's say you are sending emails to your users and you want to make sure that the email went exactly once So there interaction testing can be very useful but in most cases you should depend on the state instead of the behavior or interaction. That is one mistake.

The second mistake is let's say your service calls a payment provider and for obvious reasons you do not want to charge a real card when you're trying to test multiple times in a day. So what you do you write a mock for their client and you write it in a way that when charge method is called with these arguments it returns a success object with these fields something like data success message new balance. After that your test passes and you deploy but the whole thing fails in your production. Why? Because everything that came out of the test was what you already wrote, what you had already configured.

Everything the the structure of the success object, the field names, what happens on a duplicate charge, which errors they return and in which format, all of them were configured by you. So the rule to prevent that kind of mistake is do not mock what you do not own. So what you should do instead two things. First you wrap the whole third party in a small interface in a small wrapper function or wrapper structure whatever it is in your programming language which has your own applications structures your own success objects failure object error messages everything of your own. And then you use that wrapper in your main workflow in your production and also during testing.

And now that you own the interface, you are allowed to test it. That is the first thing you should do. Second thing is since you cannot really uh test the whole charging workflow which you can test once manually but after that testing it every time is not feasible. So what you do most of these payment providers they have their sandbox APIs right. So you write a few small tests and what are those tests for? They are to test your interface whether the request structure, the response structure, the success structure, the failure structure, everything that you're taking that you're wrapping with your own interface that matches with the actual product or not, the actual platform or not.

And those tests can be very fast. They do not uh deal with any kind of payment data or anything. So that is something that is more practical and feasible. Okay. Now let's look at the handler. The reason it was hard to test in the first place was that it reached out and it made its own dependencies out of thin air. It created the database client itself. It created the redisk client itself, established the connection itself. So when a function creates its own dependencies, it creates the thing that it depends on. As a result, you cannot really replace those external dependencies from outside from your calling function.

So what you can do instead of the function itself creating the dependencies you pass those dependencies and how do you pass it through parameters through arguments and that workflow that technique is what we call as dependency injection. And there are a lot of frameworks surrounding dependency injection. But pretty much all of them do the same thing. But as your program grows, as your back end grows, becomes more and more complex, you might need all these frameworks to to make the dependency injection workflow more developer friendly. So this is the mental model that you should have. Always push all your decisions and your calculations, your business logic, all the core part of it into different different functions.

All of them should be pure. Pure in the sense they should take values and they should return values. Then keep all the talking to the outside world to the external internet, external APIs and providers, databases, all these things into a very thin layer around them. The middle part is what we call as a functional core. And the layer around it is called a shell, the imperative shell. And once you structure it like this, the core part, the functional code in the middle, that is very easy to test. Why? Because it has nothing to mock. Everything is owned by us. So we can test every part of it. And the thin layer around it that is hard to test.

But now since it is a very thin logic layer, there is very little to test in the first place. So in this structure it becomes very easy or you are more likely to write tests if you structure your program your applications this way. Okay. Now that we have learned we can replace actual external dependencies with our own. We need to decide. We need to uh learn when and what can we actually replace with our own fakes with our own implementations instead of using the real one and when the replacing decision is right and when it is wrong. The first one, the most common one is database. And the instinct most people have here is since the database is slow, what we will do, we will wrap the Postgress layer, the database layer with something that executes all the operations inside memory using strrus, using maps or whatever data structure.

And our tests will not have any external dependencies nor have any external latency because of the database layer and all of them can be executed in a few seconds. So in programming language like NodeJS people often do in-memory replacements for this. In Go people write some fake repository layers for this. If you do Java or something then you have H2. But there is a problem in that because when you replace that database with something fake, something in memory, what you gain in return is speed, maintainability and utilization. But what are the trade-offs? What you give up in return is fidelity.

Because let's think about it, what kind of bugs, what kind of issues that actually live in our data layer? things like a wrong column name or a missing index that makes the query slow or a unique constraint that we forgot to check for or we forgot to add. A transaction that does not really do what it is expected to do. Something like a JSON operator that exists in Postgress but it does not exist in our fake implementation or the vice versa situation. And in every single one of those scenarios, the issues, the bugs actually live in the Postgress layer, in the database layer. They do not live in your logic.

And that is the exact layer that you replaced with your in-memory replacement. So what is the solution to this? The answer is to use a real database for your tests and you make it cheap. you make it very developer friendly so that you don't have to reach out for in-memory replacements and there are two techniques to do that. First one is a transaction per test. So what you do is at the start of the test you open a transaction you run everything inside it and at the end you roll back instead of doing the committing and now the database is completely untouched and when you start the next test it starts as a clean slate.

You never have to worry about corrupting the data and this is also very fast. But the only requirement for this is the code that you are running instead of taking a database handle it should be able to take a transaction handle otherwise this is not going to work. But this is something that we already discussed that in the dependency injection layer that your functions should be able to take the dependencies instead of creating them. Now the second one the second way of testing with real database is using a template database. If you are using something like posgress it lets you create a database as a copy of your existing database.

So you run all your migrations on the template and every test creates its own database from the template in a few milliseconds which is extremely fast. you will not really see the latency and because of this the tests can also run in parallel because they each have their own database and to actually use a real posgress instance for your test there is a famous library called test containers and it has the language SDKs in pretty much all the programming languages which makes it extremely easy to do all these operations using posgress database or any other database they have different different database port also.

What it does is it starts a real container from uh your test code and it gives you back the address the address of the container where you can reach the actual database. Let's see the demo of how what it actually looks like. If you see the top of that output test containers, it starts a actual Postgress 18 version instance and then it waits until it accepts a connection. And if you look at the two timestamps, the container starts at 58 seconds past the minute and it is ready at 5 seconds past the next one. So in total that is 7 seconds and it happens once for the whole package not once per test.

Now if you watch the nine tests under it, each one makes its own database by copying the template and then it runs extremely fast almost a few milliseconds. So the whole package the total time to test the whole package the container latency and all of it is just 10 and a half seconds out of which setting up the real database that costs you around 7 seconds for the whole setup and then for the next 3 seconds it runs all the tests which executes the actual testing logic. Okay, that is all about database. Now let's talk about network which is when your application or your code it calls someone else's API it calls some external API and as we learned uh in the past section for this scenario always wrap the external service or the external layer with your own interface your own API your own wrapper and then in your test layer you try to mock that or fake that but there is a second technique here which is useful when you actually want to test the HTTP call.

You want to test your own client code. You also want to interact with your headers, right? You want to set them. You want to delete them. All those operations, you want to be the one who calls the server through the HTTP instead of just calling the functions from inside. If you are using Go, it gives you this library called REST. But other languages in NodeJS, Python, they also have their own uh very famous maintained libraries for doing the same thing. What it does, it starts a actual HTTP server on an actual port in your test and then you point your client towards that. So when the request happens, it actually goes through TCP protocol like it does in production.

If you're using NodeJS, there is a famous library called MSW. it intercepts at the network layer instead of you having to replace your client. And with this setup, you can test your own handlers from the other side. Now the third thing which is time and this one causes a lot of issues because let's see a scenario. You call some function and something happens in the background and the test needs to wait for it because it is async. So in that scenario what we do we write some kind of sleep function which is available in every programming language and we sleep for let's say 100 milliseconds and then we try to check the result instead of checking instantly.

So sometimes it passes on our laptops in our local development environments and it also passes during any kind of review then when you push it to our CI continuous integration environment through GitHub action or something there it fails. Why? because the thing that you're trying to um fake which is waiting a few seconds so that the background process can finish in that machine since there is a lot of load and the machine was busy instead of taking 100 milliseconds it took 110 milliseconds. So here the issue was not because you were wrong in your logic. The problem is about the clock is about the time.

And of course the obvious fix and a naive fix is just increasing the time. But that does not really fix it permanently. If the server is busier in some other day then you'll again have to increase the time and then your tests are again slower. So that is not the right fix. There are two different solutions for this. The first one is to stop waiting for a duration and instead wait for a condition which means instead of waiting a abstract amount of time you wait for some kind of signal. So if you're using go you wait for an event that comes from a channel. If you are using NodeJS then you wait for a particular call back.

So now you are not trying to guess the time but you are waiting for the actual completion event. And the second solution is you pass the clock itself. A lot of languages, a lot of libraries offer this uh feature where you can pass an instance of a clock which means instead of checking actual time, it will depend on the past implementation of the clock of the time to check its logic and then you have complete control over the times and the timeouts. One last concept before we move on to the next thing is this whole idea that we discussed where a test which brings its own surrounding its own environment with it and that does not depend on a database that somebody set up by hand or a service that has to be running.

This environment this kind of testing is called as hermitic testing. Hermitic basically means sealed, freeze, sealed or frozen. Everything that a test needs, it starts and once it is done, it stops everything that it has started. And you need that property if you want to run your tests in any machine, whether it is your local or an SSH server or your GitHub actions environment anywhere. And this library that we saw test containers it is a way of being hermetic about a database and the injected clock technique that we saw that is being hermetic about time. So that is another technical terminology that you'll hear when talking about testing.

Okay. Now that we can write a test that survives any kind of refactoring that your team might do. Let's talk about a practice which is I have already mentioned called testdriven development also known as TDD which is a practice of writing the test writing the specification before the code itself. It is a way of working it is a workflow instead of a kind of test because it does not really do any special kind of its own tests. It still has the typical unit test, integration test and E2E tests. It's just the workflow. It's just the way of working. The only difference is when you are writing them typically instead of writing after our programming logic, after our application or back end logic, you write them beforehand.

So the whole loop of test event development has three stages or three colors. Red is you write a test for something that does not exist yet and you run it and it should fail. Okay, that is the first step. Then is green. You write the smallest amount of code that makes it pass. And here you are expected or you are allowed to write bad code or a code that does not adhere to every single best practice. But the only purpose is it should pass the test. And then in the last stage is refactor. Now that the test is passing, the test is green. You do a little bit of clean up, a little bit of restructuring.

And after that you run the test again. If it is still passing, then you have your clean code with the functionality already tested. So here the first step this phase is for the design because if you are writing the test beforehand before you write your application logic it forces you to imagine your function to use your function before it exists. So the way you would imagine it as its API how you are going to use it not how it is going to work and doing that you will find out what it should look like from the outside from the calling function instead of what it should look like from implementation side and while doing so you will realize it quickly if the test is becoming hard to write or if the test seems ugly or awkward awkward then you will realize that the interface the API that you're imagining your function will have there is a problem with it there is an issue with it so you find out instantly in a few seconds or a few minutes instead of realizing in the end and then you have to restructure the whole function expose new parameters and all so that is one advantage of writing the test beforehand which you do in the first step in the first stage of testdriven development then the second step which is for the proof Because if you imagine it the other way around, let's say you write some code and then as per the code's workings, you write a test and since you wrote the code yourself, you already know the logic and the thing that you're testing, the logic that you're testing is most probably going to pass on your first attempt.

Why? You're the one who wrote the application. You do not really learn anything. There is no way to find out whether the test actually passed because the code is right or it passed because it does not really check what it is supposed to check. So first making it red and then turning it green has a learning experience which comes with testdriven development which does not come when you write the application code first. Kent Beck, he has a whole list of these properties that he expects from a test and this is one of them. The third step that is something that most people skip in testdriven development which is once your test is green you are expected to and you must come back to the code and you have to refactor it because the reason you are allowed to write bad code or not care about best practices in second step so that you can come back to it in step three you cannot skip this part.

Okay. Now let's see what actually testdriven development looks like. We have a requirement which is a task cannot go straight from to-do to done. It has to go through doing first or in progress first. Otherwise there is no point in keeping this whole status of doing. And we have no code to enforce this yet. Okay. So let's first start with the test. Take a task that is sitting or that is in to-do. We move it to done and then we expect an error. We run it and it fails. Why does it fail? Because it was expecting an error but instead it got nil. Why? Because the move was successful. We have not really added that condition yet.

That is our phase one which is red. Now let's do the second phase which is the smallest amount of change that makes the test pass. For that we add one if statement inside the move and then we run it again. This time it is green. Okay, step one and step two is done. Now comes step three because the if statement that is a business rule and it is sitting in the middle of a method that also locks a mutx. It increases a sequence number and it also trims a history buffer. So what we do? We take it out into a function called can move. And that function takes a column that it is currently in and the column it is going to.

Nothing else. That's it. And then we run the whole package again and it is still green. We changed the code. We changed we improved the code by refactoring it after step two. And the test is still passing. That is what completes the whole test-driven development loop. There is a lot of discourse. There's a lot of argument in the ecosystem regarding test-driven development. In 2023, the author of testdriven development, Kent Beck, he wrote this short article about what the loop actually means because of all these arguments. One more resource which changes how you think about um test-driven development is this talk by Ian Cooper which is called TDD.

Where did it all go wrong? And his argument here is that people took the word unit which you already mentioned unit test to mean a class and because of that they wrote one test class per production class and they mocked every other thing every other external integration and because of that every single refactor used to break hundreds of tests and that caused a lot of people to give up on testdriven development. So his claim here was that the unit that is not about the class. The unit is the behavior because the thing that you test against is the public interface of a module. And if you do that you can if you want rearrange the whole class, rearrange the whole module without changing the interface and the tests will remain the same.

They will still pass. And if you remember the first part of this video where uh we mentioned that always try to test the state of the system instead of the behavior of the system. And this is what he means because testing the class it means you are trying to test the behavior of the module the implementation of the module instead of the state that it produced. In 2014, DHH, the author of the Ruby on Rails framework, who is also very famous these days for the Linux framework, he wrote this very controversial or very famous article called TDD is dead, long live testing. And his argument here was that he is against this test first culture and against this whole idea of designing your systems with mockability in mind and you should directly go for actual testing and all these things which led to a lot of discussions between him Kent Beck who wrote testdriven development and Martin Fowler who's very famous for system designs and all stuff.

So if you are curious about the whole ecosystem surrounding test-driven development then you can read up on those pieces. What do we do here? What is our opinion on testdriven development? Now, in my opinion, from the way I work, testdriven development is a very good tool when you know what the answer is, what it should look like, but you have not yet figured out how it should work or how to build it. For example, uh let's say you are working on a parser where it is very easy to figure out what view we want in return. Let's say you are building a JSON parser. You pass it a string and you expect a JSON object, right?

Very easy to figure out the inputs and outputs. So in that scenario, you can go with test-driven development because the figuring out your expectation is the easiest part compared to the actual implementation. Similarly, um a pricing rule in your business logic or a state machine, all these things here, test-driven development really works and it keeps you focused on what you actually want out of your logic. But in some scenarios, it is a wrong tool when you do not know what the answer is going to look like yet. You are still figuring out, you are still exploring the ecosystem, the whole space and you are trying multiple approaches to reach a best approach and you are going to depend on what is it that feels right before you decide what to stick with.

In those scenarios, which in my opinion is the case in a lot of situations when you are building features for your back end because you have to think about a lot of trade-offs, a lot of approaches, a lot of system design. So testdriven development does not really work in a lot of situations. It has its own uses but not a lot. Let's talk about one more concept called as flaky tests. What is a flaky test that I've already mentioned? Flaky test is a test that passes and fails on the same code which is very unreliable or unexpected without you changing anything any code but every time you run it you get a very different answer and a flaky test is more harmful for you as compared to you not having the test in the first place.

Why? Because if a test is missing, if you do not have a test for it, you know that there is a hole. There is something missing here. But if you have a flaky test, that changes how you and your team works because let's say um you deploy your code and it fails. The test fails in your CI continuous integration. So one of your teammates or you say that okay that test is a little flaky just run it again. It should pass. Then as expected it passes and everyone moves on which builds this behavior and because of that when someday an actual test fails which is not flaky but an actual failure in your system and the first thing someone does is they just run it again instead of instantly going for the fix.

So where exactly does this flakiness property come from? There is a study from 2014 by this person called Luo and his colleagues. What they did was they took 201 real fixes for flaky tests from different open-source projects and they try to sort them by cause and on the top there are three causes. Async await. Async await is a test that starts something and then it waits for it the wrong way. Then concurrency which means two things inside the test who are trying to race each other. And then there is test order dependency where a test only passes because another test somewhere it ran first and that left a state behind.

Those three reasons are the causes for majority of flaky tests. And interestingly we have already dealt with the first two. The async aait is the sleep problem that we saw. And what was the fix for that which was instead of trying to fake a sleep you wait for a signal you wait for a condition not the time itself. Then for concurrency we have u things like sync test. if you are using Go and other libraries and frameworks have their own primitives for synchronous testing. But the third one is new. So let's discuss that. What is test order dependency? It means that test B only passes when it runs after test A.

But if it is tried to run alone, stand alone, then it will fail. Why? usually because uh test A created some kind of database row or it set some kind of global value or it created a file that test B depends on all of these scenarios it can be anything. Why is it bad? Because if your test suite depends on a particular order every time your test suite runs in the same order it always passes. But the moment someone adds a new test in the middle or they turn on parallelism to make the test faster then it starts failing. What can you do about it? You have three options. First, you can delete it which is fine because the test was not really helping in a lot of ways and it was creating more issues for the future.

And deleting tests is a practice that pays more often than you might realize. Or second option is you can quarantine it which means that moving it out of the blocking suite into a separate one which still runs in the same way but this time you add um the owner's name the developer's name on it and a date with a reason of why this is the way so that uh you can catch it. You can document that this has to be a flaky test for this reason. Okay? Or you can add a retry which is not a good practice in tests. So in my opinion the best solution is to quarantine it with a name and a date and a reason so that the next time some issue happens the test starts failing we know that who wrote that why they wrote it and we can just communicate with them to find the right solution.

Okay, now that we have a test suite, how do we measure it? What are the metrics that we look for to find whether the test suite is good enough or not? The first number pretty much all of us reach out for is code coverage. What does it mean? Code coverage is basically the percentage of your code that was executed while your tests were running. So when you ask for the code coverage from your testing tool, the compiler or the tool it rewrites your code before building. So what it does is it breaks each function into basic blocks with no branches in them and it inserts a counter at the top of each block.

Then it runs your tests and in the end it reads the counters. So if a block has zero count it means that that function never ran. The problem with coverage as a only metric is it only records whether the function ran or not. It does not record anything about uh whether the function actually did some work or it actually checked something. So a test that calls every single function in your program in your application and it does not really assert anything. It does not really check anything that will anyway give you 100% coverage. And the second thing to know is coverage is not just one number. The most common ones are line coverage which checks whether a particular line ran or not.

There is branch coverage which checks whether each direction of each decision or each condition was taken or not. And depending on on what you are checking, they give different answers for the same code. If you take a function with one if statement and no else. If a test goes through with the condition true, it will be able to execute every single line of that function. So the line coverage will say 100%. But if the condition is false then that direction will never be taken. So the branch coverage will tell you 50. So what does the coverage percentage actually mean about the quality of your tests?

And the number everyone agrees on is 80%. In 2014, two people called Laura and Reed Holmes, they published a study at the International Conference on Software Engineering. What they did was they took large real world Java programs and they generated thousands of test suites of different different sizes. Then they measured each suit's coverage and they checked how many actual faults each one actually caught. And they use this phrase called controlling for sweet size. If you just try to correlate coverage against the number of bugs found then you will get a very strong relationship. But that is mostly because the bigger test suites they cover more of the code and they will be able to find more.

So what they did was they held the size as fixed and then they checked whether the higher coverage suite of two suites of the same size found more bugs or not. And the conclusion here was there is a low to moderate correlation between coverage and effectiveness once the size is controlled and the strength of it. It depends between various systems. So if you are using a fixed coverage value as a quality target for your test suite then that is not going to give you an effective test suite. So what should we do? Should we not respect coverage at all? Three things that we can take from that article.

First is the numbers that they published. 60% acceptable, 75 commendable and 90 exemplary. And as per the article, there is no ideal number that applies to every single product. Second thing is what is not covered by your test is more meaningful than what is covered. Which means that the report that you will get from your coverage, the true value of that report is in the red lines, not the percentage there. And the third one was the warning that the article gave us which is if you make coverage having 100% coverage as mandatory what most engineers people will do is they will write very low value tests just to hit the coverage number and they will stop improving since they have already reached the coverage number.

The article also gives us a direction to something more useful which is mutation testing. What is mutation testing? Mutation testing is a way of testing your tests. So what it does is the tool the mutation testing tool it tries to damage your code on purpose and it checks whether the tests that you have written they notice the damage or not and the share of your damage your tests your test suite is able to catch that is the score. So the tool takes your source code, your application code and it makes a couple of small changes everywhere on purpose. For example, uh if in a condition you have written greater than they will make that into greater than or equal or they will change a plus sign to a minus sign or they will replace a return value with zero.

Each of those damaged versions is what we call as a mutant. then it will run your test suite against each one of those mutants. If a test fails, then the mutant is killed and which is good because your tests were able to notice the damage. But if the test still passes, then that mutant is survived. that mutant is kept which means that you have a testing code that is not really doing the checking that it is expected to do and in the end the score that you will get is the percentage of the mutants your tests were able to kill. So this is a module it takes a list and returns one page for it and this is the test someone wrote for it.

If you read the assertions, they are like it returns something. It returns an array. It should not throw the length should be greater than zero. And now if you look at the coverage here which is a 100% of the lines because every line of that function runs during the tests. So if we go by just coverage then this job is finished. We are done with our testing. Now if we run this program which is called striker for mutation testing on the same code what it does it makes 19 small changes to the source one at a time and it runs the whole test suite against each one of those mutants and what we get as a score is 52 63%.

which means that nine out of those 19 they were able to survive our tests did not catch those damages. So if you look at the second survivor the second mutant that was able to survive the in the loop it was I less than or equal to offset plus limit. What striker did it changed it to I less than offset plus limit. So now it changes how many items the function returns but even after that every test still was able to pass. So that is the whole value of mutation testing to test our tests. Let's move on from uh measuring our test to measuring our code to code quality. The first metric that we look at here is called cyclometric complexity.

And that is a very complex sounding word which just means that cyclometric complexity which means that it is a count of the number of independent parts through a piece of code. And this technique comes from a paper written by someone called Thomas in 1976. And interestingly this is a technique uh to get the count you can actually do the computing by hand. What you do is you draw the function as a graph and every straight run of statements is a node and every jump between them is an edge. So the formula is edges minus nodes plus two. And there is a shortcut to that which is the answer is always 1 plus the number of decisions points.

one for the function itself plus one for every if condition, every loop, every case and every and or or in a condition. So if a function has no branches in it, then the count is one. If it has a if condition then the count is two. Now the opinion of most people here is the complexity number for most functions should be 10 and this is what the author himself wrote in the original paper about uh that number. As per him this is reasonable but not magical and he found this one case where the limit seemed unreasonable which is a function that is a big switch statement over many independent cases. So if you have a 20 case switch statement then the complexity will become 21 right as per the formula which is still very easy to read but it fails the general advice which is the count should be under 10.

Now our second metric because the first metric and the general advice about that metric which is it should be under 10 that clearly failed under our uh switch statement which was very easy to read which was very good code but it still failed the formula. That's why we have a second metric which is cognitive complexity. Now this metric is not very mechanical. It is not a counter. It is there to measure how hard the code is to understand as compared to how many paths it is taking. And this technique was invented by sonar source. So if we look from the perspective of this technique cognitive complexity, it does not really add anything for the switch statement itself because if you look at a flat switch statement, it is very easy to read.

It does not have any cognitive load. But every time you add any kind of nesting in there then it grows because if you see an if statement inside a loop which is again inside an if statement that becomes very hard to read and that whole situation that whole path becomes very hard to hold in your head as compared to three different if statements in a row. So if you look at both the techniques, a long flat switch statement, it scores very high on cyclomatic complexity but it scores very low on cognitive complexity. Similarly, if we look at nesting, three levels of nesting, it scores a moderate level on cyclometric complexity, but it scores very high on cognitive complexity.

So here we have a package and we have two tools. If we go by cyclometric complexity, the one that counts parts, the hubs run method, it scores eight and it comes in the third position behind the two stream handlers. But if you go by cognitive complexity, which increases as you uh increase the amount of nesting, the same method will score 19. And now it moves to the top of the list. same as the websocket handler. But instead of only going by the numbers that these tools are giving us, if you look at the code itself, there are eight ways you can enter that function. And that itself is very hard to read because how deeply it is nested.

All in all, what we get from this discussion about the code complexity using these two different tools is do not trust either tool blindly. Do not take the number as a hard rule. Instead, look at what exactly those numbers are pointing you towards and then as a team take a decision on what is the right thing to do. So take those tools as a flagging tool instead of the base of your decision. Okay. Now that we have talked about tests, we have talked about code complexity. Let's look at a few tools that that allow us to find problems for us without even running anything. And these tools are probably uh the ones you have already heard a lot of.

First one is a llinter. What is a llinter? A llinter is a tool that goes through your source code, your application code, and every time it finds a particular pattern that it thinks is a wrong pattern, then it will complain you. It will show you some kind of error. And to be clear here, it does not really check syntax. That is the job of the compiler, of your language runtime. And it also does not check the correctness of your logic, the correctness of your program, which there is no universal or reliable way to know without an actual output, an actual expectation. It only checks for patterns.

For example, if you have a comparison, you are doing some kind of if condition, but the condition that you're checking, the variable that is inside the condition that is always true, which is assigned somewhere upstream, then it will show you the error. The l will show you an error that this condition is always true. Or if uh you got an error from some kind of function call, but you did not check that error, then it will warn you that um you should check that error properly. Right? That's what a llinter means. It checks for patterns. Then we have a formatter which is the most universally known tool.

It takes your source code and it writes it in a uh standard layout typically which is decided by your whole team and that's it. It does not really change your program in any way. The whole job is just to make it more readable, make it less complex looking. Then we have type checkers. If you are using languages like Go, TypeScript, uh Rust, they all have their own type checkers typically integrated into your editor something like VS Code which warn you when you are using the types wrong. If you are using a variable and you are doing some kind of arithmetic operations on that but that variable somewhere upstream is declared as string then it will show you the error that you are trying to do arithmetic operations on top of a string but that is the job of a type checker.

There is one more thing which is called static analysis which is like a more advanced version of a llinter. So static analysis means checking code without running it. And the way they work is they start at one point and they try to follow the same value through your program and check for a few conditions. That is pretty much everything uh you can cover on a basic level about testing and code quality. Code quality is a very subjective uh matter. That is why I could not find any objective metrics, objective tools for that. So if you are in any ecosystem like if you are in a typescript ecosystem then eslint rules which is a llinter they help you to some extent to improve your code quality.

Similarly, if you are using Go, then it also has its own lint. But other than that there is no particular tool or deterministic way to decide whether a code is of high quality or

Seek backward 5s ←
Longer seek (10s) Shift ←
Play Space Seek forward 5s →
Longer seek (10s) Shift →
0:00 Auto-scroll A Copy link at current time S

26. Testing for Backend Engineers: mocks, TDD and coverage