# 25. Real-Time Backends

**Channel:** Sriniously
**Source:** https://www.youtube.com/watch?v=wUQryt697cs
**Transcript page:** https://www.withtranscript.ai/video/wUQryt697cs

## Chapters

- 0:00 — Why Real-Time Features Need Push
- 9:47 — Long Polling and Server-Sent Events
- 16:12 — WebSocket Basics and Frame Structure
- 27:06 — Connection Limits on a Single Machine
- 37:55 — Scaling Realtime Across Instances
- 46:03 — Fan-Out, Reconnect Storms, Summary

## Transcript

**[0:00]:** At this point, you have built a fairly complete back end authentication, database handler, services, repositories, caching, background jobs, and in the last two videos also object storage. And through all of it, every single thing our back end has ever done has followed exactly one shape, which is the browser asks a question and the server answers it. Now, let's talk about another requirement. Since we are building from a product angle and it's a fairly ordinary one. We have a task management application. Let's say you build that and two people are looking at the same board and one of them drags a task from to-do to in progress and the other person sitting somewhere else in the world should see the task move live. not on the page refresh, not on navigation but live. And the moment you start thinking about how to implement this feature, I mean without having the knowledge of how to implement this the right way, all the tools that are available to us, all the technologies, but if you

**[1:16]:** just think about it without any prior knowledge, then one limitation that you'll quickly realize is this one. So far whatever we have been doing it's always have been the case that the client initiates the conversation. The client starts the communication. The server cannot start the communication. It has no mechanism to speak first. It can only reply to something the browser has already asked for or it can hold a communication for a long time but the starting itself it cannot do and I said this in video five in the H video that every time the client initiates and the server responds that is the model. So at any point if the server has some news some information which the browser has not explicitly asked for then there is no way to let the information travel from server to our browser or our client. So as usual let's do the same exercise.

**[2:23]:** What is the most intuitive way to go about the solution? So if the server cannot tell us when something has changed or some new information then the only way to implement that is through the client and how which is constantly asking for some information in a loop. So every 3 seconds or so the browser the client asks the server has anything changed on this particular module or let's say in this case the example was of this board this task management board. So every 3 seconds the browser will ask the server has something changed in this board this particular board that I'm looking at with the ID 412 or something in the last 3 seconds and depending on the data volume most of the time the answer will be no occasionally in this case when the user has actually moved a task from one board to another then the answer will be yes something has changed then the server will send

**[3:28]:** either the particular change that has happened or it will just say that okay some change has happened please refetch the whole board data and rerender the whole screen. Here we have a polling demo where we have two browsers side by side and both of them are doing the polling operation that I mentioned. they are pulling every 3 seconds and as you can see from these two counters the request counter and the change counter which most of the time stays same except when let's say AAS browser it moved one task and the moment it does in the next counter of Leo's browser the unchanged counter does not increase it stays the same because some change happened so that request did not go to waste But other times every 3 seconds the poll that is happening most of the times the request counter and the unchanged counter both go up which means that you are wasting most of your requests where no change has happened.

**[4:33]:** And this whole polling setup works pinging your server every 3 seconds to ask for information. the solution works and if you have some kind of internal tooling with a few users then pulling every few seconds makes sense and is a perfectly reasonable decision for that architecture. But here since we are talking about a userfacing product front and we want to do the right solution with scales that is why we are not going to stop here. we will keep pushing the boundary of what is going to break at what level and what is the right thing to do. So the first problem that you see here that you can feel is the delay. If you poll every 3 seconds, then on average a change, whatever change happened, whether a task moved from one board to another or a task got updated or deleted, something happened, then that change waits at least 1 and

**[5:33]:** a half seconds on average before the other user or any other user sees it live on their screen. Pure arithmetic. So the average delay is always going to be half of your interval, whatever interval that you're choosing. In this case it is 3 seconds. So the average delay will be around 1.5 seconds. So what is the solution to that? Pull faster. Do not keep a delay which is visible to the user. Make it 1 second. So let's see what is going to break when we make the polling interval to 1 second. Let's say your product is doing reasonably well in the market and you have around 10,000 people with the board open at one point. So 10,000 browsers open at once. Each one is asking or each one is sending a request to your server every 3 seconds. That is 3,33 requests every second reaching your back end forever.

**[6:37]:** And each one of those requests they are proper HTTP request which means they go through your load balancer then through your authentication middleware. So in your authentication middleware you are also validating the token and then it goes through your logging middleware. So you're also writing a log line to somewhere. Then it goes to into a service layer which does queries to the database and then in the end through serialization comes back as a output as a JSON which means 3,000 database queries per second and pretty much all of them return the same answer which is nothing has changed every time at least most of the time which means that if you drop the interval to reduce the userfacing delay to make the whole interface is snappier. That in turn makes that 10,000 requests per second. Which means you made the whole user experience three times better.

**[7:37]:** But the cost, the compute or whatever resource that you are paying for it, it is now your bill that you're going to pay to your cloud infrastructure at the end of the month. That is now three times more or three times worse. Now the cost of polling scales with the number of users. It does not scale with the number of events. What does that mean? Let's say there is a board where nothing happens all day. No event is happening on that board. But still the the amount of resource that board is costing you, where nothing is happening, no event is happening, no change is happening. It is the same as a board where some event some change is happening every second which means that you're not actually paying with your resources for the actual change. You are paying for the whole communication for the requests itself which obviously feels wrong because the the thing that is bringing us value is the events itself the changes itself not the requests itself.

**[8:43]:** They are just a medium and a very wrong medium in this case. Also, if your users are on a mobile phone, then every single poll that happens through a phone, it wakes something called as the radio, which is one of the most expensive thing a phone does. So because of this your polling loop your users battery and their data is both draining at a faster pace than usual for no changes at all on the board that they are looking at. So far we have realized what is the right solution for this shift which is the server should speak first which will eliminate all these problems that we are seeing here. And the second one is we should only pay with our resources for the actual changes for the actual events and not per the user the number of browsers or the number of requests. Those are just our mediums.

**[9:39]:** They are not what is bringing us value. So that is why the rest of this video is going to be about that.

**[9:47]:** How to achieve that result, what it looks like on the network level itself, how much does it cost on resources and any other topics around. The first idea people had was very simple which is if the server can only answer a question which was what our primary limitation with the HTTP protocol was which is a client has to initiate the communication and server can only reply back. So since that is our constraint then the solution people came up to solve this whole real-time thing is ask a question and let the server take its time before answering. That's the whole shape of the very simple initial solution and in technical term we call it as long polling where what happens the browser sends a request and the server does not reply immediately. It holds that request open the request open by not sending a response back immediately sometimes for a minute just waiting.

**[10:53]:** The moment something happens, some kind of change happens, it writes the response for that change with some information and sends the response back and that request closes. The browser gets that immediately and then it starts another request, another cycle of long polling. And if you see it is actually written down in RFC 6202 along with all the things that is wrong with the solution. But to summarize the the the reason this solution is wrong is the gap that happens there after the server answers the browser has to process the whole response then open a fresh request to the server and during that whole window the server has no connection to you and according to this RFC the average latency is about one network transit but if you look at the worst case scenario it is more than three because of that gap thing. So the long polling solution it works and for a lot of years people used it to solve the real-time problem but eventually we had to move on from it to the next idea and what that next idea is

**[12:10]:** this is a little cleverer than the previous one which is this whole thing happens we are doing this long polling but what if the response just never ends right clients starts a request the browser starts a request and waits. That's it. Whenever something happens, the server keeps sending it, but it does not close that request. It just holds that request open and keeps sending data through that medium. That is exactly what we call as server send events. The browser makes a very normal ordinary looking HTTP get request. The server replies with the status 200 with an extra header of content type which has text /event dashream. Then it simply just does not close that connection that the client has opened. Every time it wants to send some information, it just writes a little bit of text in the same connection, sends it back to the client, and that connection is kept open forever.

**[13:17]:** And by forever we mean as long as the browser is still there, the tab is still there. The moment the tab closes or the browser closes or some network connection happens from the client side, then of course the request will be closed. And depending on the retry mechanism or reload mechanism on the client side, the request will be opened again. But if the client is open, if the client is working, if our platform is open on the browser side, then there will always be a connection to the server for this whole communication thing. That is the whole idea. And the interesting part is it is not a different protocol. It is still on top of HTTP. It still uses the HTTP header and the response body for all kinds of communication. And here as you can see we are starting a s request since we are using call to act as a client and communicating with this server that is running on our local and we are starting an http request

**[14:17]:** and we are predeclaring that uh we are accepting the text event stream data then the moment we start with the / ai/ event where whatever endpoint the server is listening to start this connection then we start getting all these events. that you can see each event here has an ID has the event name itself which is like task move task deleted task created etc and then the data itself so that the client the front end can take that data and directly render it on the screen by inserting a new card or something and the reason we are adding this ID here so that if at some point the connection got closed or something or the tab was refreshed or closed something happened then the client can save this last ID and it can send it back to the server that okay this is the ID of the last event that I received so send me all the

**[15:13]:** events since then instantly so that it can render all the changes on the screen there won't be any delay while it tries to catch up to the server there is also one more field called retry and that is used by the server so that it can tell the browser how long to wait before doing the whole reconnection thing and this is the whole specification of the format of server send events just four fields that we are seeing here now because the server send events this whole protocol is in the edge standard the browser itself implements the reconnection for you if the connection drops the browser waits and reconnects on its own you don't really have to write an extra code for that and when it reconnects it sends a header called last event ID that I just mentioned which carries the ID of the last event it received.

**[16:09]:** The browser itself your code does not have to do that.

**[16:12]:** Now people who are familiar with all the options that are available for realtime systems which is the most famous one that is websockets because of that some people think that server send events is like an inferior version or some kind of toy implementation in front of the actual architecture which is websockets which is not true. If you look at Uber's push platform which tells a driver about a trip that itself was built on server sent events. Similarly, if you look at LinkedIn's instant messaging feature which also has these typing indicators and read receipts, they also used server sent events. And the reason is since it is all they did not have to think about compatibility between different different devices and environments. It just worked everywhere. And in this era of agentic coding, all of us pretty much use different different kinds of CLI tools for interacting with our agents.

**[17:13]:** And every single large language model, LLM model that we use, its API streams all its tokens to your client either to your CLI, to your browser, whatever using server sent events. So when you're watching that uh text being streamed one token at a time, one word at a time, that is exactly that data field that we just saw in the demo that is being rendered. So service and events is not a inferior implementation. It has its uses. just have to see where exactly server send events fits in your architecture and where do you need something more powerful which we'll cover soon. So server send events gives us the whole server speaking first capability over plain HTTP and on top of that we also have browsers uh natively supporting reconnection and resuming any connection related errors. So what is the limitation here? Why are we still trying to look for a better solution?

**[18:23]:** Which is the name itself? If you look at the name server send events, what that means is it is one direction only. The server can talk all day like constantly. It can keep sending all the information it wants to browser to our client. But if the browser wants to communicate something to the server using the same connection or using the same real-time connection, it cannot. For example, in a task management app, if the user drags a task and it wants to let the server know about this information, that is going to be a separate post request that cannot share the same connection. And for a lot of features, for a lot of products, that is a completely fine architecture. But in this case, since we are trying to learn about all the options and what is the best option for our this feature, which is the live board feature, we want both directions on one connection.

**[19:22]:** And that is why let's look at the technology that offers us that exact feature, which is two-way communication on the same connection. So here we are using a tool called websocket to experiment with our whole websocket architecture using a server which is running on the port 7821 and it is exposing the whole websocket connection feature on the /echo endpoint and using this tool we are trying to initiate a connection to the same endpoint. The moment we start, we see a couple of info logs here which says a few things and then it says connected to this particular server at port 7821 through TCP. Right? Then if you focus on the line which says connected to websocket and the response headers that we received is upgrade websocket connection upgrade and there is a header called sec websocket accept. We'll come back to what that is. Once that is done, this whole thing is called the websocket handshake.

**[20:30]:** Once this is established, now we can start communicating with the server. Here we are seeing that a okay a task moved on board 412 and the server instantly replies what we sent to the server and then in the next line the server also sends some information back which we are rendering here. Then we send some information back and so on and so forth. the two-way connection works in the whole websocket architecture. So the thing to focus here some of the interesting things is websocket connection itself it does not start as a websocket. It starts as a normal HTTP get request and that is pretty powerful because it travels through everything that already understands HTTP which means every single device that we can imagine pretty much all of them speak HTTP by default. So the browser sends a get request with a header saying upgrade websocket and a header called sec websocket key with a 16 random bytes with base 64 encoded.

**[21:37]:** Then the server takes that key. It adds a fixed string to the end of that. Takes the S1 hash of the whole result. Then it B 64 encodes that and sends it back in a header called SEC websocket accept along with the whole status code 101 which means switching protocols from HTTP to websocket and the fixed string that I mentioned that is a constant and it is written in the specification and it sounds like it is for security but it is not because anyone can compute it. Whole job of this key is to prove that the thing that is answering it understood that you asked for a websocket and it is not just a cache that is replaying an old 200 response. So once that exchange is done the part is over the connection still stays open but it is not HTTP anymore. Everything changes.

**[22:42]:** No headers, no methods, no status codes. It is now a completely new birectional stream of data and both sides can write whenever they want. And this birectional stream of data it is also called birectional stream of frames. What is a frame? The RFC says it is a bit diagram where the first bit is FIN or FIN which says that it is the last piece of a message. Then comes three reserved bits. After that four bits of op code which says what kind of frame it is. One means text, two means binary, 8 means close and nine means ping, 10 means pong and zero means this is just a continuation of the previous message. Then one bit called mask. After that seven bits of payload length, which is a very efficient design. If your payload is under 126 bytes, the length is just that number and the whole header just cost you two bytes.

**[23:55]:** If it is bigger, you write 126 there and the real length in two more bytes. But if it is like huge, you write 127 followed by 8 bytes. So a small message just has a two byte header. Compared to our earlier long polling approach where every polling request had hundreds of bytes of headers, we can already see the kind of resource saving that is happening here. Now let's talk about that masked bit. Every single frame the browser sends to the server is what we call as masked. And what that masked means is each bite of the payload is x odd with one bite of a four byte key. And it cycles through the key as you go. And this key is sent in every single frame four bytes in front of the payload. And it is sent in clear text. It is not an encryption.

**[24:54]:** Anyone reading the whole network can read it with just one line of code. So why does it exist in the first place? The reason is while the protocol was being written, someone ran an experiment against real caching proxies that were deployed over the internet and then poisoned them. What that means is the whole attack looks something like a a malicious page would open a connection to a server that the attacker controls then does something that looks like an HTTP upgrade and then sends bytes in that connection which are carefully written to look like a fresh get request for some kind of popular file. let's say some kind of tracking script like Google Analytics or something like that which thousands of sites load then a proxy is in the middle which does not really understand this whole protocol. It sees that is just a request and a response and it caches that response and the moment it does that everyone behind that proxy gets the attacker's file when they ask for the real one.

**[26:12]:** That's the whole framing of this attack. And to counter that, what they came up was this masking. So if the browser exors everything with a fresh random key which the pages JavaScript cannot predict, then the attacker cannot choose the bytes that appear on the network. They can choose what they send, but they cannot choose what it looks like after the masking. And because of that they cannot do the fake HTTP request in the stream. And this is why the key should be and must be random. And as per the specification it says the FG for each frame. So masking in this case is not there to protect your data but it is to protect the whole arrangement between you and your user the whole communication.

**[27:06]:** Now let's talk about two more frame types because they are important. Op code 9 which we said is for ping and op code 10 which is for pong. Any of the sides either client or the server they can send a ping and the other one must send a pong back which means just a response back. That is what we call as a heartbeat. And we need this whole thing. Why? Because of TCP. If a connection dies or gets interrupted because either the user walked into a lift or their phone switched from Wi-Fi to mobile data. If something like that happens, then you do not really get anything. They do not send you anything. There is no error. There is no close, nothing like that. and your server is just there holding a socket and it it believes that the client is still there and it will keep holding on to it for a very long time because from the server's point of view from

**[28:05]:** the sockets's point of view an idle connection and a dead connection looks exactly the same because nothing is happening there. So the only way to tell them apart is to send something and see if something comes back to see if the other side is still there or not. And that is exactly why we have this ping and pong setup. In actual websocket based setups, we have these constraint where if some side sends a ping and the other side does not send a pong within 30 seconds, the connection is considered as dead and the other side can close it and it can free whatever resource it was holding. Right? So now we have persistent connections. Both directions can talk. Now, now the question is how many of these connections a particular machine can hold. So this is a plain uh Linux machine. It has 16 cores of CPU and 61 GB of free memory.

**[29:07]:** And this is a websocket server written with a few lines of go which does nothing apart from accept connections and hold on to them. There is no application logic because what we want to measure is how many connection we can hold. So let's check what are our limits exactly. If you know about file descriptors in Unix machines like Linux, Mac etc. Everything is a file which means a socket a network socket itself is a file and to interact with that you have a file descriptor. So every connection is a file as far as the kernel is concerned. There is some abstractions on top of it. So the number of connections you can hold is directly related to the number of files you can have open in your operating system. So if we run this command called ulimit- n it says that 1,024 that's how many files you can keep open.

**[30:11]:** I'm starting this server which has this limit configured and then we'll open 2,00 connections at it and we will see where does it exactly stop and it stops at 1,17 which is exactly right because we have things like a bunch of standard input standard output standard error the listening socket itself and the eole instance that the runtime is using to watch all of And there are like a seven file descriptors of overhead. And after all that we have 1,7 connections. Now let's look at what the client saw. It did not get connection refused. No kind of error at all. It just got a timeout and it was just waiting there for a handshake for a reconnection. But if you look at the server, it shows the exact reason which is too many files open and it keeps showing that reason with some kind of exponential backoff.

**[31:16]:** So the server is not down, it is up and it is very much healthy. It is still answering any kind of health check APIs we have configured but it is refusing to accept any new connections. We said that the limit is 1,024. Now let's see something else. It is the same shell and it is the same Linux machine that we saw earlier. And if we ask the go program what is the file descriptor limit is then it says 1,4,575.

**[31:52]:** And if we ask Python in the same shell, it says,24. Why? because the Go runtime itself raises its own limit when it is starting up. Here if you open the standard library of Go here the comment explains the same thing that some systems set an artificially low soft limit for compatibility with code that uses select which has a hard-coded maximum file descriptor because of the size of FD set but Go does not use select. So the go team decided that the runtime should just raise the limit for itself and let the old software as it is. And if you look at this line, it sets the new limit to the maximum minus one intentionally so that it can detect somebody else changing the limit later. That is why the number we saw was 1,48,575

**[32:53]:** not 576. So the famous quote that we hear which is you will hit a thousand file descriptors problem that exists but it is language dependent. If you are writing node or python or java then you will face that issue but if you are writing go then you have much more flexibility here. Now that we have removed that limitation, let's say we are using go and we have no limitation on the file description. So now we have 1 million descriptors. Let's see from this point how far we can go. If you try again, it stops at 28,190. Then it is dead. But this time with a completely different error which is cannot assign requested address. So to investigate this, let's ask the kernel itself which port numbers it is allowed to use when it makes an outgoing connection. And it says 32,768 up to 60,999. If you subtract and add one that is 28,232

**[34:06]:** available ports. So we got 28,190 and we have to subtract 42 because this is a shared machine and 42 of those ports are already in use by some other services. But the point is that the whole limitation here is something that you can calculate beforehand before you face this error. And this time we did not run out of memory and we did not run out of file descriptors but we ran out of port numbers where on the server not on the client. A lot of people believe that a server can only hold 65,000 connections because there are only 65,000 ports. That is not true because a T connection is not identified by a port. It is identified by four things. The source address, the source port, the destination address and the destination port. And every connection just has to be unique across all four. Our server has one address and one port.

**[35:14]:** And that never changes. So every connection to it must be different in the other two places. Our client also had one address which means that there is exactly one thing which needs to be different across every connection which is our own source port which was we just checked 28,000 of them and that is the limit which belongs to the client which also means that the fix for this is not getting a bigger server. So what we can try here, let's give the client three addresses to dial out from instead of one. And we will ask for 25,000 on each one. So 75,000 total connections. And we have zero failures. The server is holding 75,000 and 9 file descriptors, which is our 75,000 sockets and the same as before. And this is exactly why when you are trying to do load testing in your production, you'll need different machines and several of them to get to the number to get to the load that you can handle.

**[36:28]:** So now that we can actually generate the load, we have seen how what is the limitation now which is the memory. Our server was at around 7 and half MB of memory doing nothing with 75,000 connections that are currently open. It was at 722 MB. If you subtract and divide, each connection costs about 9,750

**[36:58]:** bytes. And that is our servers processes heap, its own heap per connection. It is not the total cost to the machine because the kernel has its own structures for every socket and that does not appear in the processes resident memory. And I tried to measure it three times because that's how you can reach at a definite number. It came out as first 9,51 9752 9,881 bytes. So if you multiply out 1 million connections is going to be a bit under 10 GB of heap before the kernel takes its own share. So on a 61 GB of box that is pretty much easily achievable.

**[37:55]:** So all in all what we learn from this whole thing is a million idle connections is pretty much boils down to your RAM your memory and also most of that 9 kilobytes that we saw is not the socket. It is going to be the go routine that we spawned to watch it monitor it and the buffers we gave it. And that is why people trying very large numbers. They don't use go routines per connection. Instead, they watch every socket from one place using e- pole. Now we learned that one machine can hold a lot of connection depending on how much the RAM is. But the issue is you will not have one machine in your production. You will have multiple of them because of the whole autoscaling and all architecture. And when there are multiple machines and when you also consider deployments where a container state dies and a new container comes up in all these scenarios you also have to think about dropping every single one of your users

**[39:02]:** connection and picking them up again and what happens when a machine dies and a container dies suddenly all these scenarios. So to understand that this is the same board with two instances of our server behind a load balancer. Now browser A connects and the load balancer sends it to instance one. Next browser B connects and that gets sent to instance two. Both are healthy. Both have a live web soocket. Now I move a task in browser A and nothing happens in browser B. Why? Because if you think about it, where does that connection live? Instance one has a socket in its memory, a go routine and a note somewhere that this particular socket is involved with board 412. Now instance 2 has never heard about browser A and it cannot because there are two separate process and probably in two different countries. We spent pretty much our whole playlist making our backend stateless so that any instance can serve any request.

**[40:15]:** But now in this video we're saying that a persistent connection is a state and it lives in one specific process memory and it cannot be moved and it cannot be shared. So the moment you accept a web soocket that instance is the only instance in the world that can talk to the user. So what do we do now? This is a very big contradiction to what we have been building so far. Some people suggest this solution called as sticky session. What a sticky session is a sticky session tells the load balancer to keep sending the client the same client back to the same instance and it should do the same behavior for reconnection. Also with sticky sessions, browser A keeps connecting to instance one, but that does not really solve the problem which is browser B is still on instance 2 and it still does not have any idea about browser A.

**[41:16]:** So sticky sessions is not the right solution here. It is a completely different problem that trying to solve. Our problem here is two clients at the same time. the instances themselves have to talk to each other. So the solution that most of us arrive at is going to be pubsub or publish and subscribe which is when something happens on board 412 the instance that receives it instead of trying to deliver it what it does is it publishes to a topic and something some tool in the middle radius or nats or kafka any of these popsup tools They hand that event to every subscribed instance for that topic and each one of them then looks at their own connections and they find the one watching for that board and they write the frame. So the instance that receives the change and the instance that delivers now they can be on different machines.

**[42:23]:** That is the whole architecture of making the instances talk for realtime events. We again have two browsers again have two instances but there is one difference where here this user moves a task. AA moves a task but this time Leo sees it instantly because of the whole event publish and subscribe architecture. There is a one note here which is if you're using radius so its whole popsup model the publish and subscribe model is what we call as fire and forget. So because of that if an instance is not connected at the exact moment that you publish that message is gone forever. There is no way to cue it. There is no way to delay it or retry anything just gone. So think about this scenario where uh a deployment happens and an instance goes down the container goes down and it's replacement container goes up and in the seconds that in in the seconds between them between all this happens

**[43:27]:** the one instance going down and another coming up every message that is published for the users that the particular instance was serving those messages are gone completely. So the other boards that were depending on the updates for those events the only time they see those events is when they do a refresh again. So this is a architectural decision which is if you are using radius pubsub for this you have to be aware of the limitation that radius offers you a model of delivering at most once. So if you need the alternative model which is at least once then you need something else which stores your messages and also sends acknowledgements for them which means something like ready streams or Kafka or Q something like that instead of radius popsup which brings us to the question which is what happens to a message that was published when a user was not connected because that is a very common scenario connections might drop, a phone changes its

**[44:36]:** network from Wi-Fi to mobile data, or maybe the laptop's lid is closed by someone, or the load balancer hits its idle timeout. So in a real production grade realtime system, you have to take into account all of these things and you have to think about what is your catching up strategy. Not just delivering strategy but catching up in case some kind of connection error happens, connection failure happens. And the shape of the solution is something that we have already seen in the server send events which is name all your messages or ID all your messages and make the client who is communicating with you tell you the last ID or the last number it actually received and send everything after that. In case of any kind of connection error, a platform like Uber, they do exactly this. A client connects and it says I am at sequence zero and then the service starts sending the messages with increasing sequence numbers.

**[45:41]:** If at any point the connection breaks, the client reconnects and says I was able to get up to sequence number two. So the server takes that and then it starts from three because that's where the client was errored out. This is exactly what we saw with the server send events using the header last event ID.

**[46:03]:** Now the last thing that I want to cover is what we call as fan out. We just saw above that to solve the problem of multiple instances you do the whole publish and subscribe model. But if we consider that if 30,000 people are watching the same board then one person moving one task means that your server has to perform 30,000 rightes. The actual work that your server has to perform is the number of events multiplied by the number of subscribers. Discord had wrote about this once because they had scenarios where they had communities with 30,000 people online at once and they measured that publishing a single event to one of those took somewhere between 900 milliseconds to up to 2.1 seconds for one message for a platform like Discord. core architecture is about messaging. And the solution they found for this problem was instead of doing the fan out in one place, spread out, spread the whole work across different machines who actually owns the recipients.

**[47:22]:** Let's also pay attention to the mechanism of reconnect. So think about an instance which is holding 50,000 connections and if that instant dies all the connections that it is holding all the clients they all start reconnecting at the same moment and the moment they connect all of them want their initial state which boards which tasks who is online which means 50,000 expensive requests in 1 second and this is the scenario that was written by Slack and the Fix for this was to make the connecting part cheap by keeping a cache at the edge of the network which holds the information of what a client needs to start so that whenever a reconnection load happens, a reconnection storm happens, it does not reach the origin. It does not reach the core. Okay. Now to summarize everything, we started with a server that could only reply to questions to a connection which was started from the browser.

**[48:29]:** And to beat that, we started with polling which worked. But the price it cost was the number of people watching instead of the number of things happening, number of changes happening or events happening. Then the alternative that we considered was long polling which holds a request open which again had some of these problems because of the gap between the client receiving the response processing it and starting another request. It lost some changes. So to counter that we came up with a different variation of the long polling which is server send events. So instead of closing the request after sending the response back, the server keeps the connection open. But the limitation then we faced was the client the browser itself could not send any messages back to the server and to counter that we came up with websockets which starts as a h request then it upgrades to

**[49:29]:** a websocket protocol with things like uh bite headers and masking which protects proxies. we try to measure on a machine. The first limitation that we faced was the file descriptors which is because in Unix everything is a file because of that the socket is also a file and we saw that different programming languages have different limits and if you are using go then you can go up to 1 million. Then the next limitation that we faced was a client's port range not servers limitation the client's limitation and to counter that we configured so that the client dials out from different different ports instead of a single port and in the end the limitation or the the ceiling that we faced was the memory which was around 9.6 six kilobytes of heap per connection. And then we saw the problem of persistent connection handling scenarios where our server is divided into multiple instances for different reasons like scaling and then containers restarting and all these things.

**[50:36]:** And to counter that the architecture that we came up with is popsup publish and subscribe. That's pretty much a brief about realtime systems and how clients and servers handle it using different different strategies. Now there is also a lot of things that we did not cover things like presence system to see which users are currently online in our platform and also collaborative editing where two people can change the same thing at the same time and someone has to win. If you have used platforms like notion or Google docs, they show live cursors of two people editing the same text at the same time. All of those can push the limits and the capabilities of realtime systems to a 10 hours of lecture because each one of them open a door into multiple technologies. Like if you are talking about the collaborative editing space then you have to understand algorithms like CRDT how they work over the wire and all those stuff.

**[51:44]:** But everything that we talked about in this video is the foundation on top of which we can keep adding more and more capabilities.
