In today’s world, real time data is the actual game changer. Users need real time updates to stick to their screens and engage more. User engagment and experience drives the success of any buisness. Coming from stock broking industry, I have been developing near real time APIs for quite sometime but recently got my hands on into websockets which are perfect for the real time stock market data users need.
Developing scalable websocket applications are not as easy as it may sound. It comes with its own set of challenges starting with how to implement in java using spring boot to how to optimise the processes to how to scale for millions of users. It is a journey of months altogether. So, In this blog, I would like to share all my learnings with this protocol to provide a starting point at least for those who aim to not only build a websocket application but actually make it production ready.
Technologies used : Spring boot, Java JSR356, OkHttp ( for publisher websocket connection )
Lets get started !!
What is a websocket ?
Websocket is a persistent, bi-directional and full duplex protocol which is used for client-server communication.
Let’s understand this with an example :
You go to a restaurant where for your table an assistant is assigned. You can order anything , any number of times just by registering your request with the assistant and he will take care of the rest. You need not call the different assitants every time you wish you to order something.
Similarly in websocket, client need not open a new connection every time. The websocket protocol leverages the HTTP upgrade technology to upgrade an HTTP connection to a WebSocket. After which, same connection will be persistent and can be used for two way communication unlike traditional HTTP request where client and server communicate only once and the connection gets closed.
Tips to consider while creating Websocket Server :
Spring boot is the best choice to setup a websocket server. However, I won’t be giving details of what should be the code or step by step process to actually start a websocket server because that’s the real thing to learn and dive into. Instead, I would just point down some tips that are good to consider :
- Use Java JSR-356 dependency in spring boot for implementing websocket server. Other websocket libraries, which are present such as stomp take lock on the session iterations which slows down the process of writing messages over sessions. So, native library is always the best choice !! Useful link : https://www.oracle.com/technical-resources/articles/java/jsr356.htm
- Using undertow as application server instead of tomcat. In high concurrency systems, Tomcat is relatively weak. Undertow is optimal in terms of performance and memory usage. Just introduce spring boot undertow dependency, exclude tomcat dependency and get going !!
- Whenever the undertow server starts, it doesn’t initialise the dispatcher servlet which is responsible for handling incoming HTTP request. This is called lazy initialisation. However, in real scenarios it will be good to have this initialisation done early to avoid intialisation during API call. This can be enabled just by a simple property spring.mvc.servlet.load-on-startup : 1
- Loose coupling between the producer thread vs publisher thread which sends the messages to users.
4.1 If the coupling is high the producer thread which does all the processing of incoming messages and hands over to the publisher has to wait till the publishing is finished.
4.2 This can be achieved by either maintaining a queue for all publishers ( if data is common for all ) or if the data to be sent is subscription based, better to use individual queues for each user.
5. Long running single thread acting as publisher manager to manage the threads which will send messages to user. This design helps in managing the sessions which has data to sent effectively. Using thread pool with optimal number of threads ( determined from load test ), queue size to submit tasks.
6. Use dynamic batching ( formula to adjust the batch size of sessions based on number of active sessions and the thread pool size ) to process ‘x’ no of sessions together in one thread so that each session get it’s chance faster.
7. When we send websocket message in byte buffer format one by one there is heavy performance penalty. It is recommended to batch ’n’ no. of messages present in user queue and send them at once. This is faster and gives optimal performance.
7.1 Undertow manages the threads on its own based on the number of cores allocated to the application. Task threads are 8 * no of cores and I/O threads are equal to number of cores.
7.2 When we don’t batch the XNIO- I/O threads which are limited in number are always busy which leads to latency issues and threads keep waiting for the writing to finish this could cause major delays at client side in receiving updated data.
8. Users connecting to websocket must be served with the last available data and in-memory caching comes to rescue where you can cache the incoming messages according to your use case and for every new request send first data from cache.
9. Keeping a note on GC activity and the CPU + memory utilisation in the app using Visual VM. Websocket application is CPU intensive and it generates huge amount of data, if there is a memory leak, it could lead to severe performance troubles. Hence, performance testing is must and longer GC pauses must be avoided at any cost.
More info in my blog : https://medium.com/@ramyav/handling-java-out-of-memory-issues-in-websocket-application-90ee33e936e3
10. Using message size in websocket @onMessage.. function to restrict clients from sending large chunks of data as message which could overwhelm the server.
11. Implementing rate limiting to prevent DDOS attacks on server using messages sent in uncontrolled manner from client side.
12. Keeping the connection retries with sleep (if your websocket server acts as client and consumes data from sources through websocket ) this helps to maintain CPU usage under control.
13. Accepting batch of requests to be processed in a single message to avoid performance issues during scale.
Let’s understand with this example of same assistant of restaurant:
Let’s say you decide to order 4 dishes, so if you would call assistant 4 times and tell him your order, each time he will go to chef and register your order and comeback to take next order from you this will add significant waiting time instead if you can just tell all the 4 orders at once, chef will get busy quickly in parallel to prepare them.
Following all these steps, doesn’t ensure that your application will start serving lakhs or millions customers from day 1. To reach that level of readiness, the application need to be tested through tough rounds of performance test with variable load, different infra and the number of socket connections an application could serve vary use case to use case. It depends on how much CPU intensive tasks the server does, the amount of data it process, the size of data, the collections being used, the number of clients that are expected to connect, the peak load handled by server, memory and CPU config , number of threads in server and lot more.
I can proudly say that, we did several rounds of optimisations (as mentioned above) , so many hit and trials with infra and configs to reach 10–15k connections in one server with no latency at our client side.
I hope this blog was helpful to some extent. Please share your feedback and valuable learning from the journey of building websocket applications. Thanks for reading :)
