How to write scalable backend

Is it possible to write a scalable distributed backend?

Yes of course. It is not so hard. But do not bother with the microservice architecture.

Some tips

1. Avoid communications between backends. However, there is an exception: Find the places, where CPU is burnt. Typical places like image transformation, OCR, Excel export and so on. These are good candidates for the distribution where you can use a backend to backend call. The price for the network is lower than the price for the computation. You do not have to care about the consistency – if there is some problem, you can compute it again. The computed result is stored or not. Nothing between. Consider using queues instead of direct calls.
2. Do not use locks on the backend side. Use locks on the database level. Do not use application level locking. Such locking is valid only within one instance and not the entire system. Use the database directly without ORM. You will have more control over SQL.
3. Do not use HTTP sessions. Use short live cache on the HTTP request instead. It is difficult to synchronize session content from one instance to another.
4. Assign a unique request ID for each request. You will be able to track calls from the API.
5. If you want to maintain consistency, stay with a relational SQL database. If you want to really scale the database, use sharding. But be careful, shards should be as independent as possible.
6. Be careful when writing things like scheduler. We do not want the scheduled task to run multiple times.
7. Be careful when picking items from the shared queue. If there are multiple consumers, you must ensure that the item is not taken by two instances at the same time.

See scalable monolith design.