The Problem
Imagine you are a technological startup building a next generation video communication tool. Probably you have used modern cPaaS systems like Agora or Twilio to manage video streams. Or maybe you even went into the open-source world by using let’s say Janus server.
Finally, your platform is running on a stage server and your test engineers are happy to say that the system works as expected. However, there is a small nuance, there are not enough people or testers to run the system under the load. Meaning you don’t know how the platform will behave when there are 10-20-30 users in a room. Or maybe you have smaller rooms, but the platform needs to have a lot of such rooms at the same time.
Should you go into production and test it with real users? And what happens if the system fails? How many servers do you need to pay for video conferencing? What kind of servers? Those are the questions a typical video communication startup gets before the production.
Solution thoughts
Most obvious and simplest of what could be done is to hire enough testers and ask them to run several browser tabs with your application. So you will be able to run at least one big test before going into production. However, the complication is that it’s hard to run such a test whenever you need it. It’s complex, requires many people available at the same time and moreover you need to control all of them at once. And what happens if you find a bug? You need to retest, again and again, which will become a new problem.
By analyzing previous solution example we can conclude that the load testing approach should have next important features:
- It should be automated. It’s hard to control many real testers simultaneously. What if there is a possibility to have some kind of bots?
- It should be available whenever you need it. Would it be cool to run the system any time?
- It should be relatively inexpensive. Because hiring let’s say 50 real testers for a 2 hours testing period, might cost a penny.
Our Approach
We found a way to launch a bot on a server. This bot connects to the video conferencing room as a usual WebRTC customer and starts sending/receiving video streams. Let’s say one bot corresponds to one real user.
And we have the ability to run as many bots as we like, any time we need it. We point them to the video conferencing system we need to test. And we know how to control them at once.
Such bots are quite resource hungry. They consume a lot of CPU. One decent server in the cloud can handle around 250 bots at one time. And if there is a need for more bots, it’s possible to run more servers.
Usually we use so-called “spot instances” on the cloud, which is less expensive than regular “on-demand” servers.
Here is a demo video of the testing process. You can see me and my partner Stan in one video conferencing room with 12 bots. Also you can notice HTOP (cpu load) screen from the Janus video server.
Results / Benefits
We have built a bots testing system that can be launched and connected to almost any video conferencing platform. It’s pretty inexpensive, your test engineer can launch it whenever he needs and you get the testing results fast.
If you interested to try this load testing system, ping me on sz@trembit.com
