We have scaled our stage environment to ensure the conditions and configuration match that of our product environment for all the tests, giving us confidence that these represent real world conditions.
Technical note; we use both Nightwatch and Artillery for our tests. Nightwatch replicates a user interacting with the platform, and Artillery load tests our API endpoints to ensure these are performant.
Some notable areas you might be interested in…
We are testing forms to ensure that when the demand increases, we are there to facilitate the increased demand for you. Testing begins at a lower level, with some early tests involving 30,000 responses in 1 hour and 500 responses in 5 minutes.
We have also conducted and passed tests with the same volume for forms with;
- Organisation fields
- Events fields
We’ll be ramping up this testing in the coming weeks.
Ensuring that these run in a timely manner is important to meaningful communication with students, so has been a big focus for us. We have found through testing that there are areas for improvement here, in addition to improvements we have already made to reduce instances where things don’t run as expected.
- Single broadcasts
- Concurrent broadcasts, to emulate customers running broadcasts at the same time
- High demand broadcasts with tens of thousands of subscribers
Chat widget availability
It’s going to be vital to ensure there are no performance issues with the availability of the chat widget. We’ll be testing that as volumes increase, messages arrive within chat and that the bot responds in a timely manner when involved in conversations.
Workflows starting chat conversations
We are beginning with small tests for low hundreds of conversations starting in five minutes off the back of a form responses, and are increasing this upper ceiling in the coming days. It’s the expectation that we will test around tens of thousands of new conversations within a 30 minute window – and any unexpected performance issues will be identified and resolved.
We’ll also be looking to ensure that message volume from all the different chat channels is handled well too – think SMS, email, Facebook, Twitter etc.