Continuous performance testing
Once I asked an engineering manager what they were doing to ensure that the product they were building would be able to endure peak load. After a blank stare, all I got was that they were adding features and would worry about performance later, when they development hubbub settled down.
After the product was built one of the team members ran a performance test and found out a lot of the user actions took an inordinately amount of time to complete even under a concurrent user load of just 5.
Leaving out performance considerations till the end of the development cycles is just dangerous. I have seen teams who were forced to make architectural changes right before the release and then scurry to meet release deadlines, because they never found out what the performance impact of an architectural decision they took 4 sprints ago.
That said, in one of the product teams that I was working with, found out that some of the user actions was taking an inordinate amount of time because users complained about it. This is perhaps the last thing that you want. But anyways, we agreed that we needed to know performance impact of each new feature we pushed out and each enhancement we made. We were doing weekly releases and so we knew that it would really require quite a bit of automation. Choosing performance scenarios, creating scripts, running them and analyzing the results was too much to be done manually every week.
Enter Bamboo, JMeter and Gruff. Bamboo would run the performance scripts created in JMeter along with the build every Tuesday before the build got pushed to the staging sever. We can store the results for the scenarios in MySQL and create graphs with Gruff and show them as Bamboo “artifacts” for the Tuesday builds. Yes, I know what you are thinking. Maybe we should have used the rrdtool instead of MySQL and Gruff. We tried using rrdtool and gave up. Mainly because we wanted to keep all our performance data, not just a window of it. Since we were not running very heavy loads we decided to generate the load from the Bamboo machine itself since it was a pretty decent machine. We kept the target machine isolated to make sure nothing else was done on it except performance tests.
A ruby script would trigger the JMeter scripts just after the build is complete, bundled and deployed on the target sever by capistrano. For result aggregation we used the XSLT provided by JMeter and tweaked it to produce XML instead of HTML. The aggregated data went into the MySQL database. And then we used Gruff to create the graphs and store them as Bamboo artifacts.
Here is a little sample of what it looked like in the end.
With the performance tests running with every Tuesday builds, it was now easy to see the performance impact of adding the new features or enhancements.
