Medical records and the new playing field

Recently one of our clients was looking to build a product in the healthcare domain which involved accessing and modifying medical records. As a part of the initial engagement I did a comparative study of a few EHRs, EMRs and PHRs. There are just too many of them now, each targeted towards a market of their choice. What impressed me is the amount of integrations that are possible with some of these systems. The good ones come with drug database, insurance claims process, ePrescription integration and a host of others. However, what really struck me is the fact that very few of them use modern application delivery methods. If fact a lot of them are still stuck at desktop application/client-server mode. Some have the ASP model. Few, very few have made the transition towards SaaS. Needless to say, these systems need to head towards the direction the world is headed.
The electronic revolution in medical instruments have been going on for a very long time now. But the rise in the use consumer medical instruments is pretty recent. Today we have blood glucose and pressure monitors in almost every homes. And this got me thinking – how cool it would be if these equipments can automatically send the measurements to say, the PHRs. Yes, we are looking at a new playing field if you are thinking what I’ve been thinking.

Getting started with the Cloud

There has been so much of hoo-ha about the cloud in recent days that it’s really difficult to filter out the noise. A lot of times I get this question from people – “So how do we start using the cloud?” With the slew of cloud vendors and underlying technologies, it’s really pretty easy to start using the cloud in our day-to-day lives. In this post I will talk only about the IaaS cloud, Amazon cloud components in particular. But you can choose any vendor you like.

Rapid provisioning and disposal of engineering environments

 
One of the biggest advantages of the cloud is that you can fire up an instance in the blink of an eye. OK, maybe not in the blink of an eye, but if you are using Amazon EC2, it takes about 3-5 minutes to fire up an instance. And that is probably umpteen times faster than your IT department provisioning a server for you. And this gives us the opportunity to rapidly provision engineering environments like testing, staging etc. at will. It also saves a great deal of money if you throw away the instance when you don’t need it. Say, you are at towards the end of the first sprint of the first release of the product. The product owner would need to see the demo of the working product at the end of the sprint. He would also like to play with the product a little. But you don’t want to give out your testing environment so that the product owner and the testers don’t step on each other’s toes. So what do you do. You fire up another instance of the AMI that you saved with all the configurations, pre-requisite software and libraries that’s needed to run the product and deploy the finished product to that instance right before the day of the demo. How much time would it take. Not more than 3-5 minutes, for launching the instance and maybe 10 minutes for the product if you have set up automated deployment.
When the product owner is done with playing around, you can just bring down the instance with the click of a button. It is really that simple.

Deployment of several versions of products

 
In order to keep development work on groups of features independent of each other for one of the products that we were building, we created branches on which groups of developers would work. This is an excellent strategy if you have short development cycles because if there are a group of features that need to go in simultaneously into the product, you can have a (relatively) long running branch. However these branches have to be tested independently. Also, just before the release, you will have to do away with these branches, deploy the trunk after the merge has happened and test that too. So you will need different versions of the product, residing in separate branches deployed at the same time. And that’s where the cloud comes in handy because you can provision and dispose in a jiffy.

Saving application state

 
Have you ever faced the situation when you found a huge bug in a product but the developers could not replicate it and marked the bug as invalid? I bet you have. A lot of times even saving the bug as a video as mentioned in one of my posts, does not help unearthing the real problem. That’s because, most of the times, these issues are embedded in the environment or the data. How good it would be if you could freeze the entire state of the application – session, data, environment and all, bundle it all up and gave it to the development team. Some developers will hate you for that, but I’m sure they’ll thank you later. One of the biggest advantages the cloud gives us is bundling up the entire environment. If you are using Amazon EC2, you can just bundle up the instance, data, environment, session and all into an AMI and go on your way of testing the application again.

Cloudification of a part of the application services

 
If you are in an organization where you cannot deploy anything outside the organization firewalls due to security policy or if the product itself is such that using public infrastructure like the Amazon cloud is not a viable option (one example could be IP protection), you can still use the cloud. Let me take the example of an application that indexes huge documents or maybe media amongst doing a lot of other things. Now, indexing media is very computationally intensive. And what is the cloud good for if it can’t take up the burden of computationally intensive stuff? You can keep the entire application inside the firewall and host a web-service on the cloud which listens for indexing requests and fires up an instance as soon as it gets one and processes the request. Better yet, you can enable Amazon Cloudwatch and set thresholds so that it fires up more instances when one instance is thrashing. Simple but powerful.

 


As you can see, it’s really easy to start taking the advantages of public cloud components. So go ahead and get started today.

Continuous performance testing

Once I asked an engineering manager what they were doing to ensure that the product they were building would be able to endure peak load. After a blank stare, all I got was that they were adding features and would worry about performance later, when they development hubbub settled down.
After the product was built one of the team members ran a performance test and found out a lot of the user actions took an inordinately amount of time to complete even under a concurrent user load of just 5.
Leaving out performance considerations till the end of the development cycles is just dangerous. I have seen teams who were forced to make architectural changes right before the release and then scurry to meet release deadlines, because they never found out what the performance impact of an architectural decision they took 4 sprints ago.
That said, in one of the product teams that I was working with, found out that some of the user actions was taking an inordinate amount of time because users complained about it. This is perhaps the last thing that you want. But anyways, we agreed that we needed to know performance impact of each new feature we pushed out and each enhancement we made. We were doing weekly releases and so we knew that it would really require quite a bit of automation. Choosing performance scenarios, creating scripts, running them and analyzing the results was too much to be done manually every week.
Enter Bamboo, JMeter and Gruff. Bamboo would run the performance scripts created in JMeter along with the build every Tuesday before the build got pushed to the staging sever. We can store the results for the scenarios in MySQL and create graphs with Gruff and show them as Bamboo “artifacts” for the Tuesday builds. Yes, I know what you are thinking. Maybe we should have used the rrdtool instead of MySQL and Gruff. We tried using rrdtool and gave up. Mainly because we wanted to keep all our performance data, not just a window of it. Since we were not running very heavy loads we decided to generate the load from the Bamboo machine itself since it was a pretty decent machine. We kept the target machine isolated to make sure nothing else was done on it except performance tests.
A ruby script would trigger the JMeter scripts just after the build is complete, bundled and deployed on the target sever by capistrano. For result aggregation we used the XSLT provided by JMeter and tweaked it to produce XML instead of HTML. The aggregated data went into the MySQL database. And then we used Gruff to create the graphs and store them as Bamboo artifacts.
Here is a little sample of what it looked like in the end.



With the performance tests running with every Tuesday builds, it was now easy to see the performance impact of adding the new features or enhancements.

Pre-loading the distributed cache

We have been using memcache for distributed caching in one of the products that we were working with. We had 4 application servers with 2GB of memory dedicated to memcache in each of the machines. A lot of times when we delivered new functionality or a patch, we needed to restart the memcache servers. And then for a day, the site thrashed. We got horrible response times for a lot of our pages which were supposed to be cached. We digged a little deep and found out that the cache loader which was supposed to pre-load the cache needed about 50 hours to complete. No wonder we were getting horrible response times in pages that were supposed to be cached. What we actually needed was a multi-threaded pre-loader running on two different machines. With some code optimizations and 16 threads pre-loading the pages into memcache, it took us about 3 hours.