Connecting the Dots

SVG and Javascript for interactive web graphics

Indraneel — Sun, 21 Feb 2010 17:01:54 +0000

This is something that I did quite sometime back. I hadn’t written a post about it at that time. So I thought it maybe worth the time to write a little about this.
I was handed the responsibility to come up with a front-end technology for developing a cool online game. Initially we thought of using flash but then I remembered about SVG from my earlier days when we were creating an application to convert CGM graphics to SVG so that the images and image fragments could be indexed and searched. If you are not familiar, SVG is a language for describing 2 dimensional graphics in XML. SVG is really “the” way for interactive web graphics. And after W3C came up with SVG Tiny more and more smart mobile phones have started supporting SVG as a native format for their browsers. For example – the mobile Safari browser in iPhone has native support for SVG.
One of the requirements of the game was to allow the user to draw polygons on a canvas. And then the application had to determine whether a graphic object say, a circle, is inside a polygon that is drawn by the user.
In order to give my team with a headstart I created a little prototype which lets a user click on a canvas to create a polygon. It assumes that all points are on the same polygon until the “Complete Poly” button is clicked. More polygons can then be created by clicking on different points on the canvas and then clicking the “Complete Poly” button. A red circle can be dragged to one of the polygons. And on clicking “Where?” it prints out inside which polygon the red circle is. Since this was just a prototype, I created the Javascript functions inside the SVG document itself.

SVG polygons

Interactivity is achieved by manipulating the SVG DOM at runtime by Javascript. It’s really the same as manipulating the HTML DOM with Javascript.
Here is the SVG fragment that I used for my prototype:







 Complete Poly

 Where?

Here is my Canvas. As you see it has got three rectangles. The first one is the canvas on which the polygons are drawn. On the other two I place two text nodes – “Complete Poly” and “Where?”. I use these rectangles as buttons by creating handlers for them when they are clicked.
When users click on the canvas, those points serve as the vertices of the polygon they want to draw. When they click the rectangle with id – “button1″, the polygon is complete and the subsequent clicks on the canvas are taken to be the vertices of the next polygon.
Here is how the Javascript function to add the vertices onto the polygon and draw them looks like:
function add(evt){ if (evt.target.getAttribute("id") =="button1" || evt.target.getAttribute("id") =="button2" || isDrawingDone == 1) return points = points + evt.clientX + ","+ evt.clientY + " " indices += 1;


			if(indices > 2)

			{

				var target = evt.target

			   	if ( window.svgDocument == null )

			   		svgDocument = evt.target.ownerDocument;
			    var shape = svgDocument.createElementNS(xmlns, "polygon");

			    shape.setAttributeNS(null, "id", "poly"+polyindx);

			    shape.setAttributeNS(null, "points", points);

			    shape.setAttributeNS(null, "fill", "none");

			    shape.setAttributeNS(null, "stroke", "red");

			  shape.setAttributeNS(null, "stroke-width", "3");

			  	if(polyindx > 0 )

			  	{

			  		for (var indx = 0;polyindx >= indx;indx++)

			  		{

			  			var polyid = "poly"+indx;

			  			var polyg = document.getElementById(polyid);

			  			if(polyg != null)

			  			{

			  				svgDocument.documentElement.removeChild(polyg);

} } } svgDocument.documentElement.appendChild(shape); polyindx +=1; }
As you would see from the code that each time the user clicks on the canvas, I’m drawing a polygon with all the available vertices. But I also want to erase off the previous polygons that I had drawn earlier. This is done by following a naming convention, so that the finder looks incrementally through the ids and deletes them. If I didn’t delete those polygons it would have looked like this:

But erasing the polygons poses a new problem. When I’m done with the first polygon, and I start drawing the new polygon, I want the first one to stay. So I cheat here and rename the last polygon I created breaking away from the naming convention I was following so that the finder does not find this one while trying to delete them.
And Finally when the user clicks “Complete Poly”, the following function is executed:





		function completepoly(evt)

		{

			var polyidindx = polyindx-1;

			var polyg = document.getElementById("poly"+ polyidindx);

			if(polyg != null)

			{

				currpolyindx = polyidindx*11;

				polyg.setAttributeNS(null, "id", "region"+currpolyindx);

			}

			points = "";
		}

And finally, the function – in_or_out determines inside which polygon the circle lies:
function in_or_out(evt) { isDrawingDone = 1; if(document.getElementById("txt1") != null) { var txtnode = document.getElementById("txt1"); Root.removeChild(txtnode); } var shape = document.getElementById("circ2"); var polygons = document.getElementsByTagNameNS(xmlns, "polygon");


				var indx = 0;

				var foundflag = 0;

				var numpolygons = polygons.length;

				var data="Outside";
				for(var indx=0; numpolygons>indx;indx++)

				{
			   		var points = polygons.item(indx).getAttribute("points");

					var parray = points.split(" ");

					var numvertices = parray.length;

					var vertx = new Array();

					var verty = new Array();

					var ix = 0;

					while(numvertices > ix)

					{

						vertx.push(parray[ix].split(",")[0]);

						verty.push(parray[ix].split(",")[1]);

						ix=ix+1;

					}
					if ( isPointinPoly(numvertices, vertx, verty, shape.getAttribute("cx"), shape.getAttribute("cy")))

			   			{

			   				data = document.createTextNode("Inside Region =>" + indx);

			   				foundflag = 1;

                                                        break;

			   			}

			   			else

			   			{

			   				data = document.createTextNode("Outside All Regions");

			   			}

				}

				var text = document.createElementNS(xmlns, "text");

				text.setAttributeNS(null, "font-size", "12pt");

				text.setAttributeNS(null, "x", 30);

				text.setAttributeNS(null, "y", 500);

				text.setAttributeNS(null, "id", "txt1");

				text.appendChild(data);

				Root.appendChild(text);

return; }

Pretty simple, isn’t it? This, in short was the base on which the Innovation Game – Prune The Product Tree was built.

Filed under: SVG, Technologies Tagged: Javascript, SVG

Medical records and the new playing field

Indraneel — Sun, 31 Jan 2010 17:06:35 +0000

Recently one of our clients was looking to build a product in the healthcare domain which involved accessing and modifying medical records. As a part of the initial engagement I did a comparative study of a few EHRs, EMRs and PHRs. There are just too many of them now, each targeted towards a market of their choice. What impressed me is the amount of integrations that are possible with some of these systems. The good ones come with drug database, insurance claims process, ePrescription integration and a host of others. However, what really struck me is the fact that very few of them use modern application delivery methods. If fact a lot of them are still stuck at desktop application/client-server mode. Some have the ASP model. Few, very few have made the transition towards SaaS. Needless to say, these systems need to head towards the direction the world is headed.
The electronic revolution in medical instruments have been going on for a very long time now. But the rise in the use consumer medical instruments is pretty recent. Today we have blood glucose and pressure monitors in almost every homes. And this got me thinking – how cool it would be if these equipments can automatically send the measurements to say, the PHRs. Yes, we are looking at a new playing field if you are thinking what I’ve been thinking.

Filed under: Healthcare Tagged: EHR, EMR, Healthcare

Getting started with the Cloud

Indraneel — Mon, 25 Jan 2010 21:17:44 +0000

There has been so much of hoo-ha about the cloud in recent days that it’s really difficult to filter out the noise. A lot of times I get this question from people – “So how do we start using the cloud?” With the slew of cloud vendors and underlying technologies, it’s really pretty easy to start using the cloud in our day-to-day lives. In this post I will talk only about the IaaS cloud, Amazon cloud components in particular. But you can choose any vendor you like.

Rapid provisioning and disposal of engineering environments

One of the biggest advantages of the cloud is that you can fire up an instance in the blink of an eye. OK, maybe not in the blink of an eye, but if you are using Amazon EC2, it takes about 3-5 minutes to fire up an instance. And that is probably umpteen times faster than your IT department provisioning a server for you. And this gives us the opportunity to rapidly provision engineering environments like testing, staging etc. at will. It also saves a great deal of money if you throw away the instance when you don’t need it. Say, you are at towards the end of the first sprint of the first release of the product. The product owner would need to see the demo of the working product at the end of the sprint. He would also like to play with the product a little. But you don’t want to give out your testing environment so that the product owner and the testers don’t step on each other’s toes. So what do you do. You fire up another instance of the AMI that you saved with all the configurations, pre-requisite software and libraries that’s needed to run the product and deploy the finished product to that instance right before the day of the demo. How much time would it take. Not more than 3-5 minutes, for launching the instance and maybe 10 minutes for the product if you have set up automated deployment.
When the product owner is done with playing around, you can just bring down the instance with the click of a button. It is really that simple.

Deployment of several versions of products

In order to keep development work on groups of features independent of each other for one of the products that we were building, we created branches on which groups of developers would work. This is an excellent strategy if you have short development cycles because if there are a group of features that need to go in simultaneously into the product, you can have a (relatively) long running branch. However these branches have to be tested independently. Also, just before the release, you will have to do away with these branches, deploy the trunk after the merge has happened and test that too. So you will need different versions of the product, residing in separate branches deployed at the same time. And that’s where the cloud comes in handy because you can provision and dispose in a jiffy.

Saving application state

Have you ever faced the situation when you found a huge bug in a product but the developers could not replicate it and marked the bug as invalid? I bet you have. A lot of times even saving the bug as a video as mentioned in one of my posts, does not help unearthing the real problem. That’s because, most of the times, these issues are embedded in the environment or the data. How good it would be if you could freeze the entire state of the application – session, data, environment and all, bundle it all up and gave it to the development team. Some developers will hate you for that, but I’m sure they’ll thank you later. One of the biggest advantages the cloud gives us is bundling up the entire environment. If you are using Amazon EC2, you can just bundle up the instance, data, environment, session and all into an AMI and go on your way of testing the application again.

Cloudification of a part of the application services

If you are in an organization where you cannot deploy anything outside the organization firewalls due to security policy or if the product itself is such that using public infrastructure like the Amazon cloud is not a viable option (one example could be IP protection), you can still use the cloud. Let me take the example of an application that indexes huge documents or maybe media amongst doing a lot of other things. Now, indexing media is very computationally intensive. And what is the cloud good for if it can’t take up the burden of computationally intensive stuff? You can keep the entire application inside the firewall and host a web-service on the cloud which listens for indexing requests and fires up an instance as soon as it gets one and processes the request. Better yet, you can enable Amazon Cloudwatch and set thresholds so that it fires up more instances when one instance is thrashing. Simple but powerful.

As you can see, it’s really easy to start taking the advantages of public cloud components. So go ahead and get started today.

Posted in Cloud Computing Tagged: Cloud Computing, Product Engineering

Continuous performance testing

Indraneel — Sat, 23 Jan 2010 20:07:20 +0000

Once I asked an engineering manager what they were doing to ensure that the product they were building would be able to endure peak load. After a blank stare, all I got was that they were adding features and would worry about performance later, when they development hubbub settled down.
After the product was built one of the team members ran a performance test and found out a lot of the user actions took an inordinately amount of time to complete even under a concurrent user load of just 5.
Leaving out performance considerations till the end of the development cycles is just dangerous. I have seen teams who were forced to make architectural changes right before the release and then scurry to meet release deadlines, because they never found out what the performance impact of an architectural decision they took 4 sprints ago.
That said, in one of the product teams that I was working with, found out that some of the user actions was taking an inordinate amount of time because users complained about it. This is perhaps the last thing that you want. But anyways, we agreed that we needed to know performance impact of each new feature we pushed out and each enhancement we made. We were doing weekly releases and so we knew that it would really require quite a bit of automation. Choosing performance scenarios, creating scripts, running them and analyzing the results was too much to be done manually every week.
Enter Bamboo, JMeter and Gruff. Bamboo would run the performance scripts created in JMeter along with the build every Tuesday before the build got pushed to the staging sever. We can store the results for the scenarios in MySQL and create graphs with Gruff and show them as Bamboo “artifacts” for the Tuesday builds. Yes, I know what you are thinking. Maybe we should have used the rrdtool instead of MySQL and Gruff. We tried using rrdtool and gave up. Mainly because we wanted to keep all our performance data, not just a window of it. Since we were not running very heavy loads we decided to generate the load from the Bamboo machine itself since it was a pretty decent machine. We kept the target machine isolated to make sure nothing else was done on it except performance tests.
A ruby script would trigger the JMeter scripts just after the build is complete, bundled and deployed on the target sever by capistrano. For result aggregation we used the XSLT provided by JMeter and tweaked it to produce XML instead of HTML. The aggregated data went into the MySQL database. And then we used Gruff to create the graphs and store them as Bamboo artifacts.
Here is a little sample of what it looked like in the end.

With the performance tests running with every Tuesday builds, it was now easy to see the performance impact of adding the new features or enhancements.

Posted in Performance Engineering, Product Engineering Tagged: Performance Engineering, Product Engineering

Pre-loading the distributed cache

Indraneel — Fri, 22 Jan 2010 08:20:37 +0000

We have been using memcache for distributed caching in one of the products that we were working with. We had 4 application servers with 2GB of memory dedicated to memcache in each of the machines. A lot of times when we delivered new functionality or a patch, we needed to restart the memcache servers. And then for a day, the site thrashed. We got horrible response times for a lot of our pages which were supposed to be cached. We digged a little deep and found out that the cache loader which was supposed to pre-load the cache needed about 50 hours to complete. No wonder we were getting horrible response times in pages that were supposed to be cached. What we actually needed was a multi-threaded pre-loader running on two different machines. With some code optimizations and 16 threads pre-loading the pages into memcache, it took us about 3 hours.

Posted in Product Engineering Tagged: Caching, Distributed Caching

More about email deliverability

Indraneel — Sun, 20 Dec 2009 17:31:50 +0000

Sometime ago I wrote a post about how to send emails without getting flagged as SPAM. Unfortunately I had missed out on a few things.

rDNS

One of the most important things in email deliverability is rDNS (also known as the PTR record). rDNS stands for “reverse” DNS. Here is an example how rDNS works:

A mail header says that the sender is abc@abc.com and it was sent from 111.222.333.444
The receiving mail server verifies if 111.222.333.4444 really points to abc.com by a rDNS lookup

Mail servers like AOL and Google are very very particular about rDNS i.e – they would check for reverse DNS entries for each mail received by them. Your important newsletter is almost sure to make it to the bulk email folder if you haven’t setup the reverse DNS entry for your domain.
Now if you are hosting your nice web-application on Amazon EC2, reverse DNS won’t work for you, because Amazon won’t set your reverse lookup. You will have to use a third-party email service like authsmtp or fastmail to send out mass emails with maximum deliverability.

Domain Keys Identified Mail

Domain Keys Identified Mail is a method for email authentication (as the DKIM website says). Basically DKIM allows the sender of an email to sign the email using public key cryptography. Prominent email services like Yahoo, Gmail and Fastmail implement DKIM. This is how it works in a nutshell:

The sender of the email adds a header-field named “DKIM-Signature” which contains a digital signature of the contents of the header and body of the email message
The receiving SMTP server does a DNS lookup and gets the public key for the domain.
It uses the public key to decrypt the message

You can use Javamail with DKIM for sending out that important newsletter from your web application.

Sender policy Framework

SPF is a special format DNS record which specifies which machines can send emails for that domain.

For example the owner of abc.com can determine which hosts are allowed to send emails whose sender email address ends in @abc.com
Receivers who check SPF can reject messages from unauthorized hosts before receiving the message body

And here’s what the SPF record may look like:
abc.com TXT “v=spf1 ip4:111.222.333.444 –all”

White lists, Black lists

Blacklists are lists of domain names which are known to send SPAM emails. Basically they are the lists of known offenders.
Whitelists are the opposite – lists by which an ISP allows someone to bypass spam filters when sending emails to its subscribers

So …. Get Registered

MxToolBox and ReturnPath are both good options

Posted in Product Engineering Tagged: Email Deliverability

JIRA-git integration

Indraneel — Sat, 19 Dec 2009 17:27:58 +0000

I’ve been messing around with JIRA for quite sometime now. So when one of our product teams got stuck with integrating JIRA with git, I tried to help. There was a JIRA-git plugin over here and it worked fine. Problems arose when the plugin did not pull out commits to the branches in JIRA. So I went ahead and fixed it. Now the plugin pulls up all the changes from the branches but unfortunately it does not show the branch name along with the name of the changed file in the JIRA tab. I’ll try and fix that soon. In the meantime you can download and use the modified plugin from this location.

Posted in Product Engineering Tools Tagged: Integration, Product Engineering Tools

Continuous Integration and Automated Deployment for PHP

Indraneel — Fri, 16 Jan 2009 18:11:15 +0000

I had set up the Continuous Integration for a few Ruby on Rails products including Workstreamr and PaidInterviews. It was easy with rake, the ci_reporter plugin, rcov and Bamboo. I also used Capistrano for automated deployments after each build. A new product development started a few days ago and I was called upon to setup the Continuous Integration platform again. It was different this time though. This new product was to be developed in PHP.
Rake is a general purpose build tool just like ant and maven. And Capistrano is the best deployment tool I have encountered so far. So the choice was easy. This is how I created a rake task to execute all the unit tests and produce test coverage reports with PHPUnit.

Yeah, yeah, I know, I could have done the same thing with a shell script. But when I start creating tasks for database migration and such, things are going to get messy and tracking dependencies wouldn’t be easy with shell script.

PHPUnit can also produce coverage reports. So I included the second task which would produce coverage reports.

Now for the deployment script.

And now I bundle all this up neatly in a shell script and give it to Bamboo. Here is what my build-deploy.sh script looks like:

#/bin/bash rake build:report_coverage cap remote deploy

Posted in Continuous Integration Tagged: Continuous Integration, Deployment

The PC Quest article

Indraneel — Fri, 16 Jan 2009 18:02:25 +0000

Some of my comments appeared in the January 2009 edition of PC Quest. The online version of it is available over here.

Posted in Cloud Computing

Cloud Computing – Large scale computing for everyone

Indraneel — Sun, 02 Nov 2008 18:40:35 +0000

In the beginning of 2008, New York Times ingested 405,000 very large TIFF images, 3.3 million articles in SGML and 405,000 xml files mapping articles to rectangular regions in the TIFF’s using Amazon Web Services, Hadoop and some custom code. This data was converted to a more web-friendly 810,000 PNG images and 405,000 Javascript files containing JSON in less than 36 hours.

Why is this a big deal?

NASA has been computing on far greater scale than this for a long time. NASA’s weather simulation software is computationally a lot more complex than indexing documents.
So why is this a big deal? It’s a big deal because it was neither NASA nor CERN, not even Google. It was a business who did this without buying a single machine. They rented computing power on the fly. They rented slices of a cloud.

What is cloud computing?

A few days ago a journalist said “There is clear consensus that there is no consensus on what cloud computing is”. I like to think of cloud computing as the commercialization of computing resources like CPU cycles, storage, memory etc just like public utilities like electricity, water or natural gas. At the very core of the cloud is virtualization. Virtualization is a technique in which software is used to completely simulate or emulate hardware.

Types of clouds

I see two distinct categories of clouds that vendors are selling today:

Infrastructure as a Service – IaaS vendors sell raw compute power – CPU cycles, memory, bandwidth etc. IaaS clouds are complex but with the complexity comes flexibility. Most cloud vendors allow root access to an instance. And hence, specialized knowledge is necessary to handle such flexibility.
Platform as a service – PaaS refers to those clouds which provide frameworks and infrastructure on which users can build applications. PaaS clouds are built on IaaS clouds. Most PaaS clouds are very restrictive. They generally allow users to build applications on a particular set or sets of technologies. For example Google App Engine allows users to build applications using Python only. Portability is an inherent issue with PaaS clouds, because of the lack of standards in this domain. So if you have built an application using Google App Engine and BigTable you probably won’t be ableto port the data to any other cloud without spending a huge amount of time and money.

Inside the cloud

At a very high level clouds are made up of the following layers:

Inside the cloud

At the very bottom is the hardware layer. Many cloud vendors build their clouds out of of the shelf server class software. For example Joyent uses Dell servers with quad core intel processors for their cloud. Plumbing refers to the networking elements in the cloud with all the fast router, switches and load balancers connected by fiber optic cabling. Clusters, made up of ordinary server class machines make up the skeleton of the cloud.
Storage services refer the storage provided by the cloud. Most cloud vendors offer SAN or NAS storage. Provisioning is generally on the fly and users can ask for virtually unlimited amount of storage.
As mentioned earlier, virtualization is at the very core of the cloud. Virtualization has made creation of a software machine as a clone of an existing one super fast. Think of the cluster (mentioned earlier) as one mega machine with one host OS managing all its resources. Creating virtual machines with pre-defined CPU and memory is fast and easy. Many vendors like Amazon Web Services use Xen virtualization.
Platform services are bunch of pre-installed and packaged goodies that an user of the cloud gets whenever an instance of the cloud is brought up. The LAMP stack supported by AWS and Joyent is an example of platform services.
No matter what the vendor says, if it takes more than 10-15 minutes to bring up an instance, then it is not a cloud. The web services layer is the one that enables users to templatize an instance, bring up a new instance from a template, take backups, restore from a backup etc. instantly, as and when needed.

Just in time deployment using the cloud

Deployment of products is messy business. Not so long ago, fledgling organizations had to first calculate the amount of computing resources needed for a launch, translate that into hardware requirements, call up the hardware vendors or the hosting company and wait till they provisioned the hardware and then installed and configured the software. Provisioning, installation and configuration took several days. It was a lose-lose scenario for everyone. Product success meant another cycle of calls and provisioning while the users suffered due to unresponsive software caused due to heavy load. Product failure meant huge losses due to unused hardware.
Not any more with the advent of the cloud. Now product launches can happen at the click of a button with just enough computing resources sitting behind a Virtual IP. The utilization of the resources are closely monitored. New, templatized instances of the cloud are instantiated whenever the threshold for the monitored utilization is reached. The users of the cloud pay for what they use at any instant of time. The users of the product never find it unresponsive, since computing resources are always adequate, just in time to meet the users’ needs.

Large scale computing for all

So long the ability to do large scale computing was within the reach of an elite club of businesses. Google, Amazon and Yahoo were amongst the very few in the club. Few businesses had the means to lay their hands on infrastructure of that scale. Cloud computing has changed that. Today, a ‘large’ Amazon EC2 instance with 4 EC2 compute units (which is equivalent to the capacity of 4 Opteron or Xeon processors) and 7.5 Gigs of memory costs as less as $288 per month. Users can choose from quite a few operating systems and scale up and down on the fly. Application development platforms like JBoss Enterprise Application Platform and Ruby on Rails come built into it. Clouds have opened the doors of large scale computing to virtually everyone.

Posted in Cloud Computing Tagged: Cloud Computing, product deployment