Yesterday, and today (so far), Last.fm appears to be ill in certain areas: the overall charts have disappeared from user’s profiles and the weekly charts aren’t being generated. Given that one of the primary aims of the website is to enable social networking through common musical tastes, this is a significant problem.

I’ll admit that we don’t have 20 remote servers that we work on. But we do install software to machines remotely.

For Ruby on Rails development, we have our own deployment tools. For PHP deployment we have a repeatable procedure that ensures that remote installations are predictable and reliable. We do three other things too:

• We ensure that changes work prior to deployment;
• We ensure that downtime is minimal and that it only occurs when absolutely necessary. If this means programmatically ensuring seamless transitions, we’ll do this rather than take a site down;
• We ensure that changes work after deployment, rolling back if necessary.