Performance issues

For some time now the 500px traffic has been growing on average 20% a month. Looking at the graphs a couple of months ago I projected that we will need a different hosting service in about 6-8 months. I was somewhat incorrect about it — we could survive on the current server for maybe 2-4 months, but with a pretty degraded performance. In fact, performance is already degraded — I watched the application logs and there are quite a number of requests that take over 1 second. Not good. We officially ran out of CPU this time. See that pale green area fading on the right? That’s idle CPU, which we don’t have now. Most of the resources are consumed by the database and converter, which processes photo uploads. Memory is also running out, here is another memory usage graph: We started the migration to Amazon EC2 last week. We migrated the photo storage for 500px on Amazon S3 in the summer of 2010, over the weekend I also migrated all user avatars to S3. We setup three separate instances for the application, uploader/converter and the database. We are mostly ready to flip the switch, waiting for Amazon to setup reverse DNS for our new IPs, so that our notification email get delivered to the users and not marked as spam. Please bear with us for the moment — we know the slow site sucks and we are working hard to get rid of the suck.