Engineering

It is a great time to be a web software developer. Over the last decade the components of web development which have little strategic advantage to a start up have gradually been eliminated and outsourced to such an extent that today the gap between writing code and deploying a new application is often bridged with a single click.

Whereas ten years ago deploying a new application required provisioning a new server, installing Linux, setting up MySQL, configuring Apache, and finally uploading the code, the process today has dramatically less friction. On Heroku, one powerful command line is now all that stands between a team of developers and a live application:

> git push heroku master

Let’s take a closer look at what is happening. The code residing in the repository is uploaded directly to, in this example, Heroku’s cloud platform. From that point onward, the long list of tasks involved in maintaining and fine-tuning a modern web stack are outsourced. The platform provider handles hard drive failures, exploding power supplies, denial-of-service attacks, router replacement, server OS upgrades, security patches, web server configuration … and everything in between.

The implications of this trend are bound to be far-reaching. As common infrastructure is outsourced to vendors such as Amazon, Rackspace, Google and Salesforce.com, the base of customers for hardware and stack software will become increasingly concentrated. As the platform vendors function both as curators and distributors of middle-ware for associated services such as application monitoring and error logging, new monetization opportunities will arise for those companies, such as New Relic, providing these tools.

Just as the arrival of open-source blogging platforms eliminated the intervening steps between writers and audiences, so the new breed of platforms has reduced the friction between developers and their customers.

Most importantly, though, the barriers for new private companies to compete have been permanently lowered. Today, $100 per month can buy you a billion dollar data center.

In this way you can obtain the list of the ten oldest processes:

ps -elf | sort -r -k12 | head -n 10

To sort processes by memory usage use “Shift M” when running.

Use ‘c’ to show full path for command.

For other useful ‘top’ configurations.

Stacy Smith, Intel’s CFO, has some interesting data on the tipping point for PC market penetration. As the cost of a PC in a region moves from multiple years to 8 weeks of income, the penetration changes from zero to about 15%. Once the cost drops below 8 weeks of income, the penetration rises very rapidly to 50%.

According to Smith, the cost of a PC in both India and China is now below 8 weeks of income in those countries.

Facebook isn’t often cited as a cloud computing company since the ‘Social’ moniker has proven to be stickier. It does, however, meet the common definition of ‘Cloud’ i.e. the management of the hardware is highly abstracted from its users, the infrastructure is highly elastic, a variety of services (billing, authentication etc.) are bundled, and the underlying hardware is geographically dispersed.

What is fascinating is that Facebook, more than other cloud companies, gives us a glimpse into a future where computing and storage are virtually free and ubiquitous. With $2 billion in revenue for 2010 and about 500M users, Facebook has revenue of roughly $4 per user. With some back of the envelope math, it seems likely that the variable cost for each additional user is about $1 per year. Think of the services that Facebook is providing its users for $1. Unlimited photo storage and sharing. A contact database. Email. Instant messaging. A gaming platform.

The economics in the consumer cloud are compelling. They will become more so over time and as large enterprises realize that there is no strategic value in common IT, there will be a similar shift for businesses.

One of the major issue with large data centers is power. This is applicable to both large data centers like Microsoft / Google and also to large Enterprise Data Centers which are very energy inefficient.

Definition of Power Effectiveness: Data Center Power Usage Effectiveness (PUE) is defined as the ration of data center power to IT (server) power draw. Thus a PUE of 2.0 means that the data center must draw 2 Watts for every 1 Watt of power consumed by IT (server) equipment. The ideal number would be 1.0, which means there is zero overhead. The overhead power is used by lightning, power delivery, UPS, chillers, fans, air-conditioning etc. Google claims to have achieved a PUE of 1.3 to 1.7. Microsoft runs somewhere close to 1.8. Most of Corporate America runs between 2.0 and 2.5.

A typical large data center these days costs in the range of $150 Million to $300 Million depending upon the size and location. A 15 MW data center facility is approximately $200 million. This is the capital cost so it is depreciated over time.

Most of the facility cost is power related. Anywhere  from 75% to 80% of the cost is power (pdu, chiller, ups, etc).

A typical 15MW datacenter with 50,000 servers costs about  $6.0 million per month for operating expense (excluding people cost) and the share of power infrastructure (pdu, chiller, ups, etc) is between 20% to 24% and actual power for the servers is 18% to 20%. Thus total power cost is between 38% to 44%. These numbers reflect what Microsoft / Google would do. EPA has done a study and they believe these numbers are close to 50% for inefficient data centers.

If you use Google Analytics’ Site Overlay functionality, it occasionally results in a white or gray haze over your website which prevents you from clicking on any of the links.

The good news is that your browser is the only one affected (none of your customers will see the same effect). All you have to do to fix the problem on your end is clear your cookies (specifically a cookie called GASO).

If you want to share video and visual information from your desktop you should check out Dyyno. They have combined some pretty cool video compression technology with a peer-to-peer networking layer and the result is very slick.

Their technology provides the plumbing for Xfire’s live video service. It’s still in beta but if you need a WoW fix, that’s the site to visit.

The following gems and plugins are the most popular as of Nov 12th, 2008:

• Javascript Framework: jQuery (56%), Prototype (43%)
• Skeleton: Bort
• Mocking: Mocha
• Exception Notification: Hoptoad
• Full text search: Thinking Sphinx
• Uploading: Paperclip
• User authentication: Restful_authentication (keep an eye on Authlogic)
• HTML/XML Parsing: Hpricot
• View Templates: Haml

NewRelic has a good article on the state of the Rails stack.

Every now and then it is forcefully driven home to me that Linux is not yet ready for mass adoption. I have been trying to set up my back / forward mouse buttons on Feisty Fawn. There is no reason why this should be difficult but the official instructions are alarmingly non-deterministic! Exhortations to “experiment” are just plain annoying. Plug and Play (TM) might not be perfect but it gets the job done most of the time.

I was once a respected coder. But for 5 years I’d designed ASIC’s using Verilog (where everything happens at once) and then for 5 years I’d turned to business. And it all changes in a decade. I’d let my skills lapse and in the interim C++ had morphed to Java and then suddenly CPU’s got really fast and scripting was back in vogue. 

I realized that my CS undergrad was quickly becoming worthless. Web programming was a complete mystery to me. (Whether that was really a problem is a philosophical question beyond the scope of a humble blog entry). Here is my road to recovery. In bullet point form amenable to PowerPoint and as buzzword compliant as possible.

Jan 1st, 2008: Resolve to brush up on programming skills.

Which language should I learn? Web development seems cool….what’s involved in that? Narrowed it down to a) the LAMP stack or b) Ruby on Rails. Do I want to be a) paid as a programmer or b) hip ?

I went with Hip. Rails it is.

Here are the steps (and mistakes) I took on the road to recovery:

1. Linux – I remember that: “ls -al” and all that. It’s the sine qua non for a real programmer.
2. F@(k. That’s a lot of variants of Linux. Go with Ubuntu because I’m semi-African.
3. Hmm… Windows XP is standard issue at work.
4. Get an old PC from my IT guy. Spend an entire day installing Ubuntu. Realize I’m now a web programmer so start again and install the server version. What the hell? What’s involved with web programming anyway. Will I be writing the client or the server?
5. Call college roommate who is on “tiger team” at Yahoo. He says: “Buy Pickaxe“. Sold. In a flash of environmental sympathy I buy the PDF version. It also saves $10. Print it out on corporate printer. Double sided to save the environment.
6. Need the Rails part: Buy “Agile Web Development with Rails“. We invested in an Agile software company so “agility” must be good.
7. Start reading. In the interest of time and an anxiety to see the global greeting I dispense with Linux and deploy InstantRails on Windows –> Instant gratification. (Nice to see those programmer types have dropped their antipathy towards Microsoft. I’m a web programmer. Even if it’s only on localhost. (Wow: It’s only February and I could compete with Amazon if I wanted to and if I knew where to buy all the books for my bookstore)
8. I have a bookstore up and running. No one can see it. That’s ok….how hard can deployment be.
9. March. Deploytment is hard. People don’t recommend Windows. Could I be the only person writing Ruby code in a Rails environment on Windows XP. Seems to be from my google searches.
10. Let’s reinstall Ubuntu Linux.
11. Install ruby gems. Rinse. Repeat. Rinse. Repeat. Check dependencies. Rinse Repeat. Rinse. Repeat.
12. Install MySQL. (It’s nice that I don’t need to think too much about the database. Seems like something business people should concern themselves with).
13. Stuff is working. Slow as all hell on this ancient PC but what the hell. People will wait for the page to load.
14. Becoming a problem that I can only work on my hobby at work. Can’t afford another PC at home.
15. VMware to the rescue. Downloading an Ubuntu VM on my home PC is a cinch. And hip. Which is important.
16. Realize I need a real hosting service. (Weeks of agonizing research). Settle on Dreamhost. (I love those guys!)
17. Deploy app. Hmmm…this is a f@(k1ng nightmare!
18. Passenger (mod_rails) is released a few days later. I realize I’m back on the cutting edge. Deployment is now piss easy.
19. www.assetcorrelation.com (Live as of June 1st, 2008 — 5 months start to finish)
20. Start to harass Google to show me some organic search love.

It’s been a wild ride. And not as hard as I thought. In the end, we return to the beginning. I still hate writing test benches. Hacking is still fun. And not having deadlines is the way to go.