Don't Worry, It's Safe to Power off that Server and Power It on Again

There's lot of superstition out there regarding data center best practices, and there is some amount of voodoo when it comes to powering down servers. Will the server come up when you power it on? Will the power supply fail?

This morning, I spoke with Mukesh Khattar who has studied the failure rates for the 30,000 servers in his data center. During the past three years, only 1 power supply failed. That's a failure rate of 0.001% / year. (Yeah, that's five-nines, baby!)

Mukesh is also looking at reliability rates for servers with dual power supplies and single power supplies. A power supply is designed to run at 80%-90% load, which is the case in a server with a single power supply. When you have dual power supplies in a server, each supply only runs at 40%-50% load. Since these power supplies are running below their optimum load level, they consequently generate more heat.

In our conversation, I started thinking about what you're spending for that additional reliability. With a Dell 2950, that second power supply costs $299. For 1000 servers, you've just spent $299k for those second power supplies. I'm not even counting the additional operating expense for the eletricity and cooling costs. During the 3-year depreciation for those servers, only 0.06 power supplies will fail. That's right, not even 1 sever out of that 1000 is expected to fail in 3 years due to a power-supply failure.

So, a few take-aways:

1. Don't over-provision your server hardware for failures that are extremely unlikely. You're going to drive up your capital expenses and your operating expense for a failure that's unlikely to ever occur.
2. Power off those unused servers. When you power them back on, they will come back on. Throw away your garlic cloves and salt shakers. It will be okay. Look at the data, it will set you free.
3. Take a different approach to high availability. Instead of trying to bullet-proof your hardware to prevent a failure, think about a graceful way to recover from a hardware failure.

What's Your Carbon Footprint?

Today's Mercury News had a great (and simple) worksheet that allows you to measure how you are personally contributing to global warming. There are a total of 5 inputs:

• miles driven per year
• gas mileage of your vehicle
  
• average electric usage per month (in KWh)
• average natural gas usage per month (in therms)
• miles flown each year

I used the worksheet and calculated my family's carbon footprint and came up with the following. My family's carbon footprint is 50,156 lbs of CO2 per year-- or 12,539 lbs of CO2/year per person.

The averages provided by the Mercury News are:

• Bay Area: 25,102 lbs of CO2/year
  
• California: 26,301
• Nationwide: 35,967

I don't know if comparing my family's per-individual amortization to the averages provided is fair. However, I included both cars and the family vacations instead of just my personal transportation and travels. Hmm, I wonder if I should purchase CO2 offset credits? (And I'm actually serious) One thing that I have done at home is to replace all my incandescent light bulbs with Compact Fluorescent Lights (CFL's).

Anyway, take a look at the Mercury News and their nifty calculator for yourself.

What Did Your Organization Do This Summer?

With the dog days of summer winding down, most kids will be starting (or have already started) the new school year. As part of the traditional back-to-school ritual, there's always a "what did you do during summer vacation?" session during the first week of class. Kids get to brag about all the new and cool things they did during the past several weeks. One of my son's classmates is the son of a famous CEO, and this kid always has the best stories for the summer. My son and I play golf at the local muni, whereas my son's friend plays Sharon Heights, Stanford and Pebble. (sigh)

I was attending a vendor's seminar the other day, and the speaker was asking the audience of IT personnel about new initiatives at their companies. There's lots of press out there about new trends in the industry: virtualization, server consolidation, power management, data center management...

With summer coming to a close, take a moment to reflect back on what your organization did this summer? What initiatives do you have in your organization? How are you advancing your data-center management practices? What are you doing to help top-line revenue growth or improve operating expenses? Are there simple things you can address in short order? (The famous CEO I alluded to earlier has a favorite expression: "low-hanging watermelon.")

Maybe you've already got one or two initiatives in place that are staffed and budgeted. Great! And if not, maybe it's a good time to go after some low-hanging watermelon...

Back to Virtual Reality

After a 1-week vacation in Houston, I'm back to Silicon Valley. I was briefly back in the Central time zone last week during my 3-day visit to our Minneapolis office. Earlier today, I had lunch with a former colleague from Sun Microsystems, and we were catching up on old times. Arun is now a technology evangelist in the Java group at Sun, and we were chatting about various Web 2.0 development technologies and the growth of these new communities. We eventually got around to talking blogs and comparing blog statistics. (Hey, if you blog, you know you track your blog statistics, even if you don't admit it.) Anyway, Arun's daily blog traffic is 10x my monthly blog traffic, which is pretty impressive. If you're a hardcore developer, you should check out his blog. My hats off to you, Arun!

Welcome to the Boomtown

This week, I'm literally and figuratively thousands of miles away from Silicon Valley. I'm visiting my parents in Houston, Texas, which is also my old hometown (at least from sixth grade onwards). Houston has been-- and still is-- the oil capital of the world. They even say oil differently in Houston. It's pronounced ahwl-- and there's plenty of it here. Gas is much cheaper (by almost $0.90/gallon) than in the San Francisco Bay Area. Even my kids noticed the difference at the pump! And I haven't seen a single Prius in two days. I bet Toyota's best-selling car in Texas is their Tundra Crew Cab. In Texas, the Prius might do as a golf cart, but not as your vehicle for getting around town. After all, it was already 80 degrees at the crack of dawn, and it warmed up to a balmy 94 degrees by noon. I wonder how long the air conditioner in a Prius runs from the battery? So, no surprise that the Bay Area's best-selling car is practically a no-show in Houston.

Although I'm the canonical Silicon Valley geek who lives on the leading edge of technology, my parents tend to be a little more towards the center of the bell curve when it comes to technology adoption. (Well, actually, they're somewhere in the trailing edge of the bell curve, but they do read my blog.) Earlier this year, they converted to broadband (DSL) from a dial-up Internet connection. My Dad even set up the DSL modem himself! They also have an HDTV and Dish network (satellite). And my Dad doesn't subscribe to a print newspaper. He gets his news from a variety of news sources on the Internet. This is pretty cool! This was a nice data point reaffirming my own beliefs in broadband, the Web and how technology is transforming everyday lives. Now, I just need to see when my Dad starts blogging...

Appliances in the Data Center

Recently, Google has been generating some new buzz with its search appliances. Google offers a shrink-wrapped version of its search engine in two different packages-- a 1-U rack-mounted server or a 2-U rack-mounted server. The Google Mini starts at $1,995 for a 50,000-document version and scales up to $8,995 for a version that searches up to 300,000 documents. Once you cross above the 300,000 document limit of the Mini, you step up to the pricier Google Search Appliance, which starts at $30k.

The appliances are manufactured by Dell and are distributed by Ingram. Dell touts the multi-colored Google appliance in its print ads and on its website. From the look of Google's appliance, I'm guessing it's a Dell 1950 under the hood of the mini and a Dell 2950 under the hood of the larger version. Dell's new 1900 series are much better than the their 1800 series. We have four generations of Dell hardware in our lab. From our experience, the 1800 series had some quality and reliability issues not found in their previous generations. From our recent experiences, Dell appears to have corrected these problems with their latest generation of servers.

By controlling the hardware environment, installation is greatly simplified, and the application's performance and reliability becomes more predictable. It will be interesting to see how the Google appliance fares.