Blogs

   
Michael Petrov
Co-Founder, CEO
4/30/2012
Storage Failure- a Huge Success?

Yesterday Digital Edge encountered an EMC CLARiiON CX-4 failure. It was a major component failure rather than just a disk.

Centralized enterprise storage is the most critical part of any infrastructure, and any Enterprise IT engineer would have an unpleasant feeling, a feeling that would be described as a physical pain concerning storage failure. In certain situations with a bad design "everything" can go down, and who knows for how long and how much data will be corrupted or unrecoverable.

Serious storage failure in most cases ends up with some financial consequences, job loss; money and/or reputation loss etc…

 

So in this case, EMC had an SPB controller CPU failure, half of its internal architecture. It is as if a human would have had a stroke damaging the brain’s left hemisphere, leaving the person unable to move at all. However not in our case we were up and running, here is why…

The communication from servers to SAN was configured with Multi-Path where volumes were load balanced between SPA and SPB. Each volume had a preferred path with the ability to failover to the failover path. During the failure all volumes paths with SPB preference were migrated to SPA seamlessly and applications didn’t even feel a blip.

This is a result of a good design for Fault Tolerance on each single component the SAN – Servers, Fiber, Switches, SAN itself. In theory we would feel service degradation, but didn’t notice any slowdown as the CLARiiON is a very powerful device – at least for the client’s current needs.    

 As for recovery, we received the part from EMC, and were ready to fix the failed component in 2 hours. As for industry standards it is a very quick turnaround for hardware manufacturer, its support, and supply organization.

 

Lastly, I am sure that the failed CPU was not manufactured on US soil and rather produced somewhere on the other side of the world. However shortcoming of the manufacturing quality was greatly compensated by great device, and infrastructure engineering.

   

Replies

Michael Petrov on 12/3/2012 4:02:04 PM

Alfonso.
I am not sure why you say this. CX or VNX platform has enough built in redundancy to sustain those situations. I have never seen that chassis fail.
Everything else is redundant with a good failover mechanism.
When you say that this is Tier 2 storage, are you saying this because of the power - throughput/IOPs or reliability?
I think those small systems like VNX, FAS, Compelent, 3PAR and probably dozens more are pretty good for some small size applications with specified I/O profile.
Again, the system was not down and was fixed quickly.

 
Alfonso on 12/3/2012 12:22:12 PM

Sorry to hear what happened to you, but is the proof that any chain is strong as their weakest chain.

In your case was the storage subsystem. CX-4 (no matter if it was a 120/240/... model) could be considered and was considered by EMC as a Tier 2 storage system. You (as many others got bitten) by the simple fact that you decided to run mission-critical services (who require zero downtime) on a non mission-critical piece of hardware like the CX-4 was.

If you talk to anyone competent at EMC will tell you the main differences between CX-4/VNX and Symmetrix (and of course their price difference). Happened to me on a customer (and the worse is.. they never learn).

 
Rose on 7/26/2012 8:37:34 AM

Agreed, and this is where Veeam excel with both easy to use free' versions and good diitsibutron of training videos etc.Apparently (according to @storagebod I think) Netapp will be making their VSA publically available. I think they should have done this a while ago but whenever the question has come up previously their reasoning has been around support. What if the customer configures it wrongly and then thinks the Netapp doesn't work properly? Either way, given the ease of obtaining competitors VSA's, I think Netapp need to get out there!

 
Michael Petrov on 6/26/2012 2:14:09 PM

Thank you for the reply, Carlos, However, I really think that this is a misconception of my main point which was to convey an storage malfunction within an enterprise based solution. Based on your comments, your issue is referring to a consumer related topic.

The storage device depicted in this blog is something like a Ferrari of IT, and Carlos pardon me, but your saying that for really cheap you can acceleration to a Honda Civic.
What I was trying to explain to the audience is top notch storage device, and Carlos you are referring to commercial products $99 disks.
My blog entry is referring to enterprise class storage such as 3PAR, EMC, NetApp etc… which goes from 1000 to 4500K per disk, and people pay this price because it worth it.
If someone thinks that it is senseless - the person who is thinking this way just doesn't understand the difference between enterprise and home or small business non mission critical.

 
Carlos on 6/22/2012 7:18:19 AM

Network Attached Storage (Electronics) I purchased this to have safe bpukcas of all of my photos and music. Having lost a hard drive (not backed up) last year and my external drive being at capacity, and a bit finicky lately I was concerned about the safety of my data. I purchased this as an insurance policy against losing the treasured images of my family, friends and travel I have accumulated over the years. I researched options and found this unit to be the best value to meet my needs. My needs (and wants) included: Double back up of data (RAID) Enough space to back up current drives (600GB) Network-able so I can share to laptop Out of the box this feels like a quality piece. At about 6 pounds it feels solid in your hands. Case is simple and attractive but would have preferred a black case over white. It comes with a wall-wart power supply and an ethernet cable. There is USB port but no cable is provided. One simple vertical white light is the only physical user interface. Aside from the color a very handsome package. My only issue is that the light NEVER goes out completely and is relatively bright. Consider that when deciding where to position it. Id prefer that the light have a sleep mode that would shut it off after a period of inactivity. Set up is as simple as plugging it in and patching the network cable to your router. The included software does the rest. It finds the drives and maps them to your machine. The software must be loaded onto any machine on your network that you want to have access to the drive or to backup. A simple and pain free process. Moving data to the drive is time consuming. Transfer speeds are slower than both my external and internal drives (not unexpected). Not a deal killer, just be aware that the initial data transfer will take a while. Once loaded the read/transfer speed is fast enough to stream music and movies to my laptop without issue. You'll notice that write speed was not one of my requirements as this is a back up & RAID drive. The drive has no fans so it is very quiet. Even when read/writing and sitting right on top of my desk I rarely notice it. I am a bit concerned about the long term viability considering it doesn't have fans and the venting on the top, bottom and back panels will let in a lot of dust. I have not as yet set up a MioNet account (free with purchase, otherwise $79/Yr.) so I can't comment on its usefulness or interface. In summary this unit has met my expectations for what I needed. No unpleasant surprises but no exceeded expectations. From a pricing standpoint it's a good value. Today the drives inside retail for $99 each. So you are paying $149 for a case, power supply, software, ability to network and a cable. A good deal in my book. I don't hesitate to recommend the product and deduct a star (remember thats 20%) for the Cons noted below. I sleep better now knowing my pictures and media have a better chance of surviving through my lifetime. Pros: Peace of mind knowing pics are safe Easy to set-up and network Easy to use software Quality construction Everything included Free MioNet account Cons: Slow write speed Continual dismounts Always-on status light

 

Leave a reply

Name (required)
Email (will not be published) (required)

Number from the image above
  
Latest blog posts
VNX Versions
11/10/2014
Subscribe to the blog by e-mail

Sign up to receive
Digital Edge blog by e-mail


Subscribe    Unsubscribe