I remember the day when the news broke, and I stayed awake watching the television for days.

I remember the day when airplanes filled with innocent people hit the twin towers, and the Pentagon, and nearly the White House.

I remember the day when I watched live on television, as Americans of all races and religions ran for their lives, and Heroes of all races and religions ran into the chaos to help

I remember the day when thousands of my fellow citizens, humans, people, and peers, suddenly lost their lives.

I remember the day when two buildings fell, and the whole world rose up in disgust
against evil in any form.

I remember the day when we proved that Freedom, Heroism, and Liberty cannot be buried in the rubble of a few buildings.

What do you remember?

2010 has been a year of significant change for me.  This time last year I was on my cell phone in the basement troubleshooting Exchange cluster problems at Nintendo.   Since I was one of only two people managing storage, replication, and backups there, I was perpetually on-call and as many of you who manage storage already know, the SAN (similar to the IP network) is the first to be blamed for application issues.

In January, many months of work designing and building a warm disaster recovery site culminated in a successful recovery test, proving the value of data de-duplication and SAN replication vs. tape backups.

A Career Change:

As February wrapped up I said goodbye to Nintendo after nearly 5 years there, and 12 total years working in internal IT, to make a significant change and become a Technology Consultant within EMC’s Telco, Media, and Entertainment division.

Moving from the customer side over to a manufacturer/vendor is a pretty big change.  I still have to deal with politics within IT projects, but the politics are different.  I still have to worry about financial concerns with IT projects, but those concerns are different.  I still work with customers, but they are external customers instead of internal customers.  For the customers I work with, I have become a knowledgeable consultant, a friend, and a scapegoat — anything they need me to be at the time.

My first 10 months at EMC have been a whirlwind tour.  In the midst of new hire training in Boston, followed by EMC World 2010, also in Boston, I began meeting with customers, attempting to learn about their business and environments.  Some customers want to tell you everything they can about their environment; others give up as little information as possible.

Phases of Transition:

I don’t know if this is typical of other people who move from being a customer to working for a vendor, but looking back I see distinct phases that I went through as I adjusted to this new career.

• The Fire Hose Phase – For the first couple months, in addition to the new hire training and technical training, I had to learn how to use all of the internal tools, meet my customers, and try to glean as much information as possible about their IT infrastructures.  I took lots of notes and my Livescribe pen proved its worth in short order.
• The Overcompensation Phase – My predecessor was well liked by customers and coworkers, so I set out to try and be as helpful as possible to try and build up a similar relationship with my customers.  This backfired in some ways, worked in others, and eventually taught me that I really should just focus on what my customers need and the rest will fall in place.
• The Competency Phase – As I finally settled in to the new job and got comfortable I was able to start taking on more complex requests from customers. I had a better understanding of the capabilities of EMC products and how the capabilities really mapped to business problems.  At this point I had really figured out my role within EMC as well as with EMC’s customers.

Working for EMC:

Now, as I look back at the past 10 months at EMC, I’m amazed at what I was able to accomplish coming into a sales organization for the first time.  EMC has immense amounts of training available; and the people are all extremely helpful and forgiving.  One of the things that amazed me is how accessible everyone is for a 45,000-person company.  If I need detailed technical information on Symmetrix, I can email an engineer in Hopkinton, MA and within minutes get a very detailed reply, or in many cases a call back.  In the past 6 months, I’ve had Product Managers, VP’s, Engineers, and even technical folks from other divisions on the phone, after hours, helping me get information together for my customers.

While I was getting used to my new job, my wife and I had our first child in August and even though I’d only been with EMC for 6 months at the time, my management was so helpful, covering for me longer than they really needed to and ensuring that my workload was reasonable enough to manage as I adjusted to being a new father.

I even achieved EMC Proven Professional certification along the way.  EMC has a way of giving you the tools to succeed, and then allowing you to make the decision on how and whether to use them.  It’s a competitive environment in a very positive way, where everyone wants everyone else to be successful as opposed to succeeding at another’s peril.

Looking Forward:

As this 2010 year comes to an incredible close for myself, my division, and EMC as a whole, 2011 is shaping up to be great as well.  There are some changes coming on January 1^st for my division that will affect me a little but I believe they will be positive changes overall.  Next year I hope to continue honing my skills as a blogger and in my official role as Technical Consultant.  Happy Holidays and New Year to you all.

In this new role, I’ll be helping customers overcome challenges related to storing and managing information.

I’m not sure what my future topics will be other than to say they will still be storage related.  I expect that topics will come from the challenges my customers are facing and how they can be solved with today’s technology.

I promise to be as objective as possible despite my new employer being EMC, the corporate blogging policy is quite reasonable.

As before, the opinions expressed here are my own and not those of my employer or any other person or company.  All company and product names mentioned in this blog are copyrighted by their respective companies.

There are intrinsic benefits that come with the use of thin provisioning.  First, new storage can be provisioned for applications without nearly as much planning.  Next, application owners get what they want, while storage admins can show they are utilizing the storage systems effectively.  Also, rather than managing the growth of data in individual applications, storage admins are able to manage the growth of data across the enterprise as a whole.

Thin provisioning can also provide performance benefits…  For example, consider a set of virtual Windows servers running across several LUNs contained in the same RAID group.  Each Windows VM stores its OS files in the first few GB of their respective VMDK files.  Each VMDK file is stored in order in each LUN, with some free space at the end.  In essence, we have a whole bunch of OS sections separated by gaps of no data.  If all VMs were booting at approximately the same time, the disk heads would have to move continuously across the entire disk, increasing disk latency.

Now take the same disks, configured as a thin pool, and create the same LUNs (as thin LUNs) and the same VMs.  Because thin-provisioning in general only writes data to the physical disks as it’s being written by the application, starting from the beginning of the disk, all of those Windows VMs’ OS files will be placed at the beginning of the disks.  This increased data locality will reduce IO latency across all of the VMs.  The effect is probably minor, but reduced disk latency translates to possibly higher IOPS from the same set of physical disks.  And the only change is the use of thin-provisioning.

So back to HDS Zero Page Reclaim.  The biggest problem with thin provisioning is that it doesn’t stay thin for long.  Windows NTFS, for example, is particularly NOT thin-friendly since it favors previously untouched disk space for new writes rather than overwriting deleted files.  This activity eventually causes a thin-LUN to grow to it’s maximum size over time, even though the actual amount of data stored in the LUN may not change.  And Windows isn’t the only one with the problem.  This means that thin provisioning may make provisioning easier, or possibly improve IO latency, but it might not actually save you any money on disk.  This is where HDS’s Zero Page Reclaim can help.  Hitachi’s Dynamic Provisioning (with ZPR) can scan a LUN for sections where all the bytes are zero and reclaim that space for other thin LUNs.  This is particularly useful for converting thick LUNs into thin LUNs.  But, it can only see blocks of zeros, and so it won’t necessarily see space freed up by deleting files.  Hitachi’s own documentation points out that many file systems are not-thin friendly, and ZPR won’t help with long-term growth of thin LUNs caused by actively writing and then deleting data.

Although there are ways to script the writing of zeros to free space on a server so that ZPI can reclaim that space, you would need to run that script on all of your servers, requiring a unique tool for each operating system in your environment.  The script would also have to run periodically, since the file system will grow again afterward.

NetApp’s SnapDrive tool for Windows can scan an NTFS file system, detect deleted files, then report the associated blocks back to the Filer to be added back to the aggregate for use by other volumes/LUNs.  The Space Reclamation scan can be run as needed, and I believe it can be scheduled; but, it appears to be Windows only.  Again, this will have to be done periodically.

But what if you could solve the problem across most or all of your systems, regardless of operating system, regardless of application, with real-time reclamation?  And what if you could simultaneously solve other problems?  Enter Symantec’s Storage Foundation with Thin-Reclamation API.  Storage Foundation consists of VxFS, VxVM, DMP, and some other tools that together provide dynamic grow/shrink, snapshots, replication, thin-friendly volume usage, and dynamic SAN multipathing across multiple operating systems.  Storage Foundation’s Thin-Reclamation API is to thin-provisioning what OST is to Backup Deduplication.  Storage vendors can now add near-real-time zero page reclaim for customers that are willing to deploy VxFS/VxVM on their servers.  For EMC customers, DMP can replace PowerPath, thereby offsetting the cost.

As far as I know, 3PAR is the first and only storage vendor to write to Symantec’s thin-API, which means they now have the most dynamic, non-disruptive, zero-page-reclaim feature set on the market.  As a storage engineer myself, I have often wondered if VxVM/VxFS could make management of application data storage in our diverse environment easier and more dynamic.  Adding Thin-Reclamation to the mix makes it even more attractive.  I’d like to see more storage vendors follow 3PAR’s lead and write to Symantec’s API.  I’d also like to see Symantec open up both OST and the Thin-Reclamation API for others to use, but I doubt that will happen.