You are currently browsing the category archive for the ‘problems’ category.
Short Answer: Yes!
In my dealings with customers I’ve been requesting performance data from their storage systems whenever I can to see how different applications and environments react to new features. Today I’m going to give you some more real-world data, straight from a customer’s production EMC NS480.
I’ve pulled various stats out of Analyzer for this customer’s Exchange server, which has 3 mail databases totaling about 1TB of mail stored on the NS480 via FibreChannel connect. Since this customer is not extremely large (similar to most of our customers) they are using this NS480 for pretty much everything from VMWare, SQL, and Exchange, to NAS, web/app content, and Business Intelligence systems. There is about 30TB of block data and another 100TB of NAS data. FASTCache is enabled for all LUNs and Pools with just 183GB of usable FASTCache space (4 x 100GB SSDs). So in this environment, with a modest amount of FASTCache and very mixed workload, how does Exchange fare?
Let’s first take a look at the Exchange workload itself for a 24 hour period: (Note: There were no reads from the Exchange log LUNs to speak of so I left that out of this analysis.)
Total Read IOPS for the 3 databases: (the largest peak is a result of database maintenance jobs and the smaller peaks are due to backup jobs) Here it’s tough to see due to the maintenance and backup peaks, but production IO during the work day is about 200-400IOPS. By the way, a source-deduplicating incremental-forever backup technology, such as Avamar, could drastically reduce the IO Load and duration of the nightly backup
Total Write IOPS for the 3 databases: Obviously more changes to the database occurring during the work day.
Total Write IOPS for the 3 Log files: Log data is typically cached easily in the SP cache so FAST Cache isn’t terribly required here but I’m including it to show whether there is any value to using FASTCache with Exchange logs.
Now let’s look at the FASTCache hit ratios for this same set of data: (average of all 3 DBs)
First, the Read Activity: Here you can see that aside from the maintenance and backup jobs, FASTCache is servicing 70-90% of the Read IOPs. Keep in mind that a FASTCache miss could still be a Cache Hit if the data is in SP Cache. What’s interesting about this is that it looks like the nightly maintenance job is pushing the highest load.
And the Write Activity: The beauty of EMC’s FASTCache implementation being a read/write cache, the benefit extends beyond just read IO. Here you see that FASTCache is servicing 60-80% of the writes for these Exchange Databases. That’s a huge load off the backend disks.
And the Log Writes: Since Log writes are usually not a performance problem, I would say that FASTCache is not necessary here, and the average 30% hit ratio shown here is not great. If you wanted to spend the time to tune FASTCache a bit, you might consider disabling FASTCache for Log LUNs to devote the FASTCache capacity to more cache friendly workloads.
All in all you can see that for the database data, FASTCache is servicing a significant portion of the user generated workload, reducing the backend disk load and improving overall performance.
Hopefully this gives you a sense of what FASTCache could do for your Exchange environment, reducing backend disk workload for reads AND writes. I must reiterate, since an SP Cache hit is shown as a FASTCache miss, an 80% FASTCache hit ratio does not mean that 20% of the IOs are hitting disk. To illustrate this, I’ve graphed the sum of SP Cache Hits and FAST Cache Hits for a single database. You can see that in many cases we’re hitting a total of 100% cache hits.
Most interesting is the backup window where SP Cache is really handling a huge amount of the load. This is actually due to the Prefetch algorithms kicking in for the sequential read profile of a backup, something CX/VNX is very good at.
I saw the following article come across Twitter today.
http://www.zdnet.com/blog/storage/ssd-security-the-worst-of-all-worlds/1326
In it, Robin Harris describes the issues around data recovery and secure erasure specific to SSD disks. In layman’s terms, since SSDs do all sorts of fancy things with writes to increase longevity and performance, disk erasure is nearly impossible using normal methods, and forensic or malicious data recovery is quite easy. So if you have sensitive data being stored on SSDs, that data is at risk of being read by someone, some day, in the future. It seems that pretty much the only way to mitigate this risk is to use encryption at some level outside the SSD disk itself.
Did you know that EMC Symmetrix VMAX offers data-at-rest encryption that is completely transparent to hosts and applications, and has no performance impact? With Symmetrix D@RE, each individual disk is encrypted with a unique key, managed by a built-in RSA key manager, so disks are unreadable if removed from the array. Since the data is encrypted as the VMAX is writing to the physical disk, attempting to read data off an individual disk without the key is pointless, even for SSD disks.
The beauty of this feature is that it’s set-it-and-forget it. No management needed, it’s enabled during installation and that’s it. All disks are encrypted, all the time.
- Ready to decomm an old array and return it, trade it, or sell it? Destroy the keys and the data is gone. No need for an expensive Data Erasure professional services engagement.
- Failed disk replaced by your vendor? No need for special arrangements with your vendor to keep those disks onsite, or certify erasure of a disk every time one is replaced. The key stays with the array and the data on that disk is unreadable.
If you have to comply with PCI and/or other compliance rules that require secure erasure of disks, you should consider putting that data on a VMAX with data-at-rest encryption.
Now, What if you have an existing EMC storage system and the same need to encrypt data? You can encrypt at the volume level with PowerPath Encryption. PowerPath encrypts the data at the host with a unique key managed by an RSA Key Manager. And it works with the non-EMC arrays that PowerPath supports as well.
Under normal circumstances, PowerPath Encryption does have some level of performance impact to the host however HBA vendors, such as Emulex, are now offering HBAs with encryption offload that works with PowerPath. If you combine PowerPath Encryption with Emulex Encryption HBAs, you get in-flight AND at-rest encryption with near-zero performance impact.
- Do you replicate your sensitive data to a 3rd party remote datacenter for business continuity? PowerPath Encryption prevents unauthorized access to the data because no host can read it without the proper key.
Some customers are afraid of thin provisioning…
Practically every week I have discussions with customers about leveraging thin provisioning to reduce their storage costs and just as often the customer pushes back worried that some day, some number of applications, for some reason, will suddenly consume all of their allocated space in a short period of time and cause the storage pool to run out of space. If this was to happen, every application using that storage pool will essentially experience an outage and resolving the problem requires allocating more space to the pool, migrating data, and/or deleting data, each of which would take precious time and/or money. In my opinion, this fear is the primary gating factor to customers using thin provisioning. Exacerbating the issue, most large organizations have a complex procurement process that forces them to buy storage many months in advance of needing it, further reducing the usefulness of thin provisioning. The IT organization for one of my customers can only purchase new storage AFTER a business unit requests it and approved by senior management; and they batch those requests before approving a storage purchase. This means that the business unit may have to wait months to get the storage they requested.
This same customer recently purchased a Symmetrix VMAX with FASTVP and will be leveraging sub-LUN tiering with SSD, FC, and SATA disks totaling over 600TB of usable capacity in this single system. As we began design work for the storage array the topic of thin provisioning came up and the same fear of running out of space in the pool was voiced. To prevent this, the customer fully allocates all LUNs in the pool up front which prevents oversubscription. It’s an effective way to guarantee performance and availability but it means that any free space not used by application owners is locked up by the application server and not available to other applications. If you take their entire environment into account with approximately 3PB of usable storage and NO thin provisioning, there is probably close to $1 million in storage not being used and not available for applications. If you weigh the risk of an outage causing the loss of several million dollars per hour of revenue, the customer has decided the risks outweigh the potential savings. I’ve seen this decision made time and again in various IT shops.
Sub-LUN Tiering pushes the costs for growth down
I previously blogged about using cloud storage for block storage in the form of Cirtas BlueJet and how it would not be to much of a stretch to add this functionality to sub-LUN tiering software like EMC’s FASTVP to leverage cloud storage as a block storage tier as shown in this diagram.
Let’s first assume the customer is already using FASTVP for automated sub-LUN tiering on a VMAX. FASTVP is already identifying the hot and cold data and moving it to the appropriate tier, and as a result the lowest tier is likely seeing the least amount of IOPS per GB. In a VMAX, each tier consists of one or more virtual provisioned pools, and as the amount of data stored on the array grows FASTVP will continually adjust, pushing the hot data up to higher tiers and cold data down to the lower tiers The cold data is more likely to be old data as well so in many cases the data sort of ages down the tiers over time and its the old/least used portion of the data that grows. Conceptually, the only tier you may have to expand is the lowest (ie: SATA) when you need more space. This reduces the long term cost of data growth which is great. But you still need to monitor the pools and expand them before they run out of space, or an outage may occur. Most storage arrays have alerts and other methods to let you know that you will soon run out of space.
Risk-Free Thin Provisioning
What if the storage array had the ability automatically expand itself into a cloud storage provider, such as AT&T Synaptic, to prevent itself from running out of space? Technically this is not much different from using the cloud as a tier all it’s own but I’m thinking about temporary use of a cloud provider versus long term. The cloud provider becomes a buffer for times when the procurement process takes too long, or unexpected growth of data in the pool occurs. With an automated tiering solution, this becomes relatively easy to do with fairly low impact on production performance. In fact, I’d argue that you MUST have automated tiering to do this or the array wouldn’t have any method for determining what data it should move to the cloud. Without that level of intelligence, you’d likely be moving hot data to the cloud which could heavily impact performance of the applications.
Once the customer is able to physically add storage to the pool to deal with the added data, the array would auto-adjust by bringing the data back from the cloud freeing up that space. The cloud provider would only charge for the transfer of data in/out and the temporary use of space. Storage reduction technologies like compression and de-duplication could be added to the cloud interface to improve performance for data stored in the cloud and reduce costs. Zero detect and reclaim technologies could also be leveraged to keep LUNs thin over time as well as prevent the movement of zero’d blocks to the cloud.
Using cloud storage as a buffer for thin provisioning in this way could reduce the risk of using thin provisioning, increasing the utilization rate of the storage, and reducing the overall cost to store data.
What do you think? Would you feel better about oversubscribing storage pools if you had a fully automated buffer, even if that buffer cost some amount of money in the event it was used?
So I’ve been a father for about 6 months now and over that time I’ve found that the life of a parent is essentially a series of experiments with real life consequences. Every day I wonder whether I’m doing the right thing for my daughter and when I have to make a decision about pretty much anything related to her, I must weigh the possible consequences of each option.
Case in point, I was feeding my daughter last night at 1am and my mind started wandering, as it usually does–this time about my daughter’s handedness. My wife and I believe she is left-handed, partly because we both are, and partly because she seems to favor her left hand, but it’s really too early to tell. Very early on, we found that she sleeps best at night when she’s swaddled nice and tight, but after a couple of months she stopped sleeping easily unless we kept just her left arm free, another check in the left-handed box. I now leave her left arm free every time I swaddle her, without exception, which leads me to the concern I suddenly felt last night –
Does keeping my daughters right arm tight in the swaddle stifle the development of that arm? Could she possibly be right-handed but we are only allowing the left arm to develop? Should I leave her right arm out next time? What is the consequence of continuing with the status-quo? She sleeps well with the status-quo, which means I sleep well. What is the consequence of switching arms? Would she have trouble sleeping and thus keep me up all night if I switch her arm? Or would she still sleep well and prove over time that she’s actually right-handed? Should I feed her apples or the chicken dinner? How do you handcuff a one-armed man?
The challenge here is that I don’t have a test-lab (aka: test baby) to experiment with different swaddling techniques or foods in order to answer these questions? As parents, we (the proverbial ”we”) are constantly experimenting with real-live children and hoping we don’t screw up. What’s worse is that humans are all unique, so even your first child doesn’t necessarily teach you everything you need to know for your second.
Out in the business world, there are test environments for all sorts of systems, software, electronics, cars, etc where experiments are tested and proven first, then put in place in production environments. The consequence for screwing up most of these systems? Somebody, somewhere, may not be able to buy an iPhone that day. The consequence for messing up in parenting? Ruining our children, ruining our own lives, or both. Sometimes when I have a stressful day at work, coming home to take care of my daughter puts things into perspective.
One of the customers I work with regularly has a set of key MS SQL databases totaling over 100TB in size which process transactions for their customer facing systems. If these databases are down, no transactions can be processed and customers looking to spend money will go elsewhere. The booking rate related to these databases is ~$90,000 per minute, all day, every day.
Today these databases live on Symmetrix DMX storage which has been very reliable and performant. As happens in an IT world, some of these Symmetrix systems are getting long in the tooth and we are working on refreshing several older DMX systems into fewer, newer VMAX systems. Aside from the efficiency and performance benefits of sub-LUN tiering (EMC FASTVP), and the higher scalability of VMAX vs DMX as well as pretty much any other storage array on the market, EMC has added another new feature to VMAX that is particularly advantageous to this particular customer — Federated Live Migration
Tech Refresh and Data Migrations:
Tech Refresh is a fact of life and data migrations, as a result are common place. The challenge with a data migration is finding a workable process and then shoehorning that process into existing SLAs and maintenance windows. In general, data migrations are disruptive in some way or another, and the level of disruption depends on the technology used and the type of migration. EMC has a long list of migration tools that cover our midrange storage systems, NAS systems, as well as high-end Symmetrix arrays. The downside with these and other vendors’ migration tools is that there is some level of downtime required.
There are 3 basic approaches:
- Highest Amount of Downtime:
- Take system down, copy data, reconfigure system, bring system online
- Less Downtime:
- Copy Data, Take system down, copy recent changes, reconfigure system, bring system online (EMC Open Replicator, EMCopy, RoboCopy, Rsync, EMC SANCopy, etc)
- Even Less Downtime:
- Set up new storage to proxy old storage, reconfigure system, serve data while copying in the background. (EMC Celerra CDMS, EMC Symmetrix Open Replicator)
- (Traditional virtualization systems like Hitachi USP/VSP, IBM SVC, etc are similar to this as well since you must take some level of downtime to get hosts configured through the virtualization layer, after which you can non-disruptively move data).
Common theme in all 3? Downtime!
Non-Disruptive Migrations are becoming more realistic:
A while ago now, EMC added a feature to PowerPath called Migration Enabler which is a way to non-disruptively migrate data between LUNs and/or Storage arrays from the host perspective. PowerPath Migration Enabler could also leverage storage based copy mechanisms and help with cut over. This is very helpful and I have multiple customers who have successfully migrated data with PPME. The challenge has been getting customers to upgrade to newer versions of PowerPath that include the PPME features they need for their specific environment. Software upgrades usually require some level of downtime, at least on a per-host basis, and we run into maintenance windows and SLAs again.
EMC Symmetrix Federated Live Migration (FLM)
For Symmetrix VMAX customers, EMC has added a new capability that could make data migrations easy as pie for many customers. With FLM, a VMAX can migrate data into itself from another storage array, and perform the host cutover, automatically, and non-disruptively. The VMAX does not have to be inserted in front of the other storage array, and there is no downtime required to reconfigure the host before or after the migration.
Federation is the key to non-disruptive migrations:
The way FLM works is really pretty simple. First, the host is connected to the new VMAX storage without removing the connectivity to the old storage. The VMAX is also connected to the old storage array directly (through the fabric in reality). The VMAX system then sets up a replication session for the devices owned by the host and begins copying data. This is all pretty straightforward. The smart stuff happens during the cut over process.
As you probably know, the VMAX is an Active/Active storage system, so all paths from a host to a LUN are active. Clariion, similar to most other midrange systems is an Active/Passive storage system. On Clariion, paths to the storage processor that owns the LUN are active, while paths to the non-owner are passive until needed. PowerPath and many other path management tools support both active/active and active/passive storage systems.
Symmetrix FLM leverages this active/passive support for the cut over. Essentially the old storage system is the “owning SP” before and during the migration. At cut over time, the VMAX essentially becomes the owning SP and it’s paths go active while the old paths go passive. PowerPath follows the “trespass” and the host keeps on chugging. To make this work, VMAX actually spoofs the LUN WWN and host ID, etc from the old array, so the host thinks it’s still talking to the old array. Filesystems and LVMs that signature and track LUNs based on WWN and/or Host ID are unaffected as a result. The cut over process is nothing more than a path change at the HBA level. The data copy and cut over are managed directly from the VMAX by an administrator.
Of course there are limitations… FLM initially supports only EMC Symmetrix DMX arrays as the source and requires PowerPath. From what I gather, this is not a technical limitation however, it’s just what EMC has tested and supports. Other EMC arrays will be supported later and I have no doubt it will support non-EMC arrays as a source as well.
Solving the $5 million problem:
Symmetrix Federated Live Migration became a signature feature in our VMAX discussions with the aforementioned customer because they know that for every minute their database is down, they lose $90,000 in revenue. Even with the fastest of the 3 traditional methods of migration, they would be down for up to an hour while 12 cluster nodes are rezoned to new storage, LUNs masked, drive letters assigned, etc. A reboot alone takes up to 30 minutes. 1 hour of downtime equates to $5,400,000 of lost revenue. By the way, that’s quite a bit more than the cost of the VMAX, and FLM is included at no additional license cost, so FLM just paid for the VMAX, and then some.
My recent post about Compression vs Dedupe, which was sparked by Vaughn’s blog post about NetApp’s new compression feature, got me thinking more about the use of de-duplication and compression at the same time. Can they work together? What is the resulting effect on storage space savings? What if we throw encryption of data into the mix as well?
What is Data De-Duplication?
De-duplication in the data storage context is a technology that finds duplicate patterns of data in chunks of blocks (sized from 4-128KB or so depending on implementation), stores each unique pattern only once, and uses reference pointers in order to reconstruct the original data when needed. The net effect is a reduction in the amount of physical disk space consumed.
What is Data Compression?
Compression finds very small patterns in data (down to just a couple bytes or even bits at a time in some cases) and replaces those patterns with representative patterns that consume fewer bytes than the original pattern. An extremely simple example would be replacing 1000 x “0”s with “0-1000”, reducing 1000 bytes to only 6.
Compression works on a more micro level, where de-duplication takes a slighty more macro view of the data.
What is Data Encryption?
In a very basic sense, encryption is a more advanced version of compression. Rather than compare the original data to itself, encryption uses an input (a key) to compute new patterns from the original patterns, making the data impossible to understand if it is read without the matching key.
Encryption and Compression break De-Duplication
One of the interesting things about most compression and encryption algorithms is that if you run the same source data through an algorithm multiple times, the resulting encrypted/compressed data will be different each time. This means that even if the source data has repeating patterns, the compressed and/or encrypted version of that data most likely does not. So if you are using a technology that looks for repeating patterns of bytes in fairly large chunks 4-128KB, such as data de-duplication, compression and encryption both reduce the space savings significantly if not completely.
I see this problem a lot in backup environments with DataDomain customers. When a customer encrypts or compresses the backup data before it gets through the backup application and into the DataDomain appliance, the space savings drops and many times the customer becomes frustrated by what they perceive as a failing technology. A really common example is using Oracle RMAN or using SQL LightSpeed to compress database dumps prior to backing up with a traditional backup product (such as NetWorker or NetBackup).
Sure LightSpeed will compress the dump 95%, but every subsequent dump of the same database is unique data to a de-duplication engine and you will get little if any benefit from de-duplication. If you leave the dump uncompressed, the de-duplication engine will find common patterns across multiple dumps and will usually achieve higher overall savings. This gets even more important when you are trying to replicate backups over the WAN, since de-duplication also reduces replication traffic.
It all depends on the order
The truth is you CAN use de-duplication with compression, and even encryption. They key is the order in which the data is processed by each algorithm. Essentially, de-duplication must come first. After data is processed by de-duplication, there is enough data in the resulting 4-128KB blocks to be compressed, and the resulting compressed data can be encrypted. Similar to de-duplication, compression will have lackluster results with encrypted data, so encrypt last.
Original Data -> De-Dupe -> Compress -> Encrypt -> Store
There are good examples of this already;
EMC DataDomain – After incoming data has been de-duplicated, the DataDomain appliance compresses the blocks using a standard algorithm. If you look at statistics on an average DDR appliance you’ll see 1.5-2X compression on top of the de-duplication savings. DataDomain also offers an encryption option that encrypts the filesystem and does not affect the de-duplication or compression ratios achieved.
EMC Celerra NAS – Celerra De-Duplication combines single instance store with file level compression. First, the Celerra hashes the files to find any duplicates, then removes the duplicates, replacing them with a pointer. Then the remaining files are compressed. If Celerra compressed the files first, the hash process would not be able to find duplicate files.
So what’s up with NetApp’s numbers?
Back to my earlier post on Dedupe vs. Compression; what is the deal with NetApp’s dedupe+compression numbers being mostly the same as with compression alone? Well, I don’t know all of the details about the implementation of compression in ONTAP 8.0.1, but based on what I’ve been able to find, compression could be happening before de-duplication. This would easily explain the storage savings graph that Vaughn provided in his blog. Also, NetApp claims that ONTAP compression is inline, and we already know that ONTAP de-duplication is a post-process technology. This suggests that compression is occurring during the initial writes, while de-duplication is coming along after the fact looking for duplicate 4KB blocks. Maybe the de-duplication engine in ONTAP uncompresses the 4KB block before checking for duplicates but that would seem to increase CPU overhead on the filer unnecessarily.
Encryption before or after de-duplication/compression – What about compliance?
I make a recommendation here to encrypt data last, ie: after all data-reduction technologies have been applied. However, the caveat is that for some customers, with some data, this is simply not possible. If you must encrypt data end-to-end for compliance or business/national security reasons, then by all means, do it. The unfortunate byproduct of that requirement is that you may get very little space savings on that data from de-duplication both in primary storage and in a backup environment. This also affects WAN bandwidth when replicating since encrypted data is difficult to compress and accelerate as well.
On Friday, my local gas/electric utility decided it was time to replace the gas meter and 40-year old steel gas pipe between the street and my house. I had a chance to chat with the guys a bit while they were working and I learned about a small little innovation that not only makes their work easier, it provides better uptime for natural gas customers, and most likely saves lives.
It all started when I looked out the window and saw the large hole they’d jackhammered into my driveway. At first I was a little worried about the jackhammer hitting the gas line but I they do 2-3 of these a day so I figure they must know what they are doing. Then I saw them welding–in the hole! And it turns out that they were literally welding ON the gas line. So I naturally asked, “so you had to turn off the gas to whole street to do this?” to which they replied “nope, the gas is still flowing in there.” Now some of you may know how this is achieved without large fireballs in peoples’ front yards but I was a little stunned at first. So they explained the whole deal. It turns out that the little innovation that allows them to weld a new pipe onto an in-service gas line is called a hot tap. Actually a hot tap is made with several components– a flange, a valve, a few other accessories, and a hot tapping machine.
I couldn’t find a picture that showed the same hot tapping valve they used on my gas line but the following picture from http://www.flowserve.com gives you an idea of what it does…
Flowserve "NAVAL" Hot Tapping Valve
One line shows a completed hot tap in service, and the other shows the hot-tapping tool inserted with a hand drill to drive the cutter.
Basically, they weld the valve onto an existing pipe, along with a flange to better match the contours and add some “meat” to the fitting. In the case of this picture, the hot tapping machine is inserted through the valve, sealing the opening in the valve itself, and the drill turns a magnetic cutter to cut into the working gas line. The magnetism helps to retrieve the metal shavings from the cut.
Once the hole is complete, the hot tapping machine is backed out a bit, the valve is closed, and the machine is completely removed. After that, you can attach a new pipe to the valve and open it up whenever you are ready.
The Pilchuck crew that was working on my line had an even fancier valve with a knob on top and a built-in cutter. So after they welded it on, they just screwed it down to cut the hole and unscrewed once they attached the branch line. Pretty slick since they didn’t need a separate tool to do the cut.
I was thinking about this whole process the next day and it occurred to me just how dangerous it would be to tap live gas lines. And how the idea of a hot tap is really pretty simple, but it probably saves lives. It also keeps service up for every other customer who shares the main pipeline while maintenance is performed, and I’m pretty sure it speeds up the work significantly over shutting down a gas line to cut it and inserting a T-fitting.
While I was looking for a suitable picture I found out that they do this same thing with large continental pipelines as well. There are companies that will hot yap pipes over 100″ in diameter.
This is totally unrelated to storage but I thought it was interesting.
(Warning: This is a long post…)
You have a critical application that you can’t afford to lose:
So you want to replicate your critical applications because they are, well, critical. And you are looking at the top midrange storage vendors for a solution. NetApp touts awesome efficiency, awesome snapshots, etc while EMC is throwing considerable weight behind it’s 20% Efficiency Guarantee. While EMC guarantees to be 20% more efficient in any unified storage solution, there is perhaps no better scenario than a replication solution to prove it.
I’m going to describe a real-world scenario using Microsoft Exchange as the example application and show why the EMC Unified platform requires less storage, and less WAN bandwidth for replication, while maintaining the same or better application availability vs. a NetApp FAS solution. The example will use a single Microsoft Exchange 2007 SP2 server with ten 100GB mail databases connected via FibreChannel to the storage array. A second storage array exists in a remote site connected via IP to the primary site and a standby Exchange server is attached to that array.
Basic Assumptions:
- 100GB per database, 1 database per storage group, 1 storage group per LUN, 130GB LUNs
- 50GB Log LUNs, ensure enough space for extra log creation during maintenance, etc
- 10% change rate per day average
- Nightly backup truncates logs as required
- Best Practices followed by all vendors
- 1500 users (Heavy Users 0.4IOPS), 10% of users leverage Blackberry (BES Server = 4X IOPS per user)
- Approximate IOPS requirement for Exchange: 780IOPS for this server.
- EMC Solution: 2 x EMC Unified Storage systems with SnapView/SANCopy and Replication Manager
- NetApp Solution: 2 x NetApp FAS Storage systems with SnapMirror and SnapManager for Exchange
- RPO: 4 hours (remote site replication update frequency)
Based on those assumptions we have 10 x 130GB DB LUNs and 10 x 50GB Log LUNs and we need approximately 780 host IOPS 50/50 read/write from the backend storage array.
Disk IOPS calculation: (50/50 read/write)
- RAID10, 780 host IOPS translates to 1170 disk IOPS (r+w*2)
- RAID5, 780 host IOPS translates to 1950 disk IOPS (r+w*4)
- RAIDDP is essentially RAID6 so we have about 2730 disk IOPS (r + w*6)
Note: NetApp can create sequential stripes on writes to improve write performance for RAIDDP but that advantage drops significantly as the volumes fill up and free space becomes fragmented which is extremely likely to happen after a few months or less of activity.
Assuming 15K FiberChannel drives can make 180 IOPS with reasonable latencies for a database we’d need:
- RAID10, Database 6.5 disks (round up to 8), using 450GB 15K drives = 1.7TB usable (1 x 4+4)
- RAID5, 10.8 disks for RAID5 (round up to 12), using 300GB 15K drives = 2.8TB usable (2 x 5+1)
- RAID6/DP, 15.1 disks for RAID6 (round up to 16), using 300GB 15K drives = 3.9TB usable (1 x 14+2)
Log writes are highly cachable so we generally need fewer disks; for both the RAID10 and RAID5 EMC options we’ll use a single RAID1 1+1 raid group with 2 x 600GB 15K drives. Since we can’t do RAID1 or RAID10 on NetApp we’ll have to use at least 3 disks (1 data and 2 parity) for the 500GB worth of Log LUNs but we’ll actually need more than that.
Picking a RAID Configuration and Sizing for snapshots:
For EMC, the RAID10 solution uses fewer disks and provides the most appropriate amount of disk space for LUNs vs. the RAID5 solution. With the NetApp solution there really isn’t another alternative so we’ll stick with the 16 disk RAID-DP config. We have loads of free space but we need some of that for snapshots which we’ll see next. We also need to allocate more space to the Log disks for those snapshots.
Since we expect about 10% change per day in the databases (about 10GB per database) we’ll double that to be safe and plan for 20GB of changes per day per LUN (DB and Log).
NetApp arrays store snapshot data in the same volume (FlexVol) as the application data/LUN so you need to size the FlexVol’s and Aggregates appropriately. We need 200GB for the DB LUNs and 200GB for the Log LUNs to cover our daily change rate but we’re doubling that to 400GB each to cover our 2 day contingency. In the case of the DB LUNs the aggregate has more than enough space for the 400GB of snapshot data we are planning for but we need to add 400GB to the Log aggregate as well so we need 4 x 600GB 15K drives to cover the Exchange logs and snapshot data.
EMC Unified arrays store snapshot data for all LUNs in centralized location called the Reserve LUN Pool or RLP. The RLP actually consists of a number of LUNs that can be used and released as needed by snapshot operations occurring across the entire array. The RLP LUNs can be created on any number of disks, using any RAID type to handle various IO loads and sizing an RLP is based on the total change rate of all simultaneously active snapshots across the array. Since we need 400GB of space in the Reserve LUN Pool for one day of changes, we’ll again be safe by doubling that to 800GB which we’ll provide with 6 dedicated 300GB 15K drives in RAID10.
At this point we have 20 disks on the NetApp array and 16 disks on the EMC array. We have loads of free space in the primary database aggregate on the NetApp but we can’t use that free space because it’s sized for the IOPS workload we expect from the Exchange server.
In order to replicate this data to an alternate site, we’ll configure the appropriate tools.
EMC:
- Install Replication Manager on a server and deploy an agent to each Exchange server
- Configure SANCopy connectivity between the two arrays over the IP ports built-in to each array
- In Replication Manager, Configure a job that quiesces Exchange, then uses SANCopy to incrementally update a copy of the database and log LUNs on the remote array and schedule for every 4 hours using RM’s built in scheduler.
NetApp:
- Install SnapManager for Exchange on each Exchange server
- Configure SnapMirror connectivity betweeen the two arrays over the IP ports built-in to each array
- In SnapManager, Configure a backup job that quiesces Exchange and takes a Snapshot of the Exchange DBs and Logs, then starts a SnapMirror session to replicate the updated FlexVol (including the snapshot) to the remote array. Configure a schedule in Windows Task Manager to run the backup job every 4 hours.
Both the EMC and NetApp solutions run on schedule, create remote copies, and everything runs fine, until...
Tuesday night during the weekly maintenance window, the Exchange admins decide to migrate half of the users from DB1, to DB2 and DB3 and half of the users from DB4, to DB5 and DB6. About 80GB of data is moved (25GB to each of the target DBs.) The transactions logs on DB1 and DB4 jump to almost 50GB, 35GB each on DB2, DB3, DB5, and DB6.
On the NetApp array, the 50GB log LUNs already have about 10GB of snapshot data stored and as the migration is happening, new snapshot data is tracked on all 6 of the affected DB and Log LUNs. The 25GB of new data plus the 10GB of existing data exceeds the 20GB of free space in the FlexVol that each LUN is contained in and guess what… Exchange chokes because it can no longer write to the LUNs.
There are workarounds: First, you enable automatic volume expansion for the FlexVols and automatic Snapshot deletion as a secondary fallback. In the above scenario, the 6 affected FlexVols autoextend to approximately 100GB each equaling 300GB of snapshot data for those 6 LUNs and another 40GB for the remaining 4 LUNs. There is only 60GB free in the aggregate for any additional snapshot data across all 10 LUNs. Now, SnapMirror struggles to update the 1200GB of new data (application data + snapshot data) across the WAN link and as it falls behind more data changes on the production LUNs increasing the amount of snapshot data and the aggregate runs out of space. By default, SnapMirror snapshots are not included in the “automatically delete snapshots” option so Exchange goes down. You can set a flag to allow SnapMirror owned snapshots to be automatically deleted but then you have to resync the databases from scratch. In order to prevent this problem from ever occurring, you need to size the aggregate to handle >100% change meaning more disks.
Consider how the EMC array handles this same scenario using SANCopy. The same changes occur to the databases and approximately 600GB of data is changed across 12 LUNs (6 DB and 6 Log). When the Replication Manager job starts, SANCopy takes a new snapshot of all of the blocks that just changed for purposes of the current update and begins to copy those changed blocks across the WAN.
EMC Advantages:
- SANCopy/Inc is not tracking the changes that occur AS they occur, only while an update is in process so the Reserve LUN Pool is actually empty before the update job starts. If you want additional snapshots on top of the ones used for replication, that will increase the amount of data in the Reserve LUN Pool for tracking changes, but snapshots are created on both arrays independently and the snapshot data is NOT replicated. This nuance allows you to have different snapshot schedules in production vs. disaster recovery for example.
- Because SANCopy/Inc only replicates the blocks that have changed on the production LUNs, NOT the snapshot data, it copies only half of the data across the WAN vs SnapMirror which reduces the time out of sync. This translates to lower WAN utilization AND a better RPO.
- IF an update was occurring when the maintenance took place, the amount of data put in the Reserve LUN pool would be approximately 600GB (leaving 200GB free for more changed data). More efficient use of the Snapshot pool and more flexibility.
- IF the Reserve LUN Pool ran out of space, the SANCopy update would fail but the production LUNs ARE NEVER AFFECTED. Higher availability for the critical application that you devoted time and money to replicate.
- Less spinning disk on the EMC array vs. the NetApp.
EMC has several replication products available that each act differently. I used SANCopy because, combined with Replication Manager, it provides similar functionality to NetApp SnapMirror and SnapManager. MirrorView/Async has the same advantages as SANCopy/Incremental in these scenarios and can replicate Exchange, SQL, and other applications without any host involvement.
Higher Application availability, lower WAN Utilization , Better RPO, Fewer Spinning Disks, without even leveraging advanced features for even better efficiency and performance.
Back in July I wrote about the week long sailing trip that ended after 1 day with engine failure and dramatic action. Since then our old sailboat has been stuck in Anacortes, WA while the local marine service company diagnosed and repaired the engine. My wife also delivered our first child during that time so we were a little busy anyway. They declared the engine good to go last week and I scheduled sea trials and pickup for Tuesday (8/24). We packed a cooler full of food, some clothes, sleeping bags and drove up to Anacortes to meet the boat. After a slightly expensive lunch at the marina restaurant, with masterful drinks poured by the same bartender we were served by the last time, we met the Travelift about to splash the boat.
The engine fired up just fine and sounds much better than it used to. It runs and idles smoother, doesn’t smoke, runs cooler, etc. So we headed out for sea trials in the bay and cruised around for about 45 minutes at different RPMs, heating and cooling the engine to stress it a little looking for any problems. The boat runs great! Under engine power we move about 1 knot faster than before too. I think the engine has been running poorly for quite a while before it failed. Anyway, satisfied that the boat engine was performing well, we headed back in to the dock. I paid the bill, we loaded out provisions and headed out with just enough time to make it to Deception Pass for slack tide.
In the immortal words of Captain Ron — “Well, the best way to find out is to get her out on the ocean Kitty, if anything’s gonna happen, its gonna happen out there.”
20 minutes out, a new sound develops from the engine compartment. It sounds like metal rattling–a very distinct, sharp sound. Down in the engine compartment it’s a very loud sound, an exhaust leak from somewhere. A couple phone calls and we turn back to Anacortes. We clearly aren’t making it to Deception Pass tonight. The engine is not quote right yet. Mechanic shows up and determines that the head gasket is leaking, might have been a defective gasket. But it’s solid copper and a new one is several days away. Another mechanic joins us at 8:30am the next morning and finds out that the head bolts loosened during the sea trial and subsequent motoring. He tightens then up and its running fine again. So out for another trial, then back to cool the engine and check the bolts again–still good!
So we finally leave Puget Sound’s own Bermuda Triangle for home. We pass through Deception right on time and continue
south towards Coupeville on Whidbey Island. In another moment of calamity on our eternal 3-hour tour, we are moving along at over 6 knots when the boat suddenly stops dead in the water and pitches forward. Jason who was in the galley, flies forward into the head and falls down while dishes go flying. A quick check of the depth sounder (showing 2.8ft) confirms my fears.. we hit a sand bar. It turns out the navigator (me) was too preoccupied on his cell phone dealing with plans for the night and talking to the car dealer about the Mazda’s coolant leak, to notice that we were about 100 yards outside the marked channel. Reversing the engine does nothing to help and the current is pushing us against the sand bar pretty hard.
If you read the previous post, you’ll remember that the dinghy saved the day when 
the engine failed.. Well, another notch on the dinghy’s stern is due after I threw it off the bow, mounted the Yamaha motor, and used it as a mini tugboat to spin the sailboat around into the current to push off the sand bar. I’m contemplating renaming the sailboat and dinghy to “The Problem” and “The Solution” respectively.
We made it safely to Coupeville and had a wonderful afternoon and evening. My wife and baby drove over to meet us for dinner and the next morning we shoved off early for Everett. We got a little wet on this last run due to rain but made it home safe, locked the boat down, hopped in the car and went home. It’s a series of mini-adventures I will never forget.
On the plus side, our little old sailboat is now better equipped, I have a new found respect for the dinghy, and I got to go boating once more before summer ends, even if it did cost us a lot more money than we had planned.
It’s the morning of day 2 on a 7 day sailing trip in the San Juan Islands of Puget Sound. We are 43 nautical miles from our homeport, and I’m sitting at the table watching a diesel mechanic take apart the little engine on our boat.
Over the 4th of July weekend, we spent nearly 3 full days getting the boat ready for this trip. Washed inside and out, installed new convenience items, changed the oil, checked the transmission fluid, batteries, electrical systems, etc. We taken several short and long trips with our Cal 2-29 over the past 5 years and there hasn’t been a single trip over 24 hours that didn’t require a repair of some kind. Once, the bilge pump sucked water INTO the boat and we had to re-plumb the bilge pump system with makeshift hoses available at the nearby port. Another time, while docking in Friday Harbor, my wife leaned too hard on a stanchion, causing it to break off and sending her into the cold Puget Sound water. Twice, an over-zealous helmsperson switched from reverse to forward gear while the engine was at speed and tore the flex coupling on the prop shaft in half. Both times we were close to docking so we just drifted into port and made repairs. After that we thought we had finally seen the last of the major issues for a while.
On Tuesday morning, we left too early to fuel up so I brought a 5-gallon can of Diesel on board. 35 nautical miles later that proved to be a good idea, when we almost ran out of fuel, while navigating the tight and dangerous Deception Pass. We refueled without stopping using a makeshift funnel made out of a plastic water bottle. Afterwards, the engine was clearly turning more than 2000 rpm based on sound and boat speed but the tachometer was showing 600-800 and bouncing wildly. Something to look at later since the engine seemed okay.
An hour later, on the west side of Fidalgo Island, entering the Strait of Juan de Fuca, we were planning our final destination for the day when the engine began to lose power for an unknown reason. Finally, we saw what seemed to be unusual black smoke from the exhaust. At that point we shut down the engine to check on things. We were a few hundred yards from a rock wall, which was cause for some concern, but we had a little time to assess the situation.
At first glance, the alternator belt was very loose but it didn’t make sense because the bolt that allows for adjustment had clearly not moved. It turned out that the bolt on the other end of the mounting arm, the one that secures the arm to the engine block, had sheared off and the arm was free of the engine. Since the engine is an old diesel, which does not require any power or electronic systems to run, we decided we’d try and remove the belt and go without the alternator until we can repair it. We also found a few random bolts and screws in the engine compartment.
While working to secure the belt out of the way with zip-ties we noticed the starter solenoid had pretty much fallen off of the starter, the spring was visible even. The bolts had come loose and one was missing, plus reattaching would require a lot of work due to the location of the bolts. Well, being a single cylinder small diesel, the Farymann A30M can be started with a hand crank when warm, so we secured the solenoid out of the way and figured we’d fire it up with the crank and get to a nearby marina.
Hand cranking failed to produce a running engine, and we really don’t know why, we may have needed the glow plug on which we forgot about until a long time after giving up. It was looking like we were going to have to call Vessel Assist, when I remembered a story I heard about someone pushing their sailboat with their dinghy lashed to the side of the boat near the stern. So we secured the dinghy, fired up the Yamaha 2.5hp motor, and amazingly we were moving along at 4knots just in time to move away from the rock wall that was now only about 100 yards away. An hour later we dinghy-motored our little 35 year old Cal into Flounder Bay on the northwest corner of Fidalgo Island. Some steaks, corn on the cob, and a healthy dose of Captain Morgan over the next few hours helped the mood and the day was done!
At this point we’ve found that not only was the alternator and starter solenoid loose from the engine, one of the two engine mounts was about 30 minutes of running from falling off also. It’s likely the loose engine mount added vibration, which caused the other bolts to loosen, causing more bolts to fail completely–a multi-stage failure of sorts. Today, our goal is to work with the marine service tech to get the engine put back together and tightened up, then see if the engine will run, and assess anything we find there. At $92.50 per hour, this could be a costly day.
This experience, and the previous ones we’ve had as well, reminded me that you need to be prepared for anything, especially when your life depends on it. When your customers (internal or external) depend on your IT systems, you should be prepared for anything to go wrong, and you might have to patch things together to get it going until you can fix it the right way. And that’s okay. Remember, duct tape and zip-ties can pretty much fix anything! 
And it’s only been 24 hours since the trip started.
Follow up here
