Filed under: comparisons, problems, solutions, technologies | Tags: 2007, business continuance, ccr, clariion, consistency, disaster recovery, emc, esx, exchange, ibm, mirrorview, netapp, nseries, replication, sancopy, snapmirror, snapview, vmware
Exchange Replication
Building on the redundant storage project, we also wanted to replicate Exchange to a remote datacenter for disaster recovery purposes. We’ve been using EMC CLARiiON MirrorView/A and Replication Manager for various applications up to now and decided we’d use NetApp/SnapMirror for Exchange to leverage the additional hardware as well as a way to evaluate NetApp’s replication functionality vs EMC’s.
On EMC Clariion storage, there are a couple choices for replicating applications like Exchange.
1.) Use MirrorView/Async with Consistency Groups to replicate Exchange databases in a crash-consistent state.
2.) Use EMC Replication Manager with Snapview snapshots and SANCopy/Incremental to update the remote site copy.
Similar to EMC’s Replication Manager, NetApp has SnapManager for various applications, which coordinates snapshots, and replica updates on a NetApp filer.
Whether using EMC RM or NetApp SM, software must be installed on all nodes in the Exchange cluster to quiesce the databases and initiate updates. The advantage of Consistency groups with MirrorView is that no software needs to be installed in the host; all work is performed within the storage array. The advantage of RM and SM/E is that database consistency is verified on each update and the software can coordinate restoring data to the same or alternate servers, which must be done manually if using MirrorView.
NetApp doesn’t support consistent snapshots across multiple volumes so the only option on a Filer is to use SnapManager for Exchange to coordinate snapshots and SnapMirror updates.
Our first attempt configuring SnapManager for Exchange actually failed when we ran into a compatibility issue with SnapDrive. SnapManager depends on SnapDrive for mapping LUNs between the host and filer, and to communicate with the filer to create snapshots, etc. We’d discussed our environment with NetApp and IBM ahead of time, specifically that we have Exchange CCR running on VMWare, with FiberChannel LUNs and everyone agreed that SnapDrive supports VMWare, Exchange, Microsoft Clustering, and VMWare Raw Devices. It turns out that SnapDrive 6 DOES support all of this, but not all at the same time. Specifically, MSCS clustering is not supported with FC Raw Devices on VMWare. In comparison, EMC’s Replication Manager has supported this configuration for quite a while. After further discussion NetApp confirmed that our environment was not supported in the current version of SnapDrive (6.0.2) and that SnapDrive 6.2, which was still in Beta, would resolve the issue.
Fast forward a couple months, SnapDrive 6.2 has been released and it does indeed support our environment so we’ve finally installed and configured SnapDrive and SnapManager. We’ve dedicated the EMC side of the Exchange environment for the active nodes and the IBM for the passive nodes. SnapManager snapshots the passive node databases, mounts them to run database verification, then updates the remote mirror using SnapMirror.
While SnapManager does do exactly what we need it to do, my experience with it hasn’t been great so far… First, SnapManager relies on Windows Task Scheduler to run scheduled jobs, which has been causing issues. The job will run on its schedule for a day, then stop after which the task must be edited to make it run again. This happens in the lab and on both of our production Exchange clusters. I also found a blog post about this same issue from someone else.
The other issue right now is that database verification takes a long time, due to the slow speed of ESEUTIL itself. A single update on one node takes about 4 hours (for about 1TB of Exchange data) so we haven’t been able to achieve our goal of a 2-hour replication RPO. IBM will be onsite next week to review our status and discuss any options. An update on this will follow once we find a solution to both issues. In the meantime I will post a comparison of replication tools between EMC and NetApp soon.
Filed under: solutions | Tags: 2007, ccr, clariion, emc, esx, exchange, file share witness, fsw, ibm, majority node set, mns, mscs, netapp, ontap, powerpath, raid-10, raid-dp, redundancy, vcenter, vsphere
The first application we tackled after deploying the NetApp system was Exchange 2007. We had deployed Exchange 2007 recently, running in CCR clusters on VMWare ESX. Since each node of a CCR cluster has it’s own copy of the database we wanted to put one node from each cluster onto the NetApp, leaving the other nodes on the Clariion. This environment is entirely FiberChannel, no iSCSI deployed and as such the Exchange servers are using VMWare Raw Devices for the database and log disks. This poses a problem that we didn’t discover until later which I will discuss in a future post about replicating Exchange with NetApp.
Re-Architecting the environment to fit the storage
The first thing we discovered was that neither IBM/NetApp nor EMC would support the same host HBAs zoned to multiple brands of storage. So we had to split the ESX cluster into two clusters, one on each storage platform. Luckily the Exchange environment was isolated on it’s own six node cluster so it was easy to split everything in half.
Next we learned that due to NetApp’s updated active/active mode with proxy paths in ONTap 7.3, VMWare ESX 3.x randomly selects paths when rescanning HBAs and will pick non-optimized paths to the LUNs. This still works but is not ideal as it increases IO latency, causing the Filer to send autosupport emails periodically warning of the problem. Installing the NetApp Host Utilities for ESX onto the ESX hosts themselves allows you to run a script that assigns persistent paths evenly across the HBAs. The script works as advertised but as far as I can tell you have to run the script each time you add a new LUN to the ESX server. It would be much better if it were more automated.
Actually, if you are running ESX4.0 the scenario changes since NetApp ONTap 7.3+, Clariion FLARE 26+, and ESX4 all support ALUA making this problem all but disappear and improving fabric resiliency. Unfortunately for us, ESX4 is still a bit new and hasn’t been rolled out into production yet. NetApp also released tools for vCenter 4.0 that allow you to do the path assignment and other tasks from within vCenter rather than at the command line. EMC also now has PowerPath available for ESX4.0 which will not only manage paths but load balance across all paths for increased performance and lower latency.
VirtualStorageGuy has blogged already about the NetApp/EMC/vSphere plug-ins and there is even a Powerpoint available.
Finally, during the sales process NetApp pushed their de-duplication features (A-SIS) quite a bit and stressed how much disk space we could save in a VMWare environment. During deployment we were informed that if your VMs (VMDKs and VMFS) were not properly partition aligned de-duplication wouldn’t work well or at all. Since this environment has several hundred VMs built over several years by many people, and aligning the system (C:) drive of a Windows VM is difficult, the benefit would be minimal for us. Luckily NetApp has provided tools that can scan and align VMDKs without having to repartition the disks. We have not tested this yet. Partition Alignment is a best practice for ANY SAN storage system so we can’t fault NetApp for this problem; it’s just a fact of life.
But is it REALLY Redundant?
Even with two storage systems, with independent VMWare clusters, each hosting half of the Exchange cluster environment, a problem with either array could still take down and entire Exchange cluster. This is due to the File Share Witness (FSW) component used in a Majority Node Set (MNS) cluster like Exchange CCR. The idea behind the FSW in an MNS cluster is to prevent a condition known as Split Brain. Since a MNS cluster does not have a quorum disk, it relies entirely on network communication between the nodes to determine cluster status and make decisions about which nodes should become active. In the event that the two nodes lose communication with each other, each node will check for the FSW and if it is still available, it assumes that the other cluster node is down and proceeds to bring cluster resources online (if they weren’t already). Without the FSW, both nodes would potentially go active and there could be issues with inconsistent data, etc. This is the split-brain condition.
Typically, each cluster has a single FSW on a separate server (the CAS servers in our case). With the redundancy storage model we moved to, the FSW became a single point of failure. If we put the FSW on EMC storage with NodeA, and NodeB on the IBM/NetApp storage, a problem with the EMC array could take down both the cluster node AND the FSW at the same time. The surviving cluster node on the IBM/NetApp array would go down or stay down to prevent split-brain since the FSW was not available. Moving the FSW to the IBM/NetApp array presents the same problem on opposite side of the cluster. Incidentally, we proved this problem in lab testing to be sure. The solution is to move the FSW off of BOTH arrays, to either a dedicated physical server with internal disk, or a third storage array if you have one. There was a second EMC array in production so we moved the FSW there. In the new configuration, a complete outage of any single storage array would not take down the Exchange environment.
Crude diagram of the storage redundant Exchange CCR cluster
So far this new 3-way split environment is working fine, performance on the EMC and NetApp arrays is fine for Exchange. Using the same number of disks on the NetApp array yields about twice as much usable space as the EMC due to RAID-DP vs RAID-10 but overall performance is similar. Theoretically that means we could allow for more growth of the Exchange databases but in reality that is not always the case. My next update will be about Exchange replication using SnapManager and SnapMirror and how that has effectively negated the remaining free space in the NetApp aggregate.
Filed under: Uncategorized | Tags: 3par, dmp, dynamic provisioning, hds, hitachi, inserv, netapp, ntfs, performance, reclamation, snapdrive, storage, storage foundation, symantec, thin, thin provisioning, usp, utilization, veritas, vmware, vxfs, vxvm, windows, zero page reclaim
A comment about HDS’s Zero Page Reclaim on one of my previous posts got me thinking about the effectiveness of thin provisioning in general. In that previous post, I talked about the trade-offs between increased storage utilization through the use of thin-provisioning and the potential performance problems associated with it.
There are intrinsic benefits that come with the use of thin provisioning. First, new storage can be provisioned for applications without nearly as much planning. Next, application owners get what they want, while storage admins can show they are utilizing the storage systems effectively. Also, rather than managing the growth of data in individual applications, storage admins are able to manage the growth of data across the enterprise as a whole.
Thin provisioning can also provide performance benefits… For example, consider a set of virtual Windows servers running across several LUNs contained in the same RAID group. Each Windows VM stores its OS files in the first few GB of their respective VMDK files. Each VMDK file is stored in order in each LUN, with some free space at the end. In essence, we have a whole bunch of OS sections separated by gaps of no data. If all VMs were booting at approximately the same time, the disk heads would have to move continuously across the entire disk, increasing disk latency.
Now take the same disks, configured as a thin pool, and create the same LUNs (as thin LUNs) and the same VMs. Because thin-provisioning in general only writes data to the physical disks as it’s being written by the application, starting from the beginning of the disk, all of those Windows VMs’ OS files will be placed at the beginning of the disks. This increased data locality will reduce IO latency across all of the VMs. The effect is probably minor, but reduced disk latency translates to possibly higher IOPS from the same set of physical disks. And the only change is the use of thin-provisioning.
So back to HDS Zero Page Reclaim. The biggest problem with thin provisioning is that it doesn’t stay thin for long. Windows NTFS, for example, is particularly NOT thin-friendly since it favors previously untouched disk space for new writes rather than overwriting deleted files. This activity eventually causes a thin-LUN to grow to it’s maximum size over time, even though the actual amount of data stored in the LUN may not change. And Windows isn’t the only one with the problem. This means that thin provisioning may make provisioning easier, or possibly improve IO latency, but it might not actually save you any money on disk. This is where HDS’s Zero Page Reclaim can help. Hitachi’s Dynamic Provisioning (with ZPR) can scan a LUN for sections where all the bytes are zero and reclaim that space for other thin LUNs. This is particularly useful for converting thick LUNs into thin LUNs. But, it can only see blocks of zeros, and so it won’t necessarily see space freed up by deleting files. Hitachi’s own documentation points out that many file systems are not-thin friendly, and ZPR won’t help with long-term growth of thin LUNs caused by actively writing and then deleting data.
Although there are ways to script the writing of zeros to free space on a server so that ZPI can reclaim that space, you would need to run that script on all of your servers, requiring a unique tool for each operating system in your environment. The script would also have to run periodically, since the file system will grow again afterward.
NetApp’s SnapDrive tool for Windows can scan an NTFS file system, detect deleted files, then report the associated blocks back to the Filer to be added back to the aggregate for use by other volumes/LUNs. The Space Reclamation scan can be run as needed, and I believe it can be scheduled; but, it appears to be Windows only. Again, this will have to be done periodically.
But what if you could solve the problem across most or all of your systems, regardless of operating system, regardless of application, with real-time reclamation? And what if you could simultaneously solve other problems? Enter Symantec’s Storage Foundation with Thin-Reclamation API. Storage Foundation consists of VxFS, VxVM, DMP, and some other tools that together provide dynamic grow/shrink, snapshots, replication, thin-friendly volume usage, and dynamic SAN multipathing across multiple operating systems. Storage Foundation’s Thin-Reclamation API is to thin-provisioning what OST is to Backup Deduplication. Storage vendors can now add near-real-time zero page reclaim for customers that are willing to deploy VxFS/VxVM on their servers. For EMC customers, DMP can replace PowerPath, thereby offsetting the cost.
As far as I know, 3PAR is the first and only storage vendor to write to Symantec’s thin-API, which means they now have the most dynamic, non-disruptive, zero-page-reclaim feature set on the market. As a storage engineer myself, I have often wondered if VxVM/VxFS could make management of application data storage in our diverse environment easier and more dynamic. Adding Thin-Reclamation to the mix makes it even more attractive. I’d like to see more storage vendors follow 3PAR’s lead and write to Symantec’s API. I’d also like to see Symantec open up both OST and the Thin-Reclamation API for others to use, but I doubt that will happen.
Filed under: comparisons, technologies | Tags: clariion, celerra, netapp, vmware, esx, fas, ibm, nseries, nas, san, de-duplication, filerview, navisphere, filer, controller, multistore, management, alua
In the last post, I talked about a project I am involved in right now to deploy NetApp storage alongside EMC for SAN and NAS. Today, I’m going to talk about my first impressions of the NetApp during deployment and initial configuration.
First Impressions
I’m going to be pretty blunt — I have been working with EMC hardware and software for a while now, and I’m generally happy with the usability of their GUIs. Over that time, I’ve used several major revisions of Navisphere Manager and Celerra Manager, and even more minor revisions, and I’ve never actually found a UI bug. To be clear, EMC, IBM, NetApp, HDS, and every other vendor have bugs in their software, and they all do what they can to find and fix them quickly, but I just haven’t personally seen one in the EMC UIs despite using every feature offered by those systems. (I have come across bugs in the firmware)
Contrast that with the first day using the new NetApp, running the latest 7.3.1.1L1 code, where we discovered a UI problem in the first 10 minutes. When attempting to add disks to an aggregate in FilerView, we could not select FC disk to add. We could, however, add SATA disk to the FC aggregate. The only way to get around the issue was to use the CLI via SSH. As I mentioned in my previous post, our NetApp is actually an IBM nSeries, and IBM claims they perform additional QC before their customers get new NetApp code.
Shortly after that, we found a second UI issue in FilerView. When creating a new Initiator group, FilerView populates the initiator list with the WWNs that have logged in to it. Auto-populating is nice but the problem is that FilerView was incorrectly parsing the WWN of the server HBAs and populating the list with NodeWWNs rather than PortWWNs. We spent several hours trying to figure out why the ESX servers didn’t see any LUNs before we realized that the WWNs in the Initiator group were incorrect. Editing the 2nd digit on each one fixed the problem.
I find it interesting that these issues, which seemed easy to discover, made it through the QC process of two organizations. ONTap 7.3.2RC1 is available now, but I don’t know if these issues were addressed.
Manageability
As far as FilerView goes, it is generally easy to use once you know how NetApp systems are provisioned. The biggest drawback in an HA-Filer setup is the fact you have to open FilerView separately for each Filer and configure each one as a separate storage system. Two HA-Filer pairs? Four FilerView windows. If you include the initial launch page that comes up before you get to the actual FilerView window, you double the number of browser windows open to manage your systems. NetApp likes to mention that they have unified management for NAS and SAN where EMC has two separate platforms, each with their own management tools. EMC treats the two storage processors (SPs) in a Clariion in a much more unified manner, and provisioning is done against the entire Clariion, not per SP. Further, Navisphere can manage many Clariions in the same UI. Celerra Manager acts similarly for EMC NAS. Six of one, half a dozen of the other some say, except that I find that I generally provision NAS storage and SAN storage at different times, and I’d rather have all of the controllers/filers in the same window than NAS and SAN in the same window. Just my preference.
I should mention, NetApp recently released System Manager 1.0 as a free download. This new admin tool does present all of the controllers in one view and may end up being a much better tool than FilerView. For now, it’s missing too many features to be used 100% of the time and it’s Windows only since it’s based on MMC. Which brings me to my other problem with managing the NetApp. Neither FilerView nor System Manager can actually do everything you might need to do, and that means you end up in the CLI, FREQUENTLY. I’m comfortable with CLIs and they are extremely powerful for troubleshooting problems, and especially for scripting batch changes, but I don’t like to be forced into the CLI for general administration. GUI based management helps prevent possibly crippling typos and can make visualizing your environment easier. During deployment, we kept going back and forth between FilerView and CLI to configure different things. Further, since we were using MultiStore (vFilers) for CIFS shares and disaster recovery, we were stuck in the CLI almost entirely because System Manager can’t even see vFilers, and FilerView can only create them and attach volumes.
Had I not been managing Celerra and Clariion for so long, I probably wouldn’t have noticed the above problems. After several years of configuring CIFS, NFS, iSCSI, Virtual DataMovers, IP Interfaces, Snapshots, Replication, and DR Failover, etc. on Celerra, as well as literally thousands of LUNs for hundreds of servers on Clariion, I don’t recall EVER being forced to use the CLI. CelerraCLI and NaviCLI are very powerful, and I have written many scripts leveraging them, and I’ll use CLI when troubleshooting an issue. But for every single feature I’ve ever used on the Celerra or Clarrion, I was able to completely configure from start to finish using the GUI. Installing a Celerra from scratch even uses a GUI based installation wizard. Comparing Clariion Storage Groups with NetApp Initiator groups and LUN maps isn’t even fair. For MS Exchange, I mapped about 50 LUNs to the ESX cluster, which took about 30 minutes in FilerView. On the Clariion, the same operation is done by just editing the Storage Group and checking each LUN, taking only a couple minutes for the entire process.
Now, all of the above commentary has to do with the management tools, UIs, and to some degree personal preferences, and does not have any bearing on the equipment or underlying functionality. There are, of course, optional management tools like Operations Manager, Provisioning Manager, and Protection Manager available from NetApp, just as there is Control Center from EMC (which incidentally can monitor the NetApp) or Command Central from Symantec. Depending on your overall needs, you may want to look at optional management tools; or, FilerView may be perfectly fine.
In the next post, I’ll get into more specifics about how the Exchange 2007 CCR cluster turned out in this new environment, along with some notes on making CCR truly redundant. I’ve also been working on the NAS side of the project, so I’ll also post about that some time soon.
Filed under: technologies | Tags: celerra, netapp, emc, fas, ibm, nseries, technical, comparison, nas, san, block, cifs, nfs, iscsi, fiberchannel, coexistence
I’ve been tasked recently on a project to increase availability of applications through the use of multiple/disparate storage systems. This environment has heavily invested in EMC Clariion and Celerra storage systems over the past few years and needed a non-EMC platform from which to build the second half of a redundant storage environment. For various reasons I won’t go into here, we chose IBM nSeries as that second platform. (Since the IBM system is rebranded NetApp FAS, I will refer to this as a NetApp filer.) I’ve been working on implementing the new equipment as well as integrating it into the Business Continuity strategy.
The overall strategy is to continue to use the EMC Clariion/Celerra systems for production and disaster recovery replication and split applications between and across the two storage platforms for local redundancy. The NetApp will also perform disaster recovery replication for some of the applications. Here’s a really simple diagram that might help if the description is confusing:
EMC and NetApp Redundancy
Now this may sound easy, but it is, in fact, NOT straightforward. This strategy requires close coordination with application owners and careful planning. As we move forward on this project, I’ll talk about various idiosyncrasies, caveats, and problems we’ve faced, how we got around them, and I’ll also talk a lot about the differences between the Clariion/Celerra and NetApp platforms’ features and functionality, application support, and manageability. These comparisons will include using both systems with FiberChannel connections as well as CIFS/NFS NAS, all in conjunction with DR replication and failover.
To start off, I figure we should compare some of the terminology between EMC and NetApp systems. Some terms don’t directly translate, but I matched them up as close as I could and noted where there is no equivalent. Below are two tables: one for Block Storage, and the other for NAS Storage. Click on them to see full size versions.
EMC-NetApp Block Storage Terminology
EMC-NetApp NAS Storage Terminology
In the next update, I’ll start talking about the deployment itself. The point of these articles is to discuss the differences, advantages, and disadvantages of each platform so that you can understand how each one might work in your environment. I do not intend to disparage either platform or vendor. I will try to be vendor agnostic as much as possible, and I do feel like I have a somewhat unique position of comparing new and recent hardware and firmware from both vendors, in the same production capacities, simultaneously, in the same environment. I am NOT comparing old ONTap code to new FLARE/DART code or vise-versa, nor am I comparing old Clariion CX hardware to new NetApp/IBM hardware, etc.
Stay tuned!
Filed under: problems, solutions | Tags: clariion, clone, emc, expand, expanding, lun, migration, sancopy, snapview, thin, thin provisioning
Do you have an EMC Clariion CX4 with Virtual Provisioning (thin provisioning)? Have you tried to expand the host visible (ie: maximum) size of a thin-LUN and can’t figure out how? Well you aren’t alone… Despite extremely little information available from EMC to say one way or the other, I finally figured out that you actually can’t expand a thin lun yet. This was a surprise to me, since I had just assumed that would be there. Thin-LUNs are pretty much virtual LUNs, and as such, they don’t have any direct block mapping to a RAID group that has to be maintained. And traditional LUNs can be expanded using MetaLUN striping or concatenation. Due to this restriction the host visible size of a thin-LUN cannot be edited after the LUN has been created.
But there IS a workaround. It’s not perfect but its all we have right now. Using CLARiiON’s built-in LUN Migration technology you can expand a thin-LUN in two steps.
Step 1: Migrate the thin-LUN to a thick-LUN of the same maximum size.
Step 2: Migrate the thick-LUN to a new thin-LUN that was created with the new larger size you want. After migration, the thin-LUN will consume disk space equivalent to the old thin-LUN’s maximum size, but will have a new, higher host maximum visible size.
Two Steps to expand a thin LUN
This requires that you have a RAID group outside of your thin pools that has enough usable free space to fit the temporary thick-LUN, so it’s not a perfect solution. You’d think that migrating the old thin-LUN directly into a new, larger thin-LUN would work, but to use the additional disk space after a LUN migration, you have to edit the LUN size which, again, can’t be done on thin-LUNs. I haven’t actually tested this, but this is based on all of the documentation I could find from EMC on the topic.
I’m looking into other methods that might be better, but so far it seems that certain restrictions on SnapView Clones and SANCopy might preclude those from being used. The ability to expand thin-LUNs will come in a later FLARE release for those that are willing to wait.
Filed under: Uncategorized | Tags: avamar, crab, datadomain, de-duplication, deadliest catch, discovery, emc, keith, silverpeak, wizard
I attended an event tonight put on by Optistor Technologies, a local vendor, which was centered on data de-duplication. Reps from EMC, DataDomain, and SilverPeak were there, chatting with customers about how their respective products leverage de-duplication technology to save you money. The keynote speaker was Keith Colburn, captain of the Wizard, from Discovery Channel’s Deadliest Catch and they had paired the evening with a spread of crab and prawns to snack on. Anyway, Keith admitted during his speech that he “doesn’t know a thing about WAN, re-dupe (he meant de-dupe), or any of that stuff” but I think he did an admirable job discussing the importance of picking the right vendors to work with, ones who have adequate support resources and top notch technology. He related some of the problems he’s had in the Bering Sea and how he attempts to mitigate risk as much as possible but sometimes discovers problems that he never anticipated. That’s where the tie-ins with vendor choice came together. All in all it was a fun event and I got a chance to catch up with some of the engineers and account managers I’ve dealt with in the past couple years.
Filed under: efficiency, problems, solutions, technologies | Tags: storage, deduplication, celerra, netapp, datadomain, puredisk, netbackup, PDDO, ndmp, ost, lifecycle
Okay, now that I’ve talked about backing up the datacenter with NetBackup and DataDomain, and backing up remote sites with NetBackup and PureDisk, it’s time to discuss how to get all that data offsite to protect against a catastrophic event at the datacenter.
As mentioned before we have a primary datacenter with the majority of our systems including the backup environment, and a secondary “disaster recovery” datacenter to which we replicate tier 1 applications for business continuity purposes. Since we really wanted to get away from using tapes and instead store the backups on disk in our datacenter we have a second backup environment in the DR datacenter and we replicate the backup data there.
There are several ways to replicate backup data between two sites but most of them have drawbacks..
1.) Duplicate the backup data from disk to tape and ship the tapes to the remote site to be ready for restore. This is the easiest and probably cheapest way. But there’s that pesky tape yet again with it’s media handling and shipping. And restore could take a while since you have to deal with restoring the catalog from tape, then importing the media, etc.
2.) Duplicate the backup data directly from the local disk to the disk in the second location across the WAN. This is not very feasible with any significant amount of data because every byte of data that is backed up in the datacenter has to be copied across the much slower WAN. It could take many days to duplicate a single nights’ backup. You’d also need a special Catalog backup job that wrote to a storage device across the WAN. The good here is that the backup application knows there is a second copy of the data and knows how to find it.
3.) Replicate the data with the backup storage devices’s native replication. Whether it’s PureDisk, Avamar, or DataDomain, pretty much every source-based or target-based deduplication solution has replication built in that leverages the deduplication to reduce the amount of data that traverses the WAN. The advantage here is that you can have a copy of all of your backup data in a second location in a much shorter time than a traditional copy process. If your deduplication device stores the data with 10:1 compression, then your WAN usage is reduced by 90%. The savings in practice is actually better than that. The drawback is that the backup application (hence the catalog) has no knowledge that there is a second copy of the backup data and after recovering the catalog, you would need to import all of the disk-based media which could take a long time.
4.) Leverage NetBackup Lifecycle Policies with Symantec OpenSTorage (OST) and an OST-capable backup storage system like DataDomain or PureDisk(with PDDO). Basically this has all the advantages of option #2, where it is a catalog-aware duplication, combined with the advantages of WAN bandwidth savings from option #3. Time to copy the data offsite is much shorter due to deduplication, and time to restore is very fast since the data is already in the catalog and available on the disk.
OpenSTorage (OST) is a network protocol that Symantec developed to interface with disk-based backup storage systems and DataDomain was an early adopter of OST. OST allows Netbackup to control replication between OST-capable storage systems and keep track of the replicated copies of backups in the Catalog just as if Netbackup had made both copies itself. OST is also used as the protocol to send the backup data to the storage device as opposed to CIFS/NFS or VTL. DataDomain appliances support OST as does PureDisk when used in conjunction with the PDDO option discussed earlier. In NetBackup, replication controlled by OST is called “optimized duplication” and is controlled primarily through Lifecycle Policies.
Traditionally, when creating NetBackup job policies, the administrator will specify a Storage Unit (either a disk storage unit or a tape library or drive) that the job policy will send backups to. Lifecycle Policies are treated like Storage Units as far as the Job Policy is concerned but the Lifecycle Policy includes a list of storage units, each with it’s own data retention, that the backup data must be stored onto in order for NetBackup to consider the data fully protected. Typically there is a “Backup” target which is where the actual data coming from the client is stored, followed by one or more “Duplication” targets. After the backup job completes, NetBackup will copy the backup data from the “backup” location to all of the “duplication” locations. This works with pretty much any type of storage and you can mix and match tape and disk in the same policy. Since these are duplication operations, NetBackup will read ALL of the data from the backup location, and write ALL of the data to each duplication location. This can take a long time even on the local network and trying to offsite a lot of data over the WAN is not very feasible.
With OST, the lifecycle policy operates exactly the same except that it uses “optimized duplication”, instructing the storage device to copy the file rather than performing the copy through a media server. So in the case of DataDomain, OST issues the command to the DDR, the DDR then copies the file to the second DDR in the remote site and gets all the benefits of deduplication and compression between the two. The media server doesn’t actually do any work. Once the duplication is complete, the DDR notifies NetBackup and the catalog is updated with a record of the second copy of the backup. Lifecycle Policies are fully automated, you can’t even restart a failed duplication, so in the event of a transient failure like a WAN hiccup NetBackup will retry a duplication job forever until it succeeds in order to satisfy the lifecycle policy.
As you can probably surmise, this is REALLY nice for a tape-less backup environment. Our DD690 offsites over 9TB of data every night DURING the backup window. When the last backup job completes, the offsite copies are complete within 30 minutes. And there is absolutely no management of the offsite process or duplication jobs besides configuring the lifecycle policies up front. The drawback to regular Netbackup lifecycle policies is that all duplications are taken from the initial backup copy which limits what you can do with the copies.
Enter NetBackup 6.5.4… Despite the small 6.5.3 -> 6.5.4 version number change, the 6.5.4 release had quite a few new features added. The biggest one was a revamping of the Lifecycle Policy engine to allow for nested duplications. Now you can create a copy of a backup, then create multiple copies from the copy, then create copies from the other copies. Why is this useful?
Remember when I discussed using NetBackup with PDDO to backup remote sites? Well the data backed up from the remote site is all stored in the primary datacenter and we need to get the second copy to the DR datacenter. Plus, we wanted to have a small cache of recently backed up data sitting on the remote media server for fast restore. Well, nested lifecycle’s are the key. The lifecycle writes the initial backup copy onto the media server’s local disk which is configured as a capacity-managed staging area (ie: it stores as much as it can and expires data when it needs more space for new backups). The lifecycle then creates a duplicate of the backup onto the PureDisk storage unit in the primary datacenter. Since bandwidth to the remote site is very limited we don’t want to copy it from the remote site twice so the lifecycle has a second duplication nested under the first to copy it to the DR datacenter. The source of the second copy is the primary datacenter copy, NOT the remote media server copy.
Where else can we use this? Let’s consider our tape-less datacenter backups.. We backup the clients to the DataDomain in our primary datacenter, then using a lifecycle policy and OST, create a copy on the DataDomain in the DR datacenter. If we also wanted to have a tape copy for long term archive or vaulting we could create a nested duplication to make a copy to a tape library in the DR datacenter from the disk copy that is also in the DR datacenter. Without nested lifecycle’s the only workable solution would be to create the tape in the primary datacenter. Every copy of the backup made via the lifecycle policy whether it is using OST or not is maintained by the catalog and easily used for restore. Furthermore, using OST as the protocol between Netbackup and DataDomain actually increases throughput to the DataDomain DDR systems by approximately 2X vs VTL/CIFS/NFS.
Now to the caveats.. Optimized duplication via OST is only available when you are using OST as the protocol between the media server and the storage unit. This means it doesn’t work with VTL even when the DataDomain IS the VTL. OST only works over an ethernet network which is why we skipped VTL completely and used 10gbps networks for the DDR connections. We even skipped VTL/Tape for the NAS systems, connected them directly to the 10gbps network and use 3-way NDMP to backup them up over the network, through the media servers, to the DataDomain. We get the benefit of lifecycle policies, optimized duplication, and I may have mentioned before–no pesky tape even with NDMP/NAS backups. And the interesting thing is that with the 10gbps connection, the NDMP dumps are faster than direct fiber to tape.
There were other enhancements to NetBackup 6.5.4 centered around OST functionality but the lifecycle policy improvements were huge in my opinion.
To cover the catalog replication, we run Netbackup hot catalog backups to a CIFS share that is hosted by the DataDomain. The DDR replicates that share using DataDomain native replication to the DDR in the DR datacenter where the same data is available via a similar CIFS share. Our standby Netbackup master server is already connected to the CIFS share for catalog restore and connected to the DDR via OST. A single operation restores the catalog from the replicated copy. In a real disaster we can begin restoring user data within 30 minutes from the DR datacenter.
Filed under: efficiency, problems, solutions, technologies | Tags: avamar, compression, deduplication, netbackup, networker, PDDO, performance, puredisk, storage
In a previous post I discussed the new backup environment I’ve been deploying, what solutions we picked, and how they apply to the datacenter. But I also mentioned that we had remote sites with systems we need to back up but I didn’t explain how we addressed them. Frankly, the previous post was getting long and backing up remote offices is tricky so it deserved it’s own discussion.
Now that we had Symantec NetBackup running in the datacenter, backup up the bulk of our systems to disk by way of DataDomain, we need to look at remote sites. For this we deployed Symantec NetBackup PureDisk. Despite the fact that it has NetBackup in the name, PureDisk is an entirely different product with it’s own servers, clients, and management interfaces. There are some integration points that are not-obvious at first but become important later. Essentially PureDisk is two solutions in a single product — 1:) a “source-dedupe” backup solution that can be deployed independent of any other solution, and 2:) a “target-dedupe” backup storage appliance specifically integrated with the core NetBackup product via an option called PDDO.
As previously discussed, backing up a remote site across a WAN is best accomplished with a source-dedupe solution like PureDisk or Avamar. This is exactly what we intended to do. Most of our remote site clients are some flavor of UNIX or Windows and installing PureDisk clients was easily accomplished. Backup policies were created in PureDisk and a little over a day later we had the first full backup complete. All subsequent nightly backups transfer very small amounts of data across the WAN because they are incremental backups AND because the PureDisk client deduplicates the data before sending it to the PureDisk server. The downside to this is that the PureDisk jobs have to scheduled, managed, and monitored from the PureDisk interface, completely separate from the NetBackup administration console. Backups are sent to the primary datacenter and stored on the local PureDisk server, then the backed up data is replicated to the PureDisk server in the DR datacenter using PureDisk native replication. Restores can be run from either of the PureDisk servers but must un-deduplicate the data before sending across the WAN making restores much slower than backups. This was a known issue and still meets our SLAs for these systems.
Our biggest hurdle with PureDisk was the client OS support. Since we have a very diverse environment we ran into a couple clients which had operating systems that PureDisk does not support. Both Netware and x86 versions of Solaris are currently not supported, both of which were running in our remote sites.
We had a few options:
1.) Use the standard NetBackup client at the remote site and push all of the data across the WAN
2.) Deploy a NetBackup media server in the remote site with a tape library and send the tapes offsite
3.) Deploy a NetBackup media server in the remote site with a small DataDomain appliance and replicate
4.) Deploy a NetBackup media server and ALSO use PureDisk via the PDDO option (PureDisk Deduplication Option)
Option 1 is not feasible for any serious amount of data, Option 2 requires a costly tape library and some level of media handling every day, and Option 3 just plain costs too much money for a small remote site.
Option 4, using PDDO, leverages PureDisk’s “target-dedupe” persona and ends up being a very elegant solution with several benefits.
PDDO is a plug-in that installs on a Netbackup media server. The PDDO plug-in deduplicates data that is being backed up by that media server and sends it across the network to a PureDisk server for storage. The beauty of this option is that we were able to put a Netbackup media server in our remote site without any tape or other storage. The data is copied from the client to the media server over the LAN, de-duplicated by PDDO, then sent over the WAN to the datacenter’s PureDisk server. We get the bandwidth and storage efficiencies of PureDisk while using standard NetBackup clients. A byproduct of this is that you get these PureDisk benefits without having to manage the backups in PureDisk’s separate management console. To reduce the effects of the WAN on the performance of the backup jobs themselves, and to make the majority of restores faster, we put some internal disk on the media server that the backup jobs write to first. After the backup job completes to the local disk, NetBackup duplicates the backup data to the PureDisk storage server, then duplicates another copy to the DR datacenter. This is all handled by NetBackup lifecycle policies which became about 1000X more powerful with the 6.5.4 release. I’ll discuss the power of lifecycle policies, specifically with the 6.5.4 release, when I talk about OST later.
So the result of using PureDisk/PDDO/NetBackup together is a seamless solution, completely managed from within NetBackup, with all the client OS support the core NetBackup product has, the WAN efficiencies of source-dedupe, the storage efficiencies of target-dedupe, and the restore performance of local storage, but with very little storage in the remote site.
Remote Site Backup… Done!!
For the near future, I’m considering putting NetBackup media servers with PDDO on VMWare in all of the remote sites so I can manage all of the backups in NetBackup without buying any new hardware at all. This is not technically supported by Symantec but there is no tape/scsi involved so it should work fine. Did I mention we wanted to avoid tape as much as possible?
Incidentally, despite my love for Avamar, I don’t believe they have anything like PDDO available in the Networker/Avamar integration and Avamar’s client OS support, while better than PureDisk’s, is still not quite as good as Netbackup and Networker.
Okay, so how does OST play into NetBackup, PureDisk, PDDO, and DataDomain? What do the lifecycle policies have to do with it? And what is so damned special about lifecycle policies in NetBackup 6.5.4? All that is next…
