EMC Avamar v5.0 Storage Based Backup Implementation
Closed     Case # 10048     Affiliated Job:  New Trier Township District 2031
Opened:  Tuesday, February 1, 2011     Closed:  Monday, February 28, 2011
Total Hit Count:  41470     Last Hit:  Tuesday, December 7, 2021 10:26:32 AM
Unique Hit Count:  8362     Last Unique Hit:  Tuesday, December 7, 2021 10:26:32 AM
Case Type(s):  Server, Vendor Support
Case Notes(s):  All cases are posted for review purposes only. Any implementations should be performed at your own risk.

After discussion of the future of our backup strategy utilizing Backup Exec; we finally decided to move forward with an EMC Avamar solution. EMC performed an assessment of our network and determined we would be best fit with an Avamar "DS7" grid consisting of a Utility node, Spare node and 5 Data nodes. Under a new platform of Avamar v5.0, each data node includes 3.3 TBs of available storage space.

Since we have nearly virtualized our entire server environment in our array of 5 host servers running the latest and greatest ESX 4.1 vSphere (which we have since learned of EMCs plans to do away with ESX and shift to ESXi for version 5) along with the storage of our infrastructure on an EMC CX-120 - we are good to move forward with a storage based backup solution.

We hired the help of a certified partner of EMC to perform the major install; however, the partner consulting company encountered a great number of difficulties with the hardware provisioned and after a week of troubleshooting, ended up replacing two nodes along with deploying EMC teches to return the grid to a fresh from scratch state.

Therefore, in our planned week worth of time we were only provisioned two actual days, "Day 1" for install and "Day 2" to briefly go over the high level details & job configuration. So below are lessons learned through work I have had to perform to troubleshoot and self-educate various implementations either left uncompleted or missed during the "install."

The consultant performed the install on the nodes individually, then through the "wizard" brought the grid online into a collective state; set the Avamar utility node to trust the root certificate of the vCenter server, setup the initial Avamar proxy (which captures the image level backups of a virtualized machine); imported the VM server environment into Avamar; created the requested datasets, schedules, retention policies; explained the various agents to be installed for File based (rather than image based backups which permits file exclusions, restore permissions with files etc.), SQL, Exchange; LDAP authentication into our A.D. infrastructure and SMTP notifications of "Call Home" to EMC as well as sending us server notifications.

What the consultant failed to explain or setup were a couple items we then figured out on our own. When he finished (on "Day 2") we began an "On Demand" on a couple backups to begin while keeping all others in a disabled state. And before the consultant left we had arranged a return date for him to explain the "how to" restore, review the backed up jobs (success/failure), Avamar reporting, change root/admin passwords, shutting down the grid, backing up of external NAS resources, setup of additional Avamar proxy servers (each proxy performs a single thread - to enhance backup time EMC suggests a proxy per VM host), review of the "System State" File Agent backups on Windows 2003 (Which we discovered were NTBACKUP bkf files being placed on the C volume and in some cases were consuming what little space was left)

To address the above, a great majority had to be performed on our own - simply because our consultant had already spent the 5 days budgeted on our contract building the original units only to find the failed hardware components and therefore had to spend an additional two days to reconfigure the two new nodes with the original 5 other nodes all of which were returned to factory settings after repair. Because of this, the consultant neglected to return on the date arranged and instead the questions were briefly explained through email rather than demonstrated.

Action(s) Performed:
Total Action(s): 4
Action # Recorded Date Type Hit(s) User Expand Details
10165 2/28/2011 2:37:07 PM Server 3065 contact@danieljchu.com Image Level File Level Restores Some of the lessons learned, most i  More ...
10166 2/28/2011 2:36:07 PM Server 2770 contact@danieljchu.com Add Additional Avamar Image Backup Proxies The creation of addition  More ...
10168 2/28/2011 2:35:07 PM Server 2667 contact@danieljchu.com Image Level Backup Failures Image level backups on VMs that have a   Collapse ...
Last Hit: Tuesday, December 7, 2021 10:26:20 AM

Image Level Backup Failures
Image level backups on VMs that have a block size smaller for where the VM's vmx configuration file is versus the LUN containing drives attached to the VM, for example our main file server has a 1MB block size on the LUN configured with the VMX config file also including the C drive of the VM while also consisting of a 1.5 TB vmdk attached on a LUN with a 8MB block size elsewhere. When attempting to perform a backup of these type of VMs, they fail with shapshot errors.

When errors like this occur, you may need to point the VM's snapshot location onto a LUN with the space required to create temporary snapshot files that can be provisioned the largest type of vmdk file associated with the VM. This can be any LUN even one unrelated to the VM entirely so long as the VM host has access to the specific LUN. Two lines to configure this setting in the VMX file are:
-   workingDir = "/vmfs/volumes/[Random UUID of Snapshot LUN]/vm-snapshots"
-   sched.swap.dir = "/vmfs/volumes/[Random UUID of LUN with VMX Config File]/[ORIGINAL-VM-FOLDER]"

Adjust the UUID's above to reflect either the snapshot LUN (line 1) or the LUN where the VMX file is located (line 2) with the appropriate VM folder to follow. The first line directs all working files (including the snapshot files) to that path. The second line redirects the swap file back into the same folder where the VMX configuration file is (the default and only changed when you add the first line)

After shutting down the VM, adding these two lines to the vmx file, removing from inventory the VM and adding the VM back to inventory and starting it back up - subsequent image level backups have all since been successful.
10167 2/28/2011 2:34:07 PM Server 2828 contact@danieljchu.com Backup Performance A couple things to mention, when performing back  More ...

Profile IMG: Footer Left Profile IMG: Footer Right