EMC Avamar v5.0 Storage Based Backup Implementation
Closed     Case # 10048     Affiliated Job:  New Trier Township District 2031
Opened:  Tuesday, February 1, 2011     Closed:  Monday, February 28, 2011
Total Hit Count:  52599     Last Hit:  Sunday, July 21, 2024 6:19:47 PM
Unique Hit Count:  11432     Last Unique Hit:  Sunday, July 21, 2024 6:19:47 PM
Case Type(s):  Server, Vendor Support
Case Notes(s):  All cases are posted for review purposes only. Any implementations should be performed at your own risk.

After discussion of the future of our backup strategy utilizing Backup Exec; we finally decided to move forward with an EMC Avamar solution. EMC performed an assessment of our network and determined we would be best fit with an Avamar "DS7" grid consisting of a Utility node, Spare node and 5 Data nodes. Under a new platform of Avamar v5.0, each data node includes 3.3 TBs of available storage space.

Since we have nearly virtualized our entire server environment in our array of 5 host servers running the latest and greatest ESX 4.1 vSphere (which we have since learned of EMCs plans to do away with ESX and shift to ESXi for version 5) along with the storage of our infrastructure on an EMC CX-120 - we are good to move forward with a storage based backup solution.

We hired the help of a certified partner of EMC to perform the major install; however, the partner consulting company encountered a great number of difficulties with the hardware provisioned and after a week of troubleshooting, ended up replacing two nodes along with deploying EMC teches to return the grid to a fresh from scratch state.

Therefore, in our planned week worth of time we were only provisioned two actual days, "Day 1" for install and "Day 2" to briefly go over the high level details & job configuration. So below are lessons learned through work I have had to perform to troubleshoot and self-educate various implementations either left uncompleted or missed during the "install."

The consultant performed the install on the nodes individually, then through the "wizard" brought the grid online into a collective state; set the Avamar utility node to trust the root certificate of the vCenter server, setup the initial Avamar proxy (which captures the image level backups of a virtualized machine); imported the VM server environment into Avamar; created the requested datasets, schedules, retention policies; explained the various agents to be installed for File based (rather than image based backups which permits file exclusions, restore permissions with files etc.), SQL, Exchange; LDAP authentication into our A.D. infrastructure and SMTP notifications of "Call Home" to EMC as well as sending us server notifications.

What the consultant failed to explain or setup were a couple items we then figured out on our own. When he finished (on "Day 2") we began an "On Demand" on a couple backups to begin while keeping all others in a disabled state. And before the consultant left we had arranged a return date for him to explain the "how to" restore, review the backed up jobs (success/failure), Avamar reporting, change root/admin passwords, shutting down the grid, backing up of external NAS resources, setup of additional Avamar proxy servers (each proxy performs a single thread - to enhance backup time EMC suggests a proxy per VM host), review of the "System State" File Agent backups on Windows 2003 (Which we discovered were NTBACKUP bkf files being placed on the C volume and in some cases were consuming what little space was left)

To address the above, a great majority had to be performed on our own - simply because our consultant had already spent the 5 days budgeted on our contract building the original units only to find the failed hardware components and therefore had to spend an additional two days to reconfigure the two new nodes with the original 5 other nodes all of which were returned to factory settings after repair. Because of this, the consultant neglected to return on the date arranged and instead the questions were briefly explained through email rather than demonstrated.

Action(s) Performed:
Total Action(s): 4
Action # Recorded Date Type Hit(s) User Expand Details
10165 2/28/2011 2:37:07 PM Server 3903 contact@danieljchu.com Image Level File Level Restores Some of the lessons learned, most i  More ...
10166 2/28/2011 2:36:07 PM Server 3583 contact@danieljchu.com Add Additional Avamar Image Backup Proxies The creation of addition  Collapse ...
Last Hit: Sunday, July 21, 2024 6:13:20 PM

Add Additional Avamar Image Backup Proxies
The creation of additional Avamar proxies are pretty easy, use the OVA VM Appliance file to install a new Avamar VM proxy (downloaded from the website on the Utility node), boot it and follow the prompts to set it up, once added into the Avamar configuration (part of the prompts followed after boot) they will appear in the Avamar client - two things need configured before they will function. #1, in your Policy section of the Avamar client, proceed through each of your Image level backups and check the proxy just created to be pooled from. #2, under the Administration section of the Avamar client, "Edit" the newly added Avamar proxy and check each of the "vSphere datastores to protect" in the list provided. After these two configuration updates, image level backups will now include this new proxy. I also setup a DRS rule to keep the Avamar proxies separated on the VM hosts in the 5 server cluster.

What is interesting about these image level backups, the 1st thing it does is creates a snapshot of the VM, attaches this snapshot to the active VM and attaches the live vmdk to the Avamar Image proxy. Once it completes it seems to merge this updated data found in the snapshot back into the live vmdk file.
10168 2/28/2011 2:35:07 PM Server 3392 contact@danieljchu.com Image Level Backup Failures Image level backups on VMs that have a   More ...
10167 2/28/2011 2:34:07 PM Server 3696 contact@danieljchu.com Backup Performance A couple things to mention, when performing back  More ...

Profile IMG: Footer Left Profile IMG: Footer Right