Archives For Backup

Just found this demonstration video of Veeam’s new feature coming in version 5 of their software.

I really can’t wait to start using this technology, Veeam is already my favourite backup software due to its ease of use and reliability and they continue to push the boundaries and improve the functionality they offer. With the introduction of this technology it will allow you to verify your backups every time on every VM and not just the odd one or two a month.

For more information about SureBackup please visit their website

http://www.veeam.com/surebackup/

Veeam has today released version 4.0 of its Reporter product. The new features in version 4 are as follows.

Customizable dashboards
The new, redesigned web UI (user interface) also allows you to easily create and customize your own dashboard.

Performance and events
Veeam Reporter 4.0 now collects historical performance data from vCenter to facilitate capacity planning, chargeback, trend reporting, and troubleshooting of recurring performance issues.

Capacity planning
In addition to the base report pack, Veeam Reporter 4.0 includes a new report pack for capacity planning.

Configuration management
Veeam Reporter 4.0 is the first to provide configuration reporting on vSphere ESX4 Host Profiles.

Improved change reporting—who, what, where, when
Full Business View integration. Veeam Business View now integrates directly with Veeam Reporter’s web UI so you can easily view virtual machines by business unit, department, location, purpose, service level agreement, type of server, or any other criteria you define.

Improved Visio diagrams
Diagrams of the virtual infrastructure have been redesigned to accommodate environments with thousands of virtual machines. Beta customers report being able to print and clearly read diagrams with 6000+ virtual machines.

Video

More Information

I haven’t personally used Veeam Reporter in a while but have always been impressed by the Visio intergration. For more information be sure to check out the product page on Veeam website >> http://www.veeam.com/webinars.html. As soon as I have had a chance to have a look at the latest version I will blog my findings.

Before I start this post I will start with a bit of a caveat, I don’t want this post to turn into a slagging match between backup vendors. The contents of this post is simply what I have found from my tests and experience. If you believe your backup product could easily match this functionality or better it please email me on barry(a)virtualisedreality.com with the information and I will happily test and blog my findings.

Introduction

I deal quite extensively with small and medium businesses and education from companies with 30-50 customers up to customers with over a thousand users. These customers usually have a lot in common when it comes to backups, the first is that they can’t afford to spend huge budgets on backups after virtualising their environments, second is what ever solution is delivered for backups must be reliable and easy to use and finally it must be able to complete within the backup window.

For the larger SMB’s and enterprises this can often come in the form of offsite SAN replication and archival and other such non budget friendly solutions. For most SMB’s this normally means we are having to reuse existing hardware or cheap NAS / Server storage for backup staging and archival.

The Problem

Although these customer may only have smaller budgets and numbers of employees it often doesn’t equate to the size of their data, with normally several TB of VM’s that need backing up on a regular basis. I have in fact at times found these smaller customers to have larger data usage due to less data management being in place compared to bigger customers.  With working with these kind of limitations you can often have problems writing the backup to disk quick enough to fit within the backup window, especially where full backups are involved.

The Traditional Solution

The traditional solution in this situation would be to run a full backup at the beginning of the week and follow this up with incremental backups for the rest of the week. This solution works but introduces a number of problems, the first being if your weekly full fails for what ever reason, you are then left chasing trying to get a full completed during the working week. The second is with needing to hold a full backup online each week, it makes keeping several weeks of backups online storage hungry and costly. Obviously you could archive to tape daily or weekly but this adds additional complexity and slows down the ability to be able to quickly restore VM’s and data.

Veeam

For me this is where Veeam steps in and meets the requirements, first of all Veeam Backup and Replication’s user interface is so easy to use and makes the job of configuring, monitoring and restoring your backups a breeze. One of the main points for me that makes Veeam the solution to have in these requirements is their synthetic full method.

Taken from the Veeam Backup and Replication User Guide http://www.veeam.com/veeam_backup_4_1_user_guide_pg.pdf

Synthetic backup presupposes that a full backup is performed only once. All subsequent backups are incremental: only data changed relatively the most recent version is backed up. In contrast to repeatedly performing full backups, this approach ensures a faster and less space-intensive backup.

When a full backup is performed, the resulting backup .vbk file is written to the target host. At each incremental backup, Veeam Backup & Replication 4.1 rebuilds this full backup to the most recent state of a VM and uses historical data to calculate a reverse increment.

Additionally Veeam Backup and Replication supports de-duplication and compression of backed up VM’s, instant granular recovery of VM files and Guest files. They give you the ability to take a standard full backup when required, for example if company policy insists a full is taken weekly / monthly etc. Finally they also give you the ability to be able to replicate backups to another virtual environment using the same system for no more money! This is allowing us to create a VMware SRM like environment for small customers on a tight budget! (That’s a blog post for another day!)

So when you are in a situation where you need to be backing up to low performing storage and / or if you are tight on storage space but still wish to keep as many backups online as possible Veeam fits the bill.

image

The more I read this post the more it sounds like an advert for Veeam, but I can assure you this comes from personal experience and no kind of connection with Veeam.

For me the only point where Veeam falls down is when you start archiving your jobs to tape. Veeam creates a single VBK file for the synthetic full backups, this makes it easy to create an archival job to write your backups onto tape, where you start to have a bit of difficulty is when you need to restore an individual VM or File from that tape archive. You will need to restore the whole VBK from tape first before recovering your VM or Files with Veeam. Obviously there are ways around this such as create individual jobs for each VM, but the main issue with this would be loss of de-duplication between VM’s and the amount of jobs you are then left to manage. Luckily because the amount of recovery points we are able to keep online with Veeam this situation very rarely arises. There is a conversation about tape archiving that can be found on the Veeam forum here >> http://www.veeam.com/forums/viewtopic.php?f=2&t=1441&start=0&hilit=tape+support. Veeam are also making some changes in V5 that will make backups more archive friendly.

 

Hardware

Often we will need to re-use old hardware or cheaper server based storage for the backups. It can be very easy to start running into performance issues when this is the case but sometimes they can be easily resolved.

Please check out my blog post from last year about Virtualised Backup Performance >> https://virtualisedreality.com/2009/05/21/virtualised-backup-performance/

Recently I ran into some issues using a HP DL180 G6, filled with 750Gb Midline SAS disks in a RAID5. We were getting dreadful performance whilst backing up with Veeam, with a backup of 3Tb’s of virtual machines with 100Gb compressed and de-duplicated changes taking over 30hours to complete. After some investigation it was discovered the servers P400 RAID controller although containing 256Mb cache ships with no battery as standard so the cache is disabled. You are able to override this configuration (not recommended) but as standard the cache is configured for 100% reads. So after upgrading the RAID controller to BBWC, tweaking the cache configuration for 75% writes and upgrading the firmware on all the disks (I highly recommend this on the midline SAS disks) we are now able to complete the same backup in well under 3 hours!

Conclusion

So just goes to show there is no reason cheaper hardware with the right kind of backup solutions can’t do the job, but ensure you know the limitations of your hardware and ensure everything is fully up to date before starting. Too often the backup vendors end up being the point of blame for poor performing backups and I have been guilty of this, but usually you will find a bottleneck else where that will be causing the issues.

All of the major backup vendors offer a free trial period so give them a go and come to your own conclusions.

Veeam SureBackup

March 22, 2010 — 4 Comments

So today was D-Day for the announcement of the new feature set coming from Veeam. There has been a lot of speculation coming from my blog and other on what this could be and as far as I am aware no one hit the nail on the head.

Working for a Veeam partner I was lucky enough to learn about the new functionality last week, so have been keeping my mouth closed on the subject until now.

So what is it….

From Veeam’s Website www.surebackup.com

Veeam has developed patent-pending technology that will allow you to verify the recoverability of every backup—for every virtual machine (VM), every time. It’s the latest innovation in our SureBackup™ approach to data protection.
So instead of just hoping that you can recover from a backup, you’ll actually know that you can—because you already did so as a routine part of the backup process.

Now when I was first told this I thought hmm this doesn’t sound as interesting as I thought it would, with Veeam recovery being so easy, its simple to do test recoveries to check your backups. But then as I learnt more I found out the system would would automatically do these tests for you by directly mounting the compressed backup file, to an ESX host and verifying the backup! So no more test one machine once a week and hope the others are ok, you can now test them all every backup!

How it works…

How do we do it? By running VMs directly from backups. It’s an ingenious solution to an age-old problem that leverages the power of virtualization and the innovation of Veeam. We call this potent combination Virtualization-Powered Protection™, and it significantly enhances disaster recovery so you can fully realize the promise of virtualization.

Using new patent-pending technology, Veeam publishes the contents of the backup file as a datastore that an ESX host can connect to. During recovery verification, we automatically create VMs in an isolated environment and run them directly from the backup. We start each VM, boot the OS, and confirm that everything is running normally. You can even verify that applications are functioning properly and data is intact.

A key component of this innovative technology is the ability to run a VM directly from a compressed backup file. Since you don’t have to extract the backup, running a VM from the backup is fast and requires no additional storage. Minimal host resources are required and can typically be provided by existing machines—in production, a QA lab, or even your DR (disaster recovery) site.

imageWhat this means 

No longer do you have to hope you backups are working in-between tests as they can be verified to be working not just valid every backup, the ability to be able to granularly restore elements from any applications by starting a VM from the compressed backup file. This also introduces lots of new possibilities for testing environments etc.

This new feature amongst others will be released with Veeam Backup 5 later on this year, personally I can’t wait for the release and to find out what else Veeam has done to make our life’s easier and less stressful when it comes to backups.

My further hope is that Veeam will build tools to assist us recovering granular items from Exchange and AD etc. As whilst I appreciate this new technology assists with this, nothing would beat a Veeam Restore style view of Exchange etc for mailbox / mail recovery within Veeam Backup 5.

Video

 

Links

Veeam SureBackup Website

Veeam Surebackup PDF

So that’s my quick update will blog more about it when I get a chance.

I have recently been pointed towards the following Veeam webpage http://www.veeam.com/surebackup/

image With a count down to the 22nd March, a new trademark “SureBackup” and the tag “Veeam will change the way you look at VMware backup FOREVER”

For me Veeam’s current version 4 and the previous version 3 have been big game changers in Virtualisation backups, Veeam always seems to keep ahead of the rest of the market and for me the ease of use and effectiveness has been a key selling point. Even to such a level that a Veeam Backup demo has been the final point needed when trying to sell a virtualisation solution on a couple of occasions.

So what will he announced on March 22nd? Who knows, think we can assume it will be the next evolution of Veeam backup with greater functionality.

For me my ultimate VMware backup product would contain the following

  • Speed (Veeam have already got this with the reverse incremental system and CBT!) Surely the next product couldn’t be even faster!?
  • Application aware – Veeam already have an impressive VSS implementation but for me granular recovery of Exchange, SQL and AD are a must and Backup Exec 2010 has started to address this. Imagine being able to recover an AD user, mailbox or even email as easily as you could recover a file from a Veeam Backup!
  • Tape / VTL – A lot of customer still use tape and VTL’s are growing in popularity, would it be a step to far for Veeam to move into this complex area!?
  • SAN Integration – I would love to see a product that could interact with SAN snapshot and replication functionality, but there are so many vendors ,so could we really expect to see this without a common API? Maybe Veeam could offer an API for the SAN vendors to interact with?
  • vSphere Client Integration – Being able to kick off a backup of an individual VM from the vSphere client, check backup progress or history and right click restore a VM from backup.

I don’t think for one minute some of my wish list would be addressed as some are pie in the sky but maybe a couple of them are being tackled in SureBackup!

Guess we will have to wait to find out!

Update 1

Doug Hazelman from Veeam has released a snippet of information on his blog

UPDATE: In case you’re wondering, the next version of Veeam Backup & Replication will not require a complete reinstall/re-architecture. SureBackup is an entirely new feature that will easily integrate in with existing Veeam Backup & Replication installations.

Update 2

Gostev from Veeam has given us a bit more information in the comments below, so SureBackup is a feature (Addition?) that will radically change the way we look at our VMware backups. I have taken a stab at guessing in the comments, but as Gostev says “It would be quite hard to guess though, …….because you did not realize this is even possible.” Also good to hear that Veeam Backup and Replication 5 addresses some of my wish list!

Update 3

Recently posted on twitter “RT @VeeamAtDell: The countdown has started for Dell’s partnership with @Veeam for http://www.surebackup.com” not really sure what this means. It could be a sales partnership but i’m hoping it maybe something to do with Dell Equallogic and Veeam. Maybe Veeam integration with Equallogic Auto-Snapshot Manager/VMware Edition. Probably wishful thinking 🙂

Update 4

Doug at Veeam has released a lot more information on his blog now >> http://veeammeup.com/2010/02/surebackup-blogger-briefing.html

SureBackup is NOT a new product. We’re using SureBackup to describe a set of features, some are new, some are already in place. We’re going to be introducing other terms as well that will include new features as well as existing ones. All of these features will be delivered as part of Veeam Backup & Replication version 5.0.

I look forward to learning all about it and being able to tell all my readers and customers about the latest and greatest Veeam Backup features.

Veeam Backup 4 GA

October 28, 2009 — Leave a comment

I always look forward to updates from Veeam their ability to deliver products that are a joy to use and do what they say on the tin is fantastic. Snippets about version 4 have been released practically since version 3’s release and for me seeing version 4 in action at IPExpo proved that Veeam had done it again with the vStorage API integration and the Enterprise Management Console amongst numerous other enhancements.

Veeam are the first 3rd party provider to make use of the vStorage API’s in its backup product and what this effectively means is your backups can now be completed up to 10x faster than previously using ESX4 changed block tracking and native thin-provisioned disk support.

The Enterprise Management Console means you can now manage multiple Veeam installations across your enterprise with a single view, this also means that you no longer need to connect into your backup server to view your backups, this can be done from your desktop by pointing at the Enterprise Management Console website.

I look forward to upgrading customers to version 4 and seeing what performance increase they receive from this upgrade alone, the backup window is often the biggest challenge in the backup solution.

Veeam News Release >> http://www.veeam.com/news/veeam-backup-is10-times-faster.html

Veeam Backup 4 Product Page >> http://www.veeam.com/vmware-esx-backup.html#fragment-1

5 Ways VMware vSphere Improves Backup and Recovery White Paper by Eric Siebert >> http://www.veeam.com/go/vmware-vsphere-backup-wp?ad=veeam_backup

A few pictures from Veeam’s Website

backup_schedule 

search

restore_options

 

enterprise-manager

On with the vSphere blogging competition, the winner of round 2 was announced on Friday and I was extremely surprised to see it was me! All the details can be found here http://blogs.vmware.com/vsphere/2009/10/winning-post-from-cycle-2-distributed-switch.html I would also like to point you in the direction of Hany Michael’s entry on his blog which as per usual was a fantastic source of information http://www.hypervizor.com/2009/10/vsphere-40-vnetwork-distributed-switch-vds-video-demonstration-architecture-diagram/

Onto the third round of the vSphere blogging contest VMware Data Recovery

The Problem

Often when working with SMB’s a key factor in any project is the cost, with lots of companies understanding the benefits of virtualisation the project soon gets of the ground but often is restricted by budget. This often leads to the core system being impletmeneted and features that are seen as nice to have are left until further funding can be allocated. With many companies already having licences for backup products the traditional style of backup (agent inside each VM) is left in place and the added benefits that virtualisation can bring to backup are missed.

The Solution

In steps VMware Data Recovery,

Whilst there are already many excellent backup products on the market for virtualisation, these all mean an additional cost when the budget has already been spent, Data Recovery is included with VMware vSphere Essentials Plus and from VMware vSphere advanced and up.

dp

VMware Data Recovery works on the basis of a virtual appliance that you deploy from the packaged ovf and a plug-in that you install within your vSphere client to manage and administer the backups. Setup and configuration of your first backup jobs can easily be configured within 30 minutes. VMware Data Recovery can backup a maximum of 100 VM’s and these can be written to a maximum of two datastores at once. Your data will be de-duplicated whilst it is being backed up ultimately saving you space and Data Recovery also makes use of the changed block tracking within vSphere so only has to backup the changed blocks since the last backup, a feature that we are only just seeing introduced in some of the leading products now.

Installation

Now prior to writing this blog post VMware Data Recovery was not a product I had ever used, I am pleased to report that VMware have done a fantastic job making the installation as pain free as possible.

You will first need to download Data Recovery from the download page on VMware’s website.

Steps

You need to deploy the OVF file that is on the Data Recovery ISO to your datacenter, this is done easily with vSphere using the Deploy OVF template option from the file menu.

Once you have deployed the OVF, you can add a hard disk to it to act as a datastore for your backup or skip this step until later, alternatively you are able to use a CIFS share or RDM as your datastore.

You will now need to boot up the Data Recovery appliance and configure the networking.

Once this is complete all is left is to install the Data Recovery plug-in to your vSphere client and configure the plug-in to see your appliance, before you start configuring your backups.

Below is a short video to show you through these steps if you need a little more assistance. Please excuse the poor quality as this is the first video like this I have made, also please excuse my West Country accent! It maybe easier to watch the video straight from Vimeo here >> http://www.vimeo.com/7126178

Vodpod videos no longer available.

more about “A beginners guide to VMware Data Recovery“, posted with vodpod

Things to be Aware of

Be aware of where you are storing your backups, there is little point to storing them on the same storage device as the VM’s you are backing up. For this reason a CIFs share on a physical server or NAS maybe a good idea. Data Recovery has no ability to be able to backup your VM’s to tape.

Network Destinations Must Be specified Using IP Addresses – Using DNS-resolved names to specify network destinations is not supported with Data Recovery. When adding network destinations using Configure > Destinations, enter an IP address. For example, use \\192.0.2.12\share rather than \\example\share. It has also been reported that by typing your username like There is a file level recovery option but this is only experimental, as such you should not be counting on this in a production enviroment.

Must Read Documents

All the documents relating to Data Recovery can be found at the link here >> http://www.vmware.com/support/pubs/vdr_pubs.html I highly recommend reading all the release notes and admin guide before undertaking and installation of Data Recovery.

Whilst recently experiencing performance issues running a virtualised backup on a customers site, I thought I would blog some useful steps to help diagnose issues with virtualised backup performance with products such as vRanger and Veeam a lot of these also apply to physical backups.

This is a list of things to consider / try when trying to diagnose the issues you are having, I will add to it over time but please feel free to comment further suggestions or changes.

  • Network
  • ESX
  • Storage
  • Anti-Virus / Third Party Tools
  • Veeam  Specific Settings

Network

  1. Are you using a Gb connection or 100Mb lan connection?
  2. Is the ESX host, VM and Backup server all seeing the expected network connection speed?
  3. Do a ping test between all the effected elements, are you loosing any packets? are you seeing the expected response times?
  4. Is the backup server on the same subnet as the ESX Host or VM being backed up? If not could the routing device be causing a bottle neck? Is it a 100Mb routing device or Gb? Do a tracert to ensure you are taking the expected route. Are you able to temporarly put this backup server on the same network and test backup performance?
  5. Are you using the latest drivers for your NIC?
  6. What is your backup server / VM / ESX host currently doing? Could this be causing a networking bottleneck?
  7. Try a standard file transfer between your VM and the backup server, how does this perform? Use Veeam FastSCP to do a file transfer between the ESX host and your backup server, how does this perform?

ESX

  1. Are you using VCB or Network backup? If you are using VCB try network backup, how does this effect the performance? Have you followed the relevent VCB articles on VMware’s website? Are you using a FC or iSCSI SAN? If you are using iSCSI are you expecting a performance increase by using VCB? or are you just looking to remove backup from LAN or take load away from the ESX boxes?
  2. How much memory have you reserved for the service console? Try and set it to 800Mb, does this improve the performance?
  3. Have you reserved CPU for the service console? Try increasing this from the default settings.

Storage

  1. Where are you storing your backups? What performance do you get when copying a standard file to this store? How are you connecting to this storage and what are the bandwidth limitation of this connection?
  2. Are you connecting to a NAS or SAN via iSCSI? If you are using a software iSCSI initiator how is this effecting the CPU usage of your backup server?
  3. Are the drivers and firmware for you FC card up to date?
  4. Is anything else being activly written to this device? How many backups are you trying to run at once to this store?

Anti Virus / Third Party Tools

  1. Is your anti virus causing a bottle neck for reads / writes? Is it possible to setup exclusions or temporaliy disable the AV to test? (Please ensure you don’t leave your AV permently disabled!?)
  2. Is there any other software on the server that could be causing a bottleneck for your backup?

Veeam Specific Settings

  1. If using Network mode have you enabled the SSH client on the ESX host as instructed when you added your hosts?
  2. If using full fat ESX rather than ESXi try changing the “Data Transfer Engine” under properties of your host in Veeam to “Force Service Console Agent Mode” note, the default “Automatic” mode will actually always attempt to use SSH on “fat” ESX host if it configured (previous bullet).
  3. How many backups are you running at once? Note when using network backup mode, multiple backups may slow down indvidual job rates, but the overall speed maybe quicker, test different mixes until you find a sweet spot. For VCB SAN mode though, backup storage disk speed is often a bottleneck, so unless parallel jobs write to different disks, there’s little sense to run many jobs in parallel.
  4. What commpression level is your backup set to? If currently set to best try Optimal.

Stumbled across this usefull article on problems with VMware snapshots

http://hyperinfo.wordpress.com/2008/06/12/troubleshooting-vmwareesx-snapshots/

Specifically the items below, please note I have changed the wording a content on some items from the original post now.

Locating VMs that have snapshots


Trying to find out which VMs have snapshots can be challenging. There is no centralized way to do this built into the VMware Infrastructure Client or VirtualCenter, so you should periodically check your ESX servers for old snapshots that need to be deleted. There are a few methods you can use to accomplish this.

Method 1 – use the Find command on the Service Console

  1. Login to service console.
  2. Change to your /vmfs/volumes/ directory.
  3. Type find -iname “*-delta.vmdk” -mtime +7 -ls to find snapshot files that have not been modified in 7 days or simply find -iname “*-delta.vmdk” to find all snapshot files.

and

Dealing with snapshots that do not delete properly


Occasionally, a snapshot will not delete properly leaving an active snapshot for a VM. This can happen when using VMware Consolidated Backup or when deleting snapshots through Snapshot Manager. In most cases, the snapshot will not appear in the Snapshot Manager for you to delete. The only indication that a snapshot may still exist is the presence of delta files in the VM’s directory.

If you do have a snapshot running that is not in Snapshot Manager, you can attempt to delete it one of two ways. First, create a new snapshot using the VI Client and delete all snapshots from the snapshot manager after the new one has been created. Alternatively, login to the ESX Service Console, switch to the VM’s home directory and create a new snapshot by typing vmware-cmd createsnapshot (The syntax is as follows “vmware-cmd createsnapshot name description quiesce memory” view an example here . Wait for the snapshot to be created and type vmware-cmd removesnapshots. When it completes, check to see if the delta files have been deleted. If they have, then it was successfully completed.

If the delta files weren’t deleted, check the vmx file for the VM and locate the lines starting with scsi. If the VM is configured with only one virtual disk, it is usually scsi0:0 (if .present is false, it is a non-existent drive that you can ignore). The .fileName should be using the original disk file that was created with the VM and is usually the same name as your VM. If this is the case, then your VM is not using the snapshot files. If it has a -00000# in the filename, it is currently using a snapshot file. The following makes this a little clearer: VM with no snapshots: scsi0:0.present = “true” scsi0:0.fileName = “myvmname.vmdk” VM with snapshots: scsi0:0.present = “true” scsi0:0.fileName = “myvmname-000001.vmdk”

If this is the case and the above operation failed, another option you have is to either clone the VM or clone the VM’s disk file. To clone the VM I would recommend the VM is powered down and cloned through the VI client to another VM, although this could technically be done whilst the VM is powered up if you are using ESX 3.5 Update 2 or newer.

Another method is to shutdown the VM, login to the Service Console, switch to the VM’s directory and clone the VM’s disk file by using vmkfstools and specifying the snapshot file as the source disk, i.e. “vmkfstools –i myvmname-000001.vmdk myvmnamenew.vmdk” Once it completes go into the settings for the VM, remove (don’t delete) the hard disk, add a new hard disk and browse to the newly created disk file. Power on the VM and verify everything is working before you delete the old disk and delta files.

Please note if you are struggling with a particular issue with snapshot and you have a current VMware support contract, VMware support are very good at assisting with these problems and I would suggest you log a call with them rather than trying anything you are unsure of.