Wednesday, January 19, 2011

VAAI in Vsphere 4.1 is turned on by default and can break your recoverpoint constancy groups!

Beware, VAAI in Vsphere 4.1 is turned on by default and can break your recoverpoint constancy groups!

UPDATE: Scott Lowe just wrote me back and confirmed that the version of flare code that will resolve this issue is 4.30.000.5.509

We recently had a problem where our Exchange Consistency groups in recoverpoint were all stuck at initializing 0% for several days out of the blue after running for over a year. I tried to force a re-sweep, I tried to rebuild the Constancy groups, however it was still stuck at 0% initialized. The fix from EMC support was to disable VAAI in vmware, steps are below.
We are using the following:
Vsphere 4.1
Flare 30 4.30.000.5.507
Recoverpoint 3.3 SP1
And Clariion Splitters

EMC Case Notes:
Notes: I send the customer a email asking if VAAI is on?
Just got a update from engineering stating that this is a know issue.
This case was closed and defined as a bug

From EMC Primus Article emc255099
ESX/ESXi 4.1 VAAI (vStorage APIs for Array Integration) - Hardware Acceleration features (Locking , Pre-zero, Copy) are only supported with RecoverPoint 3.3 SP1 and up that uses CLARiiON FLARE 30 type 2 patch and up and a CLARiiON splitter.

From EMC Recoverpoint Replicating Vmware on page 31:
vSphere 4.1 introduces vStorage API for Array Integration (VAAI). By
default, VAAI commands are enabled upon installation. If your
release of the RecoverPoint splitter does not support a VAAI
command, that command must be disabled in all ESX servers. Failure
to disable an unsupported VAAI command can cause data
corruption, production data being unavailable to ESX hosts,
degraded performance, and switch reboots.
For RecoverPoint support of VAAI commands, refer to Table 3 on
page 7. Use the following procedure to disable VAAI commands that
are not supported by your configuration.
To disable VAAI commands:
1. In the vSphere client inventory panel, select the host.
2. Click the Configuration tab and from the Software menu, select
Advanced Settings.
3. To disable Hardware-Assisted Locking, click VMFS3, and set the
value of VMFS3.HardwareAcceleratedLocking to 0.
4. To disable Full Copy, click DataMover, and set the value of
DataMover.AcceleratedMove to 0.
5. To disable BlockZeroing, click DataMover, and set the value of
DataMover.AcceleratedInit to 0.
6. To save the changes, click OK.
32 EMC RecoverPoint Replicating VMware
Management tasks and procedures
7. Make sure every unsupported command on every replicated ESX
Server is disabled.