Q: In VMware ESX environments, why does EQL suggest disabling Delayed ACK and Large Receive Offload (LRO)?

A: With a virtual environment, you want all data to be acknowledged and available quickly. When you use any receive offload technology, you are filling a big buffer before interrupting the CPU for processing. This “delay” can add artificial latency. As a result, ESXi v5.x will sometimes generate latency alarms on Datastores when Delayed ACK and/or LRO are enabled. In SANHQ you can see this effect. During very low I/O periods, the reported latency is high. Conversely, when under moderate load, the latency is low.

Btw. ive seen related high latency issues with Qlogic cards, in our case it was a special type of card that was the only one available for IBM pseries 520 and 550’s.

They did not allow for any management of the HBA, and the support had to recommend us switching to normal Intel NICs instead.. which worked perfectly fine for our needs. I guess its one of the dangers of having vendor specific hardware, that functionality sometimes can be limited compared to just having bought a generic qlogic HBA. Pseries servers will not boot/enable with a non-IBM labled adapter though (we tried!!)

here’s some info on how to check it and disable it using SANSURFER.

http://www.jwertheimer.com/help/qlogic/iSCSI_Help/configuring_an_HBAs_firmware_values.htm

Re: Delayed ACK. It’s not a one size fits all solution. So many variables with IO loads, etc..

I can say that it has helped many customers.

Also, you have to run a vm-support after the reboot. I’ve noticed with ESXi v5 that it doesn’t consistently set the value to “0″. So if you saw no change, it may not actually be disabled. Or only some of the LUNs will show disabled.

If you untar the vm-support file and go to the “commands” sub-directory do a #grep Delayed * | more

You’ll get a list of “nodes” that refer to the LUNs. It will end with =”0″ or =”1″. One (1) is enabled.

Sometimes I’ve had to remove the discovery address, reboot, add it back in, then select “Advanced” and scroll down to verify that “Delayed ACK” doesn’t have a check next to it.

rescan and then re-do vm-support to check the settings again.

Btw, I just found out Disabling Delayed Ack on ESX also improved IOPS (and some more on VMTN) and Delayed Ack was re-enabled by default in ESXi 5 somehow, mentioned somewhere in VMTN as well.

The mystery has been SOLVED finally after almost 1 complete month!

For details, please refer to the update I’ve added to the end of this blog article.

I was told exactly the same, and turned it off on my entire vmware enviroment, and it had no effect at all.

In my mind, something with the reporting of performance numbers has changed in the firmware between 5.0.7 and 5.1.2 that causes SANHQ to report elevated values. As previously stated, im not seeing any latency issues on the volumes, nor anything on my vmware or other directly connected hosts.

Its a shame they cannot figure out what it is, but i suppose they are limited in what statistics/diags the arrays show them.

Anyway, I do think it’s a bug in FW v5.1.x/v5.2.x, but the funny thing is Don (EQL VMware specialist) posted on VMTM saying enabling Delayed ACK will cause artificial latency, this explained why we do not have any performance issue, but only seeing the fake high latency on SANHQ.

I shall schedule a time slot for disabling Delayed Ack in ESX and report back my findings later.