Comments (0)

By admin, June 23, 2012 10:20 am

透過快照可為虛擬機器建立還原點，應用十分方便，但必須謹慎使用與管理，避免衍生額外問題

對於VMware虛擬平臺的用戶來說，快照（Snapshot）是個十分方便的工具，可為虛擬機器（VM）建立多個還原點，並在必要時將VM倒回指定的還原點狀態。

因此許多用戶都把快照作為一種VM備份手段，例如在為VM的Guest OS進行重大更新之前，先利用快照建立還原點，以備之後VM出現問題時可以倒回到更新前狀態。VMware還有許多服務或功能也都是透過快照作為中介，如VCB與VADP備份機制、Storage vMotion，與VMware Lab Manager軟體等。

VMware環境下的快照運作方式
VMware的快照屬於Copy-on-Write類型，執行快照時，系統會建立一個稱為delta.vmdk的新檔案（實際檔名通常是vmname-00001.vmdk），接下來包括新增或異動在內的所有VM寫入I/O，都會被導入到新建立的delta.vmdk檔案中，不再寫回原始vmdk檔，原始vmdk檔被轉為唯讀狀態。

每當執行一次快照，就會建立一個新的delta.vmdk檔案，並將前一次快照的delta.vmdk檔轉為唯讀，然後依靠新的delta.vmdk檔來接受VM的新寫入資料。

建立快照時，可選擇是否啟用兩個附屬功能：（1）連同VM的記憶體一同進行快照；（2）配合虛擬機器Guest OS檔案系統的靜止（Quiesce）功能。

若勾選連同VM的記憶體一同快照（如果快照時VM處於開機狀態，這選項是預設開啟），ESX主機會將VM記憶體中的資料Dump下來寫到vmdk檔中，缺點是會拉長快照執行時間。記錄VM狀態的vmsn檔案也會變得更大些。

如果要啟動另一種靜止快照，則須搭配安裝在VM上的VMware Tools使用，勾選後，VMware Tools的Sync驅動程式會讓VM的檔案系統進入靜止狀態（或者說是凍結狀態），停止應用程式對檔案系統的寫入，並將緩衝區或快取記憶體區域內的資料寫回磁碟，以得到確保資料一致性、用於備份作業的快照。

利用快照還原VM狀態的程序
要將VM倒回某個快照狀態，可有兩種操作方式：

一是在VM上按右鍵，然後從Snapshot選單中選擇的Revert to Current Snapshot項目，不過只能將VM還原到最近一次快照的狀態。第二種方法，則是利用Snapshot選單中的Snapshot Manager功能，可從展開的樹狀圖中選擇任何要回復到的快照，然後按下Go to按鈕即可。

執行倒回程序後，接下來VM將改為在所選擇的快照上執行，也就是在被選上的那個delta.vmdk檔上執行寫入動作，並捨棄當前的VM狀態。

快照操作的困難點：刪除快照

快照雖然帶來許多方便，但卻是「請神容易送神難」。由於Copy-On-Write類型的快照之間形成鎖鏈關係、彼此相依，因此不能任意刪除快照，否則就有導致其他快照不可用的問題。以VMware來說，執行快照以後，VM接下來所新增或異動的資料，都寫入在新增的快照檔案中，如果不做任何處理而直接刪除快照檔案，那也就會失去這些新增與異動資料，這顯然是不可接受的。

所以VMware雖然有刪除快照功能，但實際上被刪除的是「還原點」，而不是真的刪除那份快照檔案內所包含的資料。執行刪除動作後，只是少了那個快照還原點而已，但那份快照所包含的資料，會先被合併（Consolidate）到前一份快照中，而不是真的消失了。因此在執行刪除快照功能時，實際上進行的是「先合併、再刪除」的作業，以確保VM資料的完整性與可用性。

假設先後為VM建立了snap1、snap2與snap3等三份快照，如果選擇刪除snap2，系統會先把snap2的資料合併到snap1中，然後再刪除snap2。如果選擇刪除VM的所有快照，ESX主機就會依序將後續快照合併到前一份快照中，待預定刪除的快照資料併入原始vmdk以後，再刪除全部的快照檔案。

但這種程序也衍生出兩個問題：
● 合併作業耗時
建立快照後，如果很長一段時間都不刪除，隨著新增與寫入資料的增加，快照檔案將會變得越來越大，刪除快照時所執行的合併動作將會耗去相當多時間，甚至達數小時之譜。

● 暫存空間需求過大
刪除快照過程需要足夠的暫存磁碟空間，如果要刪除snap2，要等到snap1+snap2的程序完成後，原來的snap2檔案才會刪除，需要的最小暫存空間便等同於snap2大小。如果對VM做了多次快照，那執行刪除所有快照時，需要等系統按照由新而舊的順序，以倒推的順序進行合併後，才會實際刪除快照，累積的暫存空間非常大，甚至可能超過datastore可用空間，導致刪除失敗。

更糟的是，在ESX 4.x以前，執行刪除快照後，快照資訊便會從Snapshot Manager中移除，即使刪除失敗，也無法從vSphere Client管理介面得知。用戶常以為快照已被刪除，VM已回到原始vmdk上執行，但實際上由於快照刪除失敗，VM仍是在最後一版的快照上執行，往往等到快照變得很大、導致datastore空間大量消耗後，才發現問題，此時要解決，將會變得非常麻煩（有時得將VM遷移到有足夠空間的新datastore，然後重新執行刪除與合併）。

要避免這個問題有兩個方法，一是不要使用「刪除所有快照」功能，改為手動方式逐一刪除個別快照，如此每份快照將在合併後便行刪除，需要暫存空間大幅減少，不過操作程序也更繁瑣。

第二種方法是將系統更新到ESX 4.0 Update 2以後版本，新版的快照合併作業經過改進，首先，不是採用倒推的順序合併（從最後一份快照開始逐一往前合併），而是採用正向的合併順序，從最早的一份快照開始合併。其次，每份快照合併後便會立即刪除，因此需要的暫存空間也就大幅減少。

另外從vSphere 4.x起，就可在vCenter裡設定一個警告訊息，使vSphere Client提醒用戶當前的VM是在快照上執行，而非在沒有快照的原始vmdk上，避免過往的忽略快照刪除失敗問題。

vSphere 5.0又有所改進，新增了快照移除失敗警示功能，還在快照選單新增一個Consolidate選項，可在快照刪除失敗後，重新執行快照合併作業。

相較於正規備份作業，快照執行起來相對快速，而且還不用另外部署，十分方便，因此一些用戶有時會以快照來替代備份的角色，但VMware的快照並無法作為真正的備份使用。

首先，快照只能因應VM層次或Guest OS層級的故障，由於快照與VM位於同一臺實體設備上，如果發生實體設備層級故障，快照將與VM一同失效。而真正的備份產品，則是將備份複本保存在獨立的儲存裝置上，可應付實體設備失效的情況。

其次，備份的目的，是要相對長期的保存原始資料在多個不同時間點下的複本，一般情況每天至少要執行一次備份，其中一些備份複本往往要保存數個月甚至更長時間。若以快照充當備份使用，而讓快照保留過久、或建立過多版本的快照，將會衍生出許多麻煩的問題，包括快照占用datastore空間過大，日後刪除快照困難等。

因此較好的做法，是把快照當作臨時性的建立還原點手段，另外部署專門的備份產品來因應備份需求。

文⊙張明德

Comments (0)

By admin, June 14, 2012 7:18 pm

Currently I am also available as a freelance virtualization consultant in Hong Kong with more than 8 years of extensive hands on experience, from initial Case Analysis, to Architecture Design, to the actual Site Implementation and Post Maintenance.

If your company is looking for a guru in building your next virtual project (VMWare/Equallogic/Dell centric) or simply an additional helping hand, please do not hesitate to contact me.

The main reason is I am kind of bored with my existing client projects as most of them have been implemented, the rest is just maintaining stuff, so I am looking for some fresh challenges, oh…I prefer not to travel if possible.

Comments (1)

By admin, June 12, 2012 12:18 pm

Total 14 drives per Equallogic PS-M4110 Blade, a nice way to utilize the existing investment (ie, M1000e blade enclosures), as usual there are XV and XS (for SSD) models to choose from. PS-M4110 starts shipping in August 2012.

Some of the new Equallogic Firmware 6.0 features:

• Syncronous Replication- Real time syncronization
• Volume Unmap – Thin Provision Reclaim, I would say this is the MOST IMPORTANT FEATURE after all! We have waited for so long.
• Volume Undelete – preserve for 1 week or until the space is required

To me, Equallogic is a great product really, just like Apple’s iOS, the coming iOS 6.0 will still backward support even 3GS. For EQL, you can always enjoy the latest feature with the latest firmware on your old EQL boxes, of course, if you have a maintenance plan.

Finally, EQL will also release SANHQ 2.5 in fall 2012.

Update: July 30, 2012

Dell customers not so keen on Blade Arrays concept

BOSTON — Dell Inc. still has to convince customers that the storage blades rolled out this week at Dell Storage Forum 2012 are a good idea.

When asked about the iSCSI EqualLogic PS-M4110 Blade Arrays that Dell launched Monday, several customers said they’re unsure about the storage blade concept. A few said integrating storage, servers and networking into a condensed system is too new of an approach, while others wondered if it would have the same performance as traditional rack-mounted storage.

“The performance is not there to stick everything in a single box. You would need to stack a ton of them together just to get decent high performance, especially if you have high I/O loads,” said Tibor Pazera, a senior technology specialist at Irvine, Calif.-based Advantage Sales and Marketing LLC and a Compellent customer. “Convergence is nice for ease of deployment, but there’s not enough spindle capacity to maintain high I/O performance.”

Other customers characterize Blade Arrays as unproven.

“We get concerned about risk, partly because it’s new,” said Alex Rodriguez, vice president of systems engineering and product development at Cleveland-based IT services company Expedient Communications. “If a blade chassis has a failure, it’s gone.”

A virtualization and storage architect at a major New York-based retailer, who asked not to be identified, said he “dabbled with the idea, but it’s a bit too new for us.”

Comments (0)

By admin, May 23, 2012 10:51 pm

This article from Mainland China revealed the secret of the latest Equallogic hardware architecture.

PS6100 uses 64bits RISC CPU which provided by NetLogic XLS616, it has 4 MIPS cores with 4 threads per core at 1Ghz each (ie, 16 total threads per CPU), and 1MB L2 Cache.

In contrast, PS4100 uses its younger brother NetLogic XLS608, which has 2 MIPS cores with 4 threads per core at 800Mhz each (ie, total 8 threads per CPU), and 1MB L2 Cache, so the processing speed has been cut at least 50%.

Everything else is the same except the above processor speed and number of cores. It is also interesting to learn about the super conductor instead of traditional BBU cache for data.

Comments (0)

By admin, May 22, 2012 5:38 pm

I was told Power Connect 52xx/54xx series switches are optimized for VoIP and they are no longer recommended by Equallogic as 50% of the available buffers are reserved for VoIP unless you change the setting and REBOOT the switch.

I haven’t tried the following VoIP setting recommended by EQL as I don’t want to mess around my optimized configuration file.

Besides, I think there is an alternative way which I’ve completely disabled VoIP setting using PowerConnect 5448 GUI. There is a check for you to disable VoIP in the Switch session under the GUI.

Go to each switch and do the following:

Put the switch into admin and configuration mode.

enable
configure

Disable/remove iSCSI and VoIP optimization

voice vlan oui-table remove 00036b
voice vlan oui-table remove 00096e
voice vlan oui-table remove 0001e3
voice vlan oui-table remove 000fe2
voice vlan oui-table remove 0060b9
voice vlan oui-table remove 00d01e
voice vlan oui-table remove 00e075
voice vlan oui-table remove 00e0bb

Exit from configuration mode

exit

Save the configuration

copy running-config startup-config

When ready, type ‘reload’ to reboot

reload

Comments (5)

By admin, April 28, 2012 12:04 pm

This latest brief review compared the pros and cons of the leading backup vendors for VMware ESX. (In Chinese).

It only touched the surface, not technical oriented and in fact not detail at all, sounds more like a marketing material to me.

Comments (0)

By admin, April 27, 2012 11:20 am

VMware ESX source code has been stolen and posted online, but the company said its virtualization platform doesn’t necessarily pose an increased risk to customers.

The stolen code amounts to a single file from sometime around 2003 or 2004, the company said in a blog post.

“The fact that the source code may have been publicly shared does not necessarily mean that there is any increased risk to VMware customers,” according to the blog written by Iain Mulholland, director of the company’s Security Response Center.

The code was stolen from a Chinese company called China Electronics Import & Export Corporation (CEIEC) during a March breach, according to a posting on the Kaspersky Threat Post blog.

The code along with internal VMware emails were posted online three days ago.

VMware didn’t respond immediately to a request for more information about the impact of the breach on customers.

Eric Chiu, president of virtualization security firm Hytrust, said it’s hard to say what VMware customers should do because there’s not enough detail about how the exposed code is being used in current products.

In general, though, customers should review the security for virtual environments to address the fact that a compromised hypervisor exposes multiple virtual machines.

While the incident is reminiscent of the breach last year of RSA source code, the circumstances differ. An RSA partner was breached and that breach was used to send a malware-laced email to an RSA staffer who opened it.

In VMware’s case, the CEIEC network was hacked and finding the source code was fortuitous.

This is what VMware posted in a blog: “Yesterday, April 23, 2012, our security team became aware of the public posting of a single file from the VMware ESX source code and the possibility that more files may be posted in the future. The posted code and associated commentary dates to the 2003 to 2004 timeframe.

“The fact that the source code may have been publicly shared does not necessarily mean that there is any increased risk to VMware customers. VMware proactively shares its source code and interfaces with other industry participants to enable the broad virtualization ecosystem today. We take customer security seriously and have engaged internal and external resources, including our VMware Security Response Center, to thoroughly investigate. We will continue to provide updates to the VMware community if and when additional information is available.”

Network World (US)

Comments (9)

By admin, April 13, 2012 10:42 am

Latest 12th generation Poweredge servers offers Express Flash PCIe SSD based solution for extreme high IOPS application. Available in Q2, 2012, now you can plug a total of four PCIe SSDs to the front of R720 (PCIe doesn’t support R720xd) and the PCIe SSDs are connected to the server board PCIe 2.0 directly. There is a cable directly connect to the on board  PCIe from the backplane of the PCIe SSD tray as shown in H710p manual, more information can be found in PowerEdge R720/R720xd Technical Guide.

Btw, the only features difference between H710p and H710 are H710p has 1GB cache and supports CacheCade and FastPath where as H710 only has 512MB cache and does not support CC and FP.

This is the same case as in Poweredge R710 H700 raid card, CacheCade and FastPath are only supported with 1GB NV Cache version.

SEQUENTIAL I/O (not RANDOM) bandwidth is no longer limited by Perc Raid Card as the PCIe SSD becomes a system device now. This means you got to have a corresponding driver for the installed OS underneath, I checked VMware is on the list. Hey, doesn’t this sound exactly the same as Fusion-IO?

The maximum bandwidth is up to 2GB/s for PCIe 2.0, 1.8GB/s for Read and 1.2GB/s for write as advertised by Dell for a single PCIe SSD. Well one thing I really don’t understand is, why Dell does not use the latest PCIe 3.0 connector which offers much higher bandwidth (3.2GB/s), it’s definitely strange!

In contrast, I can reach 1.3GB/s for Read and 1.2GB/s for Write (note both are sequential) as well using a consumer grade MLC based SSD (Crucial M4 128GB) through Perc H700 (that’s PCIe 2.0) on R710 anyway. Also for Random IOPS, I can reach 55,000 IOPS easily as advertised by Crucial, so what’s the point using PCIe SSD after all?

After reading the white paper Optimizing SQL Server Storage Performance with the PowerEdge R720, I figured out these new PCIe based SSDs (175GB and 350GB) are much more reliable now with SLC NAND (I smell $$$) and they required Perc H710P (I thought it’s not using the raid card anymore). In additional, One PCIe SDD drive resulted in 10.5x more IOPS (note Random) than 16 HDDs 10K array, that simply means a single PCIe SSD is equivalent to 10.5 PS6000X or 168 10K RPM spindles, excellent, but sounds unreal to me. I couldn’t find 107,518 IOPS is based on 4k or 32k, if it’s 4k, my consumer grade Crucial M4 128GB is definitely a much cost effective solution, probably at only 1/30 the cost.

Also remember SSD performance will degrade over time due to excessive write cycle and if you stress test it over a very long period of time continuously (that’s over 24 hours), the IOPS will drop to 1/10 of it’s starting value. However a SAS 15K RPM raid EQL box will sustain all the way. (Please correct me if I’m  wrong)

Finally, something related, the SSD Dell used in their latest Equallogic is Pliant based SSD (e-MLC NAND as well), but those are 400GB.

Micron Announces 2.5″ PCIe SSD for Enterprise

Micron has announced the first 2.5″ SSD which utilizes PCIe interface instead of the more common SATA interface. The 2.5″ form factor allows for easy serviceability because the drives can be placed into the front end of servers. Traditional PCIe cards are harder to service and are not hot-swappable, requiring the server to be powered down. Furthermore, using PCIe instead of SATA eliminates the bandwidth bottleneck that SATA produces, maximum of 6Gb/s (or 750MB/s) versus 32Gb/s (or 4GB/s) for PCIe 2.0 x8.

The drive is essentially a 2.5″ version of Micron’s P320h SSD, which Micron announced in June last year. Both use 34nm SLC NAND and are based on the same in-house controller. According to Micron’s specifications, the controller is actually one of the fastest to date – providing read speeds of up to 3GB/s and write speeds of up to 2GB/s (yes, with a big B, i.e. bytes). Random 4KB read performance maxes out at 750K IOPS, while random read tops out at 341K IOPS (700GB version).

However, keep in mind that the controller features 32 channels. 2.5″ SSDs typically utilize 16 NAND packages, which means only half of the available channels would be in use with Micron’s controller. Micron couldn’t provide us with any detailed specifications or performance figures yet as this announcement was for promoting the new interface, not the actual drive, so we don’t know how much the performance differs from the FHHL sized PCIe card. The capacities are a bit smaller at 175GB and 350GB and at least random write performance should be slightly lower than what the FHHL card offers, but it’s certain that this will be one of the fastest (if not the fastest) 2.5″ SSDs.

Micron is currently sampling interested customers and an actual product announcement is going to follow later this year with more specific performance details. No pricing has been revealed, but Micron hinted that the price of the FHHL card would be over $16 per GB, which works out to be $2800 for 175GB and $5600 for 350GB if the 2.5″ version is priced similarly.

Update April 13, 2012

英特爾週四（4/12）推出910系列固態硬碟（SSD）產品，具備400GB和800GB兩種容量，並支援PCIe技術，可適用於雲端運算、虛擬化和線上交易等資料中心的儲存需求。

新產品採用最佳化的多層單元（MLC） 25奈米快閃記憶體，可提供5年內每天10次完整的硬碟寫入，與前一代標準型MCL快閃記憶體產品相比，耐用性提升了30倍。

英特爾表示，910系列具備高效能、耐久和可靠等特性，是用來取代I/O密集硬碟的理想選擇。它容易安裝，且無需修改既有的伺服器設計，可適用於儲存需要高效能處理和存取的分層和快取檔案。一顆910 SSD可取代多顆15K RPM的硬碟，不僅能節省空間和功耗，還能提升儲存擴充性，並同時降低時間延遲。

這是英特爾900系列產品的擴展，首度推出支援PCIe技術的SSD。800GB容量產品的速度為2GB/s持續序列讀取和1GB/s序列寫入，以及180,000 4K 隨機讀取IOPS（每秒輸入輸出運作）和75,000 4K隨機寫入IOPS。

樣品現已開始提供，量產時程訂於今年中。400GB容量價格為1929美元，800GB為3859美元，均提供五年保固。

Comments (4)

By admin, April 12, 2012 10:13 pm

Say if I have ordered the latest Dell Powerdge R720xd with 2 sockets Xeon-E5 CPUs and fill it up with 768GB Ram (32GB x 24 Dimms). R720 supports 4 memory channels per processor, that’s 3 dimms per channel, so 12 dimms in total per processor, or 24 dimms per 2 sockets. In contrast, R710 supports 3 memory channels and 3 dimms per channel, a total of 18 dimms per 2 sockets.

Since vSphere 5.0 Enterprise Plus Licensing Model allows 96GB vRAM per socket, so that’s 192GB vRAM per the above configuration, this leaves the remaining 576GB vRAM in question. According to their licensing formula, I need to purchase 6 more EP licenses in order to fully use the 768GB Ram. Sounds strange that I need to have 8 CPUs license while the physical server only has 2 CPUs.

Let’s put the above extreme case aside, in reality not many is going to order that super expensive 32GB and no one is going to put that much ram in mid-range server like R720 as well anyway.

So let’s focus on the optimal memory configuration for R720, I would say it’s most likely to be 192GB (in fact 256GB).

Currently, mainstream is still 8GB ram, therefore for 192GB RAM, that’s 24 Dimms (8GB x 24), but with 3 Dimms Per Channel (DPC) configuration, it will decrease the RAM speed to 1066Mhz which I want to maintain RDIMM at 1333Mhz bandwidth and 1.35V, so in order to do that, I can only use 16GB ram and for 192GB, that’s 12 Dimms, an uneven number configuration, 3 channels per side, missing one channel, this sounds REALLY ODD?!!! Isn’t it a big drop in memory performance if we do not fill up all channels on each socket? (I might be wrong on this)

So if I insist to run at 1333Mhz, the only choice I have is to have 2DPC x 4 channels, that’s 16 Dimms x 16GB = 256GB, well, it’s over 192GB limit!

I really don’t know what else I can configure the memory in order to fully use 256GB ram with 2 VMware EP license. If you have any better suggestion, please let me know. Of course, if my assumption of fully using all 4  channels proved to be wrong later, then this won’t be a problem.

That’s why people are suggesting VMware to raise the bar to 128GB per socket, this would be perfect solution then, let’s see.

Oh…I just noticed NASDAQ:VMW had a free fall on Wed as their CFO jumped off the ship, is something going on under the table?

Comments (35)

By admin, April 11, 2012 10:48 pm

Around this time December last year, I’ve been searching for a reliable, cost effective SSD for vSphere ESX environment as the one from Dell is prohibitory expensive to implement.

Besides, Dell SSDs may have poor performance problem as someone described in VMTN, those 1st generation SSD either provided by Sumsung OEM (100GB and 200GB), or Pliant based (149GB) which is much faster than Sumsung ones and of course much more expensive (over priced that is) as well. Both are e-MLC NAND.

Anyway, I’ve finally purchased a Crucial M4 SSD 128GB 2.5″ SATA 3 6Gb/s (around USD200 with 3 years warranty), here is for a list of reasons.

Then my goal is to put this SSD to Dell Poweredge R710 2.5″ tray and see how amazing it’s going to be. I know it’s not a supported solution, but no harm to try.

The Perc H700/H800 Technical Book specifically states it supports 3Gb/s if it’s a SATA SSD, but I found out this is not true, read on.

1. First thing first, I have Upgraded Crucial firmware from 0009 to 0309 as in early Jan 2012, users found out Crucial M4 SSD has a 5200 Hours BSOD problem, it’s still better than Intel SSD’s 8MB huge bug.

Correct a condition where an incorrect response to a SMART counter will cause the m4 drive to become unresponsive after 5184 hours of Power-on time. The drive will recover after a power cycle, however, this failure will repeat once per hour after reaching this point. The condition will allow the end user to successfully update firmware, and poses no risk to user or system data stored on the drive.

Something more to notice that SandForce controller based SSD has a weakness that when more and more data is stored in SSD, it’s performance will decrease gradually. Crucial M4 is based on Marvell 88SS9174 controller and it doesn’t have this kind of problem. It is more stable and the speed is consistent even with 100% full in data.

In additional, Crucial M4 Garbage Collection runs automatically at the drive level when it is idle and it has Garbage Collection which works automatically in the background in the same way as TRIM independent of the running OS. As TRIM is an OS related command, so TRIM will not be used if there is no support in the OS (ie, VMware ESX).

2. The most difficult part is actually finding the 2.5″ tray for Poweredge R710 as Dell do not sell those separately, luckily I was able to get two of them off the auction site locally quite cheap and later found out they might be Counterfeit parts, but they worked 100% fine, only the color is a bit lighter than the original ones.

3. Then the next obvious thing is to insert the M4 SSD to R710 and hopefully Perc H700 will recognize this drive immediately. Unfortunately, the first run failed miserably with both drive indicator lights OFF, as if there is no drive in the 2.5″ tray.

Check the OpenManage log, found out the drive is not Certified by Dell (Huh?) and Blocked by Perc H700 right away.

Status: Non-Critical        2359        Mon Dec 5 18:37:24 2011        Storage Service        A non-Dell supplied disk drive has been detected: Physical Disk 1:7 Controller 0, Connector 1

Status: Non-Critical        2049        Mon Dec 5 18:38:00 2011        Storage Service        Physical disk  removed: Physical Disk 1:7 Controller 0, Connector 1

Status: Non-Critical        2131        Mon Dec 5 19:44:18 2011        Storage Service        The current firmware version 12.3.0-0032 is older than the required firmware version 12.10.1-0001 for a controller of model 0×1F17: Controller 0 (PERC H700 Integrated)

4. Then I found out the reason is older H700 firmware blocked the non Dell drive access, so I have to updated Perc H700 firmware to latest (v12.10.x) using USC again. Before the upgrade, I boot into H700’s ROM and found indeed the SSD drive is not presented in the dirve pool. Anyway, the whole process took about 15 minutes to complete, not bad.

5. After the server returns to normal, the Crucial M4 128GB SSD now has light showing in the tray indicator and working correctly partly, as the indicator on the top always blinking in amber (ie, orange),   “Not Certified by Dell” indicates in OpenManage log, and this caused the r710 front panel LCD also blinking in amber.

Besides, under Host Hardware Health in vCenter, there is one error message showing “Storage Drive 7: Drive Slot sensor for Storage, drive fault was asserted”

From Perc H700 log file:
Status: OK        2334        Mon Dec 5 19:44:38 2011        Storage Service        Controller event log:

Inserted: PD 07(e0xff/s7): Controller 0 (PERC H700 Integrated)
Status: Non-Critical        2335        Mon Dec 5 19:44:38 2011        Storage Service        Controller

event log: PD 07(e0xff/s7) is not a certified drive: Controller 0 (PERC H700 Integrated)

I clear the log in OpenManage turns the front panel LCD returns to blue, but SSD drive top indicator light still blinks in amber, don’t worry, it’s just indicator showing it’s a non-dell drive.

Later, this was confirmed by a message in VMTN as well.

The issue is that these drives do not have the Dell firmware on them to properly communicate with the Perc Controllers. The controllers are not getting the messages they are expecting from these drives and thus throws the error.

You really won’t get around this issue until Dell releases support for these drives and at this time there does not appear to be any move towards doing this.

I was able to clear all the logs under Server Administrator.  The individual lights on the drives still blink amber but the main bevel panel blue.  The bevel panel will go back to amber again after a reboot but clearing the logs will put it back to blue again.  Minor annoyance for great performance.

Update: If you have OM version 8.5.0 or above, now you can disable the not a certified drive warning completely! Strange that Dell finally listen to their customers after years of complain.

In C:\Program Files\Dell\SysMgt\sm\stsvc.ini update the parameter to NonDellCertifiedFlag=no

6. The next most important thing is to do a VMFS ReScan, ESX 4.1 found this SSD immediately Yeah! and I added it to the Storage section for testing.

Then I tested this SSD with IOMeter, Wow…man! This SINGLE little drive blows our PS6000XV (14 x 15K RPM RAID10) away, 7,140 IOPS for real life 100% random, 65% read test, almost TWICE than PS6000XV!!! ABSOLUTELY SHOCKING!!!

What does this mean is A Single M4 = 28 x 15K RPM RAID10, absolutely crazy numbers!

##################################################################################
TEST NAME——————-Av. Resp. Time ms——Av. IOs/sek——-Av. MB/sek——
##################################################################################

Max Throughput-100%Read……1.4239………39832.88………1244.78

Max Throughput-100%Write……1.4772………37766.44………1180.20

RealLife-60%Rand-65%Read……8.1674………7140.76………55.79

EXCEPTIONS: CPU Util. 93.96%, 94.08, 30.26%
################################################################

So why would I spend 1,000 times more when I can get the result with a single SSD drive for under USD200? (later proved I was wrong as if you sustain the I/O process, Equallogic will stay at 3,500 IOPS and SSD will drop to 1/10 of it’s starting value)

Oh…one final good thing is Crucial M4 SATA SSD is recognized as 6Gbps device in H700, as mentioned in the very first lines, according to Perc H700 tech book, it said H700 SSD SATA interface only supports up to 3Gbps, I don’t know if it’s the latest Perc H700 firmware or actually the M4 SSD itself somehow breaks that limit.

Let’s talk something more about Perc H700 itself, most people know Dell’s Raid controller cards are LSI Megaraid OEM since Poweredge 2550 (the fifth generation) and Dell Perc H700 shares many advanced feature with its LSI Megaraid ones.

Such as CacheCade, FastPath, SSD Guard, but ONLY available in the Perc H700 1GB Cache NV Ram version.

Optimum Controller Settings for CacheCade – SSD Caching
Write Policy: Write Back
IO Policy: Cached IO
Read Policy: No Read Ahead
Stripe Size: 64 KB

Cut-Through IO = Fast Path Cut-through IO (CTIO) is an IO accelerator for SSD arrays that boosts the throughput of devices connected to the PERCController. It is enabled through disabling the write-back cache (enable write-through cache) and disabling Read Ahead.

So this means you can use LSI Megaraid Storage Manager to control your Perc H700 or H800. In my case, I found my H700 does not support any of the above as it’s only a 512MB cache version. However “SSD Caching = Enable” shows in the controller property under LSI Megaraid Storage Manager and cannot be turned off as there is no such option, I am not sure what this is (definitely it’s not CacheCade), if you know what this is, please let me know.

Then let’s move into something deeper regarding Perc H700’s bandwidth as I found the card itself can reach almost 2GB/s, this is again too good to believe!

The PERC H700 Integrated card with two x4 internal mini-SAS ports supports the PCIe 2.0 x8 PCIe host interface on the riser.

PERC H700 is x8 PCIe 2.0 (bandwidth is 500MB/s per x1 lane) with TWO SAS 2.0 (6Gbps) Ports with x4 lane, so total bandwidth for EACH lane is 500MB/s x 4 = 2,000MB/s (ie, 2GB/s).

EACH SATA III or SAS 2.0 bandwidth is 6Gbps, this means EACH drive maximum speed can produce 750MB/s (if there is such SAS drive), so it will take about SIXTEEN (16) 6Gbps 15K rpm disks (each about 120MB/s) in reality to saturate ONE PERC H700’s 2GB/s theoretical bandwidth.

A single Crucial SSD M4 is able to go over 1GB/s in both Read and Write really shocked me!

This means two consumer grade Crucial SSD M4 in RAID0 should be enough to saturate Perc H700’s total 2GB/s bandwidth easily.

From ESX Storage Performance Chart, it also shows the consistent IOPS with IOMeter’s result. (ie, over 35,000 in Seq. Read/Write).

From Veeam Monitor, showing 1.28GB/s Read and 1.23GB Write

In fact, not just me, in reality, I found out many people were able to achieve this maximum 1,600MB or 1.6GB/s. (yes, theoretical is 2GB/s) with two or more SSD under Perc H700.

Of course the newer PCIe 3.0 standard is 1GB/s per x1 line, so a x4 will give you 4GB/s,a 200% increase, hopefully someone will do a benchmark on Poweredge R720 with its Perc H710 shortly.

Some will say using a single SSD is not safe, OK, then let’s make it a RAID1, if not RAID10 or RAID50 with 8 SSD drives, with the newer Poweredge R720, you can put maximum 14 SSD to create a RAID10/RAID50/RAID60 with 2 hot-spare in a 2U, more than enough right?

The most important is the COST IS MUCH MUCH LOWER when using consumer grade SSD and it’s not hard to imagine 14 SSD in RAID10 will produce some incredible IOPS, I guess something in 50,000 to 100,000 should be able to achieve without much problem. So why use those sky high $$$ Fusion IO PCI-e cards forks?

Finally, I also did a few desktop benchmark from within the VM.

Atto:

HD Tune:

Conclusion, for a low cost consumer SSD, 100% Random RealLife-60%Rand-65%Read with 32K transfer request size, 7,000+ IOPS is simply amazing!!!