Comments (3)

By admin, December 2, 2011 8:57 am

During the Equallogic Seminar yesterday, Dell’s storage consultant introduced DPACK (Dell Performance Analysis Collection Kit) to us.

Basically it’s a Monitoring and Reporting utility that will run over time (4-24 hours) to collect data (such as CPU, Memory, IOPS, network traffic etc) from your physical or virtual servers.

The best thing is the footprint of this program is very small, about 1.5MB, still remember that crappy Dell Management Console 2.0?

The best part is you don’t need to install it at all, simply run it, key in the server or vcenter parameters, select a time range, run it, wait for a few hours, that’s it.

The output is a single file with extension .iokit, then you are supposed to use the online generator to get the DPACK result in PDF. (http://go.us.dell.com/DPACK). However, the site is currently not working as it’s still not ready according to DPACK team.

So what to do? DPACK team is kind enough, I sent the result.iokit to them and Dell helped me to generate the PDF manually.

The Overview of DPACK and Download DPACK.

Finally, I would say it’s a lite version of Veeam Monitor and Veeam Monitor does a lot more and it’s also free.

Comments (0)

By admin, December 1, 2011 4:04 pm

Despite multiple day, Veeam has finally launched it’s flagship product Veeam Backup & Replication version 6.

One of the most useful features I found is “One Click Restore”, because most of the time we do not have the root password of our client’s servers and often clients ask for mistakenly deleted files, so this enhancement is definitely the key reason for anyone to upgrade to v6.

Besides new improved replication is now 10x faster (see this appreciation post), together with throttling bandwidth capping, finally, we can replicate smoothly over our 100Mbps link between production and DR sites.

Last but not least, the scalability of backup agents (server in Veeam’s term), Veeam B&R 6.0 now utilizes proxy concept, so you have a group of backup agent/proxy or bots that will evenly distribute the loading among them. What’s best, you no longer have to login to each backup server (or agent) and you can manage all of your backup jobs on all proxies from a single panel, isn’t this just beautiful!

What about v6 now supports Hyper-V? Dohhhh…Hper-V in my own opinion is still not a mature product and it’s fine for lab use, for enterprise? No Way!

After losing a lot of sleep over the weekend working out the kinks and fixing my mistakes, v6 is running great and I can’t believe the speed increase!

Just a few examples:

• Replicating a file server with 750GB of data, down to 25 minutes total (replicating the data disk only took 6:30)!
• Replication job on my Exchange server w/ 400GB of mailbox data… 20 minutes
• Backup job with 6 VMs ranging from 50-120GB, 35 minutes
• Replication of a 60GB VM running Accounting software over a 20Mbps WAN… wait for it… 4 1/2 minutes!

The kicker here is I’m running 3-4 jobs in parallel now. My backup window has gone from 6+ hours to around 2 hours for nearly 3TB of VM data and I’m finally saturating my disk writes on the backup target. There are, of course, a few minor bugs, but I’m impressed with the quality for a first release after such major architecture changes.

Comments (1)

By admin, November 21, 2011 3:11 pm

Yes the VBK file created by transform process at the end of the week/month is enough to recover the latest VM state.

You can even extract that VBK without Veeam server present. We ship a tiny extract.exe tool (look for it in the product installation folder). No matter which media you use to backup your VBK files, just remember to put this tool there as well, and you will be able to restore data from the backup file even in 10 years from now – provided that media is still functional.

You can restore whatever restore point with the VBK. There should be some more information, such as syntax, if you do a find for “extract” in our B&R Users Guide. (Page 97 & 98)

Even if you lost the SQL backup, you could fire a new Veeam installation up (which takes less than 5 minutes) and do “Import Backup” and import your vbk/vrb/vib files and restore from those if you need other restore points.

The DB information is more about holding your job information and making sure your jobs run, you can import the data needed for restores from the files themselves.

So in other words, we  really don’t need to backup Veeam SQL data and send to DR site as this extract.exe tool will be enough even we lose the whole production site. All we need is the latest VBK file.

Comments (4)

By admin, November 21, 2011 1:52 am

Sorry, it’s really long, but I do think many will find it very interesting as it involved many things (USB 3.0 HBA, 2.5” 7200RPM external disk, Poweredge R610 PCI-e x8 slot and of course low cost Veeam Backup Solution)

I found myself running out of Veeam backup space recently. Instead of ordering an expensive DAS, I started to explore other alternative solutions that focus on low cost storage.

My Veeam Backup server is a 1U Poweredge R610 server which has 900MB internal storage (10K RPM SAS 300GB x 4 in Raid5) and it has only one PCI-e slot (Gen 2 x8) left.

The original option is to purchase a Powervault MD1200 with 12 2TB 7200RPM near-line SAS disks in Raid50 + 2 hot-spare configurations. PERC H800 raid card (with multi-path load balancing SAS Cables) is going to be used in that extra PCI-e slot. The expected performance should be around 500-600MB/s. One thing keeps delaying the client’s purchase is the cost as it is quite expensive considering there are only 16TB left (Raid50 with 10 2TB disks).

USB 3.0 really comes into play this year, giving its 10x improvement in transfer speed comparing to the old USB 2.0, it quickly captured main stream desktop and laptop market, but not in server at all.

So all I need is to find a USB 3.0 HBA with 2 or more ports, but the problem is which one?

After studied for a few days, I found out I need a latest USB 3.0 PCI-e HBA (ie, add-on card) with PCI-e GEN2 standard and each USB port has individual 5Gb/s bandwidth, so I can copy data must faster between the two USB 3.0 disks without saturating the internal bandwidth.

* Note: Bandwidth for PCI-e Gen1 is 2.5Gb/s or 250MB/s and PCI-e Gen2 is 5Gb/s or 500MB/s.

The other thing is selection of USB 3.0 disks, there are two choices: 1st one is the latest 2.5” external USB 3.0 disk that doesn’t require external power and produce very little heat. 2nd choice is to use the traditional 3.5” disk with external power supply.

In additional, you also have the choice of building your own (using USB 3.0 disk docking) or buy an existing external hard disk box either in 3.5” or 2.5”, and my preference is to go for a 7200RPM 2.5” external hard disk box that is solely powered up using standard USB 3.0 cable.

Obviously, the advantage of DIY is that you can have a higher rotation disk like 7200RPM to be placed in the disk docking because most of the pre-build external disk box are 5900RPM for 3.5” and 5400RPM for 2.5”. Of course the BIG Disadvantage is you may encounter unlimited possibilities that the disk may not work with your DIY docking or USB 3.0 box.

So I selected to play safe and I bought two disks, one is the 2.5” 7200RPM Hitachi TOURO, I wished there is a 1TB or higher 7200RPM 2.5” version, but only 750GB are available in the market currently and 1.5TB is the maximum size for normal 5400RPM 2.5” disks.

Comparing to 3.5”, disk is much bigger in size, there is even a 4TB from Seagate.

However hard disk price is really becoming more and more expensive than 2 months ago (1.5x higher in Hong Kong now than before the flooding in Thailand), so I decided to buy Lacie Minimus 2TB USB 3.0 mainly due to its price per GB and the nice looking aluminum case, actually this is important as it’s better for heat dissipation in a 24×7 data center environment.

Later I found out it’s actually a Seagate Barracuda Green Disk, Seagate ST2000DL001 (9VT156) 2TB 5900 RPM 32MB Cache SATA 3.0Gb/ s 3.5″, it’s only slightly expensive than other 2TB external disks and has 2 years warranty, but who cares.

Besides Lacie Mimimus 2TB is still the smallest in size among all external USB 3.0 disks even after 1 year of its launch back in Nov. 2010.

Oh one more thing I noticed that after the flooding in Thailand, internal 2TB harddisk is even more expensive than those external ones with box and USB2/3, e-SATA connector, Ridiculous Really! Lucily I also ready enterprise hard disks were not affected as most of those FC/SAS/SSD were made in China or Philippines.

Side topic: I didn’t know until yesterday a 3.5” SATA 2TB 7200RPM is actually faster than a 3.5” SAS 73GB 15K RPM. Mainly due to the hard disk mechanical head only requires shifting a tiny bit to read a total 73GB data comparing to a 73GB 15K RPM disk. The same theory applies to a 2.5” 5900RPM may be equivalent to a 3.5” 7200RPM disk due to the physical size and hard disk mechanical head doesn’t need to move that much in a 2.5” disk.

Back to the topic, after searching for many USB 3.0 PCI-e add-on cards, I was only able to locate one that fits all my requirements (ie, PCI-e GEN2, USB port with individual 5Gb/s bandwidth per port).

It is the latest release of RocketU series (September 2011) from HighPoint Tech (a Taiwanese storage company). Because I only need 2 USB 3.0 ports to start with, so I selected RocketU 1022A: Low-Cost 5Gb/s Dual-Port USB 3.0 HBA with PCI-e GEN 2 x1. Later I found out it uses ASMedia as the chipset which was reported better than the original NEC chipset in performance.

Highpoint claimed its RocketU product lines is the first and only one (so far) that has Dedicated 5Gb/s PER PORT performance, so I can easily get 10Gb/s with just two ports. (Only SSD may be able to saturate such high bandwidth)

I also noticed there is a white power supply connector (Molex) on top right corner which later proved I got the wrong USB 3.0 card. In additional, I have to order it on-line from Taiwan directly because none of the shops locally carry such product.

The fun part begun, I plugged in the Highpoint RocketU USB 3.0 HBA into Poweredge R610 x8 PCI-e GEN2 slot, installed the W2K8 R2 driver, OS recognized it without any problem, then connected the 2.5” 7200RPM Hitachi TOURO, there was no blue light at all, actually nothing at all. (ie, compare to USB 2.0 port, blue light will come up if I plug it into a USB 2.0 port), probably the HBA doesn’t have enough power if I did not connect the HBA to the external power, Damn!

No problem I told myself as I still got that 3.5” Lacie 2TB with external power (ha…glad I prepared ahead and had plan B), connected Lacie to USB 3.0 port, turn on the power, and guess what? Nothing happened!

Then I email Highpoint support that night and it turned out RocketU 1022A does require the external power to operate normally, OH NO!!!

So my next search is to find a way to split the power from within R610 chassis, bad luck again, there is no way at all!

I did a deep search in this area and found others actually faced the same/similar issue with no extra power connector (Molex connector) in R610/R710 for their FusionIO SSD PCI-e card which is sold by Dell as well.

For those who has R710, they are lucky and a workable solution is available for them to use this Dell Part (GP700 ASSY,CBL,PWR,BKPLN-CD-TB,R710), the split power connector, but for R610 owners, it’s just the end of the world, PERIOD!

I’ve called Dell Pro-Support several times and confirmed the above, they suggested not worth DIY myself the power splitter as it will definitely jeopardize the stability of the server and stick to USB 2.0 instead, then I gave up and used USB 2.0 for two days, but the sustained 29.8MB/s really pissed me off!

I won’t give up, ah…there is another way, there must be a USB 3.0 HBA doesn’t need any external power supply in this world, so I Googled again and located Asus U3S6 (it’s a PCI-e GEN2 x4 product as well, but no individual port with 5Gb/s) and Western Digital (it’s a PCI-e GEN1 x1 product) has such thing, bad thing is neither can be found in Hong Kong any more as they are out of stock as the products are more than 2 years old.

I visited almost all major computer malls in Hong Kong (probably more than 200 shops), 99.9% told me all USB 3.0 HBA requires an external power connector and even I found one that without, it must be a bad design that won’t work.

I almost gave up but I still have the last drop of hope that I will find one in-time!

and I DID! I found one shop finally carries one of these rare than gold card.

It’s a re-branded Syba OEM USB 3.0 HBA (SY-PEX20081, EtronTech is the chipset, proved faster than the original old NEC one as well) without the requirement of external power supply, the most amazing part is the cost, it’s only USD12.8!!!

Inside it also indicate it does support Windows Server 2008, many other USB 3.0 HBA driver only supports Windows 7, I understand they are very similar, but I dare to take the risk.

Traveling between computer malls, so many shops, data center and finding the correct one is really tiring even Hong Kong is a tiny place. If I remember correctly, I spent at least 20 hours in searching for the right products (HBAs and external disks) and testing them.

Finally, I plug it into R610 slot, reboot, installed the driver, everything went well.

Then I cross my figures and plug in that 2.5” Hitachi TOURO 750GB 7200RPM, the blue light turned on, then dimmed again and then turned off! OH NO, this means the USB 3.0 port doesn’t have enough power to support USB 3.0 2.5” disks.

Now I only have one last chance which is to test the 3.5” Lacie 2TB with external power supply.

I plug the Lacie, turn the power on, nothing coming up as USB device!!!

Does this mean GAME OVER? OF COURSE NO, I remember Veeam 5 automatically turns off disk automount to prevent accidentally erase the SAN disk when using Direct SAN backup. All I need to do is to go to Disk Management and manually give the attached USB 3.0 disk a drive name.

BINGO, 1.87TB was right there and right in front of my eyes! YEAH! I couldn’t be happier for the rest of the day!

Then I started the fun part, testing the USB 3.0 performance using IOmeter, HDTune and ATTO Express and hoping I would get the same magically 130-140MB/s like many others did.

Disappointing is the only word when I see 35MB/s in sequential read, unplug the card, re-plug again, still the same result. I suspected it’s the driver version (Jan 2011) being too old, but I couldn’t find any newer ones on Syba product page, so I googled again (Just love google more each day), found the latest one from Gigabyte web site. Search for Etron chipset USB 3.0, found it, now the latest installed driver version is Aug 2011.

After reboot, it’s still 35MB/s, only 17% faster than USB 2.0, definitely not worth upgrading and all the trouble I went through.

So I started to email Syba support, the guy suggest it may be a defective card, so I went out to the same shop again and bought a new one the next day, tested it, amazingly, this time it shots to 138MB/s from 35MB/s, almost 4 times gain in performance, I am a happy man finally!

OK, I admit this Syba HBA is a PCI-e GEN1 x1 product and my Lacie is a 5900RPM SATA 3.0Gb, not even a 6G/b product, probably the HBA is one of the lowest bandwidth one, but still it can provide maximum 250MB/s, so it will take at least two USB 3.0 ports with two USB 3.0 disks to saturate the total bandwidth. Most important it’s more than enough for Veeam backup now.

Finally, I went back to the original shop and explained the whole story AND exchanged the defect one, I kept the swapped part as a spare just in case.

So who needs a Powervault 1200 with 2TB/2TB x 12 disks now? Probably nobody will from now on, ops sorry Dell.

If I need more bandwidth or storage in the future, I will only order a HighPoint RocketU 1144A which comes with 4 ports also doesn’t require external power supply!

The most finding of the whole adventure is HOW could I miss this perfect product in the first place! It’s a PCI-e GEN2 and 4 ports with individual bandwidth of 5Gb/s and most importantly no need for external power.

With HighPoint RocketU 1144A, I will have 4 Ports x 4 TB disks= 16TB usable, comparing to Powervault MD1200 with 12 2TB disks in Raid50 + 2 hot-spare, same 16TB usable storage, but only 1/5 the cost with 13xMB/s in read and write, well, it really doesn’t take a genius to figure out the ultimate solution.

Finally, I also came across some USB 3.0 drive bays up to 8 disks in the same case with build-in Raid5/6, and guess that will complete the low cost backup target solution.

Comments (1)

By admin, November 19, 2011 11:58 pm

After almost a whole full year of scratching my head, blaming data center for not changing to the latest rack standard. I finally figured out today how to fit the longer than usual Dell Poweredge Versa Rail into the old rack.

The solution always lies in details and only if you look carefully.

I was working on something else in the data center today and suddenly I saw that disgusting Poweredge R610 rail again. It’s been sitting there for almost 1 year and I really hate to see it’s being wasted and not in line with the rest of 11^th generation Powerdge/Powervault series. I used a rail angle instead as the original R610 versa rail was too long for the rack and I couldn’t close the back door.

As I was staring at the LONG TAIL, suddenly inspiration spiked again! Yeah! How come I didn’t think of this by removing the extra long bit from the end before?

It’s really simple as that and only if you look carefully, I think this method will work for most of the 1U longer than usual versa rail.

Finally, if you know the purpose for that extra part, please do drop me a line.

Comments (1)

By admin, November 2, 2011 3:28 pm

One of my clients requires W2K3 with MSSQL 2005 and .NET 4 as well as Silverlight4, so I started to deploy the VM from Windows Server 2003 Standard SP2 template, the template itself is configured with the most basic features, there is no IIS or .NET components.

I’ve encountered a few problems that really surprised me:

When I installed IIS, it asked me for both the Windows Server 2003 and SP2 CD-ROM (it never did if I use a W2K3 without SP2), where do I get that SP2 CD huh?  After a while, I figured out the solution is simply to extract the files from the SP2 zip then it completes the whole IIS installation.

Then I encounter another problem when installing SQL Server 2005, the installation failed with error code 1063, Google helped me again, the solution is to install .NET 2.0 first as SQL Server installation files depends on this foundation.

Finally, the client requires .NET 4.0 and Silverlight 4. Luckily, I came across this blog page “Configuring Windows Server 2003 with ASP.NET 4.0 while supporting ASP.NET 2.0“.

Comments (2)

By admin, October 26, 2011 10:21 pm

I’ve asked Equallogic Support the following question today, it turns out I had wrong understanding about RAID10 before.

Client enviornment is running VMWare using EQL PS6000XV, 16 15K RPM disks (ie, 14 disks + 2 hot-spares)

In RAID10, there are 7 spindles as RAID10 only ultilizes half the disks, so there are only 7 spindles producing the atual IOPS.

In RAID50, all 14 spindles are used.

I/O pattern is mostly random (30% Read, 70% Write)

Does this mean in this case RAID50 is faster than RAID10 in terms of IOPS as there are 14 spindles in RAID50 compares to 7 spindles RAID10?

Reply from Equallogic’s US Support

The thing to remember here is that with RAID10 it will make two writes, one to each disk.

With RAID50, it will need to make the read/write/parity to all of the member disks in the raidset.

With Read/Write sequential IO, the difference would be minimal. With an environment as you described (virtual environment) where there are many small random I/Os, RAID10 will be quicker.

RAID10 will also be quicker if a drive fails whereas the performance with RAID5/50 is poor when the raidset is degraded and a rebuild is going on.

And of course, the space issue when considering RAID10 or RAID50 is a huge consideration

 

Then, I got an even better answer from a Dell storage expert in Hong Kong

Actually spindle in RAID10 is still 14, not 7; 7 is the capacity oriented view. We still consider using all disks during I/O.

But you may notice that above chart didn’t mention for Read performance? And yes, read performance for RAID 10 and RAID 50 or even RAID-5 will almost the same!  The different will mainly focus on write.

RAID 50 is two RAID-5 and stripe, the RAID parity overhead will still exist across the disks; the more disks you have in the RAID group, the more overhead and performance penalty you have!

In RAID 10, every RAID group consists with two disks only, so overhead and performance penalty is minimum.

 

Comments (0)

By admin, October 25, 2011 1:28 pm

No more worries using the pre-released or pre-production versions. As usual, it’s recommended to wait for at least another month before upgrade to the latest firmware.

I am going to try the latest SANHQ and HIT/VMware this week for sure.

Equallogic PS Series Firmware, version V5.1.2
Enhanced Load Balancing, VSSA and Thin Provision Stun

SANHQ 2.2
Live View and 95th Percentile Reporting, as well as RAID Evaluator (ie, R10 > R50 performance prediction)

HIT/VMware 3.1.1
Support for VMware Version 5, VMFS 5, Thin Provision Stun, VASA, Datastore Clusters and Storage DRS

Finally, I also noticed the release of a newer version of EqualLogic Multipathing Extension Module (MEM) V1.0.1

Update: Oct 27, 2011

I’ve upgraded the SANHQ to version 2.2 and immediately tried two of the most interesting features,

1. Live View

It does provide very useful detail ONLY when you need it, such as when you suddently found IOPS is abnormal and wanted to find out which volume is causing the problem. However to be honest, I would rather use VMTurbo’s Usage for VM table, it’s much easier and easy to find the bad VMs.

2. RAID Evaluator

I found this is the most useful one in my own environment as I may expand my Equallogic SAN capacity by changing from RAID10 to RAID50, so I definitely need to know the estimated  performance decrease if I do so, it shows me 41%, wow, that’s huge!

I recevied a false alarm email with subject “SAN HQ Notification Alert” immediately after the SANHQ upgrade completed

Caution conditions:

10/27/2011 9:02:44 PM to 10/27/2011 9:08:08 PM

Caution: Controller failed over in member eql (IT DID SCARE THE HELL OUT OF ME)

Caution: Firmware upgrade on member eql Secondary controller.
 - Controller Secondary in member eql was upgraded from firmware version to version V5.0.2 (R138185)
 - Condition already generated an e-mail message. If the condition persists, additional messages will be sent approximately every 6 hours.

Caution: Firmware upgrade on member eql Primary controller.
- Controller Primary in member eql was upgraded from firmware version to version V5.0.2 (R138185)

Comments (0)

By admin, October 21, 2011 10:09 am

This is interesting as I have just found the solution, the hotfix is not available for download, you will need to contact Microsoft or your server OEM to get the hotfix, note there are two different hotfix versions besides the 32/64bits, one is for Win7/Win2k8R2 with SP1, one is without.

Symptoms

When deploying or cloning virtual machines from a Windows 2008 R2 or Windows 7 template configured with the VMXNET3 virtual network device:

•The resulting virtual machine’s guest operating system shows the Ethernet network interface as:

- VMXNET Device #2 in Device Manager
- Local Area connection #2 in Network Properties

•The old Ethernet network interface remains present in Device Manager when Show Hidden Devices is enabled.

•The old Ethernet network interface retains its network configuration, preventing the new interfaces from reusing the previous static IP addresses. For more information see Networking Error: IP address already assigned to another adapter (1179).

Note: Upgrading the virtual hardware from version 4 to 7, or upgrading the VMware Tools, may automatically convert Flexible or VMXNET2 virtual network interfaces to VMXNET3. This issue may be observed after such an upgrade.

Also I encountered the following error when trying to open “Edit Settings” after converted the Template to VM.

Call “PropertyCollector.RetrieveContents” for object “propertyCollector” on vCenter Server “VC” failed.

The solution is simple, remove the VM from inventory and add it back again will solve the problem.

Of course, you also need to add back the extra VMX parameters if you are using OEM Windows and make sure devices.hotplug = false is added which prevents hot plug warning in OS.

* Just found out you don’t need to do this if you added the VM to Inventory from Datastore directly as the original VMX contains such information already.

Comments (4)

By admin, October 13, 2011 1:12 pm

Interesting there is a 10 years glass ceiling, it was never mentioned in their web site I guess.

If you open a license file with notepad, you will see that there are two expiration dates: one for support and one for the product. Each license is issued for 10 years, so if your support expires, you can use an existing installation until the product expiration date is reached. You will not, however, be eligible for support and product upgrades.

Thanks,
Andrey Beck
Veeam Software Support