Question about Latest Equallogic Firmware 5.1 Feature: Auto Load Balancing
I’ve opened a case with Equallogic, but so far getting no reply yet.
As I understand Equallogic Firmware 5.1 supports Auto Load Balancing which the firmware (or software) automatically reallocate hot data to appropriate group members.
In fact, there were two great videos (Part I and Part II) about this new feature on Youtube.
As we knew the best practice before FW 5.1 is to group similar generation and spindles in the same storage pool. For example, place PS6000XV with PS6000XV, PS6000E with PS6000E, etc.
Now with FW 5.1, it is indeed possible to place whatever you want in the same pool (ie, different generation and spindles in the same pool) as Auto Load Balancing will take care the rest.
Dell called this Fluid Data Technology, actually a term borrowed from its recently acquired storage vendor Compellent.
My question is in terms of performance and updated best design practice, is it still recommended by Dell Equallogic to go with the old way? (ie, Separate storage tier with similar generation and spindle speed)
Update: Dec 12, 2011
Finally got the reply from Dell, seemed the old rule still applies.
The recommendation from Dell to use drives with the same drive speed is still the same within a pool. When you mix drives of different speeds, you slow the speed of the faster drives to the speed of the slower drives. The configuration should WORK but is not optimal and not recommended by Dell.
Then my next question is what’s the use of the new feature “Fluid Data Technology” if the old rule still applies? huh?
Update: Dec 21, 2011
Received another follow up from EQL support, this really solved my confusion now.
Load balancing is across disks, controllers, cache, and network ports. That means the storage, processing, and network ports access are balanced. Not ONLY disk speed.
Dell Equallogic customers create their own configuration. It is an option for you to add disks of different speeds to the group/pool; however, the disk spin speed will change to be the speed of the slowest drive. Most customers do not have a disk spin speed bottleneck; however, most customers are also aware of the rule-of-thumb in which they keep disks of like speeds together.
http://www.cns-service.com/equallogic/pdfs/CB121_Load-balancing.pdf
Update: Jan 18, 2011
After spending a few hours at Dell Storage Community, I found the following useful information from different person, the answer is still the same.
DO NOT MIX different RPM disks in the same pool even with the latest EQL FW v5.1 APLB feature!
Yes, the new improvements to the Performance Load Balancing in v5.1.x and the sub-volume tiering performance balancing capabilities now allow for mixed drive technologies and mixed RAID policies coexisting in the same storage pool.
In your case, you would have mixed drive technologies (the PS400E, SATA 5k and the PS4100X, SAS 10k) with each member running the same RAID policies.
When the PS4100X is added to the pool, normal volume load balancing will take the existing volumes on the two PS400E’s and balance them onto the PS4100X. Once the balancing is complete and you then remove the PS400E from the group (which is the requirement to remove 1 PS400E), the volumes slices contained on this member will be moved to the remaining two members and be balanced across both members (the PS400E SATA and PS4100X SAS) at that point.
Note, however, that Sub-volume performance load balancing may not be so noticeable until the mixed pools experience workloads that show tiering and are regular in their operating behavior. Because the operation takes place gradually, this could take weeks or EVEN MONTHS depending on your specific data usage.
Arisadmin,
Hi, I’m Joe with Dell EqualLogic. Although we support having members with mixed RAID policies in the same pool, in your case this is not advisable due to the two different drive types on your two members, i.e., your PS6000E is a SATA and your PS6000XV is a SAS. Mixing different drive types in the same pool, will most likely degrade performance in the group.
If the arrays were of the same drive type, i.e., both SATA (or both SAS), then combining the two (RAID 10 and RAID 6), would not be a problem, however the actual benefits, in your case, may not be as great as expected.
In order for load balancing to work “efficiently”, the array will analyze the disk I/O for several weeks (2-3) and determine if the patterns are sequential (RAID’s 10/5/6) or random (RAID 10), and would migrate those volumes to the corresponding member.
However in a two members group this is often less efficient, hence EQL always suggest 3 members group for load balancing.
Since the array will try to balance across both member, and you may end up with 80% of the volume on one member and 20% on the other member instead of a 50/50 split.
We also support manually assigning a raid level to the volume, but this would in effect, eliminate the load balance that you are trying to achieve, since it is only a two member group.
So in summary, we don’t recommend combining different Drive types (or Disk RPM) in the same pool.
You can go to http://www.delltechcenter.com/page/Guides and review the following documents for more information:
Deploying Pools and Tiered Storage in a PS Series SAN
PS Series Storage Arrays Choosing a member RAID PolicyRegards,
Joe
This is a very popular and frequently asked question regarding what type of disks and arrays should be used in a customers environment. First, you asked about the APLB (Automatic Performance Load Balancing) feature in EqualLogic arrays.
Yes, it will move “hot blocks” of data to faster or less used disk, but its not instantaneous. This load balancing between volumes, or better known as sub volume load balancing uses an advanced algorithm that monitors data performance over time. I recommend reading the APLB whitepaper that should help you out more in better understand how this technology works.
see here: www.virtualizationimpact.com
In terms of what disks to buy, well that comes down what you are going to require in your environment. From my experience and from reading on other forums, if you are trying to push for the best performance and capacity I would look at the X series arrays or 10K series drives. You can now get 600GB 10K drives in 2.5 and 3.5 form factors (i believe) and you won’t have to worry if your 7200 drives will be able to keep up with your workload, or at least, be faster and mix them with the 15KSAS/SSD arrays. Not saying that the 7200’s won’t work, just depends on your requirements.
Hope thats some help, maybe someone else will chime in with more info too.
Jonathan
Best solution is to work with Sales and get more specific about the environment than can easily be done via a forum. You’re entire environment should be reviewed. Depending on the IOPs requirements will determine if you can mix SAS/SATA in same pool and still achieve your goals.
One thing though it’s not a good idea to mix the XVS with other non-XVS arrays in the same pool. yes the tiering in 5.1.x firmware will move hot pages but it’s not instant and you’ll lose some of the high end performance features of the XVS.
Regards,
-don
I have attend a 2.5 days lab last week, at Dell HK office.
The tutor (Mr. Cai from Dell Xiamen) said it is not necessary after FW 5.1, you can mix different grade EQL, RAID level in a pool.
Thanks for the feedback.
I am still a bit unconvinced and waiting for the official reply from a L2 EQL engineer in US.
However that’s not important now.
What’s important is how come I didn’t know about that complimentary EQL training at Dell’s HK office last week? Haha…:)
Greetings from Denmark .
Im very curious about the answer myself.
I’ll be getting a PS6100XS on Monday and I was kind of planning putting it in a pool with 3 PS6000xv running raid50 and letting it all tier out.
Btw I’m planning on doing some io tests etc on the array too if your interrested
For the Training, it is for the new user who just purchase the EQ.
I know the First training is on May, at Xiamen.
Darking,
Yes, pls do some IOMeter IOPS tests and let us know if the auto tiering really works.
Thx.
First impressions.
Solid build quality, both the disk caddies and the powersupplies/controllers seem of a sturdy construction.
Uploading firmware happends at wirespeed, compared to my old PS6000 boxes that transfer firmwars at around 40KB/s
Raid is building atm, will try to do some performance testing off hours tonight. Im a bit limited in only having 2 Ethernet cables connected atm, but i dont have time for extra wiring today.
By wirespeed i mean, it took literally 1 second to upload the 5.1.2 firmwar (the box is delivered with 5.1.1h2)
Lucky you to have a PS6100XS, it’s price is equivalent to a new Merc E300.
Btw, I would love to see some IOMeter numbers especially RealLife-60%Rand-65%Read.
Also, can you post some close up photos of the disk tray, controller, I really want to see the new chipset inside the storage controller card. As far as I know PS6100 is Quad Cores and PS4100 is Dual Cores.
However, I still don’t understand why the price for new PS4100 series is very close to PS6100, this really made PS4100 redundant. In other words, you pay only extra 1/17 more to get a PS6100, no one will even consider PS4100 series from now on.
I Will see what I can photograph tomorrow.
There is a cover over the controller modules but I’m not sure if it’s serviceable like the old ones.
I’ll do some iometer tonight. Going to use the icf file from VMware community both with directly mounted and as an small vmdk
So i did my first test.
This is from a VMDK mounted harddrive on a Server running MEM.
The array is still verifying, but im not sure if that has any influence on Performance, i will test again after its done.
Latencies seem a bit high, but i recon its firmware 5.1.2 ive seen allround greater latencies since ive upgraded, and support has not been able to find any issues with my arrays themselves. Im going to switch my Procurve 5412ZL out with my old Dell 6248’s soon, and see if that fixes my latency issues.
anyhow. here is the first results. The run of iometer was only 5 minutes, for the test tomorrow (when fully cabled) ill try and do a longer run:
SERVER TYPE: Virtual win2k8r2
CPU TYPE / NUMBER: AMD 2376 2 Cores
HOST TYPE: Dell M805
STORAGE TYPE / DISK NUMBER / RAID LEVEL: Dell PS6100XS (Raid 6 Accl) Verifying
|*TEST NAME*|*Avg Resp. Time ms*|*Avg IOs/sec*|*Avg MB/sec*|*% cpu load*|
|*Max Throughput-100%Read*|8.04|7343|229|16%|
|*RealLife-60%Rand-65%Read*|14.28|3874|30|32%|
|*Max Throughput-50%Read*|7.16|8211|256|13%|
|*Random-8k-70%Read*|12.66|4372|34|32%|
Oh, and since its not apperant from the log, i did it with a 8000MB IO test file, as to not hit cache.
I think you do need to wait for the RAID6 Accl. verification to complete as the IOPS number is a bit low.
This is when comparing to my results a year ago with a single PS6000XV (http://www.modelcar.hk/?p=2854).
I am pretty sure when your SSD caching kicks in, you will see at least 38740 or even 100,000 IOPS for RealLife-60%Rand-65%Read. Btw, I got 7,140 IOPS with just a single Crucial M4 128GB SSD recently, it’s really amazing! Think about this, a single SSD can replace 2 units of PS6000XV with 24 15K RPM spindles at only 1/1000 the cost.
Pls try again later and run it say at least 10 minutes as this will ensure the hot data flows to SSD tier. (Fluid Data Technology)
Finally may I know what brand of the EQL 400GB SSD that is? and photos of course.
And I still have one more question that always bothering me, if anyone knows the answer, pls drop me a line.
Is 24 x 10K RPM disks in RAID50 faster than 12 x 15K RPM disks in RAID10?
The Ssd is Pliant (now sandisk) LB400M disks and the harddisk are savvio 10k4
Will have sone photos to upload to flickr in 4-5 hours
I actually read about Pilant (Sandisk) product last week.
http://www.sandisk.com/enterprise-storage-solutions/lightning-products/lightning-lb-25-inch
In fact it’s e-MLC (Enterprise Grade MLC)
Btw Darking, may I know where are you from?
Located in Denmark
Ive uploaded a few shots, but my camera work is horrid. Most of the stuff is under passive cooling, and im not going to remove any of that.
http://s1185.photobucket.com/albums/z356/DarkingDK/
I did another IOPs test after the array has verified, but im not seeing a huge increase.. around 6500IOPS in random 65%
But i still have a bit to do with network ports etc, im saturating the existing ones.
It seems its more a question of my hosts beeing saturated.
I setup 2 machines a virtual and a physical, and i can hit around 7000iops Rand 65
Many thanks for the great photos! What an insider report!
I am hardware nuts who love to see the chipset and processor on all sort of SAN storage and network appliances.
I saw that 4GB (2 x 2GB) cache on each controller card. Hey, if I were you and still testing this PS6100XS unit and not ready for production, then I will put two 8GB into each controller and test the IOPS again, so you have total of 16GB cache instead of 4GB, four times the caching, but I doubt it will help much, probably 30% more IOPS. Also it may be hard to locate what type of DIMM they use and find a compatible one.
Now I finally saw the new EQL Controller (Type-12) with THE PURPLE SUPER CONDUCTOR! The whole controller board is much larger than PS6000 series. There are three heat sinks, I wonder if those are the processors. (don’t worry, I won’t ask you to open the heat sink and see if there is a Xeon or Opteron inside, lastly I learnt PS5000 is based on AMD Opteron though)
This is new stuff, that Cache to flash using super conductor, something like this, you will no longer restrict to 72 hours of time limit, but data is stored on the flash forever. In additional, is there any more BBU?
I recognize the 2.5” SAS 10K RPM 600GB disk is actually SEAGATE SAVVIO 10K4, it’s the same as my 600GB 2.5” used on R710 but with special EQL firmware. I do know there are people who put their own SATA hard disk into out of warranty EQL box and able to use it, so it seemed EQL will not block non-certified DELL drive (May be still true, may be not). Oh…that Pliant SSD looks cool! I found a single LB400M 400 SSD costs over USD4,000 already, I bet near half of the cost of PS6100XS went to these 7 units of SSD, so you better fully utilize them and make them run to 100,000 IOPS range.
The last picture really shows how slim PS6100 compares to PS6000 underneath. Your new box with 24 flashing lights definitely looks like a charming Xmas Tree.
Oh…I do think your PS6100XS can go a lot higher at least in 20,000 to 50,000 IOPS range, even a single SSD can hit 7,000 IOPS easily these days, you may want to contact EQL for deep investigation.
Last year, they helped me trouble shooting my EQL performance problem over WebEX for almost 6 hours. I truly appreciated the high quality of their technical expertise and professionalism.
BTW, Denmark is really far from where I am now, but you seemed to be just next to me. It’s very nice to share and learn EQL experience with others around the world. I really love this small but warm community.
Finally Denmark Cookies (that Blue Label) is a very popular gift during Chinese New Year in Hong Kong.
I Will try and intergrerate it Info my storagepool With 3 x ps6000xv boxes soon. They are All running raid50. Will fianally show me if it is possible to have volumes over more than 3 members, and probably help on my VDI performance.
Just have to check up with equallogic support via diag logs if everything is ready for production.
One thing I thought you might find interesting. It has 5 nic ports on the controller, port 5 is for management, but you can run it like I do with the ps6000 usingg the first 4 ports for mixed traffic.
The odd thing is.. The new vertical port failover mechanism makes port 0 and 1 be active on controller 0 and 2/3 active on controller 1.
At first I thought it was a mistake, but it seems to just work like that .
It allows for failover of if one switch dies, which did not happen on the older series controllers
Darking please let us know your PS6100XS benchmark result when you get a chance. I’m debating to get bunch of PS4100XV, one PS6100XS, or PS6100S but can’t find any benchmarks for the PS6100XS and PS6100S. Greatly appreciated!
I would not expect more than around 7000iops from the xs model with 65% random 8k.
I tested with 4k and hit around 10k.
Compared to consumer ssd’s it’s fairly low but you need to remember its a 7 disk raid6 set for the ssd’s. And that emlc drives are supposed to give good consistent io and high write endurance (the drives re rated at 7.3 Petabytes).
In sanhq I’ve seen io on the drives of around 2.000 iops a piece.
I’m sure the ps6100s can deliver a lot more 20-25k maybe because you can just make raid10
Then again it’s hard to setup test parameters. I’ve used the 8gb test from vmktree.org/iometer
This post will be read a lot for sure as people keep googling for APLB. Nice post and rose questions in my mind as well as I read through. As I was reading this post in parallel with Dell tech report TR1070 I was leaning toward what support guys told you to keep arrays in different pools. But at the end of the report it says this which changed my mind completely and now I even question the knowledge of those support guys:
by using latency as the primary criteria, the APLB does not need to explicitly evaluate any other details of the storage, such as disk type (e.g. SAS vs. SATA), spindle speed, number of disks, or EqualLogic controller type. This makes the APLB a very simple and robust mechanism that does not need to be re-trained when new hardware configurations are introduced to the EqualLogic product line. This also ensures that when unplanned events occur that may influence the ability of certain arrays to serve their workload (e.g., RAID rebuild or bad NIC) that the system automatically compensates.
I hope this will help with your tiering. For me, I will still separate my SATA from my SAS arrays, but will keep 10K SAS in the same pools as my 15K SAS.