Category: Network & Server (網絡及服務器)

Equallogic takes time to kick in the additional paths under Windows MPIO

By admin, September 17, 2010 16:02

I’ve spent almost 4 hours on-phone from mid-night to 4am in the morning trouble shooting with Dell Equallogic Consultants in US via WebEx today.

eqlpsAs we found the EQL I/O testing performance is low, only 1 path activated under 2 paths MPIO and disk latency is particular high during write for the newly configured array.

It was finally solved because we forgot the most fundamental concept after all that is Equallogic takes time to kick in the additional paths under MPIO!!! You need to wait say at least 5 mins to see the rest paths kick in.

The followings are my findings and mostly email exchange with Equallogic support. yes, it is long and boring to many, but it’s extremely useful for some who are seeking the same solutions for this problem, I wish someone put it on their blog previously, then I could sleep much better last night.

Timeline as in Descending Order:

- 2pm

We found a very interesting fact that the 2ND LINK WILL ONLY KICK IN AFTER THE 1ST LINK BEING SATURATED/OVERLOADED for a period of time, see 1.gif and 2.gif, So MPIO with Dell EqualLogic DSM (not using Microsoft Generic DSM) is actually working perfectly now and before!

1.gif showing both links are activated, I saw the 2nd link (EQL Mgt 2) suddenly kicked in (may be we opened more copy windows to iSCSI target) and it dropped out again and then come back again when needed.

2.gif shows the performance of the two active ports on EQL Iscsi target also increased by a lot. (From 45% to 80%)

So I can pretty sure the issue doesn’t exist right from the beginning, it just TAKES TIME FOR THE REST NICs (LINKs) to be activated gradually over the testing period and according to loading situation automatically. Previously, we only tested for less than 2 mins, in other words, we didn’t give enough time for MPIO intelligent logic to kick in additional paths for throughput or I/O.

- 12pm

See attached TR1036-MPIO_EQLX-DSM.pdf PS Series Best Practices
Configuring and Deploying the Dell EqualLogic™ Multipath I/O Device Specific Module (DSM) in a PS Series

MPIO DSM Load-Balance Policy

Microsoft MPIO DSM allows the initiator (server) to login multiple sessions to the same target (storage), and then aggregate that into a single device. Multiple target sessions can be established using different NICs to the target ports.

If one of the sessions fails, then another session continues to process I/O without interrupting the application.

Dell EqualLogic MPIO DSM supports following balancing policies.

• Fail Over Only: Data is sent in one path, while other paths are standby. This connection is used for routing data until it fails or times out. If the active connection fails, then one of the available paths is chosen until the former is available. This load balancing policy is the default configuration when MPIO DSM is disabled.

• Round Robin: All available paths are used to perform I/O in a rotating sequence (round robin sequence). There is no disruption in sending I/O even if any of the paths fails. Using this policy, all paths are used effectively.

• Least Queue Depth: I/O is sent to the path that has least queue length. The performance analyses for the above load balancing policies are presented in the following sections.

• EQL recommend to use Microsoft DSM with “Least Queue Depth” load balancing policy on Windows Server 2003/2003

• To fully utilize Microsoft’s MPIO capabilities, Dell EqualLogic provides MPIO DSM that is complementary to ASM for both high availability and performance.

- 11am

I found something very important on google.

Device Initialization Recall that MPIO allows for devices from different storage vendors to coexist, and be connected to the same Windows Server 2008 based or Windows Server 2003 based system. This means a single Windows server may have multiple DSM’s installed. When a new eligible device is detected via PnP, MPIO attempts to determine which DSM is appropriate to handle this particular device.

MPIO contacts each DSM, one device at a time. The first DSM to claim ownership of the device is associated with that device and the remaining DSMs are not allowed a chance to press claims for that already claimed device. There is no particular order in which the DSMs are contacted, one at a time. The only guarantee is that the Microsoft generic DSM is always contacted last. If the DSM does support the device, it then indicates whether the device is a new installation, or the same device previously installed but which is now visible through a new path.

Does this means if we see multiple DSM in MPIO, DELL Equallogic will be always used first or it’s priority is always higher than MS DSM?

- 10am

Some update I found: Even I added back with mpclaim -r -i -d “MSFT2005iSCSIBusType_0×9″

MPIO is still showing Dell Equallogic is the DSM instead of Microsoft, how can I force MPIO to select Microsoft instead of Dell Equallogic as desired? That exactly explained why there is ONLY ONE PATH (or NIC) working at the same time, but not load balancing across two NICs.

I even did a real time test, by Disabling a NIC, then all traffic automatically shifted to the 2nd NIC (or path) and vice versa. So seemed Windows Server 2008 R2 doesn’t understand Dell Equallogic DSM for MPIO. In other words, if Dell Equallogic is the DSM, then only one path is available.

I also find out from Google, that Windows Server 2008 DOES NOT add “MSFT2005iSCSIBusType_0×9″ automatically like in Windows Server 2003, we need to add it manually from MPIO GUI or CLI.

See the output.

C:\Users\Administrator>mpclaim -s -d

For more information about a particular disk, use ‘mpclaim -s -d #’ where # is
he MPIO disk number.

MPIO Disk System Disk LB Policy DSM Name
——————————————————————————-
MPIO Disk0 Disk 2 RR Dell EqualLogic DSM

C:\Users\Administrator>mpclaim -s -d 0

MPIO Disk0: 02 Paths, Round Robin, ALUA Not Supported
Controlling DSM: Dell EqualLogic DSM
SN: 6090A078C06B1219D3C8D49CF188CD5B
Supported Load Balance Policies: FOO RR LQD

Path ID State SCSI Address Weight
—————————————————————————
0000000077070001 Active/Optimized 007|000|001|000 0
0000000077070000 Active/Optimized 007|000|000|000 0

C:\Users\Administrator>mpclaim -r -i -d “MSFT2005iSCSIBusType_0×9″

So the KEY question is how can we FORCE MPIO DSM TO USE Microsoft instead of Dell Equallogic?

- 9am

1. Removed MPIO from W2K8 Feature, reboot, then remove HIT, reboot, and re-installed again, reboot, under MPIO, still no MSFT2005iSCSIBusType_0×9.

2. This time, I changed the NIC’s Flow Control to TX & RX and reading performance of EQL also increased to 99%.

I do think we need to enable Flow Control RX as well, as we saw yesterday, only writing to EQL is working at 99%, but reading from EQL is at 20%, so this proved it’s required.

3. Also, disk latency for read is very small (39ms compares to 350ms for write) when we saturated the link using multiple 16GB files, however, writing to EQL and overloading the link still gives us over 300ms disk latency. Those high number of Re-transmit % all went down from 5-6% to 1-2%.

4. No more MPIO initiator dropping out problem even without MSFT2005iSCSIBusType_0×9 in place, it may not be necessary after all?
As I installed HIT twice, MSFT2005iSCSIBusType_0×9 is not there as always, I suspect manually adding it can actually cause more problem? Or shall I remove MPIO from W2K8 Feature and just install it again manually to see if MSFT2005iSCSIBusType_0×9 pops up?

Extra Notes:

MPIO CLI Comands

mpclaim -r -i -d “MSFT2005iSCSIBusType_0×9″
(Note: HIT installation on Windows Server 2008 R2 DID NOT add this to MPIO)

mpclaim -s -d

mpclaim -s -d device_name

mpclaim.exe –v C:\Config.txt

C:\Users\Administrator>mpclaim -s -d

For more information about a particular disk, use ‘mpclaim -s -d #’ where # is
he MPIO disk number.

MPIO Disk System Disk LB Policy DSM Name
——————————————————————————-
MPIO Disk0 Disk 2 RR Dell EqualLogic DSM

C:\Users\Administrator>mpclaim -s -d 0

MPIO Disk0: 02 Paths, Round Robin, ALUA Not Supported
Controlling DSM: Dell EqualLogic DSM
SN: 6090A078C06B1219D3C8D49CF188CD5B
Supported Load Balance Policies: FOO RR LQD

Path ID State SCSI Address Weight
—————————————————————————
0000000077070001 Active/Optimized 007|000|001|000 0
0000000077070000 Active/Optimized 007|000|000|000 0

C:\Users\Administrator>mpclaim -r -i -d “MSFT2005iSCSIBusType_0×9″

Equallogic and ESX 4.1 iSCSI Setup Crack Sheet

By admin, September 16, 2010 19:17

sanhqFor the whole month, my mind is full of VMWare, ESX 4.1, Equallogic, MPIO, SANHQ, iSCSI, VMKernel, Broadcom BACS, Jumbo Frame, IOPS, LAG, VLAN, TOE, RSS, LSO, Thin Provisioning, Veeam, Vizioncore, Windows Server 2008 R2, etc.

It’s definitely like taking an extremely fast track in getting my enterprise storage degree, and after all, it worths every penny of struggling, many long nights, endless calling to Pro-Support in Hong Kong and US EQL supports.

 

Equallogic and ESX 4.1 iSCSI Setup Crack Sheet to save you typing many commands.

  1. Configure iSCSI vSwitch using GUI first and assigned multiple NICs onto the vSwitch, in my case, it’s 4 NICs.
  2. Create multiple VMKernel on this vSwitch, in my case, there are 4 VMKernel (named iSCSI 1 to iSCSI 4)
  3. Removed the extra NICs from individual VMKernel by unselecting 3 of those NICs and do this for each VMKernel.
  4. # Enable Jumbo Frame on iSCSI vSwitch using CLI
    esxcfg-vswitch -m 9000 vSwitch4
    esxcfg-vswitch -l to verify MTU=9000
  5. # Enable Jumbo Frame on each VMKernel using CLI
    esxcfg-vmknic -m 9000 iSCSI1 – iSCSI4
    esxcfg-vmknic -l to verify MTU=9000* I also enabled Jumbo Frame for VMotion as well as FT network.
  6. Go to GUI, enable software iSCSI and note down the vmhba #, in my case, it’s vmhba47.
  7. # Bind VMKernel to iSCSI Adpater using CLI
    esxcli swiscsi nic add -n vmk2 -d vmhba47
    esxcli swiscsi nic list -d vmhba47 to verify if all 4 NICs are binded with vmhba47
  8. Do a rescan of the Storage, you will see EQL volume now. Please make sure you checked “Allow Simultaneous Connection…” under EQL volume property, or multiple ESX connection to the same volume won’t work.
  9. To verify from EQL, go to group manager, then click that volume, now you see there are 8 connections with 8 different IP Addresses (ie, 2 ESX hosts, with 4 NICs each)
  10. To verify from ESX host side, go to storage, right click Manage Path, you will see there are 4 IP Addresses from EQL.

 

Just got a reply from Equallogic support team regarding my customized configuration.

The document on the web site is the supported method of setting Jumbo Frames on the switch. This is the method that we have tested and confirmed to work.

Of course, as with many things, there is typically a method of doing this through the GUI as well. The method you are following appears to work in my tests as well, but we cannot confirm if it is a viable operation as it has not been tested through our QA process.

My suggestion would be to utilize the tested method. You may also want to check with VMware directly as it is possible that the GUI method you are utilizing simply calls the CLI commands we provide, but we cannot confirm that for certain (we do not have access to their code).

(Name Removed)

Enterprise Technical Support Consultant
Dell EqualLogic, Inc.

 

Finally, test ping your destination with a large message and specify don’t fragment.

  • Linux VMs:         ping –M do –s 8000 <ip address or destination>
  • Windows VMs:    ping –f –l 8000 <ip address or destination>
  • ESX(i):                vmkping –d –s 8000 <ip address or destination>

Dell Poweredge R710 iSOE key DDR3L Broadcom Quad NICs

By admin, August 27, 2010 12:09

Finally I’ve got time to inspect each individual part thoroughly and the following is my findings. 

  1. Dell Powerdge iSCSI Offload Key for LOM NICs. Strange funny little stuff that makes a hell lot of difference for some people. Broadcom charges this for extra on their 5709 NICs (5709C not 5709S), the same apply to HP Pro-liant NICs.According to one of the EQL engineer we talked to, it is still best NOT TO use 5709C as ISOE HBA in ESX4.1 as it will lost Jumbo Frame feature and some other nice features will be gone if HAB mode is used with EQL boxes.IMG_2728

    IMG_2729

  2. DDR3 Low Voltage ECC+Buffer R-DIMM 8GB by Samsung: It’s nice to have that 20% saving, but when you add 2DPC, your nice 20% power saving (ie, 1.35V) will be disabled automatically (ie, raise to 1.5V instead), good part is you still have that 1333Mhz bandwidth with 2DPC. DDR3L 1.35V will only apply when it’s in 1DPC mode.IMG_2735What about 3DPC? Old story applies, it’s 800Mhz, tested it and proved it and if populate with 3DPC and fully filled that 18 DIMMs (ie, 144GB), it will take twice the time to verify and boot the server, so it’s better not to as you lost 40% bandwidth is very important for ESX.

    IMG_2736

    Dell’s on-line documentation has no where indicates my findings and my findings completes what I found at HP’s resources previously. Btw, why does DDR3L still need that aluminum cap for heat dissipation if it’s voltage is really that low?

    More about Samsung’s DDR3 Low-Voltage Ram

  3. Broadcom NetXtreme II 5709 Gigabit NIC w TOE & iSCSI Offload, Quad Port, Copper, PCIe-4: Nice to have two of this besides the embedded quad NICs, so total you will have 12 NICs within one server. The chipset is still BCM5709CC0KPBG, there is no iSCSI key found on the NIC, guess it’s been embedded already as well.IMG_2732

    IMG_2730

Some of my findings from past 2 months research

By admin, August 9, 2010 23:21

The new Virtual Data Center project has been keeping me really busy, the followings are some of my findings.

  • Learn that R810/M910 with 4 sockets will only use 1 memory controller instead of two, memory bandwidth will be cut by half, that sucks! Strange enough, when R810 populates with 2 sockets only, it will use both memory controllers and gain access to all 32 DIMMS. So R810/R910 is still best for 2 sockets server, that why we switch to R710 instead after read the benchmarking, it’s a waste of money to go for R810.
  • DDR3L low voltage (1.35V), we populate with 2 DPC (8GB x 12 DIMMS), guess what? Voltage will shoot to normal 1.5V! and no one at Dell pre-sales or pro-support can answer us this, I found out this fact from HP’s resources, ridiculous! Anyway, it’s still running at 1333Mhz, that’s a bless.
  • Equallogic PS6000XV 15K should be a monster, we will only be worried about adding front-end (like R710/R720/R730) and adding back-end PS6000XV/PS7000XV etc. in the future, that’s main selling point for this solution, scalable beyond imagination, this really is the whole motivation we selected EQL boxes.
  • With the release of vSphere 4.1, VAAI, iSCSI Off-load fully support on Broadcom 5709 but no Jumbo Frame, what the heck! EQL vStorage off-load improvement, Multipath plugin (EQL finally solved the big problem)
  • One box of PS6000XV is good enough for 4Gbps, absolutely no need to go for 10Gbps for the time being except you are aiming for that 200M/s increase (yes, it can only reach 650M/s at max, there is no way reaching 1000M/s in reality. We were also told by paying about 1/4 of the box, you can always upgrade to 10Gbps version of PS6010XV in the future, but for our environment, IOPS is way demanding than thoughput, so 4Gbps is more than enough, just get a PowerConnect 5448 with 48 ports, we should have lots of space to grow from here.
  • There is a special iSCSI Key need to be purchased in order to have TOE+iSCSI Off-load on R710, this is the same on HP Proliant servers.
  • Found HP Proliant’s server resources are much more professional than Dells’, but Dell’s stuff are a lot cheaper (1/3 at least), so we just have to live with that.
  • In ESX 4 or above, Thick provisioning is always recommended for performance concerned VM applications, it’s a lot faster.
  • Talked to two of the local IDC who’s going to start a cloud business, but their core technical team doesn’t seem to know what they are really doing, seemed they have a long way to go comparing with US counterparts.
  • Again, virtualization is the future and you need a good SAN to support it!
  • Talk to your inside sales manager, show your sincerity, you will be rewarded and get unbelievable discount at quarter end!
  • Dell’s EQL expert is really helpful and resourceful, thank you so much! You are really the super-hero, kick-ass type!

Current Project: VMWare vSphere with Equallogic iSCSI SAN

By admin, April 25, 2010 23:39

Currently I am involving in a massive virtualization project that I am responsible for transforming and restructuring client’s 10 years old enterprise datacenter into something Much GREEN, mainly using VMWare’s vShpere and Dell Equallogic iSCSI SAN as the their major storage solution. Each node will be deployed on the latest Dell Poweredge R815 powered by AMD’s 12-core “Magny-Cours” Opteron with 128GB memory on each server, latest technology like DRS/HA/FT/vStorage/DPM making a long time dream comes true finally. 

ps6000-storage-stack-left-powerpoint

Recently, I find myself can’t live without virtualization more and more, for example the above will result in consolidating 10 racks to 3 servers and 2 Equallogic 6000 series boxes occupying half rack only with much reliable fault tolerance and failover as well as at a fractional cost than before, just think about the electricity and rack space saving alone, making virtualization irresistible!

32-Cores on a standard 2U server is no longer a dream

By admin, March 22, 2010 23:09

With the new release of Intel’s latest Xeon CPU (Nehalem-EX) for the enterprise market, 32-Cores on a standard 2U server is no longer a dream. Think about what you can do with 32-Cores (64-Cores counting the HT) and 128GB Ram in a standard 2U for your next virtualization project! Get it? Yes, you could put almost the whole mini data center (that’s 1998 standard, say 128 servers, each with PIII 1G with 1G Ram per server) on to it, see the cost saving? Bingo!!!

intel_nehalem-ex-4-core

The New Nehalem-EX Advantage:

  • Intel Nehalem Architecture built on Intel’s unique 45nm high-k metal gate technology process
  • Up to 8 cores per processor
  • Up to 16 threads per processor with Intel® Hyper-threading
  • Scalability up to eight sockets via Quick Path Interconnects and greater with third-party node controllers
  • QuickPath Architecture with four high-bandwidth links
  • 24MB of shared cache
  • Integrated memory controllers
  • Intel Turbo Boost Technology
  • Intel scalable memory buffer and scalable memory interconnects
  • Up to 9x the memory bandwidth of previous generation
  • Support for up to 16 memory slots per processor socket
  • Advanced RAS capabilities including MCA Recovery
  • 2.3 billion transistors

My antique PIII computer is still going strong for its 11th year.

By admin, January 29, 2010 21:33

I am still using Asus P2B-D with Dual Slot 1 850Mhz, going strong for the 11th year! I still remember clearly I brought it in April 1999 in Wai Chai Computer City and I’ve upgraded it many time over the years with minimal cost. (Mostly bought second hand from Go2HK & Yahoo HK)

Seemed to me the previous generation of Asus MB is so stable, I use for Internet/Email/Photoshop mostly, I can also play MKV, HD RMVB without any problem, so it’s really more than enough.

I will post this again in 2014 where it will be its 15 years.

I have two of these P2B-D, one is running as Desktop, the other running as Network File Server with iSCSI support.

 

ASUS_p2b_ds_dual_motherboard

The Configuration:

  • The version of my Dual Slot1 PIII850Mhz  is the one using the least Watts, 17.5W each?
  • Seagate/Maxtor 120GB IDE, 8MB Cache x 2
  • SATA PCI card with Maxtor Maxline 500GB x 4, 16MB Cache on another P2B-D file server
  • IDE PCI card with Maxtor 200GB x 2, 8MB Cache, Maxtor 250GB x 2, 8MB Cache
  • USB2.0 PCI card with 4 ports
  • SoundBlaster AWE64 with Cambridge Soundworks 4.1 Digital (This is so good!)
  • Running W2K3 with IE8
  • Matrox G550 with Dell Ultrasharp 2007FP in 1600×1200 model, so sharp and nice
  • Compaq Dual network NIC
  • Logitech wireless keyboard and mouse. 

After 11 years, I am still totally satisfied with this DIY combo, no complain at all, no harddisk failure (so strange, giving the HD life is only 5 years).

It seemed to me that I really don’t need to upgrade for a long time to come.

Virtualization is the future of computing

By admin, October 22, 2009 13:33

VMwareLogo

VMWare is probably the ONLY name you need to know in virtualization world, its flagship product vSphere is the one you cannot do without. VMotion/HA/DRS/FT are the amazing tools to maintain business continuity, service stability and SLA.

A Comprehensive comparison between VMware/Microsoft Hyper-V/XEN/RH Linux

I love to play with this kind of stuff very much!

By admin, December 1, 2008 00:35

Everything fits nicely within a tiny 1/4 rack!

CIMG3600

Pages: Prev 1 2 3 ...16 17 18 19 20 21 22 23 24 25 26