Samsung DDR3 Low Voltage ECC RDIMM and DDR UDIMM

By admin, December 29, 2011 19:47

Previously I’ve mentioned the followings regarding the latest DDR3L RAM:

It’s nice to have that 20% electricity saving, but when you add 2DPC (2 DIMMs Per Channel), your nice 20% power saving (ie, 1.35V) will be disabled automatically (ie, raise to 1.5V instead). However the good part is you still get that 1333Mhz bandwidth with 2DPC. DDR3L 1.35V will only apply when it’s in 1DPC mode.

What I’ve found out today during the server upgrade is the above may not be always true.

May be it’s related to the Poweredge BIOS that I’ve updated sometimes ago as it may contain the changes now allowing 2DPC operating at 1.35V Low Voltage. (may be even 3DPC)

The server is Poweredge R610 with 1 CPU Xeon E5620 (32nm, 12M Cache, 2.40 GHz, 5.86 GT/s, max supports DDR3-1066). After I filled up all the available 6 DIMMS, (ie, 3 Channels, 2 DIMMS per channel), I saw during the rebooting screen, it showed DDR- 1066Mhz and 1.35V, This is Great! I was expecting to see 1.5V as this happens to my R710 when putting 2DPC, it automatically raised to 1.5V, probably will go down to 1.35v if I upgraded the BIOS for that R710.

Finally, I located this interesting White Paper about Samsung’s DDR3L RAM, the result shows even with 3DPC at 1.5V, you can still save up to 50% of the total power consumption by using DDR3 Low Voltage RAM. There are also many other interesting technical insides regarding the benefits using DDR3L RAM.


The other thing is the RAM Upgrade KIT somehow did’t come with Heatsink, strange.


What you are seeing in the above picture is a direct comparision of two ALMOST IDENTICAL 4GB RAM, even the IC chipset part number is almost identical except a single character difference. 

The one on top is used in Desktop (Optiplex 990 SSF), I bought it in a local computer shop for under USD20, and the other is for this server upgrade (ie, Poweredge R610).

As you can see there are two extra chips in server ram (hence ECC and Buffer), and note the “L” indicates this server RAM is a Low Voltage version.


I also noticed the desktop ram label says it’s 10600U and server one says 10600R, ha…here is the fun part, I am sure this means U-DIMM (as Un-Buffered) and R-DIMM (as Registered & Buffered), so what does it mean or why this is so important?

Well, technically you can fit a cheap U-DIMM (about USD20 per 4GB DIMM)  into any Poweredge R2xx, R6xx, R7xx, R8xx, R9xx, of course, there is a limit when using U-DIMM, maximum you can use is 24GB, so that’s about 6 DIMMS x 4GB U-DIMM.

And the saving is Huge, for some may be (as USD120 for U-DIMM vs USD420 for R-DIMM), that’s USD20 for a 4GB U-DIMM that you can buy from any local computer shop as they are simply a desktop RAM, comparing to USD70 for the equivalent R-DIMM.

Of course, I will still choose R-DIMM, you may ask for maximum 24GB, why pay that extra  USD300-315 for ECC Reg RAM?  Well…RAS and reliability is the number  one factor in data center business, and not to mention the expandability, so Why NOT?


By admin, December 28, 2011 23:34

今天中午看CCTV新聞的時候﹐突然發現了金日成的長陣靈車竟然是美國汽車改裝廠Moloney 70年代的傑作﹕Lincoln Continental Limousine。



Shrinking a Equallogic Thin Volume = Thin Reclaim on Dirty Blocks

By admin, December 22, 2011 22:28

I thought it’s only available in FW 5.1 with ESX 5.0 or above (in fact, it turns out this EQL feature is not even ready yet till Q2/Q3 2012), and I am surprised to find out the following today somewhere on-line.

Still I do not have the courage to do this  in my production, it’s just too scary, better just Storage VMotion from one datastore (ie, EQL volume) to another datastore and then remove the old dirty volume to claim space.

Shrinking is only supported in EqualLogic firmware version 3.2.x or greater. Shrinking a Thin Provisioned volume is only supported in V4 or higher. For v3.2, you can however, convert a TP volume to a standard thick volume and then resize it, assuming there is space available to do so.

CAUTION: Improperly shrinking a filesystem can result in data loss. Be very careful about shrinking volumes, and always create snapshots to fall back to in the event of a problem.

To shrink a volume you must first shrink the file system and partition. While there are a few Operating Systems, like Windows 2008 for example, which support this natively, most do not. For Windows NT, 9x,XP, W2K3, you need to use a tool such as partition magic, partition commander, or similar tool. We do not test these tools in house, so we cannot make any statements about how well they work.

WARNING: be certain that you are shrinking the volume to slightly larger than what you have reduced the filesystem/partition size to. Shrinking a volume to be smaller than the partition/filesystem “WILL” result in data loss, i.e., if you reduced the filesystem to 500G, shrink the EQL volume to 501GB.

Always create a snapshot before attempting any resize operation. Shrink the file system & partition slightly more than you intend to shrink the volume to avoid rounding or other math discrepancies.

The volume will need to be offline to shrink the volume, og off or shutdown the server(s) connected to that volume first.

The Shrink command is a CLI only option. You have to first select the volume (covered in the CLI manual available on the FW download page).

GrpName> vol sel
GrpName(volume_volume_name)> offline
GrpName(volume_volume_name)> shrink
GrpName(volume_volume_name)> online

Another Great FREE Tool from Solarwind: Storage Response Time Monitor

By admin, December 22, 2011 09:47

Just found Storage Response Time Monitor 10 mins ago, downloaded and installed it and love it at the first sight! It correctly shows me the real-time break down of each Equallogic volume  latency and what’s more, it lists the top 5 BUSIEST VMs with IOPS number for that particular volume in one single window.

Having trouble with storage latency issues? Seeing your response times getting slower and slower? Download SolarWinds Storage Response Time Monitor and start tracking those sluggish VMs. Storage Response Time Monitor shows the top host to datastore total response times and a count of additional high latency offenders. Keeping track of your storage response times has never been easier.

Storage Response Time Monitor Highlights:

  • Get at-a-glance insight into host to datastore connections with the worst response times, and the busiest VMs using those connections
  • See a breakdown of the datastore including type and device versus kernel latency


Equallogic Versus Lefthand Blog

By admin, December 22, 2011 09:32

Amazing, someone actually made such an interesting blog collecting all the related articles he could find on the net, listing out all the pros & cons of each vendor, for me, of course I bias towards  Equallogic, but it’s nice to see such in depth comparison.

What a BIG Surprise! Equallogic FW 5.1 VMware’s Thin Provision Stunning feature WORKED on ESX 4.1!

By admin, December 21, 2011 22:21

This happened two days ago and followed by a very happy ending.

Around 8AM, I received an SMS alert saying one of the email server isn’t responding, then it followed with many emergency calls from clients on that email server  (obvious…haha).

I logged into vCenter and found the email server had a question mark on top of its icon, shortly I realized the EQL volume it sits on is full as SANHQ also generated a few warning as well the previous days, but I was too busy and too over confident to ignore all the warnings and thinking it may get through over the weekend.

The solution is quite simply. First increase the Thin Provisioned EQL volume size, then extend the existing volume in vCenter, next simply answer the question on the stunned (ie, suspended) VM to Retry again, Woola, it’s back to normal again, no need to restart or shutdown the VM at all.

This is really beautiful! I was a bit depressed when knowing it will only work with ESX 5.0 previously and this was also confirmed by a Dell storage expert, then found out the The VMware’s Thin Provision Stunning feature will work with ESX 4.1 again the other day from Dell’s official document, I was completely confused as I do not know who’s right until two days ago.

FYI, I had a nasty EQL Thin Provisioned volume issue last year (EQL firmware was 4.2), the whole over-provisioned thin volume simply went offline when it reached the maximum limit and all my VMs on that volume crashed and need to restart manually even after extending the volume in EQL and vCenter.

No more worries, thank you so much Equallogic, you really made my day! :)

Finally, some those who may be interested to know why I didn’t upgrade to ESX 5.0? Why? Why should I? Normally I will wait for a major release like ESX 3.5 or ESX 4.1 before making the big move.

There is another issue as the latest Veeam Back & Replication 6.0 still have many minor problems with ESX 5.0 and frankly speaking, I don’t see many advantage moving to ESX 5.0 while my ESX 4.1 environment is so rocket solid. The new features such as Storage DRS, SSD caching or VSSA are all minor stuffs comparing with VAAI and iSCSI multipathing, thin provisioning in ESX 4.1. In additional, the latest EQL firmware features always mostly backward compatible with older ESX version, so that’s why I still prefer to stay at ESX 4.1 at least for another year.

Oh…one thing I missed that is I really do hope EQL firmware 5.1 Thin Reclaim can work on ESX 4.1, but it seemed it’s a mission impossible, never mind, I’ve got plenty of space, moving a VM off a dirty volume isn’t exactly a big deal, so I can live with it and manually create a new clean volume.

Update Jan 17, 2012

Today, I received the latest Equallogic Newsletter and it somehow also indicates this VMware Thin Provisioning Stun feature is supported with ESX 4.1, hope it’s not a typo.

Dell EqualLogic Firmware Maintenance Release v5.1.2 (October 2011)

Dell EqualLogic Firmware v5.1.2 is a maintenance release on the v5.1 release stream. Firmware v5.1 release stream introduces support for Dell EqualLogic FS7500 Unified Storage Solution, and advanced features like Enhanced Load Balancing, Data Center Bridging, VMware Thin Provisioning awareness for vSphere v4.1, Auditing of Administrative and Active Directory Integration. Firmware v5.1.2 is a maintenance release with bug fixes for enhanced stability and performance of EqualLogic SAN.

Update Mar 5, 2012

I found something new today that Thin Provisioning Stun is actually a hidden API in ESX 4.1 and apparently there are only two storage vendors support it in ESX 4.1, one being Equallogic, no wonder this feature worked even with firmware v5.0.2, as I thought at least v5.1 is required. Thanks EQL, so this gives me a bit more time to upgrade to FW5.1 or even FW5.2.

Thin Provisioning Stun is officially a vSphere 5 VAAI primitive. It was included in vSphere 4.1 and some array plugins support it, but it was never officially listed as a vSphere 4 primitive.

Out of Space Condition (AKA Thin Provisioning Stun)

Running out of capacity is a catastrophe, but it’s easy to ignore the alerts in vCenter until it’s too late. This command allows the array to notify vCenter to “stun” (suspend) all virtual machines on a LUN that is running out of space due to thin provisioning over-commit. This is the “secret” fourth primitive that wasn’t officially acknowledged until vSphere 5 but apparently existed before. In vSphere 5, this works for both block and NFS storage. Signals are sent using SCSI “Check Condition” operations.

VAAI Commands in vSphere 4.1

esxcli corestorage device list

esxcli vaai device list

esxcli corestorage plugin list

AMD Opteron 6200: 16 Cores, 1600 DDR3 4 Channels and Turbo Core

By admin, December 11, 2011 15:56


So who’s the Copy Cat? It’s hard to tell really over the past 20 years. It was Intel copied AMD’s HT and renamed it as QuickPath Interconnect (QPI), now it’s AMD’s turn to copy Intel’s Turbo Boost and renamed it as Turbo Core. Chinese has an old saying “天下文章一大抄”  described this perfectly.

The good thing is Opteron 6200 can fit nicely in the existing G34 socket with a simple upgrade in server BIOS. However, Intel’s coming E5 Xeon requires a different socket as it’s the Tick in the product map, too bad for existing Dell R710 or HP Proliant G7 customers.

Oh…it seemed to me that the latest Bulldozer is STILL a “Fake” 16 cores die as it is still a 2 x 8 cores die.




AMD推出首款16核心伺服器級處理器Opteron 6200系列,將32奈米Bulldozer架構處理器引進商用伺服器市場。

處理器廠商AMD推出全球首款16核心伺服器級處理器Opteron 6200系列(代號為Interlagos),為目前業界擁有最多核心數的伺服器級處理器,也是AMD的Bulldozer架構首度進軍伺服器市場。

AMD表示,這系列處理器的運算效能比前款Opteron 6100系列處理器(代號為Magny-Cours)提升3成,延續採用了6100使用的G34伺服器插槽,企業既有伺服器只要更新BIOS程式後,就能改用新款處理器提高運算效能。多家伺服器廠商也陸續推出搭載16核心Opteron處理器的伺服器產品,使單臺4路伺服器上看64核心。


今年10月初,AMD先推出採Bulldozer架構的桌上型電腦處理器FX系列,最多為8核心,打破桌上型電腦的核心數記錄。1個多月後,Bulldozer架構處理器進入商用伺服器市場,AMD推出全球首款16核心伺服器級處理器 Opteron 6200系列,相當於內建8個Bulldozer模組,採用32奈米製程,主要用於2或4路伺服器。

臺灣AMD資深產品經理賴榮安表示,相較於最多12核心的6100處理器,6200系列的運算能力提升將近3成,主要原因是處理器架構轉換為Bulldozer之後,核心數增多、最大記憶體規格從1,333 MHz 的DDR3轉而支援1,600MHz的DDR3記憶體,也新增Turbo Core自動超頻技術等。

Turbo Core技術類似英特爾Turbo Boost技術,同樣能提高處理器的時脈。不同之處是,Turbo Core技術提供了2種超頻模式,第一種可將處理器全部核心的時脈提升約300MHz~500MHz,第二種可選擇只將處理器半數的核心轉換為閒置狀態,另外半數核心的時脈提升約1GHz。

Opteron 6200全系列共10款,除了16核心,還提供4、8、12核心版本,時脈介於1.6GHz至2.6GHz。啟用Turbo Core技術後,時脈最多可達到2.9GHz至3.7GHz。處理器功耗介於85至140瓦。


在作業系統的支援度方面,像是Red Hat Enterprise Linux 5.7與6.1、SUSE Linux Enterprise Server 10 SP4與11 SP1、微軟Windows Server 2008 R2 SP1等都可支援Bulldozer架構處理器。不過,部分舊版本的作業系統不能支援新款處理器。例如,Linux作業系統核心2.6.31版及以下版本、Windows Server 2003 R2 SP2版本以下、Red Hat Enterprise Linux 4.x~6.0版等都難以支援Bulldozer架構處理器。


賴榮安表示,現在臺灣已有企業正在測試新款處理器。在國外,像美國國家科學基金會(NSF)位於伊利諾州的超級電腦應用中心(NCSA)宣布,將採用6200系列處理器來建置Blue Waters超級電腦。

明年推Opteron 3000系列新產品線,攻占微型伺服器市場

另外,針對低耗電的企業應用,AMD還推出6或8核心伺服器級處理器Opteron 4200系列(代號為Valencia),可用於2路以下伺服器產品,採用32奈米製程與C32插槽。

4200系列增加了核心數與處理器時脈,第三階快取為8MB,時脈介於1.6GHz至3.0GHz,透過Turbo Core自動超頻技術,時脈可達到2.8GHz至3.7GHz,開始支援DDR3-1600 MHz記憶體規格。

不過,鎖定低耗電應用的4200系列,熱設計的功耗介於35至95瓦,沒有比4100系列更少,也比不過英特爾Xeon E3-1220L僅20瓦的熱設計功耗。

到了2012年上半年,AMD才會針對低耗電應用,推出新的伺服器處理器產品線Opteron 3000系列(代號為Zurich),首款為4至8核心,採用Bulldozer架構與AM3+插槽,鎖定高密度及低耗電的1路主機代管伺服器、網路伺服器及微型伺服器等雲端供應商的應用環境。


AMD Bulldozer架構處理器的核心設計與舊款Opteron 6100系列與4100系列不同。舊款處理器的單一個核心包含1個整數運算核心與1個浮點運算核心,並具備獨立的L2快取,而英特爾處理器內單一個核心的設計,同樣包含1個整數運算核心與1個浮點運算核心,並具備獨立的L2快取,所以英特爾最高階伺服器級處理器Xeon E7具備10核心,總共包含10個整數運算核心與10個浮點運算核心。
但是,Bulldozer架構調整了處理器的核心設計,透過共享快取與浮點運算核心的方式,來增加整數運算核心的總數。以16核心Opteron 6200系列來說,單顆處理器包含8個Bulldozer模組,每1個Bulldozer模組都封裝了2個整數運算核心,每2個核心共享L2快取及1個浮點運算核心,每4個Bulldozer模組再共享L3快取,單顆處理器內含16顆核心。

採用AMD Opteron 6200系列的伺服器

HP ProLiant

型號   組態 機箱 記憶體插槽數
DL165 G7 1路 1U 24 DIMMs
DL385 G7 2路 2U 24 DIMMs
DL585 G7 4路 4U 48 DIMMs
BL465c G7 2路 刀鋒 16 DIMMs
BL685c G7 4路 刀鋒 32 DIMMs


型號   組態 機箱 記憶體插槽數
YR190-B8028-X2 1路 1U 12 DIMMs
YR190-B8238-X2 2路 1U 12 DIMMs
GT24-B8236/ GT24-B8236-IL 2路 1U 16 DIMMs
GN70-B8236-HE/ GN70-B8236-HE-IL 2路 2U 16 DIMMs
FT48-B8812 4路 4U 32 DIMMs


型號   組態 機箱 記憶體插槽數
AW2000h-AW175hq F1 2路 2U 16 DIMMs
AR 385 F1 2路 2U 24 DIMMs
AR 585 F1 4路 2U 48 DIMMs

IBM System x

型號   組態 機箱 記憶體插槽數
x3755 M3 2路 2U 32 DIMMs

Dell PowerEdge

型號   組態 機箱 記憶體插槽數
R715 2路 2U 16 DIMMs
R815 4路 2U 32 DIMMs
M915 4路 刀鋒 32 DIMMs
C6145 4路 2U 32 DIMMs

SuperMicro SuperServer 

型號   組態 機箱 記憶體插槽數
1042G-TF 4路 1U 32 DIMMs
1122GG-TF 2路 1U 16 DIMMs
1012G-MTF 1路 1U 8 DIMMs
1022GG-TF/1022G-NTF/ 1022G-URF   2路 1U 16 DIMMs
2022G-URF 2路 2U 16 DIMMs
2022G-URF4 2路 2U 24 DIMMs
2042G-6RF/2042G-TRF 4路 2U 32 DIMMs
2022TG-HIBQRF 2路 2U 16 DIMMs
2022TG-HLTRF/ 2022TG-HLIBQRF 2路 2U 8 DIMMs
2122TG-HIBQRF/ 2122TG-HTRF 2路 2U 16 DIMMs
4022G-6F 2路 直立 16 DIMMs
4042G-TRF/4042G-6RF 4路 直立 32 DIMMs
SBA-7222G-T2 2路 刀鋒 8 DIMMs
SBA-7142G-T4 4路 刀鋒 16 DIMMs

Exoto Tipo 246 F1

By admin, December 11, 2011 10:28

Pre-Order price was USD770, now it’s USD845, Exoto, why don’t you simply set the price to USD1,000?




By admin, December 11, 2011 10:06


The price of die cast model cars will become higher – 6 Dec, 2011

Autoart die cast model was first launched in 1998. At that time, the basic labor cost in China was around $1.60US per day for a ten hour working day. The rate then, was almost double what it was a decade before. There were plenty of workers waiting at the front gates of every factories looking for work. These factories provided food and shelter for the workers. These workers came from far away provinces in central China to live and work in the factories for a minimum of one year and they would return home only once a year.

Provinces in central China are the most populous. Sichuan province alone has almost 100 million people. There were few developments before the mid 2000’s and the young people in the region were forced to look for jobs in the industrial zone around the coast lines. Many of them were female workers as young as 18 years of age, the legal age of employment, and had finished eight to ten years of education. They wanted to earn some money to help their families.

Despite the low wages, they were able to save most of their income and were able to buy a small place to live in their hometown after working for a couple of years. Life was simple and the workers were happy just to work and willing to stay in the factory year after year.

Until early 2000’s, an average 1:18 scale Autoart models with 100 ~ 120 components would retail for around $50US and mass market toy grade model cars would retail at only $19.99US. The manufacturers, producers, importers, and retailers were all able to make their fair share of profits and the collectors were very active in collecting the models.

Today, a regular Autoart 1:18 scale model retails for around $150US. The price has increased as much as three times over the course of 12 years but the people involved in the manufacturing and marketing of the products are not getting their fair share of the profit margin. Many factories are now losing money and the main reason for this is that the labor cost has gone up almost ten times when calculated in US dollar. The basic worker still earns $1.60US but it is now an hourly rate instead of daily rate. Despite the higher wages, the workers are not happy with their jobs and always demand a higher pay.

They would quit as soon as another factory offers a higher pay. Inflation in China has become a major issue in recent years. Housing prices have gone up ten or even twenty times and workers can no longer buy a small place to live after working for a few years. On the other hand, the standard of living has improved greatly over the last decade. Televisions, smart phones and computers have become common household items in average families.

There are 500 million internet users and 600 millions mobile phone accounts in China, three times more than the USA. There are also more Chinese learning English than the whole America. People in China are quickly learning what the desirable things in the world are and they are no longer happy with a life that consist of having a shelter over your head and food on the table to feed the family. There are more than 18 millions new cars filling the road of China every year, 50% more than America, every Chinese wants a car instead of paddling a bicycle.

The cost of manufacturing a high quality 1:18 scale model has gone up tremendously due in big part to the cost of labor. The retail prices may have tripled in the past decade but the people involved in the making and marketing of the products are much worst off.

The mark up factor from production cost all the way to the final retail price is largely depended on the upfront investment and the estimated selling quantity. For example, designer brand clothes itself may cost only $10US to produce, but the marketing cost such as fashion show, advertising, designing fee and packaging may cost tens or even hundreds of thousands of dollars.

If the clothes are sold in hundreds of pieces, the retail price will have to be $100US or over in order to breakeven the investment. If it is mass-market clothing selling in tens of thousands of pieces, under the same production cost of the $10US, the retail price can be as low as only $30US. For a die cast model car, the main upfront investment is the tooling cost (moulds).

The licensing fee can also represent a large sum of money for some hot subjects and upfront payment is to be paid to the car makers regardless if the model is selling or not. The tooling cost for a 1:18 scale model is an investment of six figure amounts, if the selling quantity is in tens or hundreds of thousands, then the amortization of the tooling fee is only one or two dollars for each model. But if the selling quantity is only in few thousands of pieces, then the amortization of the tooling is tens of dollars for each model. The same applies to license fees; a hot subject license fee can be in the six figure amounts.

Mass market toy grade die cast models can be sold in tens or even hundreds of thousands of pieces; the amortization of tooling fees and license fees is not a major cost factor in this case. But for high quality collectable die cast models, only thousands of pieces will be sold, very seldom would it break into the five figures level. Therefore, the amortization of tooling cost and licensing fees becomes a major cost factor.

Until now, the mark up factor for all die cast model cars is set at around 2.5 ~ 3 times from production cost to retail price. It may be workable for mass market products, but it is no longer viable for high quality models. There are in fact much fewer high quality new die cast models being released into the market recently because there is no profit to be made for people involve in the business.

The factories are being squeezed to keep the prices as low as possible and they are aware that if the prices become too high, very few people will buy model cars. Even if the factories make no profit, it is still better than receiving no orders. If fewer orders come in, thousands of workers are still paid the same but with lower output. Major OEM model car factories in China are minimizing their production capacity or changing their product line because manufacturing die cast models has become too expensive and too complicated when the labor cost is so high and the production involves hundreds of processes using thousands of workers.

When the price of model cars continues to climb higher and higher, less and less people are collecting them. When there are less collectors buying the model cars, retail shops have a much harder time to survive with the declining sales and the higher cost of operations. The online shops are also taking away a big part of the retailing business because they have much lower overhead and they can sell with a lower profit margin. We are seeing model car shops closing down one by one in recent years.

Die cast model cars are classified as toys and the companies can only participate in toy related exhibitions and trade fairs where mostly toy related traders go to conduct their business. As long as people regard these die cast models as a toy, it cannot be priced too high or else people will say it is too expansive and refuse to buy them no matter how nice and delicate the model may be.

For example, a normal Swiss watch consists of less than a hundred components and can be priced for several hundred or even several thousands of dollars. Most of these so called Swiss watches are all having the cases and bands manufactured in China then shipped to Switzerland to have the movement fitted which will then qualify the watch to become Swiss made. When it is Swiss made, it can be priced ten times more than watches made in developing countries. This is the norm in the market and the general perception of consumers. Most people accept that if the watch is made in Switzerland, it has to be expansive. And if the product is made in China, it has to be cheap even though the quality is very good.

The development time for a high quality 1:18 scale model that consist of more than a hundred components, is around one year. If the component number is several hundreds of pieces, then the development time can be two years or even more. Each process during the development period requires many experienced engineers to work on the project. The whole development is far more complicated than regular watch making.

For a high quality 1:18 die cast model car that is made in China, collectors are already complaining that the prices are too high when models are retailed at $100US or over. Some collectors even commented that the workers are getting too little pay and the producers are making too much profit by raising the prices again and again. The true fact is the opposite; the producers are not getting enough money to pay for the ever increasing worker salary. If the model producers were to apply the norm of the other products as mentioned, the viable retail prices would have to be doubled or even tripled. Unfortunately, the product is regarded as toy by the general public even though some will name it as collectable. When it is a toy, it has to be cheap, especially when it is made in China. There must be a change in the product perception by the general public and the price must be higher than what it is now, otherwise, the wages of the workers are going to increase year by year; more and more factories will stay out of the business when they can no longer afford to lose more money and very few collectable die cast models will be made in the future..

合金模型的价格必將更高 – 由MC的會員vantage 翻譯











玩家早已抱怨一個高質量中國制造的1:18模型太貴,要賣 100多美金。很多玩家甚至說工人只拿了一點點錢,而生產商卻一而再而三的提价為了謀取暴利。事實卻是相反,生產商都不夠錢去付一再漲价的工人工資。如果一個模型生產商依照其他行業一般規矩定价,現在的合理零售价應該再翻1倍或2倍呢!不幸的是,這些商品被大眾認為是玩具,盡管有些人覺得他們是收藏品。只要他是玩具,他就該便宜,尤其是中國制造的。這种把合金模型認為是玩具的看法必須改變,而模型的价格必須更高。否則,隨著工人工資年复一年的增加,一個個模型厂將倒閉,因為虧損太多。將來,越來越少的高端合金會被生產。。。




AA发表的最后留言是——”价格的上涨是不可避免的,玩家数量减少也是不可避免的,只有真的懂得欣赏模型价值的玩家会继续收集AA”        背后的逻辑很简单“我就这样,爱玩不玩”
























可能AA 实在是觉得 中国大陆市场可有可无。省的国际市场流言蜚语,指指点点。确实,对于国外的经销商而言,中国的低价格是无法忍受的。






不过总之,也许AA 根本不拿中国大陆市场当回事,毕竟在中国,车模收藏的氛围还很薄弱。你爱买不买。



Why Are Model Cars Becoming Expensive – 14 Mar, 2011

The long Chinese New Year Holidays are now over and the workers are gradually coming back to the factories from their home towns. By now, their wages have increased another 20% under the new minimum wages requirements announced by the Chinese Government in February.

A decade ago, the average basic worker would receive a lump sum salary of 400 ~ 500 RMB Yuan per month. The exchange rate at that time was around 8.9 Yuan to the US Dollar, so this was equivalent to $50 ~ 60 US Dollars per month. This salary was based on an average of 60 hours per week, Monday to Saturday, working about 10 hours per day. This was the norm for the industry in China.

Since then, the minimum wages, China’s development, and China’s living standards have been advancing at an enormous rate. The cost of living in China has become much more expensive compared to only ten years ago, particularly in the housing market. Because of this, the government needed to act accordingly in order to offset the ever increasing living cost. This is when the government enforced the Labor Law in 2007 which was largely ignored previously by the industry. Under the law, the national weekly hours worked is 40 hours per week. Any overtime hours would be paid at a rate of 1.5 times on weekdays and double on weekends. Also, all of the retirement and social benefits would need to be added on top of their income. When all of these changes are applied, the average income for the basic worker increased drastically to around 1200 RMB Yuan per month. These drastic changes forced thousands of factories making products at low profits and were labor intensive to close down.

The global financial crisis erupted in late 2008, but China was able to recover quickly. By the end of 2009, everything was pretty much back to pre-crisis levels and in 2010, China’s economy continued to progress at a very quick pace. Housing prices went up at a staggering rate and food became more and more expensive. Workers earning the basic salaries could hardly keep up with supporting their basic needs. So, the wages needed to be adjusted again and again or else the workers would leave the factories and look for work elsewhere for higher paying jobs. Workers now do not want to work at a factory that will not guarantee 60 hours per week. They need the all the possible extra income generated by the overtime pay. It is not easy for factories today to utilize all the workers for 60 hours per week, especially during weekends. But since they now have no choice, it is making the cost of production much higher for factories in China.

As of today, the basic worker earns a lump sum of about 2,000 RMB Yuan per month. With the RMB Yuan exchange rate under enormous pressure form the USA and Europe, it is forced to appreciate gradually from 8.3 to 6.6 against the US Dollar. The average minimum wage now stands at around $300US Dollars per month. This means that the basic salary in US Dollars has increased almost 500% in the last ten years.

A high end die cast model car brand like Autoart is particularly hard hit when the labor cost increases. The proportion between the material cost and the labor cost is approximately 1:4 due to the hundreds of workers needed to manufacture each and every model car on the production line.

The licensing fees for the model cars are also getting higher every year. Back in the 1990’s, no car makers really paid any attention to their intellectual property and only demanded a symbolic fee when a model maker was willing to produce a model to promote their brand. When the profits from car sales became slimmer due to competition, the car makers looked at other alternatives to yield extra income and the licensing of brands became a very good source of revenue for the car makers. Licensing has become such a money maker that race organizers, oil companies, tire companies, etc, are all jumping in to get their share. Today, if a model maker decides to make a race car using a special color such as the Gulf Oil’s blue and orange, they would have to pay a fee to Gulf because of their patented color combination. Some models today can have double, triple or even quadruple licenses and the combined licensing fees can be more then 20% the ex-work price. Because of this, there are much less racing model cars being launched by model makers. It has become too expensive to make and fewer buyers would be willing to pay that price.

It was more profitable for Autoart to sell the model cars back in the early 2000’s with models retailing at around $50US then it is today with model prices retailing at triple the price. Both the model makers and the buyers were happier ten years ago when the material and energy cost were much lower and the factory workers were quite happy with earning only a fraction of today’s wages because life was much simpler and also cheaper then. Even with an increase of almost 500% in the minimum wages in such a small amount of time, the factory workers are still unhappy and will quit their jobs easily to look for work elsewhere to find higher wages.

Despite increasing the selling prices over and over again, it is still impossible to catch up with the ever increasing production cost. When prices increase, the number of collectors buying the model decreases thus reducing the quantity of models sold. This directly affects the amortization of the investment made by the model makers and it drives up the cost of production even further. Most die cast model makers, especially the high-end model makers suffer badly due to the lack of sales and the ever increasing cost of production.

To lower the production cost, it may be possible to move the production facility to another country that still offers low labor cost. However, no other countries can match the quality of workmanship done by Chinese workers. The detailing work and precise manipulation of such small parts with their fingertips is unmatched in the world. It is very difficult for other developing countries to emulate China on anything related to arts and crafts.

Even though the labor cost in China has increased so much in only a decade, China is still considered to be a low cost labor country with a current hourly rate of around $1.20US per hour. Until the mid 2000’s when the hourly rate was around $0.30US per hour, collectors were mostly spoiled with the low price models and now with such a fast increase in production cost, many collectors do not understand why the models are now at such a price.

An average 1/18 scale Autoart model car is now retailing at around $120 ~ 150US. The price is still considered to be very reasonable considering the amount of time and investment put in each model that is released. Each model consists of hundreds of parts takes about one year to develop and the production of each and every model involves hundreds of workers in the production line. Many products such as watches which also have hundreds of parts and are as complex to make demand much higher selling prices.The price of a high end model car will continue to increase as the labor cost and living standards improve in China. There will be fewer people buying model cars as prices increase, but the collectors who understand what is involved in making a high end diecast model will continue to appreciate the reasonable prices and quality being offered. They will understand that there are no other countries able to offer the same in terms of price and quality and that these are not just model cars but a work of arts.


Perception of quality – 4 Sep, 2009

There is a prevailing misconception amongst die-cast model collectors over the term “good quality”.

From a model-maker’s point of view, for a die-cast model to be deemed “good quality” it must satisfy the following basic requirements:

- the model should replicate as closely as possible the true scale of the original;
- the body should be painted evenly and in a consistent shade without ripples or an orange peel effect;
- the doors, bonnet and trunk should close tightly with any surrounding gaps fine and even;
- the plastic windscreen should be clear and without distortion;
- the headlights should be fitted to the body of the model without the attachment pins being visible;
- the chrome trim, bumpers and rocker panels should be aligned correctly.

Many collectors believe that a good quality model will inevitably be expensive and incorporate sophisticated features such as a movable gas filler cap, sun visor, glove box, roof top……etc. In fact, a model may have a relatively simple specification and sell at a standard price but still be considered good quality as long as the above criteria have been met.

Obviously, a good quality model requires more manual labor during production than a standard quality model. The cost of production is, therefore, higher. One of the most challenging tasks for a model-maker is to achieve a high quality and consistent paint finish on the body and the panels. Before proceeding, the entire body surface must be smoothed manually, area by area and panel by panel, so that the paint may be applied evenly and smoothly. When parts of the model such as the pillars are produced to the true scale, the resulting structure will be quite slim and delicate. Thus the trimming of the corners and edges must be done by careful manual filing. Any air inlet or outlet on the die-cast body can only be manually trimmed in order to achieve a genuine aperture. To have the doors and panels close perfectly with fine and even gaps requires weeks or even months of manual fine-tuning of the tooling. There are no shortcuts when striving to achieve a quality finish. No automated machine can replace trained human hands handling such delicate objects.

In order to enhance the quality, a model can be made with special intricate features provided that they look real, both in scale and texture. Common practice is to use photo-etching on metal parts to produce components such as the radiator grill, air inlet wire mesh, brake-rotors, steering wheel spokes and various emblems. The process allows the production of parts of minimal thickness with the texture and fine detail that closely resemble the real items. Moreover, real leather can be trimmed to be so thin as to permit the incorporation in the model of upholstered seats and window curtains made of real fabric. Using multi-link metal hinges can also simulate the actual movement of a real car bonnet.

There are, of course, technical constraints. For example, when the real object is made of steel 1mm thick it will be almost impossible to replicate the same in a 1:18 scale model using metal that is 18 times thinner but still retaining its three-dimensional form. There are many small features in a real car, such as wiper arms, nuts and bolts and some suspension parts, that look unreal and out of scale if they are replicated in metal. Some makers use photo-etching on metal to simulate the wiper arm, which can be made very thin, but the result is just a flat piece of metal lacking the three-dimensional form. Other makers use metal to replicate an articulated wiper arm but the resulting component often appears to be too thick and unrealistic. These practices may be considered to be overplaying the role of special features in the development of die-cast models. However, many collectors comment that a model bearing such features is of high quality and merits the high price demanded for it, despite the lack of realism and adherence to the true scale. In fact, there are high-priced models selling in the marketplace that bear intricate features on a body shell that is fundamentally out of shape and out of scale and body panels that do not meet closely and tightly. Such models are of poor quality despite the price and features.


Price adjustment – 1 Aug, 2008

China is experiencing the fastest economic growth of the modern era. This has brought with it rising living standards, massive investment in the country’s infrastructure and advances in technological achievement. Thus, Chinese consumers now make up the largest user base for mobile phones, are benefiting from a highway network rivaling that of the United States and have witnessed a Chinese astronaut being sent into space.

The downside of this economic success, however, has been the dramatic increase in the cost of raw materials to record levels due to the enormous demand. Inflation is now at its highest level for decades and the prices of foodstuffs, utilities and property rentals have risen significantly over the last eighteen months or so.

All die-cast model car makers are suffering badly due to the ever-increasing production costs. Model car manufacturing is a labor-intensive business. Since the second quarter of 2006 the average wage of an unskilled worker has more than doubled, from less than US$100 per month to more than US$200 nowadays (including food and accommodation). In the case of AUTOart, labor costs alone represent half of the product cost. Every model passes through several hundred pairs of hands during its conception. Record high oil prices have simultaneously driven the costs of energy and oil-based by-products such as plastic, paint and thinner to record levels. These increments have not been of the order of tens of per cent. Rather, the increases we have experienced have been in hundreds of percent. Furthermore, the cost of zinc used to cast the body along with the nuts and bolts needed for assembly has also risen more than threefold since the turn of the century.

A shortage of labor is no longer a major issue. Recently hundreds of factories in our region have been closing down every month due to the rising production costs described above. The recent earthquake has also made many people homeless, causing some to leave their home towns to search for work further afield. As long as we are willing to pay what they ask, we can hire sufficient labor for our needs. However, China no longer benefits from comparatively low labor costs. These are now much higher than in neighboring developing countries such as Thailand, Indonesia, Vietnam and India.

Since last year the revenue from selling our model cars has not covered our production costs. In short, we are losing money simply because our existing pricing is no longer viable, this despite a price adjustment not too long ago. We have been producing to a quality level that cannot be sustained in an era of labor and material costs that are close to double the levels we enjoyed at the turn of the century. Furthermore, the construction of our models is becoming progressively more sophisticated, resulting in a higher rejection rate and longer lead times, thus pushing the effective production costs even higher.

The high oil price is not likely to fall in the near future. There exists the general phenomenon around the world of rising prices coming up against income levels that are barely increasing. Real incomes are being squeezed, resulting in less disposable income being available for hobby items such as model cars. If sales volumes decline the amortization of our tooling costs will have to be calculated over lower quantities, thereby increasing the product cost significantly.

Regrettably, a further price increase is imminent and inevitable. This is the only way for AUTOart to continue providing the range and quality that the brand is known for. As toys our products are too expensive and inappropriate. As artifacts, however, they continue to represent terrific value for collectors who appreciate the investment, both financial and human, that go into the development of our model cars. Our dedication, experience and professionalism remain undiminished.


AUTOart Models – worth every cent! – 1 Oct, 2006

AUTOart is renowned as a maker of premium quality die-cast models. Replicating in meticulous detail a wide array of models produced by a variety of motor manufacturers, AUTOart represents authenticity down to the last detail. Recently they raised the bar for standards in model making by incorporating the long multi-color striping tampon process in the production of the recently introduced 1/18 BMW 3.0 CSL model (MSRP $114.95). Tampon printing involves the use of a soft polyurethane pad to transfer paint residue from the cavity of an etched stainless steel plate to the printed surface. It is a process that allows the printing of very fine detail on flat or curved panels. To achieve the standard required by AUTOart for the BMW, repeated reworking of the tampon prints was called for, with the high scrap rate contributing to the production cost.

The long multi-color striping tampon feature is a challenging, complicated and costly process, accounting for possibly 1/3 of the production cost of the model. To explain further, tampon printing has a width limitation. The real BMW 3.0 CSL race car has long multi-color striping all over the body. To accurately replicate this look on a model requires 155 precise tampon print transfers. Single or multiple tampon hits in one small print area in one color are fairly inexpensive and straightforward. However, to achieve 155 hits in multiple colors with precise accuracy is complex, with a high margin of error and a significant rejection rate. When a multi-color striped racing livery stretches across the car, the tampon striping process will involve multiple stamping to connect the striping and make it seamless and perfectly aligned. Even slight misalignment, particularly on curved body panels, will be obvious to the naked eye and the model will be flawed, calling for removal of the paint. This, in itself, is a painstaking process that adds to the cost. In the worst case scenario the body has to be scrapped. When the stripe is made up of three colors, the chance of misalignment is multiplied by three.

AUTOart insisted on using long multi-color tampon printing on the BMW 3.0 CSL because the end result is shiny and vibrant, true to the look of the real race car. Tampon printing also allows the color to sit securely on the surface of the paint without being scratched off easily. For premium collectible products, AUTOart believes this is the ideal method. There are similar scale models in the marketplace offered by competitors at marginally lower prices, where the long striping is applied with water decals. These are made with all the colors pre-printed on a flat piece of clear membrane attached to a piece of paper. When dipped into water, the pre-printed membrane separates from the paper and can be transferred onto the model. The water decal process is simple and less costly due to the low rejection rate and low scrap rate. However, there are disadvantages. The pre-printed colors on the clear membrane of water decals are lackluster and the membrane itself is fragile and susceptible to scratching. Furthermore, over a period of time the membrane will become brittle and experience yellowish discoloration, eventually making the model less desirable.

In addition to the long multi-color tampon printing, AUTOart’s BMW 3.0 CSL model requires a further 94 production processes for mask spraying of the colors and a further 15 processes to achieve the chrome-plating effect by hot stamping. Thus, a total of 264 production processes are involved that relate only to the color and race livery decoration of the model.

There is also the painting of the white main body after each panel and corner has been filed, polished, trimmed and buffed, the manual drilling during assembly of a total of 71 small holes for the attachment of various parts, and the milling and engraving, again manually, in ten areas. None of this can be done through automation due to critical angles and limitations in casting technology.

Evidently, a model like the BMW 3.0 CSL by AUTOart requires several hundred intricate procedures and over a year of development time before it can be made available on the world hobby market. Many collectors may not appreciate the expense and complexity involved in producing an AUTOart model. Each piece passes through the hands of hundreds of young employees with delicately trained fingers and the product is retailed at the price equivalent to roughly around one month salary of a basic worker in China. Imagine if the same model were to produce in the USA with the same processes, it would have cost more or less the same as one month salary of a basic worker in the USA. Bearing all this in mind, enjoy your AUTOart models for they are worth every cent!


The anatomy of AUTOart’s MotorSport Series: – 24 Apr, 2006

Our 1:18 Motorsport series, with sealed body panels, raised a lot of queries among the collectors why we are making die cast model cars with no engine, no opening doors and bonnets but yet still have a similar selling price to that of an AUTOart model with more features. Here are the answers:

Many top racing events, such as Formula One, LMP, DTM, WRC and Super GT (formerly known as JGTC), would not allow picture to be taken or research to be conducted on the race car’s interior by model makers due to the team’s unique and covert design of their suspension setup and engine arrangement, which would be closely guarded as a competitive advantage among their competitors. Because of this, there are no modern Formula One die cast models that are made with engine, even if they do, it is being made with assumption from limited sourcing of pictures from the press and magazines. The dimension will never be accurate and many small details are also incorrectly replicated. Other model makers went as far as placing a standard production car engine into a WRC rally car, just so that the model launched into the market may have an engine inside when the hood is opened in order to please the buyers who are mainly non collectors with little knowledge about racing cars. For this primary reason, the AUTOart Motorsport series was launched, offering modern day racing model cars that lack engine or opening features and emphasizes on perfecting the exterior features. It is meaningless to replicate an incorrect engine in a model just for the sake that the model should have an engine. For many model collectors, the most important thing for a die cast model car is to have the body shape replicated as accurately as possible to the real race car, since that is the first impression. The interior, such as engine and boot, is secondary. Unlike classic racing cars of which the engines expose all the ram pipes, wiring harnesses and small parts that become nice features on a die cast model car, modern racing car engines are mostly covered with large air intake boxes with little engine detail to review which can hardly be a nice feature in a model car. As a matter of fact, when collectors buy a model car, he or she would open the bonnets or doors once or twice and the rest of the time the model is being displayed with the doors and bonnets closed.

Another reason we are making the subject matter under Motorsport series is that sometime the car manufacture or racing teams insists the die cast model cars to be launched within a year or so of the actual cars that are still competing and only a model without opening features will be feasible to realize the project because the development process requires only seven months rather than 10-12 months for a model with full openings.

Making a model car without opening doors and bonnets will save 10% to 15% of production cost. However, the Motorsport series consist of many special features not seen in previous model making of which the production cost can easily increase to 20% to 30% more expensive than our traditional AUTOart Racing series with openings. Therefore, it is not a cost saving approach when introducing Motorsport series without opening.
Modern racing cars are getting more complicated in their shape with a lot of vents, slots, fins, cooling ducts, aerodynamic aids and etc.; it is a huge challenge to replicate these features in die cast model making which are 18 times smaller in scale due to the limitation of the injection and casting technology. In order to replicate these features in our Motorsport series as closest to the real racing car, intensive labor have to be done manually. The opening of the small slots and vents on metal bonnets and fenders require the use of trimming machine, painstakingly trimmed away the zinc metal bit by bit in order to make them become through holes. Other manufactures would attempt the short cut method and use creative decals to achieve the substandard result. Some racing models require attention to more than 20 such areas and such time consuming and labor intensive trimming process are not commonly found in die cast model retailing for less than a hundred Dollars.

Another significant feature worth noting about our Motorsport series is the stance of a racing car. Precise ride height, fender flare and clearance of the tires are critical to achieving the aggressive high performance appearance of the race car as compared to a standard street car. Replicating the correct ride height and fender clearance of a modern racing car is one of the main challenges for a die cast model maker. Zinc metal casting technology limits the thickness of the fender to a minimum of 1mm, which in real car term, is 18mm under the scale of 1:18. The actual racing car fender is less than 1mm in thickness and the rubber tires are so close to the fenders that they are almost touching each other, so in order to attain the same appearance on a model car, the metal fender lips have to be trimmed by machine manually to make it as thin as possible without breaking it so that the rubber tires can be fitted close to the fenders. This feature can never be achieved in a mass market product at which the selling price would limit the amount of manual work on trimming and polishing.

To perfect the racing livery with sharp and shinny colors, tampon printing is used in most part of the body in Motorsports series. Racing cars with complicated multi-colors livery require over two hundreds hits of tampon print, because each hit can only consist of one color on one spot not more than two inches in size. Many model makers would use a much simpler cost saving method by applying water transfer multi-color decals to avoid numerous repetition of tampon printing. The problem with decal is that the color is dull, it will have a border and it may get cracked easily and became yellowish when aged. If a clear coating is applied on the decal, the layer is too thick which would blur the fine detail of the body lines and gaps.

All the panels, edges and splitting lines are manually polished and filed to the best possible surface fineness. After applying the paint, the surface has to be buffed by real car wax so that the finish is exceptionally smooth and shinny. The process is in fact very similar to the painting of a real car.

Photo etching metal pieces are being used for delicate parts that cannot be replicated precisely by injection plastic/metal such as mesh grill, tow hook and latches. The photo etching metal pieces will give a realistic shape in the scale but they are expensive to produce which will greatly affect the production cost and therefore no mass market product will implement this feature. Motorsport series apply photo etching metal pieces in many areas in order to give the model a realistic appearance.

Also another important feature on the Motorsport series that we put high emphasis on is the wheel and brake. The brake disc rotors are made of photo etching metal, put together piece by piece and then placed on a rotating platform to manually sand in the hairline texture to simulate the exact braking surface found on the real brake rotor. AUTOart also produces wheels for real car application for use in racing and street, so these wheel experts are also utilized in the factory for the development of the model car’s wheels. This is how we can ensure that the wheels on our model cars are being replicated in the best possible scale, contour and finishes. Many car designers would emphasize the wheel is the soul of the car, making the wheels correctly and nicely on a model car is one of the most important mission for AUTOart. In summary, the introduction of Motorsport series has elevated the standard of quality of a racing die cast models car that retails within the range of a hundred dollars.

Storage I/O Control (SIOC) Causing VM to Fail

By admin, December 9, 2011 00:53

Recently, I encountered a strange situation that sharp at 2am which is the backup window (Acronis agent running inside VM), one of the VM failed to function on occasion, I have to reboot it in order to RTO. CPU on the VM went to 100% for a few hours and became non-responsive to ping gradually.

However I am still able to login to console, but cannot launch any program, reboot is ok though.

There are tons of Red Alert Error under Event Log (System), most of them are related to I/O problem, seemed harddisk on EQL SAN is having bad block or so.

Event ID: 333
An I/O operation initiated by the Registry failed unrecoverable. The Registry could not read in, or write out, or flush, one of the files that contain the system’s image of the Registry.

Event ID: 2019
The server was unable to allocate from the system nonpaged pool because the pool was empty.

Event ID: 50
{Delayed Write Failed} Windows was unable to save all the data for the file . The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.

Event ID: 57
The system failed to flush data to the transaction log. Corruption may occur.
I couldn’t’ find the exact reason during the preliminary investigation and email exchange with EQL technical support returns nothing.

Event ID: 2019
Unable to read the disk performance information from the system. Disk performance counters must be enabled for at least one physical disk or logical volume in order for these counters to appear. Disk performance counters can be enabled by using the Hardware Device Manager property pages. The status code returned is in the first DWORD in the Data section.

Then suddenly I found there is an vCenter alert saying there is a non-direct storage congestion on the volume where the VM locates. Right away I figured out it’s related to SIOC, checked the IO latency during 2AM confirmed this. It’s the SIOC throttled back  (over 30ms) or forcing the volume to use less latency during the backup windows, so this cause the backup software (Acronis) and Windows somehow crashed.

I’ve disabled SIOC on that particular volume for 3 days already, everything runs smooth so far, it seemed I have solved the mystery.

If you have encountered something like this, please do drop me a line, thanks!

Update: Dec 23, 2011

The same problem occurred again, so it’s definitely not related to SIOC. Funny thing is it happened on the same  time when scheduled antivirus and Acronis backup windows started together. So I’ve changed the Acronis backup windows to a later time  because I think these two I/O  intensive programs were competing with each other.

I do hope this is the root of the problem and will observe more.

Update: Jan 14, 2012

I think I’ve nailed the problem finally, no more crash since Dec 23, 2011. Last night I observed a very interesting fact that the VM CPU went to 90% at 2am and last for 15 mins. ah…I realized it’s the weekly scheduled virus scanning that’s causing huge I/O and latency, it even some of the service on this VM stopped responding during the busy time.

So I’ve decided to remove the weekly scan completely, it’s useless anyway.

Update: Jan 18, 2013

The above procedure (removed weekly scheduled virus scanning) does prove it’s the cause of the problem after all.

Pages: 1 2 Next