|
Importance of Disk Alignment in VMware
Some of you might already be familiar with the tremendous I/O improvements that come from a RAID disk set that has the correct alignment. But how does this translate into the virtual world and virtual disks? Well, as with many things in the virtual world, what is good practice in the physical world also makes for good practice in the virtual world.
For anyone who is not familiar with the term disk alignment, it’s an issue that has traditionally plagued physical machines (both Linux and Windows) using RAID volumes (software and hardware.) The effects are staggering, a correctly aligned disks can yield anywhere between a 10% - 40% increase in performance over a non aligned RAID volume. Just in case you missed that, I said 10% - 40% - wow!
So you might be wondering if this holds true within a virtual machine also. Well, this is a complicated question to answer because most virtual machines running on an ESX server, whether using raw disk mappings (RDM) or more traditional virtual machine disk files (VMDK) don’t use RAID. Most people create one or two VMDK’s, one for the boot and one for the data volumes, for each of their virtual machines. At first glance, you may think disk alignment within a virtual machine is not necessary, but guess again. Even though most virtual machines do not use RAID within the guest operating system, the VMDK files used by the virtual machines sit on LUNs which do use RAID. Therefore, disk alignment within your virtual machine is critical.
There is a lot of incorrect information surrounding this topic because most people fail to realize the VMDK is running from a RAID blackened LUN. So whether you’re talking about the boot or data volume, (in a virtual machine running from a LUN) you should always align the disk.
I've borrowed some text from VMware's partition alignment guide to help illustrate the point further.
"In a SAN environment, the smallest hardware unit used by a SAN storage array to build a LUN out of multiple physical disks is a called a chunk or a stripe. To optimize I/O, chunks are usually much larger than sectors. Thus a SCSI I/O request that intends to read a sector in reality reads one chunk.
On top of this, in a Windows environment NTFS is formatted in blocks ranging from 1MB to 8MB. The file system used by the guest operating system optimizes I/O by grouping sectors into so-called clusters (allocation units).
Figure 1 shows that an unaligned structure may cause many additional I/O operations when only one cluster is ready by the guest operating system."
So as you can see, alignment within the VMDK can lead to a performance hit if its not aligned.
I also want to dispel the notion that aligning the boot volume within a virtual machine is pointless. True, most good system administrators will move I/O intensive applications to a data drive, but if the application is truly I/O intensive, should it even be virtualized in the first place? Working under that assumption, for the applications and systems you have virtualized, why not give them the best possible performance? Deploying a virtual machine from a template that has already been aligned requires no additional work and that virtual machine will forever more benefit from the affects of a correctly aligned volume.
I should however point out that the 10% - 40% performance increase in the physical world doesn’t translate verbatim to the virtual world. If you’ve been running virtual machines for any length of time, you have probably noticed less I/O throughput on your virtual machine as compared to that of a physical machine. Therefore it’s safe to assume the same for the virtual machine, but don’t think for a moment that your time is wasted aligning your volumes within a virtual machine. VMware reports throughput can increase by as much as 62% (averaging about 12%) with a latency decreases of up to 33% (averaging about 10%) with proper disk alignement. That’s a nice little performance boost and one that most of us could use.
Checking VMFS Disk Alignment
Beginning with ESX v.3.x and later, VMware aligns new partitions to a 64KB boundary. The 64KB boundary will probably suffice for most, but if you want to be extra cautious, you could opt to align at 1024KB (more on that in a minute).
Although the default 64KB alignment offered in ESX v3.x and later will work for most, if your LUN's stripe size is not aligned at 64KB, then you will have to format your VMFS by hand (instructions provided in the Aligning VMFS using fdisk section below). Also, if your environment began on a pre 3.x installation (even if it has since been upgraded) you will suffer from alignment issues. Older versions of ESX, did not conform with disk alignment specifications, so you should double check any older installations.
So that we don't stray to far from the topic at hand, it's probably best to ask your SAN administrator for the correct offset of a given LUN. As long as the offset you choose is a multiple of the LUN's offset, you'll be fine. Using a staring block of 2048 (2048 = 1024KB or 1MB) should account for the stripe sizes of most configurations but not all. If you want to be sure, you must know your LUNs correct offset. Otherwise, use 2048 which should satisfy most configurations - and at only 1024KB is just a drop in the bucket!
By now I'm sure your eager to see how to check our VMFS for alignment with your LUN or RAID set. To do so, issue the following commands from the service console of your ESX host:
- If you’re not logged in at the ‘root’ user, become root first
$ whoami
thirt
$ su - root
Password: |
- List the partitions
- Fdisk will produce output similar to the following:
Disk /dev/disks/vmhba2:0:0:0: 1999.8 GB, 1999839952896 bytes
255 heads, 63 sectors/track, 243133 cylinders, total 3905937408 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/disks/vmhba2:0:0:1 128 18446744073320515964 18446744073515033727+ fb VMFS |
- Take note of the start block for each disk. The output isn't very pretty, but the start block in the above output is the second column (we highlighted it in yellow so that it may be more easily seen.) As we have already said, ESX v3.x and later align at 64K boundaries. In the above output, sector 128 = 64K. For most, this will mean your disk is aligned. However as we have already said, double check with your SAN administrator to get the correct alignment value for the given LUN.
Unfortunately, if you find your VMFS is not aligned, there is little you can do. Depending on how much work is involved, you might consider destroying the data store and recreating it through the Virtual Infrastructure Client on an ESX v3.x host or from the service console using fdisk’s expert mode. If you decide to recreate your VMFS, please be aware you will lose any data that exists on the partition, so as always backup your data first!
Aligning VMFS and Linux using fdisk
The alignment procedure for VMFS and the Linux file system (whether it be ext3, ext2, riser, xfs etc.) is much the same as any other Linux file system. We’ll use fdisk’s expert mode to create a new partition with an aligned starting block and then format the file system. You should use this procedure for not only VMFS, but for your Linux virtual machines as well (changing the type in step 7 to 83.)
- If you’re not already on the service console as the root user, begin by becoming root
$ whoami
thirt
$ su - root
Password: |
- Run fdisk on the device you would like to align. Depending on your configuration, this device will be different. Use the ‘fdisk –l’ to list all available devices; replacing ‘/dev/disks/vmhba32\:0\:0\:0’ in the command below with your device.
| $ fdisk /dev/disks/vmhba32\:0\:0\:0 |
Note that a ‘:’ is a special character in the Linux shell and must be escaped with a ‘\’
- Create a new partition
- Choose ‘p’ for a new primary partition
Command action
e extended
p primary partition (1-4)
p
|
- When asked for the partition number, enter ‘1’
Partition number (1-4): 1
|
- Use all the defaults by pressing enter at the prompts
First cylinder (1-1088, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1088, default 1088):
Using default value 1088
|
- The VMFS file system uses the type fb. Change the type to ‘fb’ on the new partition.
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fb
Changed system type of partition 1 to fb (Unknown) |
Note: If you were working with a traditional Linux partition, you would use type ‘83’ – the Linux file system.
- Enter expert mode
- Change the start block to a value that aligns with your stripe element size. I like to use a start block of 2048 (2048 = 1024KB), ESX v3.x and later default to 128 (128 = 64KB) or as we have already said, you could use a value provided by your storage admin.
Expert command (m for help): b
Partition number (1-4): 1
New beginning of data (63-8385929, default 63): 2048
|
- Commit the changes to disk
Expert command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks. |
- If you are formatting a VMFS for VMware, you should use vmkfstools to format your file system. If you are formatting a Linux file system, you should use mke2fs or a similar tool to format your file system. It goes without saying that formatting a file system will destroy any existing data on the device so make absolute certain your working with the correct disk!
- To format a newly created VMFS, use the following syntax replacing ‘/dev/disks/vmhba32\:0\:0\:0\:1’ in the command below with your device and partition number.
| # vmkfstools –C vmfs3 –b 4m /dev/disks/vmhba32\:0\:0\:0\:1 |
Note that ‘:’ is a special character in the Linux shell and must be escaped with a ‘\’
- The formatting of a Linux file system can be slightly more complex due to the many different file systems and parameters offered like journaling. We cannot cover all the different file systems and their options, but for most, ext3 is an excellent choice (includes journaling.) The following syntax will format a file system as ext3, replace '/dev/sda1' with the device and partition you would like formatted.
Preparing VMDK for Windows Installation
If you are installing Windows Server 2000, 2003 or Windows XP on a physical server or desktop, you cannot fix disk alignment at the time of installation. You can fix data partitions and other volumes that are added later (see our KB on Windows disk Alignment), but not the boot volume. Worse yet, once Windows has been installed on the drive, there is no way to fix the alignment, it must be done when the partition is created –now there’s a catch-22. This is rather frustrating knowing you’re taking a performance hit on the boot volume because it cannot be aligned.
Fortunately, in VMware, we can fix the drive alignment before the OS is ever installed! We can use fdisk from the service console and set the partition alignment before Windows is installed. Lets hear it for VMware!
- Create a new virtual machine using the virtual infrastructure client (do not power on the machine).
- From your service console, change directory to the location of the newly created virtual machine
# cd /vmfs/volumes/487bba15-14da54c8-f2a0-001e4ffbda71/WIN
/vmfs/volumes/487bba15-14da54c8-f2a0-001e4ffbda71/WIN # ls -l
-rw------- 1 root root 8589934592 Apr 9 17:49 WIN-flat.vmdk
-rw------- 1 root root 370 Apr 9 17:49 WIN.vmdk
-rw------- 1 root root 0 Apr 9 17:49 WIN.vmsd
-rwxr-xr-x 1 root root 1189 Apr 9 17:49 WIN.vmx
-rw------- 1 root root 258 Apr 9 17:49 WIN.vmxf
|
- Get the number of cylinders from the VMDK. The line begins with 'ddb.geometry.cylinders'
/vmfs/volumes/487bba15-14da54c8-f2a0-001e4ffbda71/WIN # cat WIN.vmdk
# Disk DescriptorFile
version=1
CID=41e9d328
parentCID=ffffffff
createType="vmfs"
# Extent description
RW 16777216 VMFS "WIN-flat.vmdk"
# The Disk Data Base
#DDB
ddb.virtualHWVersion = "4"
ddb.uuid = "60 00 C2 90 24 ba cd bd-f2 35 ca ab 3e 73 37 a2"
ddb.geometry.cylinders = "1044"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"
|
- Run fdisk on the WIN-flat.vmdk file
/vmfs/volumes/487bba15-14da54c8-f2a0-001e4ffbda71/WIN # fdisk ./WIN-flat.vmdk
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
You must set cylinders.
You can do this from the extra functions menu.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help):
|
- Set the number of cylinders from step #3 above. Ignore fdisk’s warning about the cylinder size being larger than 1024, this only applies to older systems and is not applicable to us.
Command (m for help): x
Expert command (m for help): c
Number of cylinders (1-1048576): 1044
The number of cylinders for this disk is set to 1044.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Expert command (m for help):
|
- Confirm your cylinder is set correctly and no partitions exist on the new disk
Expert command (m for help): p
Disk ./WIN-flat.vmdk: 255 heads, 63 sectors, 1044 cylinders
Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID
1 00 0 0 0 0 0 0 0 0 00
2 00 0 0 0 0 0 0 0 0 00
3 00 0 0 0 0 0 0 0 0 00
4 00 0 0 0 0 0 0 0 0 00
Expert command (m for help):
|
- Exit from expert mode and create a new primary partition using the defaults
Expert command (m for help): r
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1044, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1044, default 1044):
Using default value 1044
Command (m for help):
|
- Return to expert mode and change the start offset to align the disk. The start block should be a value that not only aligns the NTFS partition within the VMFS, but also aligns the VMFS with the LUN. As such, use should use the same start block you used when originally creating the VMFS partition on the LUN. If your not absolutely positive, using a start block of 2048 (2048 = 1024KB) will cover most configurations. However, you should ask your storage administrator for the LUN's alignment size to be absolutely certain.
Command (m for help): x
Expert command (m for help): b
Partition number (1-4): 1
New beginning of data (63-16771859, default 63): 2048
Expert command (m for help):
|
- Print the partition table to verify the starting offset is set correctly
Expert command (m for help): p
Disk ./WIN-flat.vmdk: 255 heads, 63 sectors, 1044 cylinders
Nr AF Hd Sec Cyl Hd Sec Cyl Start Size ID
1 00 1 1 0 254 63 1023 128 16771732 83
2 00 0 0 0 0 0 0 0 0 00
3 00 0 0 0 0 0 0 0 0 00
4 00 0 0 0 0 0 0 0 0 00
Expert command (m for help):
|
- Return from exit mode and set the filesystem type to ‘HPFS/NTF’
Expert command (m for help): r
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 7
Changed system type of partition 1 to 7 (HPFS/NTFS)
Command (m for help):
|
- Exit fdisk and write the changes to the VMDK. You can safely ignore the warning about error 25
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 25: Inappropriate ioctl for device.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
/vmfs/volumes/487bba15-14da54c8-f2a0-001e4ffbda71/WIN #
|
- Power on the virtual machine and begin the normal Windows installation process.
- When the installer gets to the partition screen, install on the existing partition (hit enter). Do not destroy or recreate the partition or you will lose the sector alignment.
If all went well, after the install has completed, you can download the Disk Alignment Check Tool from our downloads area and verify your drives alignment. For additional details on Microsoft Windows platforms and disk alignment, see our Windows Disk Alignment KB.
Note: Any new partitions created within the VM should use the procedures outline in the Windows Disk Alignment KB but with the same start offset used in step #8.
Enjoy you new optimized VM!
Add this page to your favorite website  |
Knowledge Base 
