Linux RAID Overview

In this KB, we are going to show you how to create a software RAID device in Linux.  It's important to note that we will only cover software RAID devices in this KB and will be using the term "RAID device" to infer software RAID and not hardware.

RAID stands for Redundant Array of Inexpensive Disks.  RAID was originally introduced as a way of gaining higher availability and redundancy by eliminating a single point of failure with the loss of a single volume/disk.  Since its introduction, RAID has grown in popularity and today is widely used throughout the enterprise.  For example, most SAN (storage array networks) technologies utilize a combination of hardware and software RAID techniques to present volumes or LUNs to hosts.

RAID devices provide a means of achieving higher availability by "mirroring" or copying data to multiple disks.  When data is written to the disk, RAID instructs the system to write multiple copies of the data on different disks to prevent data loss should a drive fail.

RAID can not only increase your disk's availability, but it can also achieve higher I/O throughput.  "Striping", allows you to overcome the limitations of a single disk's I/O.  RAID stripes allows you to aggregate the bandwidth of several disks by writing blocks of data horizontally across the array spreading the I/O out evenly amongst the members of the array.

Depending on your particular need, you may opt for speed, redundancy or a combination.  The different levels of RAID provide varying levels of speed and redundancy, so you can choose the setting that is right for you. Software RAID in Linux supports 7 different modes:

• Linear - Disks are appended to one another to form a larger device.  There is no striping or redundancy in linear mode.  The system simply treats the disks like an aggregate of disks, which spills over from one disk to the next as they fill up.
  
  
  
  
  
• RAID 0 - Is striping mode.  In a RAID 0 configuration, read and writes are done in parallel to all the members of the array.  Blocks of data written to the array are broken down into smaller chucks and written simultaneously to all members of the array.  The same is true with reads (but in reverse), chunks of data are simultaneously fetched from the different members of the array and assembled to build the block.  RAID 0 provides the greatest I/O but lacks redundancy.  A single drive loss in the array will corrupt the data on all the remaining volumes so use with caution.
  
  
  
  
  
• RAID 1 - Is mirroring mode.  In a RAID 1 configuration, data is mirrored between 1 or more volumes.  RAID 1 provides the highest level of availability offered in RAID, however because of the redundancy, it is also the slowest.  Because each drive contains a full copy of the data, you can loose multiple drives in the array (up-to N-1 failures) and continue to operate without a hiccup.  It should go without saying that RAID 1 also uses a significant amount of storage since multiple copies of data are stored in multiple locations.  It's also important to note that RAID 1 has very fast read times due to is ability to load-balance read requests amongst the members of the array.
  
  
  
  
  
• RAID 4 - Is striping with parity.  In a RAID 4 configuration, data is striped across the members of the array (similar to RAID 0) however parity information is also calculated providing the ability to recover from the loss of a single disk.  The redundancy of RAID 4 comes with a cost of N-1 in storage to the array (since one disk is used for parity, the total capacity of the array is reduced by 1.)  RAID 4, like RAID 0, generally provides better I/O performance since read/writes are done in parallel.  However, RAID 4 is seldom used because of the bottleneck caused by the parity disk.  Since each I/O write requires the parity information to be updated on the parity disk, the total throughput of the array can be limited by the I/O of the parity disk (Use RAID 5 instead of RAID 4.)
  
  
  
  
  
• RAID 5 - Is also known as striping with parity and very similar to RAID 4.  However in a RAID 5 configuration, the parity information is spread evenly across all the disks in the array eliminating the bottleneck of the RAID 4 parity disk.  RAID 5 offers all the features of RAID 4 but without the bottleneck of the parity disk.  Therefore, RAID 5 offers a better solution for speed with redundancy compared to that of RAID 4, but not quite the speed of Linux RAID 10.
  
  
  
  
  
• RAID 6 - Is striping with dual distributed parity.  RAID 6 works the same as RAID 5 however with dual parity.  Although less common, the additional parity information allows up-to two simultaneous disk failures of an array without data loss.  Dual parity eliminates the risk associated during the rebuild process of a RAID 5 array.  On a RAID 5 array, should another disk fail during the rebuild process, the array will be lost.  Since RAID 6 can sustain multiple drive losses within the array, the loss of a single disk does not pose a risk of data loss should another disk fail during the rebuild.  However, the added redundancy comes at a cost both in capacity and performance.  RAID 6 requires a minimum of 4 drives in the array with the total capacity reduced by N-2.  Also, because of the dual parity, write speeds can take significantly longer than that of RAID 5.
  
  
  
  
  
• RAID 10 (available in kernel v.2.6.9 and greater) could probably warrant an article all of it's own but I'll try and sum it up briefly so we can move on to the configurations.  Linux RAID 10, aka MD (multiple disk) RAID 10 can be configured in "near" or "far" configurations depending on the number of disks available in the array.   Near configurations are built from a odd number of disks and are more similar to that of RAID 1, while far configurations require even number of disks and are more similar to that of RAID 0 (the far configuration provides better performance than that of the near configuration.)  Linux MD RAID 10 should not be confused with RAID 1+0 (combination RAID levels).  MD RAID 10 in a far configuration (even number of disks in the array) will have performance characteristics similar to that of RAID 0 for reading, but half the speed for writing.  Therefore generally speaking, RAID 10 offers better performance than that of RAID 5 however comes at a total cost in capacity of N/2.
  
  
  
  
  

Now that we know a little more about the different RAID levels, lets begin with our configuration.

Configuring RAID

We are now ready to begin our first Linux software RAID configuration.  The following steps will walk you through creating a RAID 5 array created from 3 physical disks.  Let's get to it!

1. Begin by inspecting your system for available devices to create your RAID array
   
   

   [root@Linux01 ~]# fdisk -l Disk /dev/sda: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot      Start         End      Blocks   Id  System /dev/sda1   *           1          13      104391   83  Linux /dev/sda2              14        1305    10377990   8e  Linux LVM Disk /dev/sdb: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot      Start         End      Blocks   Id  System Disk /dev/sdc: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot      Start         End      Blocks   Id  System Disk /dev/sdd: 4294 MB, 4294967296 bytes 255 heads, 63 sectors/track, 522 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot      Start         End      Blocks   Id  System [root@Linux01 ~]#

   Note: You'll notice dev/sda is in use (contains operating system etc.) but /dev/sdb, /dev/sdc and /dev/sdd are available.
   
   
2. Create a primary partition of type fb on each of the devices that will be used in the RAID array.  In the example below, we will create a primary partition on /dev/sdb.  This step should be repeated for each device in the array (i.e. /dev/sdc and /dev/sdd)
   
   

   [root@Linux01 ~]# fdisk /dev/sdb Command (m for help): n Command action e   extended p   primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-522, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-522, default 522): Using default value 522 Command (m for help): t Selected partition 1 Hex code (type L to list codes): fb Changed system type of partition 1 to fb (Unknown) Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@Linux01 ~]#

   Note: The Linux RAID driver MD only supports files system type fb
   
   
3. Repeat Step #2 for the remaining devices (i.e. /dev/sdc and /dev/sdd) in the array until each device has been configured with a partition of type fb
   
   Note: If you are using a device that contains existing partitions, you may receive the following warning when exiting fdisk: "The kernel still uses the old table.  The new table will be used at the next reboot."  You can avoid a reboot by using the partprobe command (i.e. partprobe /dev/sdb). 
   
   
4. Use the RAID admin utility to create an array from the available partitions created from Step #2
   
   

   [root@Linux01 ~]# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 mdadm: array /dev/md0 started. [root@Linux01 ~]#

   
   

   Notes: If you desire a raid level other than 5, you can do so by specifying the level with the --level command line switch.  Available options include:  linear raid0 / 0 / stripe raid1 / 1 / mirror raid4 / 4 raid5 / 5 raid6 / 6 raid10 / 10 multipath / mp faulty The --raid-devices switch lets mdadm know how many devices will make up the RAID array.  In our example, we are using 3 devices so we specified the number 3 followed by the list of devices.  However, if you are using more devices, you should be sure to reflect the correct number and device list in your syntax.

   
   Optionally, you can create the array with a hot spare:
   

   [root@Linux01 ~]# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --hot-spares=1 /dev/sde1 mdadm: array /dev/md0 started. [root@Linux01 ~]#

   
   
5. Monitor the creation of the array
   
   

   [root@Linux01 ~]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdd1[3] sdc1[1] sdb1[0] 8385664 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_] [===>.................]  recovery = 15.7% (660556/4192832) finish=0.8min speed=73395K/sec unused devices: [root@Linux01 ~]#

   Note: Once the recovery has reached 100%, the array will be ready for use
   
   
6. Add the new array to the /etc/mdadm.conf so that it will be automatically activated upon reboot.  The start-up scripts will reference the mdadm.conf file upon boot and start any arrays listed within this file.  If the array is not listed in the /etc/mdadm.conf file, you will have to start the array manually. 
   
   The following was added to the /etc/mdadm.conf file to satisfy our configuration for the RAID5 array with the 3 devices (sdb1, sdc1 and sdd1):
   
   

   ARRAY /dev/md0 level=raid5 num-devices=3 devices=/dev/sdb1,/dev/sdc1,/dev/sdd1

   Note: The mdadm.conf file is used to store configuration parameters for all of your Linux software RAID devices.  The above entry will satisfy the minimum requirements in-order for mdadm to start your array.  We'll explore other parameters of this file in future kb's.
   
   
7. Overlay the new RAID device with a file-system
   
   

   [root@Linux01 ~]# mke2fs -j /dev/md0 mke2fs 1.39 (29-May-2006) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 1048576 inodes, 2096416 blocks 104820 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=2147483648 64 block groups 32768 blocks per group, 32768 fragments per group 16384 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Writing inode tables: done                            Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 39 mounts or 180 days, whichever comes first.  Use tune2fs -c or -i to override. [root@Linux01 ~]#

   
   
8. Mount the file-system and verify its functionality
   
   

   [root@Linux01 ~]# mkdir /RAID5 [root@Linux01 ~]# mount /dev/md0 /RAID5/ [root@Linux01 ~]# ls -al /RAID5/ total 28 drwxr-xr-x  3 root root  4096 Jun 11 09:33 . drwxr-xr-x 29 root root  4096 Jun 11 09:35 .. drwx------  2 root root 16384 Jun 11 09:33 lost+found [root@Linux01 ~]#

   
   
9. Update the /etc/fstab file to auto mount the RAID device upon subsequent system reboots
   
   The following was added to our /etc/fstab to support the RAID5 array we configured in this example:
   
   

   /dev/md0 /RAID5 ext3 defaults 0 0

   Note: Your device and mount point may be different depending on your configuration
   
   
10. If convenient, you can reboot your system to verify the RAID device will auto mount upon reboot

Nice job!  Please see our other RAID KB's for advanced use and other helpful RAID tips and tricks:

• Starting and Stopping RAID Arrays
• RAID Alerting
• Removing Failed RAID Devices
• Removing RAID Devices

Add this page to your favorite website

Comments

Add New Search

Adrian Bradshaw  - /etc/mdadm.conf |06/07/2009 04:53:58 Hi Firstly thanks for this series of articles - they are really superb and have helped me understand a lot more about RAID and LVM on Linux One think that I am a little confused about is your reference to /etc/mdadm.conf - I have worked through these articles and while I dont have a /etc/mdadm.conf the RAID does start and /etc/fstab mounts it on each reboot Do you know if this is just my distribution ? Centos 5.3 Thanks again, Adrian Reply | Quote

thirt |07/07/2009 12:15:33 Hi Adrian, Thanks for the feedback. Very interesting, when I wrote this article, I was working from a RHEL 5 system. I would think CentOS being such a close relative of Red Hat EL would function much the same way. A couple things you can check. Look at the man page for mdadm, (man mdadm & man mdadm.conf). Within the man page, there should be a section entitled “FILES”. The FILES section should describe the different configuration files used by mdadm. Cent’s release of mdadm might make use a configuration file located under a different directory structure. Also, I have seen some distributions use /etc/mdadm/mdadm.conf. I’m not entirely sure of Cent’s preferred location, but the man page for mdadm should certainly clue you in. You might also take a look at the RPM package. Often times, looking at the files which makeup the RPM will clue you into the files used by the package. A ‘rpm –ql mdadm’ should list all the files contained in the RPM for mdadm. You might see the configuration files listed in there as well. If all else fails, try a ‘locate mdadm.conf’ and see what that turns up. Also, I’m surprised to hear the device automatically mounts. After you created the device, did you define it within /etc/fstab or did mdadm take care of the addition for you? Let me know what you find. If I get sometime this week, I’ll fire up a copy of CentOS myself and take a look if you haven’t figured it out. Best -Tom Reply | Quote

Adrian Bradshaw  - /etc/mdadm.conf - seems its optional |11/07/2009 03:02:37 Hi Tom, thanks for getting back to me. I took your advice and it seems that /etc/mdadm.com is optional these days - its certainly best practice to use it but it wont hurt if you dont Here is a quote from about half way down http://linuxdevcenter.com/pub/a/linux/2002/12/05/RAID.html Quote: Unlike /etc/raidtab, mdadm does not rely on /etc/mdadm.conf to create or manage arrays. Rather, mdadm.conf is simply an extra way of keeping track of software RAIDs. Using a configuration file with mdadm is useful, but not required. Anyway, keep up the great work - this is a great site Reply | Quote

Joe |05/08/2009 15:42:40 This is a nice tutorial, thank you. When you discuss the mdadm.conf file, you do not mention UUIDs. Which is the preferred syntax? Reply | Quote

thirt |22/08/2009 20:22:05 Hi Joe, Excellent question. It really depends on your environment. I probably should have included a section on UUID’s in here (perhaps I’ll go back this weekend and update this article.) But to answer your question, for a system with only a few disks and/or a single array, using the UUID probably isn’t necessary (as you’ll know which disks belong to the array.) But for a system with multiple arrays and many disks, remember which disks belongs to which array can become difficult. For these systems, you’ll probably want to use the UUID. Hope this answers your question! Best, Tom Reply | Quote

Deepak Kumar |04/03/2010 07:19:21 hi........ i think fd will be used insted of fb in Hex code .. Deepak Kumar Reply | Quote

zishan  - raid 6 |13/09/2010 06:05:11 i have configured raid 5 in rhel 5.0 i want to confirm that can i configure raid 6 in rhel 5.3. do i have to install any rpm to configure raid 6. plz help to know it Reply | Quote

Rashid Iqbal |17/09/2010 06:53:59 After configuring the software raid-1 on running system but after restarting the system after configuring the software RAID-1 I receive the below error: error 15 the file cannot be found I restart the system into rescue mode but their the system cannot mount the harddisk devices under the /mnt/sysimage kindly help me to sort out this problem Reply | Quote

Biswajit  - Openoffice install to linux5.3 |12/01/2011 01:45:28 how to install or configure Openoffice in linux5.3. which OPenoffice software support for linux5.3. Plz link provide. thk for help Reply | Quote

Ravi kant sharma  - not wrking |14/03/2011 13:45:37 hiiiii....... to create raid device- mkraid /dev/md0 is not wrking in RHEL 5.3 so wat shud i do.....????????? Reply | Quote

Write comment
Name:
Email:	do not notifynotify
Website:
Title:
UBBCode:	-color-aquablackbluefuchsiagraygreenlimemaroonnavyolivepurpleredsilvertealwhiteyellow -size-tinysmallmediumlargehuge
	Please input the anti-spam code that you can read in the image.

!joomlacomment 4.0 Copyright (C) 2009 Compojoom.com . All rights reserved."