06-08-2012, 05:56 PM
[undefined=undefined]This post id old and outdated, please do NOT use this method on RC5 or higher[/undefined]
Hi there,
I've seen this question asked a few times here, and answered also, but with rather terse explanations.
So I'll try to describe with details how you can use a usb connected drive plugged into your Rpi, that will hold your system and allow you faster i/o rates, and cheaper large storage.
So first, some explanations :
Once you've imaged your SD card with the Rpi installer, you end up with 3 partitions :
The vfat (or FAT32 ) partition contains the bootstrap files, firmware, kernel and boot arguments. It HAS to stay on the SD card. This is due to the hardware of the Pi and the fact that we're bootstrapping from a GPU.
This one stays where it is.
The Swap space is used by *nix like systems to cache unused RAM data when this last one gets full. It is recommended to be twice the size of you ram when less than 1GB.
I'm not sure wether it's accurate or not, but since we're moving to a large HD, we can enlarge the original 122 MB one to reach 256MB
This one will move to HD.
The Ext4 partition contains the debian system files that serve as a software base for xbmc to run.
This one is moving too.
Separate Home folder
Raspbmc being at alpha stage, it means that you might have to go through the process of re-imaging your SD card, and recreating your filesystems a lot.
We could set up our filesystem so that when you'll upgrade you system files, all your media files, but more important, all your settings would survive an update or re-imaging.
For this, we'll have to create another partition that will old your personal 'Home' folder.
Some stuff you need to know
The raspbmc image default id infos are :
These commands should be run from your default user location. To make sure you're there :
Most of the commands described here need the 'sudo' prefix in order to work !
If you get a 'permission denied' or 'command not found' error message, it is likely that you forgot the 'sudo' prefix.
Let's go
What you need :
IMPORTANT : The steps described here will DESTROY the data that resides on your hard drive ! Make sure to backup everything important. I will refuse to be held responsible for any file loss.
You've been warned. If you agree to this disclaimer and to take your responsibilities, just breath once.
Preamble
An important step is to disable udevd before connecting your USB drive. Udev is the daemon that's in charge of detecting new devices. It will prevent us from going to a lot of hassle with it trying to mount our new filesystems :
We can also stop xbmc :
Step 1 : Setup your hard-drive
First, we need to set up your hard drive with the right file system. To achieve this, we're going to use some nice and powerful *nix programs. Fear not of the CLI, for it is a mighty full tool !
Unfortunately, you'll have to use another computer to SSH into your Rpi, since there is no option to boot on CLI, without XBMC launching for the moment ( Could be a good idea ? )
Once you ssh'd into you Rpi, you can check your hard-drive is there by listing present devices :
It should issue a line saying ''/dev/sda''. If otherwise, you might want to check your HD is spinning, USB connections, etc.
You can also try to increment the last letter in 'sda', in the alphabetic order - ie 'sdb', 'sdc', etc.
Once you made sure /dev/sda is your HD alright, we're going to partition it this way :
- one SWAP part. sized 256 MB
- one / part. sized 8 GB ( yeah we've got lot of space )
- one /home partition that will fit the remainder of the disk.
Let's unmount all partitions on sda :
As much people won't feel confortable without a GUI, I'm going to use the 'cfdisk' utility :
You should now see a nice text user interface that lists what's on your disk right now, with a menu on the bottom of the screen.
- Select 'New' then hit return. Choose 'Primary', then enter the size of the swap in MB : 256, and choose to place it on the beginning of the disk.
Once you've done that, select 'Type' in the bottom menu and hit return. You should now see a list of possible types to choose from. This is a numbered list, so to choose SWAP type, we shall enter '82', and validate.
- Select the free space with the down arrow key, and do it again, this time with 8000Mb size, and type Linux : '83'.
- Repeat, with default proposed size.
You should now write down what partitions the different device files refers to, just in case.
It should be :
Once done, you MUST select 'Write' in the bottom menu and confirm by answering 'yes' to the question that you'll make sure to read and check.
You're back at the cli. Now, we have defined our partitions, but the filesystems (fs) were not created yet. That's what we're going to do with the mkfs utilies. The swap uses a different utility.
Build the SWAP fs :
Build the Ext4 fs :
If you get an error message telling you '/dev/sdX' doesn't exist, you can unplug, then replug your USB drive. Then check that 'dev/sdX' is accessible by entering :
It should be noted that you do not want to see any /dev/sdaX in the ouptut of
yet. If you do, you have to unmount then by issuing :
( The wildcard is literal ! )
That's it for this step. We now have a nice setup hard drive.
To check for this, we can issue the command :
that should print something like :
Step 2 : Copy the system files
In order to copy the files from the SD to the USB HD, we'll have to do some mounting .
We create the directories that will hold our mounts , then mount the corresponding partitions :
Beware ! These directories will disappear upon reboot, as they are place in the '/tmp' folder.
Should I use 'dd' or 'rsync' ?
I've seen several answers that mentioned the use of 'dd' commands to copy your system files.
I personaly don't think it's a good idea, nor the good tool.
I'll just copy what I said on another thread :
Unless there is a good reason for using 'dd', I would recommend using rsync.
'dd' copies a disk bit-by-bit, thus it's very slow, plus it can be very dangerous to play with when you don't know what you're doing.
I'll add that if you want to duplicate a mounted filesystem you're working off, you have to use the rsync method.
Let's install that sweet utility :
A. Personal folder and user settings
We'll begin easy, and copy the user settings on our home, reimaging-proof partition, with rsync first :
We use 3 options here :
-a : archive mode : it will preserve owner,group,permissions,links and devices information, and perform recursively. This is REQUIRED.
-v : Because we want more output of the progress
-h : Because we want human readable output !
Let's get rid of this one for now :
B. System files
We'll continue a little harder now .
Since the filesystem we want to copy the files from is running ( your Rpi is on right ?), we'll have to remount it elsewhere in order to rsync it without falling into a deadly recursive loop.
Plus, we don't want to rsync what's in the '/home/pi' folder sinc we just rsynced it :
EDIT : And obviously, we don't want to rsync the '/tmp' folder, since we're working there :/
Thanks to Baldjedi for bringing that up ! We also can exclude the '/boot' folder, as it won't be of any use on the HD.
We remount the FS :
We rsync, without '/home' , '/tmp', '/boot' contents :
This is where you could optionally use the '-u' option, so that when you upgrade the system files from the SD card next time, only files that were modified are updated.
We yet have to recreate the folders we excluded during rsync :
You can check all went well by issuing :
It should show a use of at least a few hundreds MB.
You can also :
which should answer something like:
Yay, we're done with that !
Step 3 : Setup the new Swap file and System files
In order for Gnu/Linux to find our newly created Swap file and system files, we have to tell him where it is.
For that, we have to declare it in the file '/tmp/sys/etc/fstab'.
You should have something that looks like :
Edit the three last lines so that they look like :
Once this done, save by hitting 'Ctrl'+'O', then 'Return'.
You can then quit by hitting 'Ctrl'+'X'.
You can unmount all these now :
You might see an error related to /tmp/home not being mounted. This is of no importance here.
We just want to make sure everything's unmounted.
Step 4 : Tel the Rpi where to find what
In order for the Rpi to boot on the right partition, we have to tell it in the '/boot/cmdline.txt' file :
Edit :
To look like this:
It should be noted here that we've modified the 'root=' argument, so that it looks for our file system on '/dev/sda2'. That's the partition we set up in Step 1. We could also use another
technic to refer to our partitions by their UUID. Let me know if you'd like to see me describe this here.
Step 5 : Reboot !
Hopefully, if you followed all these steps carefully, after issuing :
Your Rpi should boot fine, and you should notice some improvements in terms of reactivity.
A new issue of :
should echo something like :
You can check that the swap file in use is our new one by issuing
It should answer with :
What if I want to upgrade raspbmc ?
If you were in the need to upgrade raspbmc, you'd simply reimage the sd card, that would boot on the system files that are on the SD card. You would then have to reproduce Step 2.A, Step 3 and Step 4.
It should be mentionned that you do not need to do this if you upgrade with a nightly build, as it is my understanding that it will only replace the xbmc binaries in /opt.
You could even use the '-u' option on rsync the first time you copy system files. That way, only the files that were modified since last rsync gets updated.
I've been careful not to let any typo or mistake slip in but you always forget some. Be sure to let me know if you find one, or have any suggestions/remarks.
Should also mention english is not my native language, hence the numerous mistakes and bad grammar.
Hope it's useful !
Thanks to BaldJedi and r8rae for their corrections, suggestions and commitment !
EDIT : Corrected some typos.
After some time running the RC2 with this setup, here are my observations :
- On SD card, RPi used to freeze from time to time ( on a hourly basis), with no apparent reason, and had to hard reboot. This behaviour seems to be gone, as the Rpi is running since yesterday without freezing.
- It seems that network shares access is faster (SMB and NFS). This might due to the fact that the cache files are no longer written on the SD, but on the HD, hence faster I/O and caching ... ?
I should have mentioned that I use a 4g class 10 SDHC card, that is supposed to offer à 10MB/S min rate.
Sam's Note: Remember, kernel-vfat-latest.tar.gz overwrites cmdline.txt, where you edited -- so be sure to chattr +i it -- cheers
Hi there,
I've seen this question asked a few times here, and answered also, but with rather terse explanations.
So I'll try to describe with details how you can use a usb connected drive plugged into your Rpi, that will hold your system and allow you faster i/o rates, and cheaper large storage.
So first, some explanations :
Once you've imaged your SD card with the Rpi installer, you end up with 3 partitions :
- One vfat sized 63 MB
- One swap partiton sized 122MB
- One Ext4 partition that fits the remaining space of your SD card.
The vfat (or FAT32 ) partition contains the bootstrap files, firmware, kernel and boot arguments. It HAS to stay on the SD card. This is due to the hardware of the Pi and the fact that we're bootstrapping from a GPU.
This one stays where it is.
The Swap space is used by *nix like systems to cache unused RAM data when this last one gets full. It is recommended to be twice the size of you ram when less than 1GB.
I'm not sure wether it's accurate or not, but since we're moving to a large HD, we can enlarge the original 122 MB one to reach 256MB
This one will move to HD.
The Ext4 partition contains the debian system files that serve as a software base for xbmc to run.
This one is moving too.
Separate Home folder
Raspbmc being at alpha stage, it means that you might have to go through the process of re-imaging your SD card, and recreating your filesystems a lot.
We could set up our filesystem so that when you'll upgrade you system files, all your media files, but more important, all your settings would survive an update or re-imaging.
For this, we'll have to create another partition that will old your personal 'Home' folder.
Some stuff you need to know
The raspbmc image default id infos are :
Code:
login : pi
pasword: raspberryThese commands should be run from your default user location. To make sure you're there :
Code:
pi@raspbmc: ~$ cd ~Most of the commands described here need the 'sudo' prefix in order to work !
If you get a 'permission denied' or 'command not found' error message, it is likely that you forgot the 'sudo' prefix.
Let's go
What you need :
- A Raspberry Pi (duh)
- A SD Card with Raspbmc set up
- A USB Hard drive with an external PU ( or USB key )
IMPORTANT : The steps described here will DESTROY the data that resides on your hard drive ! Make sure to backup everything important. I will refuse to be held responsible for any file loss.
You've been warned. If you agree to this disclaimer and to take your responsibilities, just breath once.
Preamble
An important step is to disable udevd before connecting your USB drive. Udev is the daemon that's in charge of detecting new devices. It will prevent us from going to a lot of hassle with it trying to mount our new filesystems :
Code:
pi@raspbmc:~$ sudo service udev stopWe can also stop xbmc :
Code:
pi@raspbmc:~$ sudo initctl stop xbmcStep 1 : Setup your hard-drive
First, we need to set up your hard drive with the right file system. To achieve this, we're going to use some nice and powerful *nix programs. Fear not of the CLI, for it is a mighty full tool !
Unfortunately, you'll have to use another computer to SSH into your Rpi, since there is no option to boot on CLI, without XBMC launching for the moment ( Could be a good idea ? )
Once you ssh'd into you Rpi, you can check your hard-drive is there by listing present devices :
Code:
pi@raspbmc:~$ ls /dev/sdaIt should issue a line saying ''/dev/sda''. If otherwise, you might want to check your HD is spinning, USB connections, etc.
You can also try to increment the last letter in 'sda', in the alphabetic order - ie 'sdb', 'sdc', etc.
Once you made sure /dev/sda is your HD alright, we're going to partition it this way :
- one SWAP part. sized 256 MB
- one / part. sized 8 GB ( yeah we've got lot of space )
- one /home partition that will fit the remainder of the disk.
Let's unmount all partitions on sda :
Code:
pi@raspbmc:~$ sudo umount /dev/sda*As much people won't feel confortable without a GUI, I'm going to use the 'cfdisk' utility :
Code:
pi@raspbmc:~$ sudo cfdisk -z /dev/sdaYou should now see a nice text user interface that lists what's on your disk right now, with a menu on the bottom of the screen.
- Select 'New' then hit return. Choose 'Primary', then enter the size of the swap in MB : 256, and choose to place it on the beginning of the disk.
Once you've done that, select 'Type' in the bottom menu and hit return. You should now see a list of possible types to choose from. This is a numbered list, so to choose SWAP type, we shall enter '82', and validate.
- Select the free space with the down arrow key, and do it again, this time with 8000Mb size, and type Linux : '83'.
- Repeat, with default proposed size.
You should now write down what partitions the different device files refers to, just in case.
It should be :
- /dev/sda1 == SWAP (256MB)
- /dev/sda2 == / (8 GB)
- /dev/sda3 == /home (remainder of the disk)
Once done, you MUST select 'Write' in the bottom menu and confirm by answering 'yes' to the question that you'll make sure to read and check.
You're back at the cli. Now, we have defined our partitions, but the filesystems (fs) were not created yet. That's what we're going to do with the mkfs utilies. The swap uses a different utility.
Build the SWAP fs :
Code:
pi@raspbmc:~$ sudo mkswap /dev/sda1Build the Ext4 fs :
Code:
pi@raspbmc:~$ sudo mkfs.ext4 /dev/sda2Code:
pi@raspbmc:~$ sudo mkfs.ext4 /dev/sda3If you get an error message telling you '/dev/sdX' doesn't exist, you can unplug, then replug your USB drive. Then check that 'dev/sdX' is accessible by entering :
Code:
pi@raspbmc:~$ ls /dev/sdXIt should be noted that you do not want to see any /dev/sdaX in the ouptut of
Code:
pi@raspbmc:~$ df -hyet. If you do, you have to unmount then by issuing :
Code:
pi@raspbmc:~$ sudo umount /dev/sda*( The wildcard is literal ! )
That's it for this step. We now have a nice setup hard drive.
To check for this, we can issue the command :
Code:
pi@raspbmc:~$ sudo fdisk -l /dev/sdathat should print something like :
Code:
pi@raspbmc:~$ sudo fdisk -l /dev/sda
Disk /dev/sda: 200.0 GB, 200049647616 bytes
255 heads, 63 sectors/track, 24321 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000baa4f
Device Boot Start End Blocks Id System
/dev/sda1 1 12814 524288 82 Linux swap / Solaris
/dev/sda2 12814 13834 8192000 83 Linux
/dev/sda3 13834 24322 84243456 83 LinuxStep 2 : Copy the system files
In order to copy the files from the SD to the USB HD, we'll have to do some mounting .
We create the directories that will hold our mounts , then mount the corresponding partitions :
Code:
pi@raspbmc:~$ sudo mkdir /tmp/sys && sudo mount /dev/sda2 /tmp/sysCode:
pi@raspbmc:~$ sudo mkdir /tmp/home && sudo mount /dev/sda3 /tmp/homeBeware ! These directories will disappear upon reboot, as they are place in the '/tmp' folder.
Should I use 'dd' or 'rsync' ?
I've seen several answers that mentioned the use of 'dd' commands to copy your system files.
I personaly don't think it's a good idea, nor the good tool.
I'll just copy what I said on another thread :
Unless there is a good reason for using 'dd', I would recommend using rsync.
'dd' copies a disk bit-by-bit, thus it's very slow, plus it can be very dangerous to play with when you don't know what you're doing.
I'll add that if you want to duplicate a mounted filesystem you're working off, you have to use the rsync method.
Let's install that sweet utility :
Code:
pi@raspbmc:~$ sudo apt-get install rsyncA. Personal folder and user settings
We'll begin easy, and copy the user settings on our home, reimaging-proof partition, with rsync first :
Code:
pi@raspbmc:~$ sudo rsync -avh /home/pi/ /tmp/home/pi/We use 3 options here :
-a : archive mode : it will preserve owner,group,permissions,links and devices information, and perform recursively. This is REQUIRED.
-v : Because we want more output of the progress
-h : Because we want human readable output !
Let's get rid of this one for now :
Code:
pi@raspbmc:~$ sudo umount /tmp/homeB. System files
We'll continue a little harder now .
Since the filesystem we want to copy the files from is running ( your Rpi is on right ?), we'll have to remount it elsewhere in order to rsync it without falling into a deadly recursive loop.
Plus, we don't want to rsync what's in the '/home/pi' folder sinc we just rsynced it :
EDIT : And obviously, we don't want to rsync the '/tmp' folder, since we're working there :/
Thanks to Baldjedi for bringing that up ! We also can exclude the '/boot' folder, as it won't be of any use on the HD.
We remount the FS :
Code:
pi@raspbmc:~$ sudo mkdir /tmp/sd_sys && sudo mount --bind / /tmp/sd_sysWe rsync, without '/home' , '/tmp', '/boot' contents :
Code:
pi@raspbmc:~$ sudo rsync -avh --exclude 'home/pi' --exclude 'tmp/' --exclude 'boot/' /tmp/sd_sys/ /tmp/sys/This is where you could optionally use the '-u' option, so that when you upgrade the system files from the SD card next time, only files that were modified are updated.
We yet have to recreate the folders we excluded during rsync :
Code:
pi@raspbmc:~$ sudo mkdir /tmp/sys/boot /tmp/sys/tmpYou can check all went well by issuing :
Code:
pi@raspbmc:~$ df -h /dev/sda2It should show a use of at least a few hundreds MB.
You can also :
Code:
pi@raspbmc:~$ ls -lah /tmp/syswhich should answer something like:
Code:
total 89K
drwxr-xr-x 23 root root 4.0K Jun 7 09:15 .
drwxr-xr-x 23 root root 4.0K Jun 7 09:15 ..
drwxr-xr-x 2 root root 4.0K Jun 7 21:38 bin
drwxr-xr-x 2 root root 1.0K Jan 1 1970 boot
drwxr-xr-x 12 root root 3.2K Jan 1 1970 dev
drwxr-xr-x 76 root root 4.0K Jun 8 15:18 etc
drwxr-xr-x 3 root root 4.0K Jun 7 00:45 home
drwxr-xr-x 12 root root 4.0K May 30 03:25 include
drwxr-xr-x 11 root root 4.0K Jun 7 23:18 lib
drwx------ 2 root root 16K Jun 7 08:51 lost+found
drwxr-xr-x 10 root root 4.0K Jun 7 00:46 media
drwxr-xr-x 2 root root 4.0K May 7 15:28 mnt
drwxr-xr-x 4 root root 4.0K Jun 7 09:21 opt
dr-xr-xr-x 70 root root 0 Jan 1 1970 proc
drwx------ 4 root root 4.0K Jun 7 22:20 root
drwxr-xr-x 2 root root 4.0K Jun 7 00:49 sbin
drwxr-xr-x 4 root root 4.0K Jun 7 00:49 scripts
drwxr-xr-x 2 root root 4.0K Jul 21 2010 selinux
drwxr-xr-x 3 root root 4.0K Jun 7 00:45 srv
drwxr-xr-x 12 root root 0 Jan 1 1970 sys
drwxrwxrwt 6 root root 4.0K Jun 8 15:17 tmp
drwxr-xr-x 10 root root 4.0K Jun 7 00:39 usr
drwxr-xr-x 13 root root 4.0K Jun 7 00:39 varYay, we're done with that !
Step 3 : Setup the new Swap file and System files
In order for Gnu/Linux to find our newly created Swap file and system files, we have to tell him where it is.
For that, we have to declare it in the file '/tmp/sys/etc/fstab'.
Code:
pi@raspbmc:~$ sudo nano /tmp/sys/etc/fstabYou should have something that looks like :
Code:
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 0
/dev/mmcblk0p2 none swap sw 0 0
/dev/mmcblk0p3 / ext4 defaults,noatime 0 0Edit the three last lines so that they look like :
Code:
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 0 # SD card bootstrap files
/dev/sda1 none swap sw 0 0 # New swap file
/dev/sda2 / ext4 defaults,noatime 0 0 # New System files
/dev/sda3 /home ext4 defaults,noatime 0 0 # New User personal folderOnce this done, save by hitting 'Ctrl'+'O', then 'Return'.
You can then quit by hitting 'Ctrl'+'X'.
You can unmount all these now :
Code:
pi@raspbmc:~$ sudo umount /tmp/sys /tmp/home /tmp/sd_sysYou might see an error related to /tmp/home not being mounted. This is of no importance here.
We just want to make sure everything's unmounted.
Step 4 : Tel the Rpi where to find what
In order for the Rpi to boot on the right partition, we have to tell it in the '/boot/cmdline.txt' file :
Code:
pi@raspbmc:~$ sudo nano /boot/cmdline.txtEdit :
Code:
dwc_otg.lpm_enable=0 root=/dev/mmcblk0p3 rootfstype=ext4 rootwait quietTo look like this:
Code:
dwc_otg.lpm_enable=0 root=/dev/sda2 rootfstype=ext4 rootwait quietIt should be noted here that we've modified the 'root=' argument, so that it looks for our file system on '/dev/sda2'. That's the partition we set up in Step 1. We could also use another
technic to refer to our partitions by their UUID. Let me know if you'd like to see me describe this here.
Step 5 : Reboot !
Hopefully, if you followed all these steps carefully, after issuing :
Code:
pi@raspbmc:~$ sudo rebootYour Rpi should boot fine, and you should notice some improvements in terms of reactivity.
A new issue of :
Code:
pi@raspbmc:~$ df -hshould echo something like :
Code:
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 7.7G 826M 6.5G 12% /
tmpfs 62M 0 62M 0% /lib/init/rw
udev 10M 168K 9.9M 2% /dev
tmpfs 62M 4.0K 62M 1% /dev/shm
/dev/sda3 80G 264M 75G 1% /home
/dev/mmcblk0p1 63M 6.0M 58M 10% /bootYou can check that the swap file in use is our new one by issuing
Code:
pi@raspbmc:~$ sudo swapon -sIt should answer with :
Code:
Filename Type Size Used Priority
/dev/sda1 partition 249948 0 -1What if I want to upgrade raspbmc ?
If you were in the need to upgrade raspbmc, you'd simply reimage the sd card, that would boot on the system files that are on the SD card. You would then have to reproduce Step 2.A, Step 3 and Step 4.
It should be mentionned that you do not need to do this if you upgrade with a nightly build, as it is my understanding that it will only replace the xbmc binaries in /opt.
You could even use the '-u' option on rsync the first time you copy system files. That way, only the files that were modified since last rsync gets updated.
I've been careful not to let any typo or mistake slip in but you always forget some. Be sure to let me know if you find one, or have any suggestions/remarks.
Should also mention english is not my native language, hence the numerous mistakes and bad grammar.
Hope it's useful !
Thanks to BaldJedi and r8rae for their corrections, suggestions and commitment !
EDIT : Corrected some typos.
After some time running the RC2 with this setup, here are my observations :
- On SD card, RPi used to freeze from time to time ( on a hourly basis), with no apparent reason, and had to hard reboot. This behaviour seems to be gone, as the Rpi is running since yesterday without freezing.
- It seems that network shares access is faster (SMB and NFS). This might due to the fact that the cache files are no longer written on the SD, but on the HD, hence faster I/O and caching ... ?
I should have mentioned that I use a 4g class 10 SDHC card, that is supposed to offer à 10MB/S min rate.
Sam's Note: Remember, kernel-vfat-latest.tar.gz overwrites cmdline.txt, where you edited -- so be sure to chattr +i it -- cheers