OSMC forum

Full Version: [Howto] Configure and boot to a USB key/hard-drive
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5 6 7 8 9 10 11 12
[undefined=undefined]This post id old and outdated, please do NOT use this method on RC5 or higher[/undefined]


Hi there,

I've seen this question asked a few times here, and answered also, but with rather terse explanations.
So I'll try to describe with details how you can use a usb connected drive plugged into your Rpi, that will hold your system and allow you faster i/o rates, and cheaper large storage.

So first, some explanations :

Once you've imaged your SD card with the Rpi installer, you end up with 3 partitions :
  • One vfat sized 63 MB
  • One swap partiton sized 122MB
  • One Ext4 partition that fits the remaining space of your SD card.

The vfat (or FAT32 ) partition contains the bootstrap files, firmware, kernel and boot arguments. It HAS to stay on the SD card. This is due to the hardware of the Pi and the fact that we're bootstrapping from a GPU.
This one stays where it is.

The Swap space is used by *nix like systems to cache unused RAM data when this last one gets full. It is recommended to be twice the size of you ram when less than 1GB.
I'm not sure wether it's accurate or not, but since we're moving to a large HD, we can enlarge the original 122 MB one to reach 256MB Smile
This one will move to HD.

The Ext4 partition contains the debian system files that serve as a software base for xbmc to run.
This one is moving too.

Separate Home folder

Raspbmc being at alpha stage, it means that you might have to go through the process of re-imaging your SD card, and recreating your filesystems a lot.

We could set up our filesystem so that when you'll upgrade you system files, all your media files, but more important, all your settings would survive an update or re-imaging.

For this, we'll have to create another partition that will old your personal 'Home' folder.

Some stuff you need to know

The raspbmc image default id infos are :

Code:
login : pi
pasword: raspberry

These commands should be run from your default user location. To make sure you're there :

Code:
pi@raspbmc: ~$ cd ~

Most of the commands described here need the 'sudo' prefix in order to work !
If you get a 'permission denied' or 'command not found' error message, it is likely that you forgot the 'sudo' prefix.

Let's go

What you need :
  • A Raspberry Pi (duh)
  • A SD Card with Raspbmc set up
  • A USB Hard drive with an external PU ( or USB key )

IMPORTANT : The steps described here will DESTROY the data that resides on your hard drive ! Make sure to backup everything important. I will refuse to be held responsible for any file loss.
You've been warned. If you agree to this disclaimer and to take your responsibilities, just breath once.


Preamble

An important step is to disable udevd before connecting your USB drive. Udev is the daemon that's in charge of detecting new devices. It will prevent us from going to a lot of hassle with it trying to mount our new filesystems :

Code:
pi@raspbmc:~$ sudo service udev stop

We can also stop xbmc :

Code:
pi@raspbmc:~$ sudo initctl stop xbmc


Step 1 : Setup your hard-drive

First, we need to set up your hard drive with the right file system. To achieve this, we're going to use some nice and powerful *nix programs. Fear not of the CLI, for it is a mighty full tool !

Unfortunately, you'll have to use another computer to SSH into your Rpi, since there is no option to boot on CLI, without XBMC launching for the moment ( Could be a good idea ? )

Once you ssh'd into you Rpi, you can check your hard-drive is there by listing present devices :

Code:
pi@raspbmc:~$  ls /dev/sda


It should issue a line saying ''/dev/sda''. If otherwise, you might want to check your HD is spinning, USB connections, etc.
You can also try to increment the last letter in 'sda', in the alphabetic order - ie 'sdb', 'sdc', etc.

Once you made sure /dev/sda is your HD alright, we're going to partition it this way :

- one SWAP part. sized 256 MB
- one / part. sized 8 GB ( yeah we've got lot of space )
- one /home partition that will fit the remainder of the disk.

Let's unmount all partitions on sda :

Code:
pi@raspbmc:~$  sudo umount /dev/sda*


As much people won't feel confortable without a GUI, I'm going to use the 'cfdisk' utility :

Code:
pi@raspbmc:~$  sudo cfdisk -z /dev/sda

You should now see a nice text user interface that lists what's on your disk right now, with a menu on the bottom of the screen.

- Select 'New' then hit return. Choose 'Primary', then enter the size of the swap in MB : 256, and choose to place it on the beginning of the disk.
Once you've done that, select 'Type' in the bottom menu and hit return. You should now see a list of possible types to choose from. This is a numbered list, so to choose SWAP type, we shall enter '82', and validate.
- Select the free space with the down arrow key, and do it again, this time with 8000Mb size, and type Linux : '83'.
- Repeat, with default proposed size.

You should now write down what partitions the different device files refers to, just in case.

It should be :
  • /dev/sda1 == SWAP (256MB)
  • /dev/sda2 == / (8 GB)
  • /dev/sda3 == /home (remainder of the disk)

Once done, you MUST select 'Write' in the bottom menu and confirm by answering 'yes' to the question that you'll make sure to read and check.

You're back at the cli. Now, we have defined our partitions, but the filesystems (fs) were not created yet. That's what we're going to do with the mkfs utilies. The swap uses a different utility.

Build the SWAP fs :
Code:
pi@raspbmc:~$  sudo mkswap /dev/sda1

Build the Ext4 fs :

Code:
pi@raspbmc:~$  sudo mkfs.ext4 /dev/sda2
Code:
pi@raspbmc:~$  sudo mkfs.ext4 /dev/sda3

If you get an error message telling you '/dev/sdX' doesn't exist, you can unplug, then replug your USB drive. Then check that 'dev/sdX' is accessible by entering :

Code:
pi@raspbmc:~$  ls /dev/sdX

It should be noted that you do not want to see any /dev/sdaX in the ouptut of

Code:
pi@raspbmc:~$  df -h


yet. If you do, you have to unmount then by issuing :

Code:
pi@raspbmc:~$  sudo umount /dev/sda*

( The wildcard is literal ! )

That's it for this step. We now have a nice setup hard drive.
To check for this, we can issue the command :

Code:
pi@raspbmc:~$ sudo fdisk -l /dev/sda

that should print something like :

Code:
pi@raspbmc:~$ sudo fdisk -l /dev/sda

Disk /dev/sda: 200.0 GB, 200049647616 bytes
255 heads, 63 sectors/track, 24321 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000baa4f

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1           1           12814      524288   82  Linux swap / Solaris
/dev/sda2           12814       13834     8192000   83  Linux
/dev/sda3           13834       24322    84243456   83  Linux

Step 2 : Copy the system files


In order to copy the files from the SD to the USB HD, we'll have to do some mounting .
We create the directories that will hold our mounts , then mount the corresponding partitions :

Code:
pi@raspbmc:~$ sudo mkdir /tmp/sys && sudo mount /dev/sda2 /tmp/sys

Code:
pi@raspbmc:~$ sudo mkdir /tmp/home && sudo mount /dev/sda3 /tmp/home

Beware ! These directories will disappear upon reboot, as they are place in the '/tmp' folder.

Should I use 'dd' or 'rsync' ?

I've seen several answers that mentioned the use of 'dd' commands to copy your system files.
I personaly don't think it's a good idea, nor the good tool.

I'll just copy what I said on another thread :

Unless there is a good reason for using 'dd', I would recommend using rsync.
'dd' copies a disk bit-by-bit, thus it's very slow, plus it can be very dangerous to play with when you don't know what you're doing.


I'll add that if you want to duplicate a mounted filesystem you're working off, you have to use the rsync method.

Let's install that sweet utility :

Code:
pi@raspbmc:~$ sudo apt-get install rsync

A. Personal folder and user settings


We'll begin easy, and copy the user settings on our home, reimaging-proof partition, with rsync first :

Code:
pi@raspbmc:~$ sudo rsync -avh /home/pi/ /tmp/home/pi/

We use 3 options here :

-a : archive mode : it will preserve owner,group,permissions,links and devices information, and perform recursively. This is REQUIRED.
-v : Because we want more output of the progress
-h : Because we want human readable output !

Let's get rid of this one for now :

Code:
pi@raspbmc:~$ sudo umount /tmp/home

B. System files

We'll continue a little harder now .
Since the filesystem we want to copy the files from is running ( your Rpi is on right ?), we'll have to remount it elsewhere in order to rsync it without falling into a deadly recursive loop.

Plus, we don't want to rsync what's in the '/home/pi' folder sinc we just rsynced it :

EDIT : And obviously, we don't want to rsync the '/tmp' folder, since we're working there :/
Thanks to Baldjedi for bringing that up ! We also can exclude the '/boot' folder, as it won't be of any use on the HD.

We remount the FS :

Code:
pi@raspbmc:~$ sudo mkdir /tmp/sd_sys && sudo mount --bind / /tmp/sd_sys

We rsync, without '/home' , '/tmp', '/boot' contents :

Code:
pi@raspbmc:~$ sudo rsync -avh --exclude 'home/pi' --exclude 'tmp/' --exclude 'boot/' /tmp/sd_sys/ /tmp/sys/

This is where you could optionally use the '-u' option, so that when you upgrade the system files from the SD card next time, only files that were modified are updated.


We yet have to recreate the folders we excluded during rsync :

Code:
pi@raspbmc:~$ sudo mkdir /tmp/sys/boot /tmp/sys/tmp

You can check all went well by issuing :

Code:
pi@raspbmc:~$ df -h /dev/sda2

It should show a use of at least a few hundreds MB.

You can also :

Code:
pi@raspbmc:~$ ls -lah /tmp/sys

which should answer something like:

Code:
total 89K
drwxr-xr-x 23 root root 4.0K Jun  7 09:15 .
drwxr-xr-x 23 root root 4.0K Jun  7 09:15 ..
drwxr-xr-x  2 root root 4.0K Jun  7 21:38 bin
drwxr-xr-x  2 root root 1.0K Jan  1  1970 boot
drwxr-xr-x 12 root root 3.2K Jan  1  1970 dev
drwxr-xr-x 76 root root 4.0K Jun  8 15:18 etc
drwxr-xr-x  3 root root 4.0K Jun  7 00:45 home
drwxr-xr-x 12 root root 4.0K May 30 03:25 include
drwxr-xr-x 11 root root 4.0K Jun  7 23:18 lib
drwx------  2 root root  16K Jun  7 08:51 lost+found
drwxr-xr-x 10 root root 4.0K Jun  7 00:46 media
drwxr-xr-x  2 root root 4.0K May  7 15:28 mnt
drwxr-xr-x  4 root root 4.0K Jun  7 09:21 opt
dr-xr-xr-x 70 root root    0 Jan  1  1970 proc
drwx------  4 root root 4.0K Jun  7 22:20 root
drwxr-xr-x  2 root root 4.0K Jun  7 00:49 sbin
drwxr-xr-x  4 root root 4.0K Jun  7 00:49 scripts
drwxr-xr-x  2 root root 4.0K Jul 21  2010 selinux
drwxr-xr-x  3 root root 4.0K Jun  7 00:45 srv
drwxr-xr-x 12 root root    0 Jan  1  1970 sys
drwxrwxrwt  6 root root 4.0K Jun  8 15:17 tmp
drwxr-xr-x 10 root root 4.0K Jun  7 00:39 usr
drwxr-xr-x 13 root root 4.0K Jun  7 00:39 var

Yay, we're done with that !

Step 3 : Setup the new Swap file and System files

In order for Gnu/Linux to find our newly created Swap file and system files, we have to tell him where it is.

For that, we have to declare it in the file '/tmp/sys/etc/fstab'.

Code:
pi@raspbmc:~$ sudo nano /tmp/sys/etc/fstab

You should have something that looks like :

Code:
proc            /proc           proc    defaults         0       0
/dev/mmcblk0p1  /boot           vfat    defaults         0       0
/dev/mmcblk0p2  none            swap    sw               0       0
/dev/mmcblk0p3  /               ext4    defaults,noatime 0       0

Edit the three last lines so that they look like :

Code:
proc            /proc           proc    defaults                0       0
/dev/mmcblk0p1  /boot           vfat    defaults         0       0  # SD card bootstrap files
/dev/sda1       none            swap    sw                   0       0 # New swap file
/dev/sda2       /               ext4    defaults,noatime   0       0 # New System files
/dev/sda3       /home           ext4    defaults,noatime 0       0 # New User personal folder

Once this done, save by hitting 'Ctrl'+'O', then 'Return'.
You can then quit by hitting 'Ctrl'+'X'.

You can unmount all these now :

Code:
pi@raspbmc:~$ sudo umount /tmp/sys /tmp/home /tmp/sd_sys

You might see an error related to /tmp/home not being mounted. This is of no importance here.
We just want to make sure everything's unmounted.

Step 4 : Tel the Rpi where to find what

In order for the Rpi to boot on the right partition, we have to tell it in the '/boot/cmdline.txt' file :

Code:
pi@raspbmc:~$ sudo nano /boot/cmdline.txt

Edit :

Code:
dwc_otg.lpm_enable=0 root=/dev/mmcblk0p3 rootfstype=ext4 rootwait quiet

To look like this:

Code:
dwc_otg.lpm_enable=0 root=/dev/sda2 rootfstype=ext4 rootwait quiet

It should be noted here that we've modified the 'root=' argument, so that it looks for our file system on '/dev/sda2'. That's the partition we set up in Step 1. We could also use another
technic to refer to our partitions by their UUID. Let me know if you'd like to see me describe this here.

Step 5 : Reboot !


Hopefully, if you followed all these steps carefully, after issuing :

Code:
pi@raspbmc:~$ sudo reboot


Your Rpi should boot fine, and you should notice some improvements in terms of reactivity.

A new issue of :

Code:
pi@raspbmc:~$ df -h


should echo something like :

Code:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             7.7G  826M  6.5G  12% /
tmpfs                  62M     0   62M   0% /lib/init/rw
udev                   10M  168K  9.9M   2% /dev
tmpfs                  62M  4.0K   62M   1% /dev/shm
/dev/sda3              80G  264M   75G   1% /home
/dev/mmcblk0p1         63M  6.0M   58M  10% /boot

You can check that the swap file in use is our new one by issuing

Code:
pi@raspbmc:~$ sudo swapon -s

It should answer with :

Code:
Filename                Type        Size    Used    Priority
/dev/sda1                               partition    249948    0    -1

What if I want to upgrade raspbmc ?

If you were in the need to upgrade raspbmc, you'd simply reimage the sd card, that would boot on the system files that are on the SD card. You would then have to reproduce Step 2.A, Step 3 and Step 4.

It should be mentionned that you do not need to do this if you upgrade with a nightly build, as it is my understanding that it will only replace the xbmc binaries in /opt.

You could even use the '-u' option on rsync the first time you copy system files. That way, only the files that were modified since last rsync gets updated.

I've been careful not to let any typo or mistake slip in but you always forget some. Be sure to let me know if you find one, or have any suggestions/remarks.
Should also mention english is not my native language, hence the numerous mistakes and bad grammar.

Hope it's useful !

Thanks to BaldJedi and r8rae for their corrections, suggestions and commitment !


EDIT : Corrected some typos.

After some time running the RC2 with this setup, here are my observations :

- On SD card, RPi used to freeze from time to time ( on a hourly basis), with no apparent reason, and had to hard reboot. This behaviour seems to be gone, as the Rpi is running since yesterday without freezing.

- It seems that network shares access is faster (SMB and NFS). This might due to the fact that the cache files are no longer written on the SD, but on the HD, hence faster I/O and caching ... ?

I should have mentioned that I use a 4g class 10 SDHC card, that is supposed to offer à 10MB/S min rate.

Sam's Note: Remember, kernel-vfat-latest.tar.gz overwrites cmdline.txt, where you edited -- so be sure to chattr +i it -- cheers
Although I'm waiting until final release to try this out, I saw that you mentioned about doing this in another post and I was going to PM you for the how-to info, no need now, great help, thanks!
Deffinately a keeper!
Is this line correct?

sudo rsync -avh --exclude '/home/pi/' /tmp/sd_sys /tmp/sys/

after entering this command it appears to end up in a loop...each time adding a sd_sys/tmp/sys. after 5 mins the lines appearing in terminal look like

sd_sys/tmp/sys/sd_sys/tmp/sys/sd_syd/tmp/sys.......
@AB can you look into the above please? then once we have some positive feedback from the whole tut I will stick this thread.

Would also appreciate it if you can be our official HOWTOer, very impressed with this comprehensive guide.
(06-09-2012 01:37 PM)BaldJedi Wrote: [ -> ]Is this line correct?

sudo rsync -avh --exclude '/home/pi/' /tmp/sd_sys /tmp/sys/

after entering this command it appears to end up in a loop...each time adding a sd_sys/tmp/sys. after 5 mins the lines appearing in terminal look like

sd_sys/tmp/sys/sd_sys/tmp/sys/sd_syd/tmp/sys.......

My mistake ! I forgot the trailing slash after 'sd_sys' and excluding the 'tmp' folder !
I also excluded the '/boot' folder, as we don't need it on the HD.
Should be :

Code:
sudo rsync -avh --exclude 'home/pi' --exclude 'tmp' --exclude 'boot' /tmp/sd_sys/ /tmp/sys/

To get rid of all the folder that appeared due to that mistake, you can do :

Code:
sudo rm -rf /tmp/sd_sys/*

Let me know if that does the trick.
Was getting another loop...tmp/sys/tmp/sys/tmp/sys.... but noticed you have edited your post since i tried lol
(06-09-2012 02:23 PM)BaldJedi Wrote: [ -> ]Was getting another loop...tmp/sys/tmp/sys/tmp/sys.... but noticed you have edited your post since i tried lol

Yeah, sorry about that, I am having a hard time trying to figure out the right path to exclude since a trailing slash can change the meaning of the command, plus, it depends on where you're running the commands from :/

I think it's ok now, can you give it a try ?
No probs....have to try later as rebooting Pi now giving kernel panic.. going to have to re-install Smile
(06-09-2012 02:02 PM)mikeyb5753 Wrote: [ -> ]@AB can you look into the above please? then once we have some positive feedback from the whole tut I will stick this thread.

Would also appreciate it if you can be our official HOWTOer, very impressed with this comprehensive guide.

Well let's first see how many people I lead to a kernel panic before officialising anything ! Smile

I'd be glad to be of any use to this great project, and if my knowledge and will to share it can serve raspbmc users, it's a great joy for me.
Did a clean install of raspbmc and went through the instructions again.

The files appeared to trf to hard drive ok....no more looping. However, getting kernel panic when rebooting the Pi.
Pages: 1 2 3 4 5 6 7 8 9 10 11 12
Reference URL's