How to clone SD card

[wonka68]

Hi,

Despite working in IT for the past 20 years, I'm a linux noob, and have been determined to learn it for ages, so this little device is my first proper chance of picking up linux in anger. So please forgive me for what will seem like stupid questions.  Ultimately, my aim is to make a combined always-on print server, and file sharer (like the pogoplug).

I just received my sheevaplug multiboot yesterday and decided that before I start playing with it, I should backup and clone the SD card. I've read a few posts on cloning in this forum, and one or two others, but am still puzzled.

Someone in this thread suggested using a USB Image Tool. So having overcome various errors, I thought I'd managed it, but it doesn't seem to boot up. The blank SD card I have is 8GB, but not exactly the same size as the NewIT one (which may explain that failure).

I've considered booting up Ubuntu on my home PC, and copying the card that way, but there are 2 partitions?  Would I need to do an FDISK from Ubuntu?  And is it so straight forward as to copy the files?  Is there some sort of MBR on an SD card?

Or, alternatively, in this thread, Marcus suggests

• Remove SD card; boot from NAND.
• Mount USB stick and insert bootable SD card
• dd if=/mmcblk0 of=/mnt/my-usb-stick/my-sd-image.img
• Wait. The completion message for dd will indicate the number of bytes copied and the transfer speed.
• Upon completion, remove SD card and insert a new (blank) SD card
• dd if=/mnt/my-usb-stick/my-sd-image.img of=/mmcblk0

But I find parts of this confusing. Why the use of a USB stick? How do you mount a USB stick?  What does "dd" command do?  (OK I could google that).  I'm guessing the USB stick is to store an image... if so, does it need to be minimum 8GB?  Formatted?

As soon as I master this, I'll write it up and post, so other noobs can use it too.

[NewIT_Marcus]

Or, alternatively, in this thread, Marcus suggests

• Remove SD card; boot from NAND.
• Mount USB stick and insert bootable SD card
• dd if=/mmcblk0 of=/mnt/my-usb-stick/my-sd-image.img
• Wait. The completion message for dd will indicate the number of bytes copied and the transfer speed.
• Upon completion, remove SD card and insert a new (blank) SD card
• dd if=/mnt/my-usb-stick/my-sd-image.img of=/mmcblk0

But I find parts of this confusing. Why the use of a USB stick? How do you mount a USB stick?  What does "dd" command do?  (OK I could google that).  I'm guessing the USB stick is to store an image... if so, does it need to be minimum 8GB?  Formatted?

As soon as I master this, I'll write it up and post, so other noobs can use it too.

Just to expand on this, of course there's no point in using a USB stick that has less capacity than the SD card. It doesn't matter if your USB device is a memory stick or hard drive, it just needs to be a device with sufficient capacity to contain a complete image of the SD card.

We like the idea of cloning the entire card, because it puts everything necessary for recovery into a single file. Alternative solutions could include copying each partition individually, but in that case you might need to preserve additional information about the size and format of each partition. The dd method preserves all that information.

You can dd from a larger device to a smaller, but not vice versa. So if your SD card is larger than the supplied card, you can use it as a "duplicate". If it's smaller, you oughtn't to try.

But copying each partition individually would work. There isn't any MBR or equivalent to worry about. You shouldn't blindly copy commands that you don't understand, but you're just about there with the final bits of research that you need to decide the strategy that you prefer.

[DaveH]

I did this recently, but had the benefit of two available independent SD slots in my card reader, so for me the duplication was

dd if=/dev/sde of=dev/sdf

where sde and sdf happened to be the devices assigned to the two on my system. Your mileage may vary.

The dd command is a very low-level copy command (try 'man dd' for more information on what it does)

It takes ages to read /write 8GB though.

[wonka68]

I had another attempt at this last night.  I ran a Ubuntu 10.04 LiveCD on my laptop, which has a built-in card reader.  I thing I've got the hang of the "dd" command, and starting to grasp the file system & mounting. I ended up backing up the SD image onto my laptop's hard disk.

The "dd" operation onto my blank SD card reported a failure though, it ran out of space. This seems to be because the SD card is actually 7.9GB, not a full 8. (Actually, I scanned it in the Ubuntu LiveCD disk utility this morning, and the partitions do seem to be present, along with the various directories. I shall try booting off it tonight).

So, bearing in mind I have a non-standard size SD card, there are bound to be problems restoring the images, so should I just try copying the files across onto the blank SD card?

[NewIT_Marcus]

I had another attempt at this last night.  I ran a Ubuntu 10.04 LiveCD on my laptop, which has a built-in card reader.  I thing I've got the hang of the "dd" command, and starting to grasp the file system & mounting. I ended up backing up the SD image onto my laptop's hard disk.

The "dd" operation onto my blank SD card reported a failure though, it ran out of space. This seems to be because the SD card is actually 7.9GB, not a full 8. (Actually, I scanned it in the Ubuntu LiveCD disk utility this morning, and the partitions do seem to be present, along with the various directories. I shall try booting off it tonight).

So, bearing in mind I have a non-standard size SD card, there are bound to be problems restoring the images, so should I just try copying the files across onto the blank SD card?

Now that you know how to make an SD card image, you have a method for making backups onto your laptop.

If you want to duplicate the supplied SD card to your own (our current batch is also less than 8G), you'll need to find another way of copying the file systems. You have 3 choices

• copy each partition on the plug to another device (ie USB stick or drive), then transfer the contents to your partitioned SD card.
• mount the image file that you created on your laptop and extract the contents that way (also to your partitioned SD card)
• use your SD card as-is, after fixing the file system. (use fsck)

Once you have prepared your smaller card the way you want it, you could then use it as the "master", and copy it's image to the supplied Ultima card. It's really a matter of how you want to use the 2 cards - either as an alternative boot device, or as a safety backup. So far as a safety backup goes, that is just as easily (and probably more conveniently) achieved using an image file on your laptop. Our newer SD card images provide the size of the image file within the filename. (You should be able to download a 4G image and use fsck to expand it up to the full capacity).

[wonka68]

OK so I thought I'd go with your 3rd option... use fsck to fix the file system. I also tried it without the "-r" option:

Code:

root@ubuntu:~# fsck -r /dev/mmcblk0p2
fsck 1.41.4 (27-Jan-2009)
e2fsck 1.41.4 (27-Jan-2009)
The filesystem size (according to the superblock) is 1956344 blocks
The physical size of the device is 1939448 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort<y>? yes

This seems pretty obvious, but I thought fsck -r would repair the partition table.

By the way, during bootup, there's a failure to find a file... but I guess that could be copied after repair?? Or is it inside an image?:

Code:

EXT3-fs error (device mmcblk0p2): ext3_find_entry: reading directory #481441 offset 0
FATAL: Could not load /lib/modules/2.6.32.7/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.32.7/modules.dep: No such file or directory
 * Starting early crypto disks...       
[ OK ]
FATAL: Could not load /lib/modules/2.6.32.7/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.32.7/modules.dep: No such file or directory

[NewIT_Marcus]

By the way, during bootup, there's a failure to find a file... but I guess that could be copied after repair?? Or is it inside an image?:

Code:

EXT3-fs error (device mmcblk0p2): ext3_find_entry: reading directory #481441 offset 0
FATAL: Could not load /lib/modules/2.6.32.7/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.32.7/modules.dep: No such file or directory
 * Starting early crypto disks...       
[ OK ]
FATAL: Could not load /lib/modules/2.6.32.7/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.32.7/modules.dep: No such file or directory

Not sure of the options for fsck, but in principle it should be doable.

The missing modules ... that shouldn't be happening. Did you upgrade the kernel? If so, you should have upgraded/updated modules too. The sheeva.with-linux scripts should deliver both.

[wonka68]

Thanks for your help Marcus.

No, I didn't upgrade the kernel, I'm a complete linux noob   

I pressed ahead with fsck, and it did attempt the repair. There were quite a few pages of output, and a warning at the end that there are still problems in the file system. The upshot is that the MMC card is no longer recognised at bootup.

My gut feeling is that fundamentally, the partition probably needs resizing (resize2fs?), not that I know what I'm talking about...

So I think I shall abandon option 3, and go for one of the other options - i.e. manually clone.  Basically, I think I'll have to: copy the SD contents to a laptop folder, do an fdisk (for the 2 partitions), makefs, then copy the files across.

[wonka68]

OK I formatted/fdisked etc the SD card, and copied the files back onto it, but got some sort of failure:

=======================================================================
        |  \/  | __ _ _ ____   _____| | |
        | |\/| |/ _` | '__\ \ / / _ \ | |
        | |  | | (_| | |   \ V /  __/ | |
        |_|  |_|\__,_|_|    \_/ \___|_|_|
 _   _     ____              _
| | | |   | __ )  ___   ___ | |_
| | | |___|  _ \ / _ \ / _ \| __|
| |_| |___| |_) | (_) | (_) | |_
 \___/    |____/ \___/ \___/ \__|
 ** MARVELL BOARD: SHEEVA PLUG LE

U-Boot 1.1.4 (Jul 14 2009 - 06:46:57) Marvell version: 3.4.16

U-Boot code: 00600000 -> 0067FFF0  BSS: -> 006CF120

Soc: 88F6281 A0 (DDR2)
CPU running @ 1200Mhz L2 running @ 400Mhz
SysClock = 400Mhz , TClock = 200Mhz

DRAM CAS Latency = 5 tRP = 5 tRAS = 18 tRCD=6
DRAM CS[0] base 0x00000000   size 256MB
DRAM CS[1] base 0x10000000   size 256MB
DRAM Total size 512MB  16bit width
Flash:  0 kB
Addresses 8M - 0M are saved for the U-Boot usage.
Mem malloc Initialization (8M - 7M): Done
NAND:512 MB

CPU : Marvell Feroceon (Rev 1)

Streaming disabled
Write allocate disabled


USB 0: host mode
PEX 0: interface detected no Link.
Net:   egiga0 [PRIME], egiga1
Hit any key to stop autoboot:  3  2  1  0
SDHC found. Card desciption is:
Manufacturer:       0x03, OEM "SD"
Product name:       "SU08G", revision 8.0
Serial number:      2155484868
Manufacturing date: 3/2009
CRC:                0x00, b0 = 0

2822228 bytes read
## Booting image at 00800000 ...
   Image Name:   Linux-2.6.32.7
   Created:      2010-02-10  21:21:03 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2822164 Bytes =  2.7 MB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
Linux version 2.6.32.7 (newit@gemini) (gcc version 4.4.1 (Sourcery G++ Lite 2009q3-68) ) #5 PREEMPT Wed Feb 10 21:09:39 GMT 2010
CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE), cr=00053177
CPU: VIVT data cache, VIVT instruction cache
Machine: Marvell SheevaPlug Reference Board
Memory policy: ECC disabled, Data cache writeback
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 130048
Kernel command line: console=ttyS0,115200 root=/dev/mmcblk0p2 rootdelay=5
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 256MB 256MB = 512MB total
Memory: 513152KB available (5104K code, 1034K data, 140K init, 0K highmem)
SLUB: Genslabs=11, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Hierarchical RCU implementation.
NR_IRQS:114
Console: colour dummy device 80x30
Calibrating delay loop... 1192.75 BogoMIPS (lpj=5963776)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
NET: Registered protocol family 16
Kirkwood: MV88F6281-A0, TCLK=200000000.
Feroceon L2: Enabling L2
Feroceon L2: Cache support initialised.
bio: create slab <bio-0> at 0
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
cfg80211: Using static regulatory domain info
cfg80211: Regulatory domain: US
   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
   (2402000 KHz - 2472000 KHz @ 40000 KHz), (600 mBi, 2700 mBm)
   (5170000 KHz - 5190000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
   (5190000 KHz - 5210000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
   (5210000 KHz - 5230000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
   (5230000 KHz - 5330000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
   (5735000 KHz - 5835000 KHz @ 40000 KHz), (600 mBi, 3000 mBm)
cfg80211: Calling CRDA for country: US
Switching to clocksource orion_clocksource
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 16384 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
JFS: nTxBlock = 4010, nTxLock = 32083
msgmni has been set to 1002
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
serial8250.0: ttyS0 at MMIO 0xf1012000 (irq = 33) is a 16550A
console [ttyS0] enabled
brd: module loaded
loop: module loaded
sata_mv sata_mv.0: version 1.28
sata_mv sata_mv.0: slots 32 ports 2
scsi0 : sata_mv
scsi1 : sata_mv
ata1: SATA max UDMA/133 irq 21
ata2: SATA max UDMA/133 irq 21
NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
Scanning device for bad blocks
Bad eraseblock 858 at 0x000006b40000
Bad eraseblock 4001 at 0x00001f420000
Creating 3 MTD partitions on "orion_nand":
0x000000000000-0x000000100000 : "u-boot"
0x000000100000-0x000000500000 : "uImage"
0x000000500000-0x000020000000 : "root"
MV-643xx 10/100/1000 ethernet driver version 1.4
mv643xx_eth smi: probed
net eth0: port 0 with MAC address 00:50:43:01:52:b5
libertas_sdio: Libertas SDIO driver
libertas_sdio: Copyright Pierre Ossman
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
orion-ehci orion-ehci.0: Marvell Orion EHCI
orion-ehci orion-ehci.0: new USB bus registered, assigned bus number 1
orion-ehci orion-ehci.0: irq 19, io mem 0xf1050000
orion-ehci orion-ehci.0: USB 2.0 started, EHCI 1.00
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver ums-datafab
usbcore: registered new interface driver ums-freecom
usbcore: registered new interface driver ums-jumpshot
usbcore: registered new interface driver ums-sddr09
usbcore: registered new interface driver ums-sddr55
mice: PS/2 mouse device common for all mice
rtc-mv rtc-mv: rtc core: registered rtc-mv as rtc0
i2c /dev entries driver
cpuidle: using governor ladder
cpuidle: using governor menu
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
mmc0: mvsdio driver initialized, lacking card detect (fall back to polling)
Registered led device: plug:green:health
mv_xor_shared mv_xor_shared.0: Marvell shared XOR driver
mv_xor_shared mv_xor_shared.1: Marvell shared XOR driver
mv_xor mv_xor.0: Marvell XOR: ( xor cpy )
mmc0: host does not support reading read-only switch. assuming write-enable.
mmc0: new high speed SDHC card at address b15d
mmcblk0: mmc0:b15d SU08G 7.40 GiB
 mmcblk0: p1 p2
mv_xor mv_xor.1: Marvell XOR: ( xor fill cpy )
mv_xor mv_xor.2: Marvell XOR: ( xor cpy )
ata1: SATA link down (SStatus 0 SControl F300)
mv_xor mv_xor.3: Marvell XOR: ( xor fill cpy )
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
oprofile: using timer interrupt.
TCP cubic registered
NET: Registered protocol family 17
lib80211: common routines for IEEE802.11 drivers
rtc-mv rtc-mv: setting system clock to 2010-06-05 10:12:47 UTC (1275732767)
Waiting 5sec before mounting root device...
ata2: SATA link down (SStatus 0 SControl F300)
kjournald starting.  Commit interval 5 seconds
EXT3 FS on mmcblk0p2, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with writeback data mode.
VFS: Mounted root (ext3 filesystem) on device 179:2.
Freeing init memory: 140K
Warning: unable to open an initial console.
Kernel panic - not syncing: No init found.  Try passing init= option to kernel.
[<c00307f8>] (unwind_backtrace+0x0/0xdc) from [<c04031e4>] (panic+0x48/0x11c)
[<c04031e4>] (panic+0x48/0x11c) from [<c002b5e8>] (init_post+0xd4/0xfc)
[<c002b5e8>] (init_post+0xd4/0xfc) from [<c0008440>] (kernel_init+0xe0/0x110)
[<c0008440>] (kernel_init+0xe0/0x110) from [<c002c40c>] (kernel_thread_exit+0x0/0x8)

=======================================================================

I followed the advice in this thread, although I just kept it simple, didn't try to replace the boot image or anything else:
http://www.newit.co.uk/forum/index.php/topic,309.0.html

Any ideas what went wrong??

[NewIT_Marcus]

OK I formatted/fdisked etc the SD card, and copied the files back onto it, but got some sort of failure:

Any ideas what went wrong??

You didn't say which card this is. I assume that it is your smaller card.

Start again:

• Partiton the card; small partition for kernel (say 4M - 15M, we use 4M) + larger partition for rootfs
• Boot from NAND
• Mount SD card partitions
• cp partition #1 to USB or eSATA or network device
• cp partition #2 to USB or eSATA or network device
• unmount SD card
• Insert alternate SD card
• cp back to partitions #1 & #2

[wonka68]

Yes, it is the smaller (7.9G) card.

I formatted (etc) it on Ubuntu 64-bit 10.04 LiveCD... it went smoothly... but I guess I need to try this on the sheevaplug, as you say.

[wonka68]

OK, this is now resolved.

I now have an alternative SD card, formatted and booting up the sheevaplug. I've written up everything I did to get it cloned. See here:
http://sheevaplug.jottit.com/backup

Thanks to Marcus for his patience, and suggestions.

It's dead easy this linux malarkey  ;-)

[NewIT_Marcus]

Congratulations.

The method you ended up using is essentially copying of each and every file. In some ways that fundamentally different from cloning (which was, after all the subject of the original post / question). Cloning and copying have different characteristics and advantages and disadvantages.

Most commonly we use cloning (for the SD cards that we sell), because it is a one-step process and perfect for the job. As a backup solution, however, it wouldn't necessarily be perfect, because the one thing that it doesn't do is to test the integrity of the files or file system.

Having said that, if cloning is the way you want to go, I can tell you now that not all SD cards are created equal. You knew that already, but our experience has also been that the same product in an identical package does not necessarily mean an identical capacity. I don't know how many batches we have gone through, but I know that for our Ultima cards, our 4G cards have come with 3 different capacities, and the 8G cards with 2 different capacities.

If it is important to you to have cards with identical capacities we would recommend that you buy a number of same-model cards in one purchase. If you wish to buy cards identical to the Ultima cards that we supply with the Sheevaplug, they are available in our shop ...