在 Linux 上快速建立 空白檔案 / 大檔案 的方法

建立 swap 置換空間的或是跑一些測試時候有時會需要建立一個超大檔案(這邊指的是空白檔案),最常用的方法應該是 dd,例如從 /dev/zero 讀空白資料出來寫到我們的目的地檔案 ./dd_1G,一次寫入 1MB 的資料,總共寫入 1024 塊,就可以產生一個 1GB 的檔案:

$ time dd if=/dev/zero of=./dd_1G bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.92408 s, 181 MB/s

real 0m6.052s
user 0m0.000s
sys 0m0.660s

在我的電腦上大概花了6秒鐘左右,檔案大的話花的時間會更多,例如 10G 或 100G 之類的,以空檔案來說,其實 dd 有更快的做法,直接用 seek 跳到指定的位置:

$ time dd of=./dd_1G_fast bs=1 count=0 seek=1G
0+0 records in
0+0 records out
0 bytes (0 B) copied, 3.4469e-05 s, 0.0 kB/s

real 0m0.001s
user 0m0.000s
sys 0m0.001s

0.001 秒 … 可以想像成把檔案頭尾的起點終點設定完就結束了,中間的內容跳過不管他,所以會很快,不過這樣產生的檔案不見得什麼工具都接受,要拿來當 swap 的話就沒辦法了,會出現:

$ sudo swapon /swap
swapon: /swap: skipping - it appears to have holes.

這邊可以參考 swapon 的 manpage:

NOTES
You should not use swapon on a file with holes. Swap over NFS may not work.

除了 dd,我們可以考慮使用 – fallocate / truncate

$ time fallocate -l 1G fallocate_1G
real 0m0.002s
user 0m0.000s
sys 0m0.002s

$ time truncate -s 1G truncate_1G
real 0m0.001s
user 0m0.000s
sys 0m0.001s

truncate 產生出來的檔案拿來給 swap 使用也會遇到上述 swap 不接受的問題,fallocate 就沒有這個問題,只是可能不適用於所有檔案系統,已知支援:btrfs, ext4, ocfs2 及 xfs,如果跑在 ext3 上面會出現:

fallocate: /swap: fallocate failed: Operation not supported

如果不考慮速度的還有一個工具 – xfs_mkfile,Debian / Ubuntu based GNU/Linux 可以透過 apt-get install xfsprogs 來安裝,雖然他是 xfs 的 utility 但也可以在其他檔案系統上面使用:

$ time xfs_mkfile 1024m xfs_mkfile_1G
real 0m10.810s
user 0m0.000s
sys 0m0.110s

比較一下產出的檔案有什麼差異?光看大小跟 checksum 看起來是沒差~ 只是使用起來可能會有上面提到的差異

peter ~/test $ ls -l
total 4194320
-rw------- 1 peter peter 1073741824 Jan 20 17:13 dd_1G
-rw------- 1 peter peter 1073741824 Jan 20 17:13 dd_1G_2
-rw------- 1 peter peter 1073741824 Jan 20 17:16 dd_1G_fast
-rw------- 1 peter peter 1073741824 Jan 20 17:13 fallocate_1G
-rw------- 1 peter peter 1073741824 Jan 20 17:13 truncate_1G
-rw------- 1 peter peter 1073741824 Jan 20 17:14 xfs_mkfile_1G

peter ~/test $ md5sum *
cd573cfaace07e7949bc0c46028904ff dd_1G
cd573cfaace07e7949bc0c46028904ff dd_1G_2
cd573cfaace07e7949bc0c46028904ff dd_1G_fast
cd573cfaace07e7949bc0c46028904ff fallocate_1G
cd573cfaace07e7949bc0c46028904ff truncate_1G
cd573cfaace07e7949bc0c46028904ff xfs_mkfile_1G

這時候要用 du -k --apparent-size & du -k 才會看得出來,有興趣的人可以試試看,結論是要產生 swap file 的話用 fallocate 會是比較快速又實際的方法

Online disk defragment on Linux

Though we rarely defragment the filesystem on unix-like os, sometimes we still need it.

The demonstrate is on LinuxMint 17.1 Cinnamon 64-bit which is based on Ubuntu 14.04 LTS.

Okay, let’s check the fragmentation first!

Here is a very easy method to check the fragment on ext4 online:
$ sudo fsck -fn /dev/sdXY, replace XY for yourself, for example, $ sudo fsck -fn /dev/sda1

The output will look like this:

fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
Warning! /dev/sda1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda1: 247/121920 files (0.4% non-contiguous), 66062/487424 blocks

(BTW, this method could also work on ext2, ext3)

and on xfs:
$ xfs_db -c frag -r /dev/sdXY, for example,$ xfs_db -c frag -r /dev/sda1

The output will look like this:

actual 297843, ideal 286076, fragmentation factor 3.95%

And defrag a partition now!

This is the command to defrag ext4 online:
$ sudo e4defrag -v /dev/sdXY, for example, $ sudo e4defrag -v /dev/sda1
(BTW, device /dev/sdXY can be replaced by a regular file or a directory, this is supported in e2fsprogs 1.42.9)

The output will look like this:

ext4 defragmentation for device(/dev/sda1)
[2/238]/boot/System.map-3.2.0-4-686-pae: 100% [ OK ]
[3/238]/boot/vmlinuz-3.16-0.bpo.2-686-pae: 100% [ OK ]
[5/238]/boot/grub/video_cirrus.mod: 100% [ OK ]
[6/238]/boot/grub/usb_keyboard.mod: 100% [ OK ]
[8/238]/boot/grub/locale/pl.mo: 100% [ OK ]
[9/238]/boot/grub/locale/nl.mo: 100% [ OK ]

.
.
.

[223/238]/boot/grub/password_pbkdf2.mod: 100% [ OK ]
[224/238]/boot/grub/videotest.mod: 100% [ OK ]
[225/238]/boot/grub/ntldr.mod: 100% [ OK ]
[226/238]/boot/grub/udf.mod: 100% [ OK ]
[227/238]/boot/grub/halt.mod: 100% [ OK ]
[228/238]/boot/grub/minix.mod: 100% [ OK ]
[229/238]/boot/grub/gfxterm.mod: 100% [ OK ]
[230/238]/boot/config-3.2.0-4-686-pae: 100% [ OK ]
[231/238]/boot/memtest86+_multiboot.bin: 100% [ OK ]
[232/238]/boot/memtest86+.bin: 100% [ OK ]
[233/238]/boot/System.map-3.16-0.bpo.2-686-pae: 100% [ OK ]
[234/238]/boot/initrd.img-3.2.0-4-686-pae: 100% [ OK ]
[235/238]/boot/initrd.img-3.16-0.bpo.2-686-pae: 100% [ OK ]
[237/238]/boot/config-3.16-0.bpo.2-686-pae: 100% [ OK ]
[238/238]/boot/vmlinuz-3.2.0-4-686-pae: 100% [ OK ]

Success: [ 234/238 ]
Failure: [ 4/238 ]

And here is the command to defrag xfs online:
$ sudo xfs_fsr -v /dev/sdXY, for example, $ sudo xfs_fsr -v /dev/sda1
(device /dev/sdXY by a file, but a directory is not supported yet in xfs_fsr version 3.1.9)

/ start inode=0
ino=1020
extents before:2 after:1 DONE ino=1020
ino=2000
extents before:3 after:1 DONE ino=2000
ino=2270
extents before:2 after:1 DONE ino=2270
ino=2272
extents before:2 after:1 DONE ino=2272
ino=2497
extents before:2 after:1 DONE ino=2497
ino=2498
extents before:2 after:1 DONE ino=2498
.
.
.

Not very hard, but useful, if the data on the target to defrag is important, I’ll respectfully suggest you to backup first, in case there is an accident like losing electrical power.