在 Linux 上快速建立 空白檔案 / 大檔案 的方法

建立 swap 置換空間的或是跑一些測試時候有時會需要建立一個超大檔案(這邊指的是空白檔案),最常用的方法應該是 dd,例如從 /dev/zero 讀空白資料出來寫到我們的目的地檔案 ./dd_1G,一次寫入 1MB 的資料,總共寫入 1024 塊,就可以產生一個 1GB 的檔案:

在我的電腦上大概花了6秒鐘左右,檔案大的話花的時間會更多,例如 10G 或 100G 之類的,以空檔案來說,其實 dd 有更快的做法,直接用 seek 跳到指定的位置:

0.001 秒 … 可以想像成把檔案頭尾的起點終點設定完就結束了,中間的內容跳過不管他,所以會很快,不過這樣產生的檔案不見得什麼工具都接受,要拿來當 swap 的話就沒辦法了,會出現:

這邊可以參考 swapon 的 manpage:

NOTES
You should not use swapon on a file with holes. Swap over NFS may not work.

除了 dd,我們可以考慮使用 – fallocate / truncate

truncate 產生出來的檔案拿來給 swap 使用也會遇到上述 swap 不接受的問題,fallocate 就沒有這個問題,只是可能不適用於所有檔案系統,已知支援:btrfs, ext4, ocfs2 及 xfs,如果跑在 ext3 上面會出現:

fallocate: /swap: fallocate failed: Operation not supported

如果不考慮速度的還有一個工具 – xfs_mkfile,Debian / Ubuntu based GNU/Linux 可以透過 apt-get install xfsprogs 來安裝,雖然他是 xfs 的 utility 但也可以在其他檔案系統上面使用:

比較一下產出的檔案有什麼差異?光看大小跟 checksum 看起來是沒差~ 只是使用起來可能會有上面提到的差異

這時候要用 du -k --apparent-size & du -k 才會看得出來,有興趣的人可以試試看,結論是要產生 swap file 的話用 fallocate 會是比較快速又實際的方法

Online defragment on Linux

Though we rarely defragment the filesystem on unix-like os, sometimes we still need it.

The demonstrate is on LinuxMint 17.1 Cinnamon 64-bit which is based on Ubuntu 14.04 LTS.

Okay, let’s check the fragmentation first!

Here is a very easy method to check the fragment on ext4 online:
$ sudo fsck -fn /dev/sdXY, replace XY for yourself, for example, $ sudo fsck -fn /dev/sda1

The output will look like this:

fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
Warning! /dev/sda1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda1: 247/121920 files (0.4% non-contiguous), 66062/487424 blocks

(BTW, this method could also work on ext2, ext3)

and on xfs:
$ xfs_db -c frag -r /dev/sdXY, for example,$ xfs_db -c frag -r /dev/sda1

The output will look like this:

actual 297843, ideal 286076, fragmentation factor 3.95%

And defrag a partition now!

This is the command to defrag ext4 online:
$ sudo e4defrag -v /dev/sdXY, for example, $ sudo e4defrag -v /dev/sda1
(BTW, device /dev/sdXY can be replaced by a regular file or a directory, this is supported in e2fsprogs 1.42.9)

The output will look like this:

ext4 defragmentation for device(/dev/sda1)
[2/238]/boot/System.map-3.2.0-4-686-pae: 100% [ OK ]
[3/238]/boot/vmlinuz-3.16-0.bpo.2-686-pae: 100% [ OK ]
[5/238]/boot/grub/video_cirrus.mod: 100% [ OK ]
[6/238]/boot/grub/usb_keyboard.mod: 100% [ OK ]
[8/238]/boot/grub/locale/pl.mo: 100% [ OK ]
[9/238]/boot/grub/locale/nl.mo: 100% [ OK ]

.
.
.

[223/238]/boot/grub/password_pbkdf2.mod: 100% [ OK ]
[224/238]/boot/grub/videotest.mod: 100% [ OK ]
[225/238]/boot/grub/ntldr.mod: 100% [ OK ]
[226/238]/boot/grub/udf.mod: 100% [ OK ]
[227/238]/boot/grub/halt.mod: 100% [ OK ]
[228/238]/boot/grub/minix.mod: 100% [ OK ]
[229/238]/boot/grub/gfxterm.mod: 100% [ OK ]
[230/238]/boot/config-3.2.0-4-686-pae: 100% [ OK ]
[231/238]/boot/memtest86+_multiboot.bin: 100% [ OK ]
[232/238]/boot/memtest86+.bin: 100% [ OK ]
[233/238]/boot/System.map-3.16-0.bpo.2-686-pae: 100% [ OK ]
[234/238]/boot/initrd.img-3.2.0-4-686-pae: 100% [ OK ]
[235/238]/boot/initrd.img-3.16-0.bpo.2-686-pae: 100% [ OK ]
[237/238]/boot/config-3.16-0.bpo.2-686-pae: 100% [ OK ]
[238/238]/boot/vmlinuz-3.2.0-4-686-pae: 100% [ OK ]

Success: [ 234/238 ]
Failure: [ 4/238 ]

And here is the command to defrag xfs online:
$ sudo xfs_fsr -v /dev/sdXY, for example, $ sudo xfs_fsr -v /dev/sda1
(device /dev/sdXY by a file, but a directory is not supported yet in xfs_fsr version 3.1.9)

/ start inode=0
ino=1020
extents before:2 after:1 DONE ino=1020
ino=2000
extents before:3 after:1 DONE ino=2000
ino=2270
extents before:2 after:1 DONE ino=2270
ino=2272
extents before:2 after:1 DONE ino=2272
ino=2497
extents before:2 after:1 DONE ino=2497
ino=2498
extents before:2 after:1 DONE ino=2498
.
.
.

Not very hard, but useful, if the data on the target to defrag is important, I’ll respectfully suggest you to backup first, in case there is an accident like losing electrical power.