How to install ruby gem packages without Internet?

git_stats is a good tool to analyze the statistics of a git repo, there is an issue tomgi/git_stats#45 How to Install Offline in that project, I think this is a good question, sometimes the speed or quality of Internet is not so fast and stable, or we want to install a gem package on a server that can only access internal network only, so I tried to figure it out, that’s the solution:

Milly/gem-fetch-dependencies is a ruby script posted on gist, it can help us fetch gem packages with the dependency, we can’t just fetch the package we want to use via gem fetch , because it doesn’t handle the dependency issue, that’s why we need this script, download the script(raw) first:

https://gist.github.com/Milly/909564/raw/c3c921e78b6736197558ee44efb84a495f4c1505/gem-fetch-dependencies

and we can easily fetch the gem packages with dependencies with this script like this:
$ ruby gem-fetch-dependencies fetch --dependencies

Once we fetched all the packages, you copy them to everywhere you want, and install them locally:
$ gem install -f --local *.gem

Online defragment on Linux

Though we rarely defragment the filesystem on unix-like os, sometimes we still need it.

The demonstrate is on LinuxMint 17.1 Cinnamon 64-bit which is based on Ubuntu 14.04 LTS.

Okay, let’s check the fragmentation first!

Here is a very easy method to check the fragment on ext4 online:
$ sudo fsck -fn /dev/sdXY, replace XY for yourself, for example, $ sudo fsck -fn /dev/sda1

The output will look like this:

fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
Warning! /dev/sda1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda1: 247/121920 files (0.4% non-contiguous), 66062/487424 blocks

(BTW, this method could also work on ext2, ext3)

and on xfs:
$ xfs_db -c frag -r /dev/sdXY, for example,$ xfs_db -c frag -r /dev/sda1

The output will look like this:

actual 297843, ideal 286076, fragmentation factor 3.95%

And defrag a partition now!

This is the command to defrag ext4 online:
$ sudo e4defrag -v /dev/sdXY, for example, $ sudo e4defrag -v /dev/sda1
(BTW, device /dev/sdXY can be replaced by a regular file or a directory, this is supported in e2fsprogs 1.42.9)

The output will look like this:

ext4 defragmentation for device(/dev/sda1)
[2/238]/boot/System.map-3.2.0-4-686-pae: 100% [ OK ]
[3/238]/boot/vmlinuz-3.16-0.bpo.2-686-pae: 100% [ OK ]
[5/238]/boot/grub/video_cirrus.mod: 100% [ OK ]
[6/238]/boot/grub/usb_keyboard.mod: 100% [ OK ]
[8/238]/boot/grub/locale/pl.mo: 100% [ OK ]
[9/238]/boot/grub/locale/nl.mo: 100% [ OK ]

.
.
.

[223/238]/boot/grub/password_pbkdf2.mod: 100% [ OK ]
[224/238]/boot/grub/videotest.mod: 100% [ OK ]
[225/238]/boot/grub/ntldr.mod: 100% [ OK ]
[226/238]/boot/grub/udf.mod: 100% [ OK ]
[227/238]/boot/grub/halt.mod: 100% [ OK ]
[228/238]/boot/grub/minix.mod: 100% [ OK ]
[229/238]/boot/grub/gfxterm.mod: 100% [ OK ]
[230/238]/boot/config-3.2.0-4-686-pae: 100% [ OK ]
[231/238]/boot/memtest86+_multiboot.bin: 100% [ OK ]
[232/238]/boot/memtest86+.bin: 100% [ OK ]
[233/238]/boot/System.map-3.16-0.bpo.2-686-pae: 100% [ OK ]
[234/238]/boot/initrd.img-3.2.0-4-686-pae: 100% [ OK ]
[235/238]/boot/initrd.img-3.16-0.bpo.2-686-pae: 100% [ OK ]
[237/238]/boot/config-3.16-0.bpo.2-686-pae: 100% [ OK ]
[238/238]/boot/vmlinuz-3.2.0-4-686-pae: 100% [ OK ]

Success: [ 234/238 ]
Failure: [ 4/238 ]

And here is the command to defrag xfs online:
$ sudo xfs_fsr -v /dev/sdXY, for example, $ sudo xfs_fsr -v /dev/sda1
(device /dev/sdXY by a file, but a directory is not supported yet in xfs_fsr version 3.1.9)

/ start inode=0
ino=1020
extents before:2 after:1 DONE ino=1020
ino=2000
extents before:3 after:1 DONE ino=2000
ino=2270
extents before:2 after:1 DONE ino=2270
ino=2272
extents before:2 after:1 DONE ino=2272
ino=2497
extents before:2 after:1 DONE ino=2497
ino=2498
extents before:2 after:1 DONE ino=2498
.
.
.

Not very hard, but useful, if the data on the target to defrag is important, I’ll respectfully suggest you to backup first, in case there is an accident like losing electrical power.

Fix inconsistent/broken pkgng on FreeBSD

Environment:
FreeBSD 10.1-RELEASE-p8 i386

Problem:

  • FreeBSD repository is up-to-date & all repositories are up-to-date, but when I try to install some package, it’ll try to fetch the non-latest version(so … 404 not found)
  • Keep getting message like this : pkg: cached package bind-tools-9.10.2: size mismatch, fetching from remote
  • pkd version said that there are some out-date packages, but pkg upgrade said every thing is up-to-update

pkg update -> not working
pkg clean + pkg update -> still not working
reboot -> of course not working …

So I did this to clean its cache:
$ sudo rm -rf /var/cache/pkg/*
and force update
$ sudo pkg update -f

Fortunately, it works fine now.

Error: Read-only file system on Linux

Today I found that there is a strange situation, in certain mount point on my web server (Debian GNU/Linux 7.8), every process with write behavior will get message like this:

touch: cannot touch `test’: Read-only file system

I don’t think I mount it as read-only, so I check it:
$ mount | grep srv
And I got:

/dev/sdb1 on /srv type ext4 (ro,relatime,data=ordered)

Very strange … there was something wrong, could I remount it with r/w? I tried:
$ sudo mount -o remount,rw,relatime,data=ordered /srv
Hmmmm …

mount: cannot remount block device /dev/sdb1 read-write, is write-protected

Okay, seems I need to check the filesystem …
$ sudo umount /srv
$ sudo fsck.ext4 /dev/sdb1

The process didn’t take long time, just within 3 mins, and that’s the result:

e2fsck 1.42.5 (29-Jul-2012)
/dev/sdb1: recovering journal
/dev/sdb1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb1: 796453/122077184 files (3.8% non-contiguous), 321192367/488281498 blocks

Let’s mount it again:
$ sudo mount -o rw,relatime,data=ordered /srv
Oh, I didn’t get any error message this time, let’s check it again:
$ mount | grep srv
And …

/dev/sdb1 on /srv type ext4 (rw,relatime,data=ordered)

Okay, nice … very easy this time, I’ve got some disks with destroyed partition before, event I can’t not recognize its partition type, though I fixed it, but I didn’t record the process that time, ha.

Virtualization related notes

Guest vs Host:

  • Host – usually runs on physical hardware, lower level.
  • Guest – runs on the virtual/virtualized environment, upper level.

Virtualization types:

  • Full virtualization –  virtualize all the devices!
    • Can run almost all the operating systems without any modifications.
    • Emulate all the devicesn.
    • Slower than Paravirtualization and Operating-system-level virtualization.
    • Software Emulation (Without Hardware-Assisted-Virtualization)
      • Very Slow.
      • Need to do some jobs like binary translation or software instruction decode, will have a heavy overhead, is very inefficiet.
    • Paravirtualization on HVM
      • Full virtualization with paravirtualization drivers.
  • Paravirtualization(PV) – use modified kernel to interact with the special interface
    • Guest knows it’s a guest on the host, the guest will communicate with hypervisor.
    • Use hypercall (call to hypervisor) as its system call.
    • Hard(almost impossible) to modify the kernel of closed-source operating systems like Windows to use this method.
    • Faster than full virtualization but slower than operating-system-level virtualization.
  • Hardware-Assisted-Virtualization (HVM, HAV)
    • Using help from hardware capabilities.
    • Faster than Software Emulation.
    • Technic examples
  • Operating-system-level virtualization – don’t really virtualize the devices
  • Partial virtualization – need be confirmed, not a usual type

Type-1 vs type-2 hypervisor:

Wikipedia also use Application/Environment and OS level to distinguish different types of virtualization:

  • Application-level
    • Sandbox
  • Environment-level
    • Containers
  • OS-level
    • Hypervisors

Common integrated virtualization solutions:

Resources and references:

The paravirtualization spectrum

source : https://blog.xenproject.org/2012/10/31/the-paravirtualization-spectrum-part-2-from-poles-to-a-spectrum/

Still learning, hope that there are not too many wrong things here … comments to point out mistakes/weak points are welcome!