How to install ruby gem packages without Internet?

git_stats is a good tool to analyze the statistics of a git repo, there is an issue tomgi/git_stats#45 How to Install Offline in that project, I think this is a good question, sometimes the speed or quality of Internet is not so fast and stable, or we want to install a gem package on a server that can only access internal network only, so I tried to figure it out, that’s the solution:

Milly/gem-fetch-dependencies is a ruby script posted on gist, it can help us fetch gem packages with the dependency, we can’t just fetch the package we want to use via gem fetch , because it doesn’t handle the dependency issue, that’s why we need this script, download the script(raw) first:

https://gist.github.com/Milly/909564/raw/c3c921e78b6736197558ee44efb84a495f4c1505/gem-fetch-dependencies
[ruby]#!/usr/bin/ruby

require ‘rubygems’
require ‘rubygems/commands/fetch_command’

class Gem::Commands::FetchCommand
def add_version_option_with_fetch_depends
add_version_option_without_fetch_depends
add_option(‘-y’, ‘–[no-]dependencies’,
"Fetch dependent gems") do |value, options|
options[:dependencies] = value
end
end

def get_all_gem_names_with_fetch_depends
@dependent_gem_names || get_all_gem_names_without_fetch_depends
end

def execute_with_fetch_depends
execute_without_fetch_depends
if options[:dependencies] then
@dependent_gem_names = get_dependent_gem_names
options[:version] = nil
execute_without_fetch_depends
end
end

[:add_version_option, :get_all_gem_names, :execute].each do |target|
feature = "fetch_depends"
alias_method "#{target}_without_#{feature}", target
alias_method target, "#{target}_with_#{feature}"
end

private
def get_dependent_gem_names
version = options[:version] || Gem::Requirement.default
request_gem_names = get_all_gem_names_without_fetch_depends.uniq
to_do = fetch_specs_by_names_and_version(request_gem_names, version)
seen = {}

until to_do.empty? do
spec = to_do.shift
next if spec.nil? or seen[spec.name]
seen[spec.name] = true
deps = spec.runtime_dependencies
deps.each do |dep|
requirements = dep.requirement.requirements.map { |req,| req }
all = !dep.prerelease? and
(requirements.length > 1 or
(requirements.first != ">=" and requirements.first != ">"))
result = fetch_spec(dep, all)
to_do.push(result)
end
end

gem_names = seen.map { |name,| name }
gem_names.reject { |name| request_gem_names.include? name }
end

def fetch_specs_by_names_and_version(gem_names, version)
all = Gem::Requirement.default != version
specs = gem_names.map do |gem_name|
dep = Gem::Dependency.new(gem_name, version)
dep.prerelease = options[:prerelease]
fetch_spec(dep, all)
end
specs.compact
end

def fetch_spec(dep, all)
specs_and_sources, errors =
Gem::SpecFetcher.fetcher.fetch_with_errors(dep, all, true,
dep.prerelease?)
if platform = Gem.platforms.last then
filtered = specs_and_sources.select { |s,| s.platform == platform }
specs_and_sources = filtered unless filtered.empty?
end
spec, source_uri = specs_and_sources.sort_by { |s,| s.version }.last
spec
end
end

load `which gem`.rstrip
[/ruby]

and we can easily fetch the gem packages with dependencies with this script like this:
$ ruby gem-fetch-dependencies fetch --dependencies

Once we fetched all the packages, you copy them to everywhere you want, and install them locally:
$ gem install -f --local *.gem

Online disk defragment on Linux

Though we rarely defragment the filesystem on unix-like os, sometimes we still need it.

The demonstrate is on LinuxMint 17.1 Cinnamon 64-bit which is based on Ubuntu 14.04 LTS.

Okay, let’s check the fragmentation first!

Here is a very easy method to check the fragment on ext4 online:
$ sudo fsck -fn /dev/sdXY, replace XY for yourself, for example, $ sudo fsck -fn /dev/sda1

The output will look like this:

fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
Warning! /dev/sda1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda1: 247/121920 files (0.4% non-contiguous), 66062/487424 blocks

(BTW, this method could also work on ext2, ext3)

and on xfs:
$ xfs_db -c frag -r /dev/sdXY, for example,$ xfs_db -c frag -r /dev/sda1

The output will look like this:

actual 297843, ideal 286076, fragmentation factor 3.95%

And defrag a partition now!

This is the command to defrag ext4 online:
$ sudo e4defrag -v /dev/sdXY, for example, $ sudo e4defrag -v /dev/sda1
(BTW, device /dev/sdXY can be replaced by a regular file or a directory, this is supported in e2fsprogs 1.42.9)

The output will look like this:

ext4 defragmentation for device(/dev/sda1)
[2/238]/boot/System.map-3.2.0-4-686-pae: 100% [ OK ]
[3/238]/boot/vmlinuz-3.16-0.bpo.2-686-pae: 100% [ OK ]
[5/238]/boot/grub/video_cirrus.mod: 100% [ OK ]
[6/238]/boot/grub/usb_keyboard.mod: 100% [ OK ]
[8/238]/boot/grub/locale/pl.mo: 100% [ OK ]
[9/238]/boot/grub/locale/nl.mo: 100% [ OK ]

.
.
.

[223/238]/boot/grub/password_pbkdf2.mod: 100% [ OK ]
[224/238]/boot/grub/videotest.mod: 100% [ OK ]
[225/238]/boot/grub/ntldr.mod: 100% [ OK ]
[226/238]/boot/grub/udf.mod: 100% [ OK ]
[227/238]/boot/grub/halt.mod: 100% [ OK ]
[228/238]/boot/grub/minix.mod: 100% [ OK ]
[229/238]/boot/grub/gfxterm.mod: 100% [ OK ]
[230/238]/boot/config-3.2.0-4-686-pae: 100% [ OK ]
[231/238]/boot/memtest86+_multiboot.bin: 100% [ OK ]
[232/238]/boot/memtest86+.bin: 100% [ OK ]
[233/238]/boot/System.map-3.16-0.bpo.2-686-pae: 100% [ OK ]
[234/238]/boot/initrd.img-3.2.0-4-686-pae: 100% [ OK ]
[235/238]/boot/initrd.img-3.16-0.bpo.2-686-pae: 100% [ OK ]
[237/238]/boot/config-3.16-0.bpo.2-686-pae: 100% [ OK ]
[238/238]/boot/vmlinuz-3.2.0-4-686-pae: 100% [ OK ]

Success: [ 234/238 ]
Failure: [ 4/238 ]

And here is the command to defrag xfs online:
$ sudo xfs_fsr -v /dev/sdXY, for example, $ sudo xfs_fsr -v /dev/sda1
(device /dev/sdXY by a file, but a directory is not supported yet in xfs_fsr version 3.1.9)

/ start inode=0
ino=1020
extents before:2 after:1 DONE ino=1020
ino=2000
extents before:3 after:1 DONE ino=2000
ino=2270
extents before:2 after:1 DONE ino=2270
ino=2272
extents before:2 after:1 DONE ino=2272
ino=2497
extents before:2 after:1 DONE ino=2497
ino=2498
extents before:2 after:1 DONE ino=2498
.
.
.

Not very hard, but useful, if the data on the target to defrag is important, I’ll respectfully suggest you to backup first, in case there is an accident like losing electrical power.

Fix inconsistent/broken pkgng on FreeBSD

Environment:
FreeBSD 10.1-RELEASE-p8 i386

Problem:

  • FreeBSD repository is up-to-date & all repositories are up-to-date, but when I try to install some package, it’ll try to fetch the non-latest version(so … 404 not found)
  • Keep getting message like this : pkg: cached package bind-tools-9.10.2: size mismatch, fetching from remote
  • pkd version said that there are some out-date packages, but pkg upgrade said every thing is up-to-update

pkg update -> not working
pkg clean + pkg update -> still not working
reboot -> of course not working …

So I did this to clean its cache:
$ sudo rm -rf /var/cache/pkg/*
and force update
$ sudo pkg update -f

Fortunately, it works fine now.

Error: Read-only file system on Linux

Today I found that there is a strange situation, in certain mount point on my web server (Debian GNU/Linux 7.8), every process with write behavior will get messages like this (That’s the result from command touch test):

touch: cannot touch `test’: Read-only file system

I don’t think I mount it as read-only, so I check it with(the mount point is /srv):

$ mount | grep srv

And I got:

/dev/sdb1 on /srv type ext4 (ro,relatime,data=ordered)

Very strange … there was something wrong, could I remount it with r/w? I tried:

$ sudo mount -o remount,rw,relatime,data=ordered /srv

Hmmmm …

mount: cannot remount block device /dev/sdb1 read-write, is write-protected

Okay, seems I need to check the filesystem …

$ sudo umount /srv
$ sudo fsck.ext4 /dev/sdb1

The process didn’t take a very long time, just within few mins, and that’s the result:

e2fsck 1.42.5 (29-Jul-2012)
/dev/sdb1: recovering journal
/dev/sdb1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb1: 796453/122077184 files (3.8% non-contiguous), 321192367/488281498 blocks

Let’s mount it again:

$ sudo mount -o rw,relatime,data=ordered /srv 

Oh, I didn’t get any error message this time, let’s check it again:

$ mount | grep srv

And … it’s good again

/dev/sdb1 on /srv type ext4 (rw,relatime,data=ordered)

Okay, nice … it’s very easy this time, I’ve got some disks with damaged partition before, event I can’t not recognize its partition type, though I fixed it, but I didn’t record the process that time, haha.