loopkid: Category Linux

predefined character classes in grep

Stefan — Sun, 06 Jul 2008 15:54:00 +0200

Many regex implementations have “macros” for various character classes. In Perl, for example, \d matches any digit ([0-9]) and \w matches any “word character” ([a-zA-Z0-9_]). Grep uses a slightly different notation for the same thing: [:digit:] for digits and [:alnum:] for alphanumeric characters. (BSD)

Finally, certain named classes of characters are predefined within bracket expressions, as follows. Their names are self explanatory, and they are [:alnum:], [:alpha:], [:cntrl:], [:digit:], [:graph:], [:lower:], [:print:], [:punct:], [:space:], [:upper:], and [:xdigit:]. For example, [[:alnum:]] means [0-9A-Za-z]. (grep man page)

Was unter Ruby

line = "length 1450"
puts line if line =~ /\d{4}/

heißt wird also unter der bash mit grep zu

export line="length 1450"
echo $line | egrep [[:digit:]]{4}

was zwar die regulären Ausdrücke unnötig aufbläht, aber immerhin die gleiche Funktionalität zur Verfügung stellt.

beautiful network monitoring

Stefan — Fri, 27 Jun 2008 17:10:00 +0200

Monitoring networks with tcpdump works fine, but even in quiet mode tcpdump outputs too much information if you’re interested in application layer protocols like HTTP or IMAP. A nice alternative on the command line is ngrep which has a much more readable output. ngrep filters all tcp packets with an empty data part and strips the header of non-empty tcp packets. The only beauty flaw in my eyes is the dot ngrep inserts for every tab and for every carriage return. In my opinion it should offer an option to specify the tabulator size and just ignore the carriage returns.

But see for yourself, here the tcpdump output of an IMAP session,

$ sudo tcpdump -i en1 -A -s 0 -qtn port imap
tcpdump: verbose output suppressed,
use -v or -vv for full protocol decode
listening on en1, link-type EN10MB (Ethernet),
capture size 65535 bytes
IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 0
E..@..@.@.......P..N.|...7.n........Go.............
/!..........
IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 0
E..<..@.9...P..N.......|.....7.o...............
w.A./!......
IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 0
E..4..@.@.......P..N.|...7.o...............
/!..w.A.
IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 21
E..I..@.9...P..N.......|.....7.o...........
w.A./!..* OK Dovecot ready.

IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 0
E..4.P@.@..j....P..N.|...7.o...............
/!..w.A.
IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 33
E..U.8@.@..`....P..N.|...7.o...............
/!.2w.A.1 LOGIN foo@bar.net password

IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 0
E..4..@.9...P..N.......|.....7.......<.....
w.F'/!.2
IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 17
E..E..@.9...P..N.......|.....7......c......
w.F./!.21 OK Logged in.

IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 0
E..4.0@.@.......P..N.|...7.....?.....D.....
/!.>w.F.
IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 14
E..B.#@.@.......P..N.|...7.....?...........
/!.cw.F.1 LIST """%"

IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 0
E..4..@.9...P..N.......|...?.7.............
w.H./!.c
IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 513
E..5..@.9...P..N.......|...?.7.............
w.H./!.c* LIST (\HasChildren) "." "Trash"
* LIST (\HasNoChildren) "." "Sent"
* LIST (\HasNoChildren) "." "Spam"
* LIST (\HasNoChildren) "." "Sent Messages"
* LIST (\HasNoChildren) "." "Drafts"
* LIST (\HasNoChildren) "." "Spamtraining"
* LIST (\HasNoChildren) "." "Hamtraining"
* LIST (\HasNoChildren) "." "Spamtesting"
* LIST (\HasNoChildren) "." "Hamtesting"
* LIST (\HasNoChildren) "." "Deleted Messages"
* LIST (\HasNoChildren) "." "Spamverdacht"
* LIST (\HasNoChildren) "." "INBOX"
1 OK List completed.

IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 0
E..4]'@.@.8.....P..N.|...7.....@...........
/!.cw.H.
IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 10
E..>.L@.@..c....P..N.|...7.....@.....N.....
/!..w.H.1 LOGOUT

IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 19
E..G..@.9...P..N.......|...@.7.............
w.IP/!..* BYE Logging out

IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 0
E..4.I@.@..p....P..N.|...7.....S.....%.....
/!..w.IP
IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 24
E..L..@.9...P..N.......|...S.7.............
w.IP/!..1 OK Logout completed.

IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 0
E..4C#@.@.R.....P..N.|...7.....l...........
/!..w.IP
IP 192.168.2.22.50556 > 80.237.145.78.143: tcp 0
E..4.;@.@.......P..N.|...7.....l...........
/!..w.IP
IP 80.237.145.78.143 > 192.168.2.22.50556: tcp 0
E..4..@.9...P..N.......|...l.7.......j.....
w.IQ/!..

20 packets captured
28 packets received by filter
0 packets dropped by kernel

compared with the corresponding ngrep output.

$ sudo ngrep -d en1 -W byline port imap
interface: en1 (192.168.2.0/255.255.255.0)
filter: (ip) and ( port imap )
####
T 80.237.145.78:143 -> 192.168.2.22:50556 [AP]
* OK Dovecot ready..

##
T 192.168.2.22:50556 -> 80.237.145.78:143 [AP]
1 LOGIN foo@bar.net password.

##
T 80.237.145.78:143 -> 192.168.2.22:50556 [AP]
1 OK Logged in..

##
T 192.168.2.22:50556 -> 80.237.145.78:143 [AP]
1 LIST """%".

##
T 80.237.145.78:143 -> 192.168.2.22:50556 [AP]
* LIST (\HasChildren) "." "Trash".
* LIST (\HasNoChildren) "." "Sent".
* LIST (\HasNoChildren) "." "Spam".
* LIST (\HasNoChildren) "." "Sent Messages".
* LIST (\HasNoChildren) "." "Drafts".
* LIST (\HasNoChildren) "." "Spamtraining".
* LIST (\HasNoChildren) "." "Hamtraining".
* LIST (\HasNoChildren) "." "Spamtesting".
* LIST (\HasNoChildren) "." "Hamtesting".
* LIST (\HasNoChildren) "." "Deleted Messages".
* LIST (\HasNoChildren) "." "Spamverdacht".
* LIST (\HasNoChildren) "." "INBOX".
1 OK List completed..

##
T 192.168.2.22:50556 -> 80.237.145.78:143 [AP]
1 LOGOUT.

#
T 80.237.145.78:143 -> 192.168.2.22:50556 [AP]
* BYE Logging out.

##
T 80.237.145.78:143 -> 192.168.2.22:50556 [AFP]
1 OK Logout completed..

28 received, 0 dropped

tcpdump: packets dropped by kernel

Stefan — Wed, 25 Jun 2008 09:49:00 +0200

tcpdump is a really nice tool, but it may render useless with it’s default settings.

$ sudo tcpdump -i en1

In my case led to

80 packets captured
7705 packets received by filter
6794 packets dropped by kernel

To solve the issue I had to turn off address translation.

$ sudo tcpdump -i en1 -n

Now the results look as expected.

6941 packets captured
7011 packets received by filter
0 packets dropped by kernel

memory allocation error in aptitude

Stefan — Tue, 17 Jun 2008 23:33:00 +0200

If you encounter the following error while searching with aptitude or apt-cache

terminate called after throwing an instance of 'std::bad_alloc'
what(): St9bad_alloc
Aborted

you might be able to resolve the situation easily with a

sudo aptitude clean

which at least helped in my case.

documentation style

Stefan — Fri, 08 Feb 2008 17:46:43 +0100

Meine Dokumentationen sehen oft nicht anders aus als Auszüge aus der .bash_history zusammen mit Listings oder Diffs von Konfigurations-Dateien und einigen Erklärungen dazu. Zum Beispiel warum eine bestimmte Version einer Software eingesetzt werden musste oder dergleichen.
(Arne Brodowski)

Es ist immer schön, wenn man das Gefühl hat nicht allein da draußen zu sein.

wiki is a foreign word for ruby

Stefan — Sat, 17 Nov 2007 16:59:32 +0100

Wieso gibt es eigentlich in Ruby im Gegensatz zu PHP oder Phython eigentlich keine ausgereifte Wikiengine ? Als die einzigen halbwegs ernstzunehmenden Kandidaten erscheinen mir Instiki und Pandora, wobei ersteres immer noch keine Löschfunktion für einzelne Wiki-Seiten oder Wiki-Bereiche besitzt und bei letzterem die Entwicklung eingefroren scheint.

finding large directories

Stefan — Tue, 30 Oct 2007 23:49:25 +0100

Every once in a while there is the need to clear out the system. The question that inevitably pops up is: Where has all the disk space gone ? The solution of course lies in locating the largest directories. With WhatSize and OmniDiskSweeper there are two great tools on Mac OS X. There are also feasible Tools on Linux like Filelight, but if you’re administering a remote server and you don’t want to use remote x11 access you need a solution that can be controlled either on the command line or via browser. Of course you could use sturdy old du.


# Find all directories with the largest directories on top
$ sudo du -Sx / | sort -nr | less
# Find directories over 100MB with the largest directories on top
$ sudo du -Sh / | grep ^[1-9][0-9][0-9][0-9\.]*M | sort -nr

Or you could hack together some fancy perl script, but wouldn’t it be nice to have an interactive graphical visualisation through which you could comfortably navigate ? This is where Philesight pops in, which offers both command line and browserbased control. Putting it up on Debian is a breeze. Just install cairo and berkley db and their ruby bindings and then get rolling.


$ wget http://zevv.nl/code/philesight/philesight-2035.tgz
$ tar xvfz philesight-2035.tgz
$ sudo apt-get install libcairo2 libcairo-ruby
$ sudo apt-get install libdb4.2 libdb4.2-ruby1.8
# Generate index
$ sudo ./philesight --db base.db --index /
# Check if image is correctly generated
$ ./philesight --db base.db --path / --draw base.png

Firing up the web interface is just three small steps further. First configure the CGI-File.


# philelight.cgi
db = "./base.db"
default_path = "/"

Then create a reasonably named subdomain and after that just add a few lines to your lighttpd configuration file (or the corresponding lines to the webserver of your choice).


$HTTP["host"] =~ "philesight.yourdomain.net" {
  cgi.assign = ( ".cgi" => "/usr/bin/ruby" )
  alias.url = ( "/" => "/path/to/philesight/philesight.cgi")
  server.document-root = "/path/to/philesight"
}

That’s all there’s to it. Have fun.