Know your shell and GNU coreutils

A friend of mine was frustrated by the fact that in his bash, the `pwd` command gave him the following `unwanted` results:

When he is in the directory `/usr/a`, pwd gives ‘/usr/a/’. In this directory, if he has a symbolic link ‘b’ pointing to /sys/b, then after using `cd b`, pwd gives him `/usr/a/b/` instead of `/sys/b`.

What he wants is the physical/absolute path name as the result, but he can’t get that in bash. (Later we know that we have to use `pwd -P`, but this trick is not stated in `man pwd`, this is in `man bash`) He tried to consult man page, but man pwd redirects him to the man page of pwd in the coreutils, while using it in shell doesn’t give him the expected result indicated by the pwd command in coreutils . Strange, isn’t it?

The problem here is that for commands like ‘pwd’ (and ‘cd’, ‘echo’, etc.), your shell will not actually start a process to execute these commands, instead, your shell has build in functions to do the job. If you consult the man page of pwd, there is one line in the man page stating:

NOTE: your shell may have its own version of pwd, which usually supersedes the version described here. Please refer to your shell’s documentation for details about the options it supports.

Aha! Blame the shell !!!

Why your shell wants to replace the pwd from GNU? Well, the advantage this kind of replacement is that now you have efficiency, because your shell can execute the build in code without forking a new process. The downside is that, now you put your bet on the shell. If the shell behaves differently than you’ve thought, you are screwed.

This is exactly what we want a platform independent tool set. For every Linux distribution, GNU coreutils is always included. If you force your shell to use the pwd from coreutils by typing /bin/pwd, you are insured that even on different platforms with different shells, all results will be the same.

Some shell-dependent features:
alias, echo, cd, printf, background/foreground process (& syntax), pipe and IO redirecting, env variable setting/accessing.

Suffice it to say that you should know your shell’s behavior under some circumstances, especially when you want a consistent sh shell scripts behavior.

Happy hacking!

–EOF–

Using Arduino IDE without an Arduino

Arduino is essentially an AVR microcontroller interfaced with an USB/Serial chip to compter. It’s not hard to build your own arduino, and there are a lot of resources on the internet about this. See this.

However, all these tricks requires you to buy an USB interface such that you can program your arduino. That small chip or cable usually costs you more than 20 dollars. For a cheap person like me, I don’t want to do that. However, the arduino IDE is really handy. I love to program in it, even though I use VIM a lot, I still want to use this IDE to program my AVR. So I really want to use this IDE such that I can upload the file to the arduino with one click. However, I don’t want to invest any money on the buying an arduino, since I know it’s just a uC with interface, and I usually use my uC in a naked way.

I know that the IDE uses avr-gcc and avrdude.Avr-gcc compiles any thing into a standard hex file to the avr uC. Thus, the IDE didn’t do anything special here. The avrdude program supports way more programming methods to an AVR chip than the Arduino way (usb/serial). Thus, it is possible to make a cheap ass ISP programmer to the uC, and then modify the IDE such that it calls avrdude with the function you want.

To make things easier, I give a very simpler version to make a cheap arduino (I will call it naked avr chip with battery included):

An AVR uC, for instance, ATMEGA8L ($3 here? My uncle in China mailed me many. Pick the uC with suffix L, meaning a 3V voltage is high enough for it to work, you don’t need 5V. For using 5V, you might have to buy a 7805 regulator, which I don’t want to.)

Some LED to test it. (Radio Shack, around $3 for 20, any LED will be good. They are cheap and robust, you don’t need transistors to protect them, or your uC)

A 2-AA battery box (Radio Shack, less than $2)

A breadboard and some jump wire  (ebay, 20$ for both, free shipping, find the best deal)

You don’t need a 20Mhz clock and capacitors around it, you can just use the clock inside the AVR chip.

—————–

If you want to DIY an DB-25 ISP programmer cable:

A DB-25 connector (male, you plug it to your computer’s parallel port)

Some wire

To see these simple steps to make a parallel port programmer, see this link:

You can also buy one online, for like $8.

Connect your ISP to your AVR correctly (you can find the instructions easily online), you don’t need any resistor or any capacitor whatever on the breadboard, seriously. Again, your AVR is robust enough to handle things under 3V voltage AA battery power. 3V voltage won’t destroy your LED either, so don’t even bother connecting your LED to a transistor (I know it’s suggested, but I just want to show you that you can be lazy here. Every transistor/capacitor on the breadboard will scare away half hobbists ).

OK, here is the real hack.

1. To burn the bootloader down, choose your board, and your cable (w/ parallel programmer, depends on your ISP cable), you can see that there is a menu item for it. You can see if it works. Mine doesn’t, because my cable is called stk200, and arduino assumes another interface. What I did was go to your hardware/tools folder, you will find avrdude there. I changed avrdude to avrdude1 and then write a bash script called avrdude and calls avadude. In this way, I use echo $* to see the command line passed to avrdude. I then changed the fifth option to -t stk200, which is my ISP programmer protocol.

Actually it’s not that important to burn bootloader, especially everytime you actually use ISP programmer instead of using the bootloader. I write it here because I hacked it and in case you want to burn the bootloader for your AVR in Arduino environment.

2. To upload. I tried to click the “Upload to uC with I/O” button in the IDE. It complains that there is no Serial port. Of course there is not, because I don’t use it . I also found the exception stack saying something about AvrdudeUploader. Haha, that’s it. So, I checked the souce code out from arduino website using:

svn checkout http://arduino.googlecode.com/svn/trunk/ arduino-read-only

Then, you edit app/AvrdudeUploader.java, you will find a function called “uploadViaBootloader“, that’s exactly the function that IDE calls to upload the program. Here, comment out the String protocol = line, change it to something like

String protocol = “your_protocol”;

Assign it to the protocol you use ( for instance, I use stk200, or the parallel cable protocol). Remove the line with comment “don’t erase” (yes, erase it , remove the whole “if (Preference.get…)” chuck, and remove the link that contains “upload.speed“, then go to the build/linux folder, ececute the make and dist script. It will create a folder called arduino-0016 something, and you can go inside to the lib/ director, copy all the “.jar” files to the install director of your arduion install directory.

Now, do everything you want on your naked avr, using arduino IDE

(I quick tip, Arduino uses pin 13 as the default LED pin. It it used as SCK pin for the programmer, so, when you test “Blink”, you’d better use another pin, otherwise, unplug your programmer from the uC. The programmer draw sink the current such that you won’t see the blinking light.

I will post some photos later about my hack.

Happy hacking.

How to fix your firefox (and other software) library not found error in Ubuntu

I recently crashed my Firefox for no reason. I guess I accidentally removed the ia32-libs. However, when you install Firefox using apt-get, ia32-libs is not listed as the dependency. Therefore, no matter how hard you try, if you don’t have lib-ia32, you can’t use firefox, even though

 apt-get install firefox

works fine.

Anyway, here are some generic methods I’ve used to fix the “library not found” problem in using firefox and other pieces of software. In short, you have to use two family of tools.

First step: Find the missing library file using ldd, objdump and nm

The first family of tools is related you the dynamic linked library, it includes ldd, objdump and nm.

Objdump and nm are from gnu binutls and ldd is from gnu libc. Usually, when your software complains about not finding a symbol or a missing library, you can generally use ldd to see the libraries required by your software. For instance, if your software executable file is called aaa and it is dynamically linked against some missing libraries, you can just use

ldd aaa

to see what are the libraries used by your software. You can use grep to filter the information you want.

If there is a “.so hell” problem (you can find the library find you can’t find the symbol), you have to use objdump to see where is the missing symbol. Usually, you can google the missing symbol to decide the library that contains it. Anyhow, you have to first locate the library. “nm” servers a similar function. Read the man file and you will know how to use those three tools.

Second step: locate the library file

The second family of tools helps you to locate and get the library. They are apt-file. apt-get, gcc and Google.

There are several possibilities for a library missing. The first is that you accidentally removed that. For this, the only problem is to find the package name that contains the missing library file. To do this, on Ubuntu, we can use a tool called apt-file. It searches all the packages in the apt repository and output all packages that contains the file name you entered. For install, if you want to find libxyz.so, you can just use

apt-file search libxyz.so

and it will output the package names that contains libxyz.so. There might be multiple packages, and usually it’s easy to try and decide which one is missing as if you’ve already had it on your machine, apt-get will tell you.

Usually, you can fix your problem by installing the missing packages. But sometimes you have all latest package installed, and your software still complains about the undefined symbol. This is possible because your software may rely on an older version of the library and in the new version, some symbols get removed. For open source software, try to decide the version of the library file first, and then download the source code and compile it. Recently I crashed my gnome window manager, and it requires an older version of libxcb. I have to download the older version to let it work.If it is not the open source component, try to get a copy of the library file. Usually I will use google to find that file. Google is very good at finding files you need.

That’s it.

Generate Ambient White Noise on Linux

Here is the link for the software:

http://pessimization.com/software/whitenoise/

It supports frequency cutoff. I’ve been using it for a whole day, it is wonderful.

Access WUSTL vpn on Ubuntu 8.10

I have to say, WUSTL has a very good network, but it’s technical support sucks a lot.

1. To use the VPN of WUSTL Engineering school, you can go to this page and download the vpn client.

2. You can download the CISCO client for Linux, which is called vpnclient-linux-x86_64-4.8.02.0030-k9.tar.gz.

3. Then you use

tar -zxvf vpnclient-linux-x86_64-4.8.02.0030-k9.tar.gz

to extract the contents out.

4. Naturally, in the extracted folder, you want to use

./vpn_install

to install the VPN client. However, it does not work.

5. You have to go to this page to download a patch file (the second one).

6. Copy the patch file to the directory that has vpn_insatll.  Issue

patch -p1 < 4.8.02-64bit.patch

to patch the file.

7. Use

./vpn_insatll

to install the VPN client. Using the default parameters is recommended.

8.  Use

chmod -R o-w /etc/CiscoSystemsVPNClient/
/etc/init.d/vpnclient_init start

to start the kernel service for VPN.

9. Since wustl EIT didn’t bother giving you a profile, you have to create it of your own. Go to
/etc/CiscoSystemsVPNClient/Profiles, make a duplicated copy of sample.pcf as wustl.pcf via

cp {sample,wustl}.pcf

10. Edit the wustl.pcf and make the following changes (if you are using CSE VPN. For further information, visit http://eit.engineering.wustl.edu/help/VPN_setup.asp ):


Host=csevpn.seas.wustl.edu
GroupName=cse
Username=your_user_name

11. Use

vpnclient connect wustl

to connect the VPN.

-EOF-

Some LaTeX tricks you might want to know

1. TeX does not support multi-line comments. You can do a trick like this:

\newcommand{\comments}[1]{}

Then you can use \comments{ something } to make multi-line comments.

% I learned this from my advisor.

2. Every time you use \cite{}, make sure you use tilde (~\cite). This is from TeXbook. A tilde will prevent the line break between the text and the citation brackets. 

3. If you want to extract text from LaTeX, you can use 

detex your_file

Use 

sudo apt-get install texlive-extra-utils

to install tetex. 

4. For spelling check for your LaTeX file, you can use 

aspell -c your_file.tex

Hacklog: Convert PDF to TXT for free on Linux

If you Goolge pdf2txt, you will find a website called http://www.pdf2txt.com/ where each copy of their software costs you $35.

I don’t know how can they make that shit up and still have business. Here is what you can do for free:

ps2ascii your_pdf_file.pdf out.txt

DONE

OK, what’s the cost???

To install a Linux, you might need a CD-R(everybody has it) and a CD-R disk ($0.7).

1. You go to Ubuntu.com and download it.

2. You  install it on your windows, yes, on your windows, for free.

3. After installing it, reboot your computer. Go to Linux.

4. Copy your pdf file to Linux use a flash disk. Just insert it to your computer with Linux, and you will find the disk icon on your disktop. Drag the file to your home folder on the Desktop(drag it to the “Home shaped icon” on your desktop).

5. Click Application on the Menu, choose Accessories -> Terminal

6. In Terminal, type

ps2ascii your_pdf out.txt

DONE.

Total cost: 0.7 dollars. Additionally, you get the most advanced operation system on your computer. Try to poke around, you will love it.

Hacklog: Fix some problems in Intel Lab Data

The Intel Lab Data is perhaps the most well-known data set in the sensor network community. For an introduction of it and downloading, you can click here. 

As any kind of real world data, they are noisy. For instance, in the introduction, it says that

1. It contains 54 nodes reading
2. The epoch value is unique. 

Neither of these two assertions are true. Actually it has 58 nodes (1-58), and the epoch value is not unique. To verify this, let’s assume the data is stored as data.txt

cat data.txt | tr " " "\t" | cut -f 2 | sort -n |uniq 

will give you a list of 1-58 instead of 1-54. 

Also, you can use

grep "59745\ 25" data.txt 

Here is what you get. Note that the 4th field is the epoch value and the 5th is the node index. Thus the pair is not unique (even the data introduction said so…)

2004-03-19 18:50:50.067531 59745 25 22.1346 43.2191 104.88 2.42416

2004-04-01 10:47:48.86101 59745 25 122.153 -4 1847.36 2.12114

There are also some funny nosiy. For example, some lines don’t even have 8 fields. There are missing data also (there might be no record for any epoch value and any node)

Thus, if you want to get a clean version of the data for further use, you have to do extensive work to bring sanity to the data. 

Here is what I did:
awk 'NF == 8 {print}' data.txt > temp.txt
awk '$4<55 {print}' temp.txt > 8field54nodes.txt

Then, to make the epoch-id unique (so that you can align the measurement data across different sensors), you have to choose a very short time window. I choose “2004-03-08″. Thus, you use

grep "2004-03-08" ../8field54nodes.txt | awk '{print $3, $4, $5}' > oneday.txt

There are possible processing I am going to omit here. For instance, some nodes has significantly less samples than others, we might want to pick a subset of nodes. Also, we need to align the data according to epoch and fill the missing items in the epoch-node-measure table. I have a whole bunch scripts for that, but I will omit them here. 

-EOF-

Draw Finite State Machine with Graphviz

Graphviz is a very useful tool for graph visualization . The language in graphviz is called dot. It is from Bell Lab. You can simple describe the graph in a human-friendly language in a txt file. Dot will translate the description into eps, png, or svg etc.

Here I share some tricks in drawing a finite automata for my homework. 

The finial graph looks like this:

There are three tricks:

1. Draw the graph horizontally instead of vertically. Use 

rankdir=LR;

2. Draw an edge without a source node (to denote the initial state “1″). You create a dummy node called null with text as it’s shape. But you don’t actually put any text in it. 

null [shape = plaintext label=""]
null -> 1

3. Draw a double circle

2 [shape=doublecircle];

The sample dot file for the above graph

digraph G {
rankdir=LR;
null [shape = plaintext label=""]
null -> 1
node [shape=ellipse];
2 [shape=doublecircle];
1 -> 1 [label = "a"];
1 -> 2 [label = "b"];
}

Hacklog: use trial version proprietary software forever

Warning: this is a made-up story, for instruction and educational purposes only.

On my Linux, I have a piece of proprietary software. I have to use it for my daily research work.

Every time, this software checks the license file to see if the license has expired and if I am using it on the exact same machine because it is a single machine license. When the license expired last year, I was forced to renew the license at a very expensive price. This year, I am not willing to do it anymore. Thus, I did a little bit of a hack (yes, it is illegal) to get around the license protection. Here is the story.

At the end of last year, the license expired. But I had to use that software that day, and I cannot change the time of the server (I am not the root, and a lot of services on that server rely on the correct time setup). I was forced to hack the code to make it work.

First, I tried to figure out how the program worked by using `strace’ and `ltrace’. Here is a sample from my ltrace (let’s call the software that I am going to hack “evil”)

>ltrace evil

In the output, I found a suspicious line:

time(NULL) = 1234037514

I knew that this software would check the expiration information. Thus, I thought this software must use time(NULL) library call to get the current time and make a comparison with the expiration date specified in the encrypted license file. I happened to have a piece of code which can give programs fake time. Thus, I used my previous code to wrap this software as a child process. This worked very well for a while.

At the very beginning of this semester, when I used this software again, it stopped working again. I spent about 5 hours debugging my “faketime” code to see why and did not get any clue. Then, I remembered that this software would check if it was the correct machine! I used ltrace again, and found the following two suspicious lines:

popen("/sbin/ifconfig", "r")
strncmp("a0:b3:..", "A2:B4...") = 1

Aha! I immediately realized that the MAC address had been changed for the server for some unknown reasons. I asked the sysadmin to help me to change the MAC address to what I wanted, and he did it. However, it did not help. Turned out that they upgraded the Linux on that server and the latest version of Linux would print the letters in the MAC address in lower case (like a0:b2:c2…), and that software requires upper case (I guess an early version of ifconfig had upper case letters in MAC address, so it worked fine at that time).

Once I knew this, what I did was very simple. I used

sudo apt-file search ifconfig

on my own Debian box to get the package name for the ifconfig package. It is called net-tools. I used

sudo apt-get source net-tools

to get the source code. After poking around the source code for a while, I found a file, interface.c, which was responsible for the MAC address output. The actual printing code reads,

if (hw->print != NULL && (! (hw_null_address(hw, ptr->hwaddr) &&
hw->suppress_null_addr)))
printf(_("HWaddr %s  "), hw->print(ptr->hwaddr));

Now everything became crystal clear. I used the code lines below to replace the above code:

if (hw->print != NULL && (! (hw_null_address(hw, ptr->hwaddr) &&
hw->suppress_null_addr)))
// Eric Xu
{
char* temp = hw->print(ptr->hwaddr);
int i;
for (i=0; temp[i]; i++)
temp[i] = toupper(temp[i]);
printf("%d\n", i);
printf(_("HWaddr %s  "), temp );

}

I recompiled the code, and persuaded the sysadmin to use my version of the ipconfig. Bingo, the software worked again.

If you think about this experience on a high level, you will realize that the OS really controls the software. We can always let the software run in a trap box by feeding it with fake time or other fake information. Thus, any software-only safety solution cannot really be safe, because it can be cheated on the OS level. Moreover, on M$ Windows, you cannot really intercept the OS API call. On Linux, you can do everything freely because you have the source code.

You should embrace open source, in an evil way