• use a date-based directory structure for the experiments you perform
• keep a lab-notebook containing documentation and code for each experiment (How about Org-mode?)
• create generate scripts that work on original data and a general runner script for the complete experiment

His work flow is applies mostly to *nix enviroments. One thing I’d like to add: Use symbolic links! Due to huge amounts of data and backup strategies, it might be impractical to have all the project’s data in a single directory. Create subdirectories of your data and result directories according to the naming convention and link them to other network drives using the ‘ln -s’ command.

import sys, re
import csv
class CommentedFile:
    def __init__(self, f, commentstring="#"):
        self.f = f
        self.commentstring = commentstring
    def next(self):
        line = self.f.next()
        while line.startswith(self.commentstring):
            line = self.f.next()
        return line
    def __iter__(self):
        return self

tsv_file = csv.reader(CommentedFile(open("inputfile.txt", "rb")),
                      delimiter='\t')
for row in tsv_file:
    print row[2] # prints column 3 of each line

sed -n -e '3p' -e '5p' -e '11,15p' textfile.txt

Here is the good news for everybody who wants to use an ANYCOM Bluetooth USB adapter (200, 250, 500) on Windows 7. The Vista driver runs just fine under Windows 7. And I got the driver ( “anycom-bluetooth-usb200-250-500-vista-v6-1-0-4700.exe”). If anybody needs it, feel free to send me an email (see About). It may be worth noting that Windows 7 complains about not being able to correctly install Bluetooth devices like a headset (Plantronics Voyager 510 for me), while in fact you you only need the correct driver for the adapter.

Update: After brisk demand I decided to allow you to download the driver directly from this website. Of course, no warranty whatsoever provided.

#!/bin/bash
paramset="foo.txt 1 --with-graphics
bar.txt 8 --no-graphics
flock.txt 4 --with-graphics"

echo "$du" | while read file p1 p2 ; do
./myProgram $file -t $p1 --verbose $p2
done

We here run the program myProgram three times (for each line in the multi-line string). Every line contains three white-space separated values (words), to which we assign the variable names file, p1, p2 in the loop header. Note that the last variable (in this case p2) always contains all remaining words of a given line if there are more words then variables.

The set of parameters can also be stored in a file. In that case, replace the loop header with cat params.txt | while read file p1 p2 ; do. If the script is not working properly, examine the Input-Field-Separator (IFS) variable, which should be set to IFS=" ".

The key tool here is the wonderful org-mode extension Org-R by Dan Davison. It allows to include code fragments that create figures or tables directly into your document. That way, everything will be in one place.  Searching, au revoir.

Introduction to Org-R

Luckily, Dan wrote a comprehensive tutorial on how to use org-R. If your planning to give it a try, I suggest to look over his words first. However, he forgot to mention two things you’ll need to enable the org-R goodness. Insert

(add-to-list ‘load-path “~/site-lisp/org-mode/contrib/lisp”)
(require ‘org-R)

into your .emacs and edit the directory to point into your org-mode directory. Also, you need to have a R instance running (invoke M-x R) in the directory of your document. Try some of the examples given in his tutorial in a new .org document now.

What happens behind the scenes here is that org-R translates the inline commands into a R command sequence. For example, infile:x.csv is converted into x = read.table(...) . The +RR prefix introduces direct R commands, which are executed before the +R commands (except infile:). The result is then inserted as a table or as a link to a graphic. Note that many actions rely on conventions:  outfile:x.png will create a png image. Reading from a datafile will only work if the file suffix is .csv. And its contents will always be stored in a variable x. If you are not sure what is happening behind the scenes, the command showcode:t shows you the created R-code and is hence a great tool for debugging.

Using figures

A typical fragment for creating a figure that is also exported into LaTeX/PDF/HTML properly is

#+CAPTION: Distribution of foot sizes among coputer scientists
#+LABEL: fig:SizeDistribution
#+ATTR_LaTeX: scale=0.5
[[file:images/distanceDist.png]]
#+R: infile:”tmp.csv” #+R: action:density columns:1 args:(:lwd 4)
#+R: outfile:”images/distanceDist.png”
#+R: inline:t

The first three lines give the image a caption and a reference, and define the size of the image in the paper. Invoking C-c (or org-apply) in the second part will (re-)create the image. The command #+R: inline:t is required for the resulting link to the image having the same format as the LaTeX exporter recognizes for figures.

Keeping your document clean

The main intend of org-R obviously was to enable org-mode users to perform quick calculations and create simple graphs from data that originates from org-tables. Undeniable, org-r is very good at that. However, our objective shall rather be to centralize and de-fragment drafting of papers containing complex figures. When it comes to more elaborate graphs and datasets, the usefulness of  the predefined actions vanishes and longer, more complex R code is required. You might rather want to work in ESS/R-mode, where you have all the coloring and intendation niceness, and you do not have to precede every line with an +RR:.

The alternative is to source the functions from a separate .R file, for example you could use

#+CAPTION: Distribution of finger lengths among computer scientists
#+ATTR_LaTeX: scale=0.5
#+LABEL: fig:LengthDistribution
[[file:images/distanceDist.png]]
#+R: infile:”tmp.csv”
#+RR: source(”rcode.R”)
#+RR: myDensity(x)
#+R: outfile:”images/distanceDist.png”
#+R: inline:t

with the following code (imagine a longer fragment) in rcode.R

myDensity <- function(x) {  
plot(density(x[,1]), col=”chartreuse4″, lwd=5, xlim=c(0,2000))
}

That way, you avoid immense R code pollution in your draft, yet you have all the information (data-files, linked code and parameters) in one place. Happy drafting!

alias lsd="ls -l|awk '/^d/ {print \$9}'"

It displays all subdirectories underneath the current directory. The goodness in this variant is that it gives you the pure names and that you can use it in loops without hassle :

for d in `lsd`; do
mv $d/resultfile.dat $d_result.dat;
rmdir $d;
done