Mar 092008

I created a little script that retrieves all the sequences for a particular trace archive query and stores everything into a file, that’s got to be useful to more than me, it uses the file provided by NCBI.
here it goes (updated 04/01/2008 for mac os X compatibility):


#first count the number of sequences

seqCount=`$queryScript query count $QUERY`
fileCount=$(( $seqCount/40000 ))
echo found $seqCount sequences
echo I will create $(($fileCount + 1)) files named dataN.tgz
while [[ $i -lt $(($fileCount + 1)) ]]
echo downloading data$i.tgz;
(echo -n "retrieve_tgz fasta 0b"; $queryScript "query page_size 40000 page_number $i binary $QUERY")| $queryScript > data$i.tgz
# fasta can be changed to all / xml
# use "text" instead of "binary" if doing an xml query
i=$(( $i + 1 ))

rm $OUTFILE -f
for file in `ls data*.tgz`;
echo processing $file;
tar -zxf $file --to-stdout >> $OUTFILE;
echo $QUERY > $OUTFILE.query
seqsInFile=`cat $OUTFILE | grep ">" | wc | tr -s " " | cut -f2`
echo $seqsInFile sequences found in the final file
tar -czf $OUTFILE.tar.gz $OUTFILE*

Mar 082008

Some follow up with running blat on Unfortunately, using blat on smaller chunks of DNA, like chromosomes wasn’t the way to go: I quickly rearranged my qsub submissions so that individual chromosomes maybe searched instead of whole genomes at once. twoBitInfo, which retrieves the information related to the chromosomes in a 2bit file, was my friend for that, however I obtained the same “killed” error message for the twoBitInfo utility, hinting seriously at some compilation issues. I went back on track with my efforts to cross compile blat for amd64 on my RHEL4 box but that still gave me error message on the grid engine.

By the way, i disgress a bit but unless you didn’t realize, testing those binaries isn’t the most fun thing to do if you don’t have a solaris box setup, as each time you need to reupload the binaries and scripts and test them live, while at the same time wasting 1cpu/hour. Also somehow lately jobs are taking forever to start up, even though the job id increments only by one (hinting that there were no other jobs running during my wait time).

Seeing how my efforts at cross compilation failed miserably, I decided that my next move would be to try and compile blat natively… After running grid engine jobs solely for compiling blat (a job that takes 5 seconds on my machine, a bit of an overkill to use the cluster but thank god i’m not using all 5000 cpus) I managed to compile the blatSuite successully. The major bit to “porting” was the error message:

pscmGfx.c: In function `colinearPoly':
pscmGfx.c:390: warning: implicit declaration of function `isinf'
gmake[1]: *** [pscmGfx.o] Error 1
gmake: *** [topLibs] Error 2

it seems the isinf function is a source of headache for people porting to solaris, some suggestions ( are to change the compiler flags to gnu99, but that was to no avail. I resolved it by removing pscmGfx.o from the makefile, since it’s not used by blat. the mods I made to the common make file follow, some changes maybe useless but I did not take the time to test:
CFLAGS=-L/usr/sfw/lib/amd64 -L/usr/lib -lnsl -lsocket -lresolv
HG_INC=-I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -I/usr/include -I/usr/sfw/lib/gcc/i386-pc-solaris2.10/3.4.3/include/

more on benchmarking later…

update: Benchmarking estimate tell me that it will cost 730$ to repeat mask and map a whole library’s worth of end sequences (about 280 Megabases onto the human genome)

Mar 042008

Repeat masking now works thanks to a set of perl and bash scripts I created; I can input any fasta formatted file I want, and specify some repeat masker parameters together with the size of the work unit in bytes; the fasta file gets chopped into small work units of size close to specified size and the jobs get distributed on the grid, and processed at lightning speed. I haven’t figured out costs yet, but I will post results back here when I scale up with the searches. The only current caveat is that each RepeatMasker process requires to have its own database. At run time, each process has to recreate a database, I had to hack the code in RepeatMasker a bit so that each process is able to do this; I calculated that the impact of this change should cost about $0.002/cpu/hour, which is fortunately very little. Another down side to this little caveat is that on a 1000 cpu job the database will end up chewing space, that is 3.5 Gb of space will be used for temp storage of the databases, that’s pretty important given the Sun Grid Engine only allows up to 10Gb of storage space…

Next step: getting blat to work; my first experience is that the scripts I developed in parallel with RepeatMasker’s scripts are running fine however the blat process seems to be killed out of the blue, I suspect it’s because blat is taking too much memory. I contacted tech support for this, since I don’t see a way out of that on my own.

 Posted by at 11:57 am
Feb 282008

I unearthed one of my long ago written program for KOMP to present at ucsf to an audience who is interested in knocking out micro-rnas. I had to present it today, it seems things went well, although Lu who was here as well seems to say I wasn’t bragging enough, and that I’m not trying to sell my stuff enough. I have to agree: I hate to oversell something, and that’s how I’m made I guess; but I admit too, sometimes it’s nice to hear from someone (usually a good salesman) that what you’re buying into is such a nice product, and you’re already in love with it before you get a chance to see all the unironed bugs and lack of functionality… At least you’ve been in love for some time…