Apr 052008

Microsoft / business objects have changed the way Crystal Reports based applications can be deployed. It used to be complicated enough, now things have changed and apparently the documentation is scarce on the subject. I was sitting in a cafe a couple days ago trying to upgrade the source of one of my apps to visual studio 2008, I realized the crystal reports merge modules were not found anymore and wasn’t able to find any info about this in the documentation… god I hate that, everything should be findable in the docs (either the search function or the content have something to do with it, but somehow msdn docs can be unsettling sometimes). I found the solution to the problem as soon as I got connected to the internet.

It used to be be that you could deploy a Crystal Reports based project using merge modules, the new way to deploy crystal reports under visual studio 2008 is to use a redistributable msi package located under “C:Program FilesMicrosoft SDKsWindowsv6.0ABootstrapperPackagesCrystalReports10_5CRRedist2008_x86.msi”. I haven’t figured out a visual studio UI supported way to deploy this package at the same time as my application, and it looks as if, for now, one needs to deploy crystal reports separately. I will post my solution to this problem when I find out how to do this.

Mar 202008

Today I was just approved for the iphone sdk developer program, which makes me one of a “select few” developers that can do native development on the iphone. I was pretty sure my application was going to be denied since there is nothing special about me or my application which could have made a difference. Anyways that’s very good news since that will give me a little head start to continue developing my app (for the moment I’m not giving any details about my app until I have something presentable and I know it’s going somewhere – or nowhere). After succesfully installing osX on my pc I was able to quickly port all my code over to the iphone sdk and I must say I was pretty impressed with it; it has all a developer could dream of to create apps really productively…

Anyways I haven’t been able to touch the sweet stuff yet as I’m still waiting on my order to be processed but will try to get back here to post some update.

Mar 182008

Today I tried to install the Windows Vista Service Pack 1, but couldn’t go through when the installer said that certain language packs are incompatible with Service pack 1. Right enough I installed Italian and French languages last year when I installed Windows Vista and never used them since (!). Trying to uninstall the pack in programs and features it seems impossible to remove them, my way out of that was to run the lpksetup tool and I was able to uninstall both languages (one at a time with a reboot in between) and free up 1.6Gb of precious space at the same time.

Mar 112008

I was seriously considering buying a mac (and some apple stocks seeing how they almost converted me) in order to get started with the iphone sdk and continue development of my app, but I came across this article on dual booting a pc with macosX leopard, ultimately this is going to be cheaper for me. And I’m pretty excited that i’m going to get a true unix environment on my pc (and delete cygwin).

update: I’m getting the “chain0 booting error” and am not sure how to fix it, however i can still boot in osX using F8 + rd=disk0s1 at the boot prompt for the atkos disk. Overall the osX experience is not so nice on my pc, since neither wifi nor lan cards are functionning, also the vertical screen resolution is limited to 1024 instead of 1280 and the screen is shifted to the left uncentered. I am now testing parallels to see if running with Vista will improve things (that’s unlikely imho).

update2: I have since then resolved my booting and network problems on my vaio sz120p; I had overlooked the customize button at install which lets you add extensions to your install. Here are the steps that I took to make it all work:

-my macOS partition space was too small so I had to move my windows partition to free some space up, at first I tried using the windows partitioning software, but it is very limited in that it does not move files within the partition and as such is practically useless for most situation; I then realized that partition magic is not working any more under windows Vista. My last but not least option was to use the Gparted which is an awesome tool to manage almost all your repartitioning tasks.

-I reinstalled macOsX making sure I enable the proper extensions, including the macOs bootloader which I didn’t have in the first place. Now my boot problem was all resolved, no more”chain0 booting error”. A side effect to that is that I do not see anymore that macOsX recognizes my pc as a laptop, maybe I picked the wrong kernel.

-I managed to get the Marvell Yukon 88E8036 network card working by following instructions from this post which also works under MacOsX leopard…

-Next step for me is to get the full screen resolution working as well as the wifi network card

Mar 092008

I created a little script that retrieves all the sequences for a particular trace archive query and stores everything into a file, that’s got to be useful to more than me, it uses the file provided by NCBI.
here it goes (updated 04/01/2008 for mac os X compatibility):


#first count the number of sequences

seqCount=`$queryScript query count $QUERY`
fileCount=$(( $seqCount/40000 ))
echo found $seqCount sequences
echo I will create $(($fileCount + 1)) files named dataN.tgz
while [[ $i -lt $(($fileCount + 1)) ]]
echo downloading data$i.tgz;
(echo -n "retrieve_tgz fasta 0b"; $queryScript "query page_size 40000 page_number $i binary $QUERY")| $queryScript > data$i.tgz
# fasta can be changed to all / xml
# use "text" instead of "binary" if doing an xml query
i=$(( $i + 1 ))

rm $OUTFILE -f
for file in `ls data*.tgz`;
echo processing $file;
tar -zxf $file --to-stdout >> $OUTFILE;
echo $QUERY > $OUTFILE.query
seqsInFile=`cat $OUTFILE | grep ">" | wc | tr -s " " | cut -f2`
echo $seqsInFile sequences found in the final file
tar -czf $OUTFILE.tar.gz $OUTFILE*

Mar 082008

I never knew but I just found out a few days ago: St Nicholas is in fact Santa… and that’s not all: Barese people (people from the south of Italy near a city named Bari) – at least the ones that I know of – love to make gifts and are experts at it and that’s normal ! that’s something that had me feel a little insecure with my upbringing (which was particularly the opposite).

I was testing a new Google feature (the time line view is new to me at least) which lets you track how a certain subject has evolved across the centuries. I typed in the search the word Christmas and after exploring a bit I noticed the city of Bari (Lucia’s hometown) was mentioned ! I quickly wondered what does Bari got to do with Christmas ??! after reading a bit I read about the story of San Nicholas, whose Relics are deposited in the Santa Nicholas church in Bari after they were stolen about 1000 years ago. He was famous for attempting to give money anonymously to a poor merchant who couldn’t afford to pay the dowry for his 3 daughters, at which point he was caught and ever since renowned for being the symbol of gift giving. The tradition of Santa Nicholas was later fused with the Christmas celebrations.

Altogether this reading came as a revelation to me and explained a lot of things and why perhaps why Lu is so good at making gifts and remembering whoever’s wishes, I guess she’s taking her hometown’s traditions to heart!

Mar 082008

Some follow up with running blat on Unfortunately, using blat on smaller chunks of DNA, like chromosomes wasn’t the way to go: I quickly rearranged my qsub submissions so that individual chromosomes maybe searched instead of whole genomes at once. twoBitInfo, which retrieves the information related to the chromosomes in a 2bit file, was my friend for that, however I obtained the same “killed” error message for the twoBitInfo utility, hinting seriously at some compilation issues. I went back on track with my efforts to cross compile blat for amd64 on my RHEL4 box but that still gave me error message on the grid engine.

By the way, i disgress a bit but unless you didn’t realize, testing those binaries isn’t the most fun thing to do if you don’t have a solaris box setup, as each time you need to reupload the binaries and scripts and test them live, while at the same time wasting 1cpu/hour. Also somehow lately jobs are taking forever to start up, even though the job id increments only by one (hinting that there were no other jobs running during my wait time).

Seeing how my efforts at cross compilation failed miserably, I decided that my next move would be to try and compile blat natively… After running grid engine jobs solely for compiling blat (a job that takes 5 seconds on my machine, a bit of an overkill to use the cluster but thank god i’m not using all 5000 cpus) I managed to compile the blatSuite successully. The major bit to “porting” was the error message:

pscmGfx.c: In function `colinearPoly':
pscmGfx.c:390: warning: implicit declaration of function `isinf'
gmake[1]: *** [pscmGfx.o] Error 1
gmake: *** [topLibs] Error 2

it seems the isinf function is a source of headache for people porting to solaris, some suggestions ( are to change the compiler flags to gnu99, but that was to no avail. I resolved it by removing pscmGfx.o from the makefile, since it’s not used by blat. the mods I made to the common make file follow, some changes maybe useless but I did not take the time to test:
CFLAGS=-L/usr/sfw/lib/amd64 -L/usr/lib -lnsl -lsocket -lresolv
HG_INC=-I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -I/usr/include -I/usr/sfw/lib/gcc/i386-pc-solaris2.10/3.4.3/include/

more on benchmarking later…

update: Benchmarking estimate tell me that it will cost 730$ to repeat mask and map a whole library’s worth of end sequences (about 280 Megabases onto the human genome)

Mar 062008

The standard solaris and opteron binaries for blat don’t work on’s grid engine. the processes don’t get a chance to start and are killed on the spot.

the error message is pretty cryptic too:

/var/tmp/spool/r130c23z1/job_scripts/295234: line 52: 7090 Killed "$BLAT" "$DBPATH" "$CHUNK" "$OUTFILE" $PARAMS -dots=`echo $(($CHUNKSIZE/10))`

I will have to recompile them from the sources with different architecture flags.

update: recompilation won’t be necessary, it seems blat is also being spontaneously killed locally on my server, a behavior I had forgotten about. It seems that blatting individual chromosomes instead of whole genomes will be the way to go.

Mar 042008

Repeat masking now works thanks to a set of perl and bash scripts I created; I can input any fasta formatted file I want, and specify some repeat masker parameters together with the size of the work unit in bytes; the fasta file gets chopped into small work units of size close to specified size and the jobs get distributed on the grid, and processed at lightning speed. I haven’t figured out costs yet, but I will post results back here when I scale up with the searches. The only current caveat is that each RepeatMasker process requires to have its own database. At run time, each process has to recreate a database, I had to hack the code in RepeatMasker a bit so that each process is able to do this; I calculated that the impact of this change should cost about $0.002/cpu/hour, which is fortunately very little. Another down side to this little caveat is that on a 1000 cpu job the database will end up chewing space, that is 3.5 Gb of space will be used for temp storage of the databases, that’s pretty important given the Sun Grid Engine only allows up to 10Gb of storage space…

Next step: getting blat to work; my first experience is that the scripts I developed in parallel with RepeatMasker’s scripts are running fine however the blat process seems to be killed out of the blue, I suspect it’s because blat is taking too much memory. I contacted tech support for this, since I don’t see a way out of that on my own.

 Posted by at 11:57 am
Feb 282008

I unearthed one of my long ago written program for KOMP to present at ucsf to an audience who is interested in knocking out micro-rnas. I had to present it today, it seems things went well, although Lu who was here as well seems to say I wasn’t bragging enough, and that I’m not trying to sell my stuff enough. I have to agree: I hate to oversell something, and that’s how I’m made I guess; but I admit too, sometimes it’s nice to hear from someone (usually a good salesman) that what you’re buying into is such a nice product, and you’re already in love with it before you get a chance to see all the unironed bugs and lack of functionality… At least you’ve been in love for some time…