Memo's Island Blog : Mehmet Süzen

XML and command line parsing in C++/C

2011-09-18T05:48:00.000-07:00

Dynamic languages like python, tcl or alike are taking over in driving many C/C++ codes I/O operations and option parsing due to quick and easy implementation, however it is still possible to use libraries/APIs for parsing options via command line (or XML ) quite efficiently in C/C++. Here is the list of some API/libraries:

Xerces is Apache's XML parser for C++.
GCCXML is gcc extention for XML parsing in C++.
Commons CLI is Apache's command line parsing API.
Argtable is ANSI C command line parser.
getop is gnu's commang line parser.
tclap is templatized C++ command line parser.

Operating in Multiple Repository within ant

2011-05-14T05:23:00.001-07:00

Ant script can not change its base directory after it is invoked, for that reason if you are handling two repositories under same directory it would be wise to write a secondary wrapper script.

R Package install order automation

2011-04-16T12:29:00.001-07:00

There is a bioconductor tool to generate install order for packages [here].
Initially tool generates graph of CRAN package dependencies (ignoring suggests, because otherwise graph will be acyclic).

Usage Example:

source("http://www.bioconductor.org/biocLite.R")
biocLite("pkgDepTools")
require('pkgDepTools')
deps <- makeDepGraph("http://cran.r-project.org", type="source")
a<-getInstallOrder("plyr",

deps,needed.only=FALSE)
a$packages
[1] "iterators" "itertools" "plyr"

R-plugin for vim

2011-04-09T04:54:00.000-07:00

vim is a powerful text editor and R is a powerful environment for statistical computing task. To harness both powers a plug-in for vim is developed. One needs python enabled vim and conqueterm.

Ant task with wildcard expansion for svn.

2011-04-08T02:23:00.000-07:00

Ant is a popular tool among Java developers but one can use it for some other generic tasks. For example using svn as a task inside an ant script is provided by a library. However wildcard expansion won't work under this task. One way to over come this to use an exec task as follows (in windowz):

<exec executable="cmd.exe">
  <arg line=" /C svn --force delete pathtofiles\*.dat
             --username=username --password=password">
  </arg>
</exec>

Python Crash tutorial for experienced programmers

2011-01-18T17:09:00.000-08:00

Phython has already became a de-facto scripting language, even taking over Perl (depends who do you ask, of course). Recently I have began to use it and like its approach to OO/D. Here is my very short crash Python notes [txt], in 10 minutes, you can learn the basic facilities.

Monitoring IP traffic on a host against denial of service

2011-01-05T23:57:00.000-08:00

Long time ago I have given some tips for Cisco routers IoS capability to limit SYN half-open TCP connections [here]. Recently a nice python script [here], provides a mechanism to monitor similar information on regular bases and block IP addresses for a certain period of time automatically. It uses combination of tools : netstat, iptables and python modules. However, tuning thresholds must be done carefully to prevent blocking legitimate traffic.

Package Bibentry

2010-12-25T04:49:00.000-08:00

Bibtex is a powerful tool for referencing in scientific publications and many other type of academic documents. One nice package (bibentry) that enables us to use bibtex entries within the text. Simply add the package as follows:

\usepackage{bibentry}

When you need to refer to needed bibtex key information, just use:

\bibentry{needed_bibtex_key}

Reverse 32-bit Hexadecimal Value (with C)

2010-09-27T16:13:00.000-07:00

A 32-bit hexadecimal value such as 0xABCD1234 may need to be reversed as 0x4321DCBA. This might be needed. The following is a naive implementation of reversing a hexadecimal. It would be interesting quiz for undgrad CS student to write a n-bit version of this.

signed int reverse_hex(signed int num) {
/* stupid reverse hex */
int rev= 0x00000000;
int digit= 0x00000000;
int mask1=0x0f000000;
int mask2=0x00f00000;
int mask3=0x000f0000;
int mask4=0x0000f000;
int mask5=0x00000f00;
int mask6=0x000000f0;
int mask7=0x0000000f;

digit=num << 28;
rev=num << 20;
rev=rev & mask1;
rev=digit+rev;
digit=(num << 12) & mask2;
rev=digit+rev;
digit=(num << 4) & mask3;
rev=digit+rev;
digit=(num >> 4) & mask4;
rev=digit+rev;
digit=(num >> 12) & mask5;
rev=digit+rev;
digit=(num >> 20) & mask6;
rev=digit+rev;
digit=(num >> 28) & mask7;
rev=digit+rev;
return(rev);
}

However, the above procedure is not so usual while it is reversed by chunks of 4-bit. More realistic situation is reversing from between big and little endian representation. Such as, 0xABCD1234 would be reversed as 0x3412CDAB, so byte ordering matters. The following is the C function doing this. Similarly n-byte version of
this function will be left as a further exercise.

signed int reverse_hex_byte(signed int num) {
/* stupid reverse hex */
int rev= 0x00000000;
int digit= 0x00000000;
int mask1=0x000000ff;
int mask2=0x0000ff00;
int mask3=0x00ff0000;
int mask4=0xff000000;

/* Move 1st byte */
digit=num >> 24;
rev=digit & mask1;
/* Move 2nd byte */
digit= num >>8;
digit=digit & mask2;
rev=digit+rev;
/* Move 3rd byte */
digit= num <<8;
digit=digit & mask3;
rev=digit+rev;
/* Move 4rd byte */
digit= num <<24;
digit=digit & mask4;
rev=digit+rev;
return(rev);
}

Vodafone (telsim Cyprus) 3G with Ubuntu

2010-08-25T05:31:00.000-07:00

Sometime ago I have given some instructions on how to connect to movistar,Spain
3G network [here]. The similar connection can be made for Vodafone (telsim Cyprus)
with the following changes:

1. Modem USB settings command should be read:

modprobe usbserial vendor=0x12d1 product=0x1001

2. In wvdial.conf entry for telsim should look like:

[Dialer telsim]
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Init3 = AT+CGDCONT=1,"IP","edge.kktctelsim.com"
Modem Type = Analog Modem
Baud = 115200
New PPPD = yes
Modem = /dev/ttyUSB0
ISDN = 0
Dial Command = ATDT
Phone = "*99***1#"

More efficient diag() function

2010-04-23T05:06:00.000-07:00

GNU R's diag() function provides a tool to produce diagonal matrices, however for
large vectors this is quite memory inefficient. Here is a version which uses at least
25% less memory;

mydiag<- function(A) {
#
# Given a a vector generate
# a diagonal matrix
# (Consume at least 1/4 less memory then R's diag)
#
A <- as.vector(A) ;
B <- matrix(0,nrow=length(A),ncol=length(A)) ;
row <- nrow(B) ;
col <- ncol(B) ;
size <- col*row ;
row <- row+1 ;
B[seq(1,size,row)] <- A ;
return(B)
}

Repeat chunks of a sequence arbitrary times with GNU R

2010-03-17T07:25:00.000-07:00

Repeating arbitrary size chunks multiple times in a given sequence with preserving order of chunks would be needed in certain computations. For example in matrix multiplication that is reduced to a 1D problem. Mapping your sequence into a matrix and repeating certain column or generating another sequence in the appropriate loop are straight forward solutions indeed. However these approaches will suffer immensely in memory consumption with increasing sequence size and would generate the original problem it seek to solve. More efficient and the general way to achieve repeating chunks is solved with the following R code:

manyrep <-function(a,f,n) {
i <- as.integer(length(a)/f);
q <- f
for(j in 1:i) {
a<-append(a,a[(q-f+1):q],after=q);
q<-q+2*f
p <- 1;
while(p <= (n-1)) {
a<-append(a,a[(q-2*f+1):(q-f)],after=(q-f));
q<-q+f ;
p<-p+1
}
}
return(a)
}

For example if sequence a = {1,2,3,4}, frequency or chunk size f=2 and number of repetition n=1, hence manyrep(1:4,2,1) the output should look like {1,2,1,2,3,4,3,4}. If n was 2 output would be {1,2,1,2,1,2,3,4,3,4,3,4}.

Append to a vector in regular intervals efficiently in R

2010-03-10T12:29:00.000-08:00

In many instances one may need to append a character or a value in a given sequence, usually represented as a vector or array. It is pretty easy to achieve this naively. However, the point is not to consume any more memory. This is one of the simple solutions I can propose in GNU R.

# This function appends a word into vector a in every f element
appendfreq<-function(a,f,word) { i <- as.integer(length(a)/f-1); q<-f; for(j in 1:i) { a<-append(a,word,after=q); q<-q+f+1; } ;return(a); }

GNU R Garbage Collection

2010-01-12T03:27:00.000-08:00

On GNU R if you delete your objects by

rm(list=ls())

This action won't free any memory associated with those objects. For this reason, R provides a utility for garbage collection, simply apply

gc()

This will free not used allocations. If you like to apply this whenever R allocates memory, apply this:

gctorture(on = TRUE)

Consult with R Internals manual for further details.

Groebner Bases and Cryptography

2009-12-10T00:47:00.000-08:00

You may have already heard the brilliant technique called Groebner bases to solve multivariate system of equations. But unfortunately it is pointed out that it isn't possible to use this in cryptography: [Why you cannot even hope to use Groebner Bases in Public Key Cryptography?].

Crash Notes for Connecting T30 running Ubuntu 8.04.3 to MoviStar (Spain) 3,5G Network (GPRS)

2009-08-05T04:08:00.000-07:00

Short description on how to connect Spanish movistar 3.5G service with your ubuntu, here

Deleting extraneous files from destination: rsync

2008-10-15T18:33:00.000-07:00

There were some reported non-functioning --delete flag when using rsync in syncronizing two filesystems/locations. This may occur due to failures or errors during rsync tries to sync zero size files. One simple work around for this is using --ignore-errors flag. Some examples for taking backup of user directory:

rsync -vaz -e ssh --delete --ignore-errors user@host:/home/user/ local_dir/
rsync -vaz --delete --ignore-errors /home/user/ /media/disk/backup_dir/

TCL Bad Code Error

2008-08-14T04:18:00.001-07:00

Using tcl extension would be quite reasonable to write a flexable code. How ever if you use parse command set and TCL_OK as a return, be sure that type of your function is int, otherwise tcl complains with a strange run time error, saying bad code. So, be sure about the correct return type.

Traffic Utilization on router high CPU usage.

2008-07-21T18:21:00.000-07:00

Long time ago, I was trying to address traffic utilization issues on border routers. This is the summary of my communication with nsp mailing list [Traffic Utilization Summary].

Crash LAMP Installation: Apache 2.2.6, PHP 5.2.6, mod_perl 2.0

2008-07-21T16:34:00.000-07:00

Recently I need to install LAMP on my laptop to brush up my LAMP skills. Actually, LAMP compilation on *nix like system is not that difficult. I've tried to compile almost bare minimum things one must to do in order to have running web server with php, perl, mod_perl and MySQL support. Here is the short document titled Crash Notes on installing LAMP System with Apache 2.2.6, PHP 5.2.6, mod_perl 2.0. Notes contain installation instructions, configuration and minimum testing. Example (httpd.conf)

Prototype Codes: memoproto

2008-03-18T12:01:00.000-07:00

There are many nice books out there on Programming, such as ANSI C programming by Kernigan & Richie. Usually they lack complete working examples, and rather they tend to give parts of codes that are related to topic they describe. For that reason, I've just started to collect many working full examples in various languages. It is located on Google's code repository. [link]

Simple syncronization via ssh with rsync.

2008-03-06T18:31:00.000-08:00

One can syncronize two different locations by using rsync via ssh or without ssh.
Here are the steps:

1. get from the other location
rsync -avuzb dir/mydir/ .

2. put into other location
rsync -Cavuzb . dir/mydir/

If one of the location is reachable via ssh then one ca write

rsync -avz -e ssh remoteuser@remotehost:/remote/dir /this/dir/

Look at the man pages for more details. Actually -e flag is not needed.

CPU Timing in a C code

2008-02-20T08:45:00.000-08:00

Examine this simple example :



#include < stdio.h >
#include < stdlib.h >
#include < time.h >
#include < math.h >

int main();

int main() {
  long int i,a;
  clock_t cputime,cputime1;
  double timing,timing0;
   cputime= clock();
    for(i=0;i<10000000;i++){
       a=pow(4,2);
       a=pow(4,2);
       a=pow(4,2);
       a=pow(4,2);
       a=pow(4,2);
      }

   cputime1= clock();
   timing0=((double) cputime1-cputime);
   timing=((double) cputime1-cputime) / CLOCKS_PER_SEC;
   printf("timing =%12.9f clock per sec=%d timing0=%12.9f\n",
          timing,CLOCKS_PER_SEC,timing0);
   exit(0);
}

Latex Poster: A working full example

2008-02-14T04:03:00.000-08:00

You must use Latex to generate your posters, because it looks beautiful. Here is fully working and self contained example [latex_poster].

Simple scripts: Compressing Files & Cluster Queue Jobs Termination

2008-02-13T05:54:00.000-08:00

If you would like to compress your data files or any type recursively you can use this simple perl script [datcompress].

Sometimes it is annoying to delete all your PBS jobs one by one, so simple perl script helps [deleteall].

How to benchmark communication cost in the beowulf (parallel system)

2008-02-12T03:12:00.000-08:00

Quite long time ago, I've proposed one simple way of estimating
communication cost of an MPI program (or multiple-instruction)
appeared in the beowulf mailing list. Here is the outline of
the procedure.
[original post]

Definition Communication cost (Comm) is the ratio between the real
time used by all processors (processes actually if you use MPI)
and global real cpu time (GRU).

 Comm=GRU/GR = (GR-GU-GS)/GR

Undefined symbols are as follows:

Comm= Communication cost
GR=global real time
LR=local worked real time 
LU=local user
LS=local system
GS=global system
GU=global user
n=# of procs
GU=n*LU
GS=n*LS
GR=n*LR

In the actual MPI program this measure's implementation could be outlined
as follows

 BEGIN PROGRAM Myprogram
      begin distribution
       initial_timings=timings;
                ...do whatever is your distributed computing....
       end_timings=timings;
      end distribution
       if(myrank == 0) {compute communication cost}        
END Myprogram

BEGIN PROC timings
    for(n=0;n < numproc;numproc) {
       if(n==myrank) { measure timing of myrank and send to rank 0 }
     }
END PROC