Don Unix - Unix Tips n Tricks

SVN : Merge a branch back into the trunk

2010-02-19T11:52:00.002+01:00

How to copy a branch called V2 into the trunk :

I. find the first revision "V2"

in V2 directory type :
[pene@donunix] svn log --stop-on-copy

note the latest revision number displayed (for us it's 270)

II. find the latest V2 revision

in V2 directory type :
[pene@donunix]svn up

note the revision number displayed (for us it's 1603)

III. merge the branch into the trunk

in V2 directory type :
[pene@donunix]svn merge -r270:1603 https://my.svn.server/path/code/branch/V2

and commit:
[pene@donunix]svn ci -m "Merge V2 branch back into the trunk"

[Perl] a one-line command to create a random number

2010-02-16T16:21:00.002+01:00

Here is how to create a random number between 0 and 100 with this simple perl one-line command :

[pene@donunix] perl -e "print int(rand(100))"

grep -v in Perl one-line

2010-02-01T11:45:00.002+01:00

How to create the grep -v command in Perl, here is the clue :

[pene@donunix] perl -ne "! /your_string/ && print" file.txt

And if you want to write the output directly into the input file (file.txt), just type this :
[pene@donunix] perl -i.bak -ne "! /your_string/ && print" file.txt

SVN : How to restore an older version of my current file ?

2009-11-09T17:26:00.002+01:00

I want to restore an older version (from the SVN revision number 1289) of my current file fic.pl, what can I do ?

Here is the tips (NB my current revision is 1492) :

[npene@donunix] svn merge -r 1492:1289 fic.pl

[Perl] One liner calculus

2009-07-29T14:13:00.003+02:00

In the following example file « calc.txt », I want to multiply each number by 2/3 :

199,119
326,147
385,345
250,330
157,218
67,241
402,175

Here is the solution in one line with Perl :

[pene@donunix] perl -pi.bak -e 's|(\d+),(\d+)|int($1*2/3).",".int($2*2/3)|e' calc.txt

How to convert a string to upper or lower case ?

2009-07-20T11:34:00.001+02:00

[pene@donunix] var=`echo $var | tr [a-z] [A-Z]`

Twitter : display your rate limit per hour

2009-06-20T09:56:00.004+02:00

[pene@donunix] curl -u twitter_name:twitter_password http://twitter.com/account/rate_limit_status.xml

N.B. : For information, the default rate limit is 100 requests per hour.

grep: advanced use

2009-04-20T15:51:00.005+02:00

How to display all lines with the word text or the word name?
Here is the answer:

[pene@donunix] grep -P 'text|name' file.txt

find and only print files located in subdirectories

2009-04-06T15:54:00.007+02:00

[pene@donunix] find . -maxdepth 2 -mindepth 2

SSH : Public keys generation

2009-03-19T09:54:00.005+01:00

The purpose of this short post is to explain how being connected to an ssh server without typing the password, how ?
With the generation of public ssh keys.

I. Client : Generation of public keys

First (just for the first time) you have to genere the 2 keys :
[donunix@client] ssh-keygen -t dsa

During the dsa key generation, ssh-keygen ask you to enter a path phrase (it replace the password system). type [enter] if you doesn't what to genere a path phrase.

The 2 keys are in the ~/.ssh directory :
[donunix@client] ls ~/.ssh/id_dsa*
id_dsa id_dsa.pub

id_dsa is the private key
id_dsa.pub is the public key to send too the server

Client : send to the server the public keys

copy the public keys to the server :
[donunix@client] scp .ssh/id_dsa.pub user@server:.ssh/client.pub

Server : Record the client public keys

Connect to the server :
[donunix@client] ssh donunix@server

record the keys on the .ssh/authorized_keys file :
[donunix@server] cd .ssh
[donunix@server] cat client.pub >> autorized_keys

And that's all folk ;)
You can now run ssh sessions without typing your password.

[Dvorak keyboard] Launch a XKB keyboard file

2009-03-05T16:00:00.006+01:00

I want to learn the french version of the Dvorak Simplified Keyboard.
To toggle my keyboard from azerty to Dvorak-bepo configuration, I need to launch the dvorak-bepo.xkb file.

So here is, how to set your keyboard in Dvorak-bepo :
[pene@donunix] xkbcomp -w0 fr-dvorak-bepo.xkb :0

And if you want to go back, with the classic French configuration :
[pene@donunix] setxkbmap fr

Enregistrer maintenant

Create a different colored prompt for each users and hosts

2009-03-02T16:44:00.007+01:00

I've just coded this script to avoid mistakes when you're using more than one host and user login :

#!/usr/bin/ksh
#######################################
# Program : Prompt-Color
#
# Version : V1.0
# Date : 2009-03-02
# Author : N. PENE
#
# Inputs : none
#######################################
hostname=`hostname`
username=`whoami`

#-- classic colors vars :
BLACK='\[\033[0;30m\]'
RED='\[\033[0;31m\]'
GREEN='\[\033[0;32m\]'
BROWN='\[\033[0;33m\]'
BLUE='\[\033[0;34m\]'
PURPLE='\[\033[0;35m\]'
CYAN='\[\033[0;36m\]'
GRAY='\[\033[0;37m\]'
DEFAULT='\[\033[0m\]'
#-- bolded colors vars :
BBLACK='\[\033[1;30m\]'
BRED='\[\033[1;31m\]'
BGREEN='\[\033[1;32m\]'
BBROWN='\[\033[1;33m\]'
BBLUE='\[\033[1;34m\]'
BPURPLE='\[\033[1;35m\]'
BCYAN='\[\033[1;36m\]'
BGRAY='\[\033[1;37m\]'
BDEFAULT='\[\033[1;0m\]'

cprompt=$DEFAULT
cborder=$DEFAULT
cpath=$DEFAULT

#-- username color :
case $username in
"npene" ) cuser=$CYAN;;
"donunix" ) cuser=$BLUE;;
"fenice" ) cuser=$PURPLE;;
"donpene" ) cuser=$BROWN;;
"root" ) cuser=$GREEN;;
esac

#-- hostname color :
case $hostname in
"host1" ) chost=$BRED;;
"host2" ) chost=$BBLUE;;
esac

PS1="$cborder[$cuser\u$cborder@$chost\h$cborder: $cpath\W$cborder] $cprompt"
export PS1

convert a pdf file to png files

2009-03-01T19:51:00.002+01:00

[pene@donunix] convert -resize 540x700 fic.pdf fic%04d.png

audio files : increase the volume by +2db

2009-03-01T19:47:00.004+01:00

[pene@donunix] mp3gain -r -d 2.0 mysong.mp3

print the exact date, one week ago

2009-02-17T17:05:00.002+01:00

[pene@donunix] date +'%Y %m %d' --date '1 week ago'

NB : The output format of this command is : YYYY MM DD (2009 02 10)

Script : keep your connection on a remote shell

2009-02-11T16:49:00.003+01:00

A script to launch if you doesn't want to be disconnected by timeout on your remote ssh connection :

#!/usr/bin/ksh
#####################################################################
# Program : Keep-Hand
#
# Version : V2.0
# Date : 2009-02-11
# Author : N. PENE
#
# Inputs : none
#####################################################################

while [ 1 -eq 1 ] ; do

#-- check if the code is attached to an xterm :
term=`ps -p $$ | cut -d " " -f 2`
if [ $term == "?" ] ; then
exit
fi

#-- wait 4min :
sleep 240
echo -e ".\c"
done

SVN : remove all deleted files

2009-02-11T11:31:00.003+01:00

Hi folks,
Today a new Subversion tips: How to remove from the SVN base deleted files (by rm command).

[pene@donunix] svn status | grep '^\!' | sed 's/! *//' | xargs -I% svn rm %

ncftpput : copy a directory

2008-12-08T16:56:00.003+01:00

Here is my reminder, to record one for all the true ncftpput syntax:

[pene@donunix] ncftpput -R -u remote_login -p remote_pass remote_host remote_directory/ local_directory

kill all process unattached to a terminal

2008-11-20T10:00:00.003+01:00

I need to kill a lots off process running called "keep-hand.ksh".
The problem: I don't want to kill "keep-hand.ksh" process attached to a Terminal (TTY).
The challenge: Doing it, in one line!

My answer:
[pene@donunix] ps -u pene | grep keep-hand.ksh | grep ? | cut -d "?" -f 1 | xargs kill -9

[NCO] extract a smaller area in a netCDF file

2008-10-15T15:53:00.002+02:00

I want to create a smaller area with these new dimentions :
For the x coordinate : [1,300]
For the y coordinate : [ 250,800]

[pene@donunix] ncks -dx,,200 -dy,250,800 fic_in.nc fic_out.nc

[NCO] concatenate 2 netcdf files with differents variables

2008-10-15T15:38:00.002+02:00

In two steps :

[pene@donunix] ncecat fic_in1.nc fic_out.nc
[pene@donunix] ncecat -A fic_in2.nc fic_out.nc

Make a list of your twitter friends

2008-09-06T12:29:00.004+02:00

Make lists of your twitter friends and followers :

friends :

[pene@donunix] curl -u twitter_name:twitter_password -d status="friends" http://twitter.com/statuses/friends.xml | grep screen_name | sort | sed -e 's/ * $.*$<\/screen_name>/\1/' > friends.xml

followers :

[pene@donunix] curl -u twitter_name:twitter_password -d status="followers" http://twitter.com/statuses/followers.xml | grep screen_name | sort | sed -e 's/ *$.*$<\/screen_name>/\1/' > followers.xml

How to post on twitter from a Unix console !

2008-07-25T10:33:00.003+02:00

Its really simple to do it with curl :
[pene@donunix] curl -u twitter_name:twitter_password -d status="$1" http://twitter.com/statuses/update.xml

and, if like me you're in front of the proxy problem :
[pene@donunix] curl -x http://proxyurl:proxyport -U proxy_username -u twitter_name:twitter_password -d status="$1" http://twitter.com/statuses/update.xml

[Perl] make changes form line 3 to line 20

2008-05-23T16:35:00.002+02:00

How to substitute all pattern1 to pattern2 from the 3rd line to the 20st line?
Perl has the solution... and in one line :D

[pene@donunix] perl -p -i -e "s|pattern1|pattern2| if 3 .. 20" file.txt

Find all running process named "firefox"

2008-05-09T01:49:00.004+02:00

Find for the user pene all running process named "firefox"

[pene@donunix] ps -u pene | grep firefox

[Shell-Script] create a lock system for your code

2008-04-28T09:47:00.004+02:00

Here is the code for making a lock system in your shell-script program.
The main utility of the lock system is to be sure to not running your code while another process of the same code is running.

The main principe is to create a lock file when the code is running, and check the existance of this file each time you're launching it.

NB : the variable $exec is the name of the script you want to execute.

#-- Parameters:
exec="./my-script.sh"
NAME="my-script"
LOCK="/tmp/${NAME}.lock"
out="/tmp/${NAME}-$(/bin/date '+%Y%m%d').out"

#-- Error Function:
error () {
echo "$1" 1>&2
exit 1
}

#-------------------------
# Launch the code
#-------------------------
if [ -f "$LOCK" ]
then
if [ "$(ps -p `cat $LOCK` | wc -l)" -gt 1 ]; then
#-- process is still running
echo "Quit at start: lingering process `cat $LOCK`"
error "Daemon already running (not launched)!"
exit 10
else
#-- process not running, but lock file not deleted
echo "process not running, but lock file not deleted"
rm $LOCK
echo $$ >> "$LOCK"
fi
else
echo "create lock"
echo $$ >> "$LOCK"
echo "done (pid=$$)"
# launch the code:
ksh ${exec} >> $out 2>&1

fi

rm $LOCK

Find out Linux version

2008-04-14T09:16:00.000+02:00

With LSB (Linux Standard Base) you can know the name of your linux distribution ,the version and the version code name :

[pene@donunix] lsb_release -a

[SVN] Add Recursively

2008-04-10T09:54:00.002+02:00

Here is the command to add recursively new files in Subversion :

[pene@donunix] svn status | grep '^\?' | sed -e 's/^? *$.*$/\1/;s/ /\\ /g' | xargs svn add

Number of characters in my file (without end_of_line/return characters)

2008-04-02T16:49:00.004+02:00

I have some problem with the wc command.
If I write just 3 characters in a file, wc -m give to me 4 characters instead of 3.
For example :
[pene@donunix] echo 123 | wc -m
4

To explain quickly this, wc add the "end of line" character. So, if I don't want it for my echo 123, I have to write this :
[pene@donunix] echo -e "123\c" | wc -m

And if I want to exclude all the end_of_line/return characters of my wc -c, I have to write this wonderful command :
[pene@donunix] cat fic.txt | xargs -i% echo -e "%\c" | wc -c

[svn] file : restore older versions

2008-04-01T15:42:00.002+02:00

To restore the version of the file.sh from the lastest commit, type this :
[pene@donunix] svn revert file.sh

To get the version form the release number 214 :
[pene@donunix] svn update -r 214 file.sh

How many files on all the subdirectories

2008-03-18T18:40:00.003+01:00

I want to know the total number of files in all the subdirectories from the courant path.
Here is the clue :

[pene@donunix] dir -1 | xargs -i% ls %/ | wc -l

NB : In my subdirectories there is only files, not directories.

remove all files, exept *.gif files

2008-03-03T15:02:00.002+01:00

To keep just *.gif files from the courant directory, just type this :
[pene@donunix] ls * | grep -v \.gif | xargs rm

List and sort by sizes all directories and files (in human readable format)

2008-02-26T10:16:00.003+01:00

This command takes more time to process than the classic du . | sort -n , but in the result sizes are printed in human redable format:

[pene@donunix] du . | sort -n | awk '{print $2}' | xargs du -hs

search from the current directory and replace html files content

2008-02-20T09:28:00.003+01:00

How replace the expression expr1 by expr2 in all the html files located from the current directory and all his sub-directories :

[pene@donunix] find . -name "*.html" -print0 | xargs -0 perl -p -i -e 's|expr1|expr2|g'

Find big files

2008-02-11T11:38:00.002+01:00

This line find and print all the files greater than 20Mo in the /home/pene directory :
[pene@donunix] find /home/pene -type f -size +20000k -exec ls -sh {} \;

Dynamic prompt line

2008-02-06T11:43:00.000+01:00

Two lines to put on your .bashrc file to have a nice dynamic prompt line.

export MAX_PATH_LENGTH=25

export PROMPT_COMMAND='PS1="\\[\\033[32m\\][\u@\h:\`if [[ \$? = "0" ]]; then echo "\\[\\033[35m\\]"; else echo "\\[\\033[31m\\]"; fi\` \`if [[ `pwd|wc -c|tr -d " "` > $MAX_PATH_LENGTH ]]; then echo "\\W"; else echo "\\w"; fi\`\\[\\033[32m\\]]\$\[\033[0m\] "; echo -ne "\033]0;`hostname -s`:`pwd`\007"'

NB : The color of the path change if the last command written is unknown or wrong, and shortens for long directories path.

[Shell-Script] Loops for each element of a tabular

2008-01-30T11:21:00.000+01:00

Because, I have some difficulties to record this information in my mind :)
In this code we create a tabular and then do a loops on each element to print the result:

tab[0]="Don"
tab[1]="Unix"
tab[2]="Unix Tips n Tricks"

for var in ${tab[@]}; do
echo $var
done

[Vim] highlight a search or not, that is the question

2008-01-21T13:53:00.000+01:00

As default, when you make a search, Vim highlight the result.
This option can sometimes be irritating, especialy when you search "=" or "," characters.

So, here is a line to put on your .vimrc (Vim default configuration file) :
noremap ‹F7› :set hls!‹cr›:set hls?‹cr›
So now, with the [F7] key you can highlight or not a search result. Usefull, isn't it?

[Vim] open n files with vertical split

2008-01-16T16:07:00.000+01:00

How to open 3 files with a vertical split :
[pene@donunix] vim -O fic1 fic2 fic3

To switch between each files press [ctrl+W] + a direction key.
If you want to open another file in your Vim session write in [Escape] mode :
:vsplit fic4 for vertical split
:split fic5 for horizontal split

[Vim] completion

2008-01-15T10:31:00.000+01:00

The completion is a very usefull option from one of the best code editor : Vim.

To complete a variable cl_ just press [crtl]+N or [ctrl]+P to go forward or backward in the list of variables beginning by cl_.

Another powerfull completion option is the completion of paths and files names, to complete a path like /home/pene/ press [crtl]+X once, and then [crtl+F] or [ctrl]+P to go forward or backward in the list of files and directories.

find the full path of an executable

2008-01-08T20:17:00.000+01:00

If we supposed our executable is named prg and located in the ./bin dir, it's possible to exctract the full path with the following command :

[pene@donunix] which bin/prg | xargs dirname

Import a new project to Subversion

2008-01-07T08:38:00.000+01:00

Before all, you have to create thoses directories :
[pene@donunix] mkdir -p origin/branches origin/tags origin/trunck

then copy your project in the origin/trunk directory :
[pene@donunix] cp -r myproject/* origin/trunk

And to finish, import the project to your subversion base :
[pene@donunix] svn import origin https://svn.b11.fr/svnroot/ -m 'initial version'

replicate standard output to a specific file

2008-01-04T10:02:00.000+01:00

The tee command replicates the standard output to both your terminal and an output file (here output.txt) :

[pene@donunix] ps -u pene | tee output.txt

how to mout an iso file with Linux

2008-01-03T10:19:00.000+01:00

[pene@donunix] mount -o loop myisofile.iso /mnt/tmp/

How to take a screenshot

2008-01-02T18:36:00.000+01:00

The screenshot is saved in the screenshot.png file :

[pene@donuix] xwd -root | convert - screenshot.png

print (only) the name of file for a grep command

2007-12-26T16:09:00.000+01:00

Search the occurence occur in *.pro files, and print only the name of files where the occurence was found.

[pene@donunix] grep occur *.pro | sed -e 's/$.*\.pro$:.*/\1/' | uniq

transform a path onto a variable in a list of files

2007-12-26T15:30:00.000+01:00

in this code, we change in all IDL *.pro files, the path ../DATA onto a variable named data_toolbox.
For more precisions, the occurence '../DATA is replaced by data_toolbox+'
Here is the code :
[pene@donunix] perl -p -i.bak -e "s|\'\.\./DATA|toolbox_data+\'|g" *.pro

NB : Before making the changes, perl create a copy of the IDL files with the extension .bak.

display directories size and sort it by size

2007-12-21T15:29:00.000+01:00

[pene@donunix] du . | sort -n

list files and sort it by numbers of lines

2007-12-20T11:57:00.000+01:00

[pene@donunix] ls * | xargs wc -l | sort -n

a shell-script template

2007-12-18T10:48:00.000+01:00

#!/bin/bash
#####################################################################
# Program :
#
# Version :
# Date    :
# Author  :
#
# Inputs  :
#
# Notes   :
#
#
#####################################################################

prg=$0

#--------------------------------------------------------------------
# Setup Global Variables
#--------------------------------------------------------------------


#--------------------------------------------------------------------
# Test for prerequisites
#--------------------------------------------------------------------

#--------------------------------------------------------------------
# Do something
#--------------------------------------------------------------------

How to extract an rpm file

2007-12-17T10:23:00.000+01:00

Here is a command to extract an rpm file :
[pene@donunix] rpm2cpio fichier.rpm | cpio --extract

number of *.png files in my directory

2007-12-14T17:04:00.000+01:00

Everthing is in the title :
[pene@donunix] ls *.png | wc -l

ls : list directory contents sorted by size

2007-12-14T16:54:00.000+01:00

[pene@donunix] ls -1sSh .

This command list the common directory content and give in result the size of each files sorted by... size :D
It can be intersting to create an alias of this command.

Perl one line : how to create a html thumb page

2007-12-14T15:16:00.000+01:00

Here is a Perl one-line command to create an html thumb page.
Then in the html thumb page you will be able to visualise gifs from a specific directory :

The code :
[pene@donunix] ls -1 *.jpg *.gif *.png > index.html
[pene@donunix] perl -p -i.bak -e 's/^(.*)\.(.{3})$/<a href="$1\.$2"><img src="$1\.$2" width="320" border="0" alt="$1" ><\/a><br\/>/g' index.html

Well done, you've created an index.html file in your pictures directory.