SRCS – Shell Revision Control System

I wrote a script to simulate a Version Control System. It is simulating the basic core RCS function, like check out a file with a specified version, check in a file, show all log metadata of a file, diff two versions, etc.
It’s logic:
1. For a totally new file, first run ‘init’ command to add a placeholder in the repository. Without ‘init’, user is not able to do other SRCS activities.
2. After ‘init’, user can run ‘ci’ to check in a new version of file in the repository. The Version 0 is empty and it is for placeholder. The versions of the check in are starting with 1, 2, 3,…,10,…
3. User can use ‘co’ to check out a latest version of the file from the repository or provide a ‘version’ in the command to check out a specified version.
4. User can use ‘diff’ command to diff two versions existing in the repository or diff one version with the local copy.
5. User can use ‘log’ command to print all of the metadata info pertaining to a file including its check in comments and version information.

How it work:
1. It depends on linux ‘diff’ and ‘patch’ command.
2. For every file in the repository, it will have a reversion control file RCS/filename,v. For example, for file admin/sql/test.sql, SRCS will maintain its metadata in admin/sql/RCS/test.sql,v
3. The root directory of the repository is defined in parameter _PROD

In SRCS – Check in file – It implements the initial and check in a file.
In SRCS – check out file – It implements check out a file with a specified version.
In SRCS – log – It implements a ‘log’ command to allow user to print all the metadata for a file.
In SRCS – diff – It implements a ‘diff’ command to show diff for two versions.

The script: https://github.com/luohuahuang/SRCS/blob/master/srcs.sh


_PROD="/u01/camel/12.0"
_DEBUG="on"
_LOG="_srcs.log"

fname_srcs="$_PROD/$2"
fname=$(basename "${fname_srcs}")
fdir=`echo "${fname_srcs}" | sed 's/'"$fname"'$//'`
fver="$fdir/RCS/${fname},v"

function DEBUG()
{
	[ "$_DEBUG" == "on" ] && $@ || :
}

function checkout()
{
	version=`tail -3 $fver | egrep "^version#" | sed 's/^version#//'`
	cp $fname_srcs ${fname}.tmp.$1
	for (( v=$version; v>$1; v-- ))
	do
		let delta=$v-1
		_l1=`nl "$fver" | egrep "version#$delta" | sed "s/version#$delta//" | sed 's/^\s*|\s*$//g'`
		_l2=`nl "$fver" | egrep "version#$v" | sed "s/version#$v//" | sed 's/^\s*|\s*$//g'`
		sed -n "${_l1},${_l2}p" "$fver" | patch ${fname}.tmp.$1 -i - >> /dev/null
	done
}

if [ $1 == "diff" ]; then
	##
	# diff a file with two versions.
	# This command will diff the v5 and v6 of admin/sql/test.sql: srcs.sh diff admin/sql/test.sql 5 6
	# This command will diff the v5 and local copy of admin/sql/test.sql: srcs.sh diff admin/sql/test.sql 5
	##
	echo "Diff $2 $3 $4 from SRCS..." | tee -a $_LOG
	if [ "$#" -lt 3 ]; then
		echo "Please provide at least one revision for diff...<FAILED>" | tee -a $_LOG
		exit 1
	fi
	if [ ! -f "$fver" ]; then
		echo "$fname doesn't exist in SRCS...<FAILED>" | tee -a $_LOG
		exit 1
	else
		if egrep -v "^\s*$" $fver | wc -l | egrep "^0" > /dev/null ; then
			echo "$fname is just initialled in SRCS...<FAILED>" | tee -a $_LOG
			exit 1
		else
			if ! cat $fver | egrep "^version#$3" > /dev/null ; then
				echo "Version $3 doesn't exist...<FAILED>" | tee -a $_LOG
				exit 1
			else
				checkout $3
			fi
		fi
		if [ "$#" -lt 4 ]; then
			echo "Diff $2 $3 from SRCS with the local copy..." | tee -a $_LOG
			if [ ! -f "$fname" ]; then
				echo "$fname doesn't exist in local directory...<FAILED>" | tee -a $_LOG
				exit 1
			fi
			diff ${fname}.tmp.$3 $fname | tee -a $_LOG
			echo "Diff $2 $3 from SRCS with the local copy...<SUCCESSED>" | tee -a $_LOG
		else
			if ! cat $fver | egrep "^version#$4" > /dev/null ; then
				echo "Version $4 doesn't exist...<FAILED>" | tee -a $_LOG
				exit 1
			fi
			checkout $4
			diff ${fname}.tmp.$3 ${fname}.tmp.$4 | tee -a $_LOG
			echo "Diff $2 $3 $4 from SRCS...<SUCCESSED>" | tee -a $_LOG
		fi
	fi
elif [ $1 == "init" ]; then
	##
	# initial a file in revision repository
	# this command will initial a placeholoder of admin/sql/test.sql in revision repository: srcs.sh init admin/sql/test.sql
	##
	date | tee -a $_LOG
	echo "Initial $2 in SRCS..." | tee -a $_LOG
	mkdir -p "$fdir/RCS"
	if [ -f "$fver" ]; then
		echo "$2 exists in SRCS already...<FAILED>" | tee -a $_LOG
		echo "Initial $2 in SRCS...<FAILED>" | tee -a $_LOG
		exit 1
	fi
	touch "$fver"
	if [ $? -gt 0 ];then
		echo "Initial $2 in SRCS...<FAILED>" | tee -a $_LOG
		exit 1
	fi
	echo "Initial $2 in SRCS...<SUCCESSED>" | tee -a $_LOG
	exit 0
elif [ $1 == "ci" ]; then
	##
	# check in a file in revision repository
	# this command will check in a new version of admin/sql/test.sql in revision repository: srcs.sh ci admin/sql/test.sql
	##
	echo "Checkin $2 in SRCS..." | tee -a $_LOG
	if [ ! -f "$fname" ]; then
		echo "$fname doesn't exist in local directory...<FAILED>" | tee -a $_LOG
		exit 1
	elif [ ! -f "$fver" ]; then
		echo "$fname doesn't exist in SRCS. Run init command to init it firstly...<FAILED>" | tee -a $_LOG
		exit 1
	else
		if egrep -v "^\s*$" $fver | wc -l | egrep "^0" > /dev/null ; then
			version=1
		else
			version=`tail -3 $fver | egrep "^version#" | sed 's/^version#//'`
			let version=1+$version
		fi
			echo "Input comment for this checkin: "
			read comment
			touch "${fname_srcs}"
			diff $fname $fname_srcs >> $fver
			echo "comment#$comment" >> $fver
			echo "version#$version" >> $fver
			echo "" >> $fver
			cp $fname $fname_srcs
			echo "Checkin $2 in SRCS...<SUCCESSED>" | tee -a $_LOG
	fi
elif [ $1 == "co" ]; then
	##
	# check out a file with a specified version from revision repository
	# this command will check out the latest version of admin/sql/test.sql from revision repository: srcs.sh co admin/sql/test.sql
	# this command will check out the v5 of admin/sql/test.sql from revision repository: srcs.sh co admin/sql/test.sql 5
	##
	echo "Checkout $2 $3 from SRCS..." | tee -a $_LOG
	if [ ! -f "$fver" ]; then
		echo "$fname doesn't exist in SRCS. Run init and ci command to version control it firstly...<FAILED>" | tee -a $_LOG
		exit 1
	else
		if egrep -v "^\s*$" $fver | wc -l | egrep "^0" > /dev/null ; then
			echo "$fname is just initialled in SRCS. Run ci command to version control it firstly...<FAILED>" | tee -a $_LOG
		elif ! cat $fver | egrep "^version#$3" > /dev/null ; then
			echo "Version $3 doesn't exist...<FAILED>" | tee -a $_LOG
			exit 1
		elif [ "$#" -lt 3 ]; then
			cp $fname_srcs $fname
			echo "Checkout $2 $3 from SRCS...<SUCCESSED>" | tee -a $_LOG
		else
			version=`tail -3 $fver | egrep "^version#" | sed 's/^version#//'`
			cp $fname_srcs $fname
			for (( v=$version; v>$3; v-- ))
			do
				let delta=$v-1
				_l1=`nl "$fver" | egrep "version#$delta" | sed "s/version#$delta//" | sed 's/^\s*|\s*$//g'`
				_l2=`nl "$fver" | egrep "version#$v" | sed "s/version#$v//" | sed 's/^\s*|\s*$//g'`
				sed -n "${_l1},${_l2}p" "$fver" | patch $fname -i - >> /dev/null
			done
			echo "Checkout $2 $3 from SRCS...<SUCCESSED>" | tee -a $_LOG
		fi
	fi
elif [ $1 == "log" ]; then
	##
	# print all the metadata for a file including a complete version history and the checkin comments from revision repository
	# this command will print the metadata for admin/sql/test.sql in revision repository: srcs.sh log admin/sql/test.sql
	##
	echo "Log for $2 from SRCS..." | tee -a $_LOG
	if [ ! -f "$fver" ]; then
		echo "$fname doesn't exist in SRCS...<FAILED>" | tee -a $_LOG
		exit 1
	else
		if egrep -v "^\s*$" $fver | wc -l | egrep "^0" > /dev/null ; then
			echo "$fname is just initialled in SRCS and has no revision...<FAILED>" | tee -a $_LOG
		else
			logs=`cat $fver | egrep "^version#|comment"`
			echo "$logs" | tee -a $_LOG
		fi
	fi
else
	##
	# when all options fail
	##
	echo "$1 is not a supported parameter in SRCS..." | tee -a $_LOG
	exit 1
fi

SRCS – diff

I am writing a script to simulate a Version Control System.
In SRCS – Check in file – It implements the initial and check in a file.
In SRCS – check out file – It implements check out a file with a specified version.
In SRCS – log – It implements a ‘log’ command to allow user to print all the metadata for a file.

In this post I will implement a diff command lists file changes.

For example,
The following command will provide a list of differences between file admin/sql/test1.sql revisions 5 and 6:
srcs diff admin/sql/test1.sql 5 6
The following example, you receive a list of differences between revision 5 and the local copy:
srcs diff admin/sql/test1.sql 5


_PROD="/u01/camel/12.0"
_DEBUG="on"
_LOG="_srcs.log"

fname_srcs="$_PROD/$2"
fname=$(basename "${fname_srcs}")
fdir=`echo "${fname_srcs}" | sed 's/'"$fname"'$//'`
fver="$fdir/RCS/${fname},v"
	
function DEBUG()
{
	[ "$_DEBUG" == "on" ] && $@ || :
}

function checkout()
{
	version=`tail -3 $fver | egrep "^version#" | sed 's/^version#//'`
	cp $fname_srcs ${fname}.tmp.$1
	for (( v=$version; v>$1; v-- ))
	do
		let delta=$v-1
		_l1=`nl "$fver" | egrep "version#$delta" | sed "s/version#$delta//" | sed 's/^\s*|\s*$//g'`
		_l2=`nl "$fver" | egrep "version#$v" | sed "s/version#$v//" | sed 's/^\s*|\s*$//g'`
		sed -n "${_l1},${_l2}p" "$fver" | patch ${fname}.tmp.$1 -i - >> /dev/null
	done
}

if [ $1 == "diff" ]; then
	echo "Diff $2 $3 $4 from SRCS..." | tee -a $_LOG
	if [ "$#" -lt 3 ]; then
		echo "Please provide at least one revision for diff...<FAILED>" | tee -a $_LOG
		exit 1
	fi
	if [ ! -f "$fver" ]; then
		echo "$fname doesn't exist in SRCS...<FAILED>" | tee -a $_LOG
		exit 1
	else
		if egrep -v "^\s*$" $fver | wc -l | egrep "^0" > /dev/null ; then 
			echo "$fname is just initialled in SRCS...<FAILED>" | tee -a $_LOG
			exit 1
		else
			if ! cat $fver | egrep "^version#$3" > /dev/null ; then
				echo "Version $3 doesn't exist...<FAILED>" | tee -a $_LOG
				exit 1
			else
				checkout $3
			fi
		fi
		if [ "$#" -lt 4 ]; then 	
			echo "Diff $2 $3 from SRCS with the local copy..." | tee -a $_LOG
			if [ ! -f "$fname" ]; then
				echo "$fname doesn't exist in local directory...<FAILED>" | tee -a $_LOG
				exit 1
			fi
			diff ${fname}.tmp.$3 $fname | tee -a $_LOG
			echo "Diff $2 $3 from SRCS with the local copy...<SUCCESSED>" | tee -a $_LOG
		else 
			if ! cat $fver | egrep "^version#$4" > /dev/null ; then
				echo "Version $4 doesn't exist...<FAILED>" | tee -a $_LOG
				exit 1
			fi
			checkout $4
			diff ${fname}.tmp.$3 ${fname}.tmp.$4 | tee -a $_LOG
			echo "Diff $2 $3 $4 from SRCS...<SUCCESSED>" | tee -a $_LOG
		fi
	fi
fi

Test Result:

-bash-3.2$ sh srcs.sh diff admin/sql/test2.sql
Diff admin/sql/test2.sql   from SRCS...
Please provide at least one revision for diff...<FAILED>
-bash-3.2$ sh srcs.sh diff admin/sql/test1.sql
Diff admin/sql/test1.sql   from SRCS...
Please provide at least one revision for diff...<FAILED>
-bash-3.2$ sh srcs.sh diff admin/sql/test2.sql
Diff admin/sql/test2.sql   from SRCS...
Please provide at least one revision for diff...<FAILED>
-bash-3.2$ sh srcs.sh diff admin/sql/test2.sql 3
Diff admin/sql/test2.sql 3  from SRCS...
test2.sql is just initialled in SRCS...<FAILED>
-bash-3.2$ sh srcs.sh diff admin/sql/test2.sql 3 5
Diff admin/sql/test2.sql 3 5 from SRCS...
test2.sql is just initialled in SRCS...<FAILED>
-bash-3.2$ sh srcs.sh diff admin/sql/test1.sql 6
Diff admin/sql/test1.sql 6  from SRCS...
Diff admin/sql/test1.sql 6 from SRCS with the local copy...
2,7c2,3
<  I am Version 6
< ahoghgoh
< gegoruaon
<  hello version6
< haxialg xn
< this line is new in version 6
---
> ### I am Version 1
> ### This line should be the same
Diff admin/sql/test1.sql 6 from SRCS with the local copy...<SUCCESSED>
-bash-3.2$ sh srcs.sh diff admin/sql/test1.sql 5
Diff admin/sql/test1.sql 5  from SRCS...
Diff admin/sql/test1.sql 5 from SRCS with the local copy...
2,5c2,3
<  I am Version 5
<  hello version5
< haxialg xn
< this line is new in version 5
---
> ### I am Version 1
> ### This line should be the same
Diff admin/sql/test1.sql 5 from SRCS with the local copy...<SUCCESSED>
-bash-3.2$ sh srcs.sh diff admin/sql/test1.sql 53
Diff admin/sql/test1.sql 53  from SRCS...
Version 53 doesn't exist...<FAILED>
-bash-3.2$ sh srcs.sh diff admin/sql/test1.sql 3
Diff admin/sql/test1.sql 3  from SRCS...
Diff admin/sql/test1.sql 3 from SRCS with the local copy...
2,4c2,3
<  I am Version 3
< ### This line should be the different
< this line is new in version 3
---
> ### I am Version 1
> ### This line should be the same
Diff admin/sql/test1.sql 3 from SRCS with the local copy...<SUCCESSED>
-bash-3.2$ sh srcs.sh diff admin/sql/test1.sql 2
Diff admin/sql/test1.sql 2  from SRCS...
Diff admin/sql/test1.sql 2 from SRCS with the local copy...
2c2
< ### I am Version 2
---
> ### I am Version 1
4d3
< this line is new in version 2
Diff admin/sql/test1.sql 2 from SRCS with the local copy...<SUCCESSED>
-bash-3.2$ sh srcs.sh diff admin/sql/test1.sql 1
Diff admin/sql/test1.sql 1  from SRCS...
Diff admin/sql/test1.sql 1 from SRCS with the local copy...
Diff admin/sql/test1.sql 1 from SRCS with the local copy...<SUCCESSED>
-bash-3.2$ sh srcs.sh diff admin/sql/test1.sql 6 4
Diff admin/sql/test1.sql 6 4 from SRCS...
2,7c2,4
<  I am Version 6
< ahoghgoh
< gegoruaon
<  hello version6
< haxialg xn
< this line is new in version 6
---
>  I am Version  4
>  hello version4
> this line is new in version 4
Diff admin/sql/test1.sql 6 4 from SRCS...<SUCCESSED>
-bash-3.2$

SRCS – log

I am writing a script to simulate a Version Control System.
In SRCS – Check in file – It implements the initial and check in a file.
In SRCS – check out file – It implements check out a file with a specified version.

In this post I will implement a ‘log’ command to allow user to print all the metadata for a file, including a complete version history and the checkin comments.


_PROD="/u01/camel/12.0"
_DEBUG="on"
_LOG="_srcs.log"

fname_srcs="$_PROD/$2"
fname=$(basename "${fname_srcs}")
fdir=`echo "${fname_srcs}" | sed 's/'"$fname"'$//'`
fver="$fdir/RCS/${fname},v"

function DEBUG()
{
	[ "$_DEBUG" == "on" ] && $@ || :
}

if [ $1 == "log" ]; then
	echo "Log for $2 from SRCS..." | tee -a $_LOG
	if [ ! -f "$fver" ]; then
		echo "$fname doesn't exist in SRCS...<FAILED>" | tee -a $_LOG
		exit 1
	else
		if egrep -v "^\s*$" $fver | wc -l | egrep "^0" > /dev/null ; then
			echo "$fname is just initialled in SRCS and has no revision...<FAILED>" | tee -a $_LOG
		else
			logs=`cat $fver | egrep "^version#|comment"`
			echo "$logs" | tee -a $_LOG
		fi
	fi
fi

Test result:

-bash-3.2$ sh srcs.sh log admin/sql/test2.sql
Log for admin/sql/test2.sql from SRCS...
test2.sql is just initialled in SRCS and has no revision...<FAILED>
-bash-3.2$ sh srcs.sh log admin/sql/test3.sql
Log for admin/sql/test3.sql from SRCS...
test3.sql doesn't exist in SRCS...<FAILED>
-bash-3.2$ sh srcs.sh log admin/sql/test1.sql
Log for admin/sql/test1.sql from SRCS...
comment#initial checkin for test1.sql
version#1
comment#hello version 2
version#2
comment#okie, version 3
version#3
comment#i am version 4.
version#4
comment#hello hello version 5
version#5
comment#version 6 here...
version#6
-bash-3.2$

MISC – May 19th, 2014

1# nl – number lines of files. Write each FILE to standard output, with line numbers added.  With no FILE, or when FILE is -, read standard input.

-bash-3.2$ cat file1
This is file 1
I am a file 1
-bash-3.2$ nl file1
1 This is file 1
2 I am a file 1
-bash-3.2$

2# sed -n “$start,${end}p” filename
Output certain lines from a file

-bash-3.2$ nl file2
     1  This is file 2
     2  I am a file 2
     3  line 3 - 123
     4  line 4 - 123
     5  line 5 - 123
-bash-3.2$ sed -n '2,4p' file2
I am a file 2
line 3 - 123
line 4 - 123
-bash-3.2$

3# In Shell, if you want to pass variables into a pattern, the best practice is to use a double quotes instead of the single ones, so the $variables can be expanded.

-bash-3.2$ export start=2
-bash-3.2$ export end=4
-bash-3.2$ sed -n '$start,${end}p' file2
sed: -e expression #1, char 14: unterminated `s' command
-bash-3.2$ sed -n "$start,${end}p" file2
I am a file 2
line 3 - 123
line 4 - 123
-bash-3.2$

4# Three typical ways to do for in Shell,

for (( v=10; v>1; v-- ))
do

done

for i in {1..5}
do
   echo "Welcome $i times"
done

for f in $FILES
do

done

5# Output all of the lines except the last N lines.

head -n -N file
-bash-3.2$ nl file2
1 This is file 2
2 I am a file 2
3 line 3 - 123
4 line 4 - 123
5 line 5 - 123
-bash-3.2$ head -n -2 file2
This is file 2
I am a file 2
line 3 - 123
-bash-3.2$
6# output all of the lines except the first N lines.
<pre>
tail -n +(N+1) file
-bash-3.2$ nl file2
1 This is file 2
2 I am a file 2
3 line 3 - 123
4 line 4 - 123
5 line 5 - 123
-bash-3.2$ tail -n +3 file2
line 3 - 123
line 4 - 123
line 5 - 123
-bash-3.2$

SRCS – check out file

I just started up writing a script to simulate a basic Source Version Control tool. In http://luohuahuang.com/2014/05/17/srcs-startup/ I prepared an experimental script to do initial and checkin activities. Now in this post I will try the check out function.
Basically the check out function will support:
1# If the version is the latest, just copy the latest from repository to local.
2# otherwise check out the specified version.

I will use Linux patch command to apply the diff content to the latest file to get the specified version.
The logic:
1# copy the latest file to local.
2# run patch command to apply diff in a loop way until it reaches the specified version. For example,

-bash-3.2$ cat 2.sql
hello world
no wrong v4
new line in v4
select * from table2;
-bash-3.2$ vi diff.txt
-bash-3.2$ cat diff.txt | patch 2.sql -i -
patching file 2.sql
-bash-3.2$ cat 2.sql
hello world
no wrong
new line in v3
select * from table2;

Note:
-i patchfile or –input=patchfile
Read the patch from patchfile. If patchfile is -, read from standard input, the default.

So in the loop, I will extract the diff content from filename,v and output it into standard input for patch command.

The logic:

1. if it checks out a file doesn’t have history in repository. Failed.

2. if it checks out the file without specifying a version or with the latest version. Check out the latest version.

3. if it chceks out the file by specifying a version. If the version doesn’t exist, Failed. Otherwise check out the specified version.

 


_PROD="/u01/camel/12.0"
_DEBUG="on"
_LOG="_srcs.log"

fname_srcs="$_PROD/$2"
fname=$(basename "${fname_srcs}")
fdir=`echo "${fname_srcs}" | sed 's/'"$fname"'$//'`
fver="$fdir/RCS/${fname},v"

function DEBUG()
{
	[ "$_DEBUG" == "on" ] && $@ || :
}

if [ $1 == "co" ]; then
	echo "Checkout $2 $3 from SRCS..." | tee -a $_LOG
	if [ ! -f "$fver" ]; then
		echo "$fname doesn't exist in SRCS. Run init and ci command to version control it firstly...<FAILED>" | tee -a $_LOG
		exit 1
	else
		if egrep -v "^\s*$" $fver | wc -l | egrep "^0" > /dev/null ; then
			echo "$fname is just initialled in SRCS. Run ci command to version control it firstly...<FAILED>" | tee -a $_LOG
		elif ! cat $fver | egrep "^version#$3" > /dev/null ; then
			echo "Version $3 doesn't exist...<FAILED>" | tee -a $_LOG
			exit 1
		elif [ "$#" -lt 3 ]; then
			cp $fname_srcs $fname
			echo "Checkout $2 $3 from SRCS...<SUCCESSED>" | tee -a $_LOG
		else
			version=`tail -3 $fver | egrep "^version#" | sed 's/^version#//'`
			cp $fname_srcs $fname
			for (( v=$version; v>$3; v-- ))
			do
				let delta=$v-1
				_l1=`nl "$fver" | egrep "version#$delta" | sed "s/version#$delta//" | sed 's/^\s*|\s*$//g'`
				_l2=`nl "$fver" | egrep "version#$v" | sed "s/version#$v//" | sed 's/^\s*|\s*$//g'`
				sed -n "${_l1},${_l2}p" "$fver" | patch $fname -i - >> /dev/null
			done
			echo "Checkout $2 $3 from SRCS...<SUCCESSED>" | tee -a $_LOG
		fi
	fi
fi

Test data:

Below data is recorded by check in script posted in http://luohuahuang.com/2014/05/17/srcs-startup/ in the filename,v file. It basically records all of the diff changes.
####################################################################

### This is test1.sql file
### I am Version 1
### This line should be the same
###

comment: initial checkin for test1.sql

####################################################################

### This is test1.sql file
### I am Version 2
### This line should be the same
this line is new in version 2
###

comment: hello version 2

####################################################################

### This is test1.sql file
I am Version 3
### This line should be the different
this line is new in version 3
###

comment: okie, version 3

####################################################################

### This is test1.sql file
I am Version 4
hello version4
this line is new in version 4
###

comment: i am version 4.

####################################################################

### This is test1.sql file
I am Version 5
hello version5
haxialg xn
this line is new in version 5
###

comment: hello hello version 5

####################################################################

### This is test1.sql file
I am Version 6
ahoghgoh
gegoruaon
hello version6
haxialg xn
this line is new in version 6
###

version 6 here…

Test Result:

-bash-3.2$ sh srcs.sh co admin/sql/test1.sql 4
Checkout admin/sql/test1.sql 4 from SRCS…
Checkout admin/sql/test1.sql 4 from SRCS…<SUCCESSED>
-bash-3.2$ cat test1.sql
### This is test1.sql file
I am Version 4
hello version4
this line is new in version 4
###
-bash-3.2$ rm test1.sql
-bash-3.2$ sh srcs.sh co admin/sql/test1.sql 2
Checkout admin/sql/test1.sql 2 from SRCS…
Checkout admin/sql/test1.sql 2 from SRCS…<SUCCESSED>
-bash-3.2$ cat test1.sql
### This is test1.sql file
### I am Version 2
### This line should be the same
this line is new in version 2
###
-bash-3.2$ sh srcs.sh co admin/sql/test1.sql 1
Checkout admin/sql/test1.sql 1 from SRCS…
Checkout admin/sql/test1.sql 1 from SRCS…<SUCCESSED>
-bash-3.2$ cat test1.sql
### This is test1.sql file
### I am Version 1
### This line should be the same
###
-bash-3.2$ sh srcs.sh co admin/sql/test1.sql 0
Checkout admin/sql/test1.sql 0 from SRCS…
Version 0 doesn’t exist…<FAILED>

From the test result, the check out script is working!

So far, the scripts can simulate below version control functions:
1. initial a file in the version repository.
2. check in a file into the repository and the repository records the changes.
3. check out a file by specifying a version.
4. It is only for text-based files.

 

Todo list:

1. Lock and unlock
2. Access control
3. branching
4. .etc

I should complete this in later posts.

SRCS – Check in file

Today I got some spare time so I tried to simulate a basic Source Versions Control Tool by using linux patch command. Basically my target is, the tool could be able to handle version control on a standalone env. for a single user, like check in a file, check out a file, check out a file with dedicated version, lock and unlock, log review, etc. Of course, multi-users and CS mode will be supported in future.

This afternoon I just tried to figure out how patch command work. Basically,

Let’s see an example of patch command,

-bash-3.2$ cat file1
This is file 1
I am a file 1
-bash-3.2$ cat file2
This is file 1
This is file 2
I am a file  2
-bash-3.2$ diff file1 file2 > diff.txt
-bash-3.2$ cat diff.txt
2c2,3
< I am a file 1
---
> This is file 2
> I am a file  2
-bash-3.2$ rm file2
-bash-3.2$ cp file1 file2
-bash-3.2$ patch file2 < diff.txt
patching file file2
-bash-3.2$ cat file2
This is file 1
This is file 2
I am a file  2
-bash-3.2$

Currently I just composed an experimental script. The script can do below thing:
1. for a new file, run ‘srcs init file’ to initial a placeholder in the repository.
2. check in a new file and note down the diff in the RCS/file,v file.

For example, for new file admin/sql/test/1.sql, I run ‘srcs init admin/sql/test/1.sql’ it will help me create a control file under $_PROD/admin/sql/test/RCS/1.sql,v.


_PROD="/u01/camel/12.0"
_DEBUG="on"
_LOG="_srcs.log"

fname_srcs="$_PROD/$2"
fname=$(basename "${fname_srcs}")
fdir=`echo "${fname_srcs}" | sed 's/'"$fname"'$//'`
fver="$fdir/RCS/${fname},v"

function DEBUG()
{
	[ "$_DEBUG" == "on" ] && $@ || :
}

if [ $1 == "init" ]; then
	date | tee -a $_LOG
	echo "Initial $2 in SRCS..." | tee -a $_LOG
	mkdir -p "$fdir/RCS"
	if [ -f "$fver" ]; then
		echo "$2 exists in SRCS already...<FAILED>" | tee -a $_LOG
		echo "Initial $2 in SRCS...<FAILED>" | tee -a $_LOG
		exit 1
	fi
	touch "$fver"
	if [ $? -gt 0 ];then
		echo "Initial $2 in SRCS...<FAILED>" | tee -a $_LOG
		exit 1
	fi
	echo "Initial $2 in SRCS...<SUCCESSED>" | tee -a $_LOG
	exit 0
fi

if [ $1 == "ci" ]; then
	echo "Checkin $2 in SRCS..." | tee -a $_LOG
	if [ ! -f "$fname" ]; then
		echo "$fname doesn't exist in local directory...<FAILED>" | tee -a $_LOG
		exit 1
	elif [ ! -f "$fver" ]; then
		echo "$fname doesn't exist in SRCS. Run init command to init it firstly...<FAILED>" | tee -a $_LOG
		exit 1
	else
		if egrep -v "^\s*$" $fver | wc -l | egrep "^0" > /dev/null ; then
			version=1
		else
			version=`tail -3 $fver | egrep "^version#" | sed 's/^version#//'`
			let version=1+$version
		fi
			echo "Input comment for this checkin: "
			read comment
			touch "${fname_srcs}"
			diff $fname $fname_srcs >> $fver
			echo "comment#$comment" >> $fver
			echo "version#$version" >> $fver
			cp $fname $fname_srcs
			echo "Checkin $2 in SRCS...<SUCCESSED>" | tee -a $_LOG
	fi
fi

Output:

-bash-3.2$ srcs.sh ci admin/sql/test1/2.sql
Chekcin admin/sql/test1/2.sql in SRCS...
2.sql doesn't exist in SRCS. Run init command to init it firstly...<FAILED>
-bash-3.2$ srcs.sh init admin/sql/test1/2.sql
Sat May 17 01:17:40 PDT 2014
Initial admin/sql/test1/2.sql in SRCS...
Initial admin/sql/test1/2.sql in SRCS...<SUCCESSED>
-bash-3.2$ srcs.sh ci admin/sql/test1/2.sql
Chekcin admin/sql/test1/2.sql in SRCS...
Input comment for this checkin:
this is the 1st checin for 2.sql. Good luck
Chekcin admin/sql/test1/2.sql in SRCS...<SUCCESSED>
-bash-3.2$ cat /u01/camel/12.0/admin/sql/test1/2.sql
hello world
yes correct
select * from table2;
-bash-3.2$ cat /u01/camel/12.0/admin/sql/test1/RCS/2.sql,v
1,3d0
< hello world
< yes correct
< select * from table2;
comment#this is the 1st checin for 2.sql. Good luck
version#1
-bash-3.2$ vi 2.sql
-bash-3.2$ srcs.sh ci admin/sql/test1/2.sql
Chekcin admin/sql/test1/2.sql in SRCS...
Input comment for this checkin:
okie v2 is coming.
Chekcin admin/sql/test1/2.sql in SRCS...<SUCCESSED>
-bash-3.2$ cat /u01/camel/12.0/admin/sql/test1/2.sql
hello world
no wrong
new line in v2
select * from table2;
-bash-3.2$ cat /u01/camel/12.0/admin/sql/test1/RCS/2.sql,v
1,3d0
< hello world
< yes correct
< select * from table2;
comment#this is the 1st checin for 2.sql. Good luck
version#1
2,3c2
< no wrong
< new line in v2
---
> yes correct
comment#okie v2 is coming.
version#2

In the coming days, I will implement check out files with dedicated versions and add access control to it.

Script to monitor tablespace usage and send alert

This script can be used to connect to database, check its tablesapce at regular intervals and send email if it finds any tablespace usage > the threshold.

#! /usr/local/bin/perl 

use POSIX;
use Time::localtime;

################################################################################
## TABLESPACE_NAME, TOTAL_GB, USED_GB, FREE_GB, PCT_USED.
################################################################################
while(1){
my $SQL = "init_tb_pct.sql";
system("rm init_tb_pct.sql");
#change this loc to your dir and ensure it is writable
my $report_loc = "/home/luhuang/public_html/upload/STQA_TB_Report";
my $datestr = GetDate();
my $filename = "STQA_TB_Reprt_" . "$datestr" . ".txt";
my $REPORT = "$report_loc" . "/" . "$filename";

my $log = "I_am_running.log";
system("echo 0 >> $log");

open (SQLS, ">$SQL");
print SQLS "set feedback off\n";
print SQLS "set pages 1000\n";
print SQLS "spool $REPORT \n";
print SQLS "select  a.tablespace_name Tablespace_Name,(sum(b.bytes)/count( distinct a.file_id||'.'||a.block_id))/1024/1024/1024  Total_GB,(sum(b.bytes)/count( distinct a.file_id||'.'||a.block_id) -  sum(a.bytes)/count( distinct b.file_id ))/1024/1024/1024 Used_GB,(sum(a.bytes)/count( distinct b.file_id ))/1024/1024/1024  Free_GB,100 * ( (sum(b.bytes)/count( distinct a.file_id||'.'||a.block_id )) - (sum(a.bytes)/count( distinct b.file_id ) )) / (sum(b.bytes)/count( distinct a.file_id||'.'||a.block_id)) PCT_USED from sys.dba_free_space a, sys.dba_data_files b where a.tablespace_name = b.tablespace_name group by a.tablespace_name, b.tablespace_name;\n";
print SQLS "spool off\n";
print SQLS "/\n";
print SQLS "exit;\n";
close SQLS ;
#update your connection string here
my $cmd = "sqlplus system/xxx\@'(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = slcxxx.us.xxxx.com)(PORT = 1520)))(CONNECT_DATA =(SERVICE_NAME = xxfdb )(INSTANCE_NAME = xxxb )))' \@$SQL";
system("$cmd");

open (SQLS, "$REPORT");
my @lines = <SQLS>;
my $count = 0;
my $trigger = 0;
foreach $line (@lines) {
	if ($count > 1){
		$line =~ s/^\s+|\s+$//g;
		if (length($line) > 2) {
			my ($TABLESPACE_NAME, $TOTAL_GB, $USED_GB, $FREE_GB, $PCT_USED) = split(/\s+/, $line);
			if (($PCT_USED > 90) && ($trigger < 1)) {
				send_mail($filename);
				$trigger = 1;
			}
		}
	}
	$count = $count + 1;
}

close SQLS ;
if($trigger eq 0){
	system("rm $REPORT");
}
sleep 900;
}

sub GetDate {
    my $tm                   = localtime;
    my ($year, $month, $day) = ($tm->year+1900, ($tm->mon)+1, $tm->mday);
    my ($hour, $min, $sec)   = ($tm->hour, $tm->min, $tm->sec);

    my $date = sprintf("%4s%2s%2s", ${year},${month}, ${day});
    my $time = sprintf("[%2s:%2s:%2s]", ${hour}, ${min}, ${sec});
    my $date_time = sprintf("%4s%2s%2s%2s%2s%2s", ${year},${month}, ${day}, ${hour}, ${min}, ${sec});
    $date =~ s/ /0/g;
    $time =~ s/ /0/g;
    $date_time =~ s/ /0/g;

    return $date_time;
}

sub send_mail
{
(my $filename) = @_;
$to1="luohua.huang\@gmail.com";
open(MAIL,'|/usr/lib/sendmail -t -f $FROM');
print MAIL "Mime-Version: 1.0\n";
print MAIL "Content-type: text/html\; charset=\"us-ascii\"\n";
print (MAIL "To: ${to1} \r\n");
print MAIL "Subject: <Action Required> Tablesapce Report \n";
print MAIL "Customize your report here";
close (MAIL);

}

 

Scripts to Update WLS DataSource for EBS 12.2.x

On EBS R12.2 we need to reconfigure the EBS datasource if we change the password of apps schema. Its’ sequences,

1. Change system password,

2. Shutdown mid-tier services,

3. Change product schema password,

4. Change applsys password,

5. Startup WLS adadmin console only,

6. Logon WLS console UI to reconfigure its datasource,

7. Startup oacore managed server,

8. Startup all the mid-tier services.

 

With the help of WebLogic wlst script, I wrote a script to automate this.


# | DESCRIPTION
# | This .sh script will take care of steps described on xxxx
# | You might update the passwords parameters defined below to meet your requirement before running this script.
# |
# | USAGE
# | sh change_apps_pwd.sh
# |
# | PLATFORM
# | EBS R12.2.X
# |
# | CONTACT
# | luohua.HUANG@gmail.com

# ======================================================================
# Set Parameters
# ======================================================================
printf "Enter your ORACLE_SID (e.g: aux123db): "
read ORACLE_SID

APPS_OLD="apps"

APPS_NEW="newapps123"
SYSTEM_NEW="newmanager123"
PRODUCT_NEW="newebs123"

WEBLOGIC_PWD="welcome1"

RUN_FS="/u01/R122_EBS/fs1"
ORACLE_HOME="/u01/R122_EBS/11.2.0"

HOST_NAME=`hostname -a`
FULL_HOST_NAME=`hostname -f`

WORKING_DIR=`pwd`
LOGFILE="${WORKING_DIR}/change_apps_pwd.log"

# ======================================================================
# Source db tier and update system password
# ======================================================================
source ${ORACLE_HOME}/${ORACLE_SID}_${HOST_NAME}.env
printf "********* Changing SYSTEM password ********* \n";
DBSHUTDOWN=`sqlplus '/as sysdba'<<!
 alter user SYSTEM identified by ${SYSTEM_NEW};
exit
!`
printf "$DBSHUTDOWN \n" >> $LOGFILE;

# ======================================================================
# Source Apps tier
# ======================================================================
source ${RUN_FS}/EBSapps/appl/APPS${ORACLE_SID}_${HOST_NAME}.env
PORT=`grep s_wls_adminport $CONTEXT_FILE | sed "s/.*\">//" | sed "s/<.*//"`

# ======================================================================
# Stop mid-tier services
# ======================================================================
printf "********* Running adstpall ********* \n";
{ echo apps; echo ${APPS_OLD}; echo ${WEBLOGIC_PWD}; } | sh adstpall.sh -nopromptmsg >> $LOGFILE;
while ps -ef | grep -i fndlib | grep -v "grep" > /dev/null
do
 sleep 30
 printf "********* Waiting for mid-tier services come down ********* \n";
done

# ======================================================================
# Change Product Schema Passowrd
# ======================================================================
printf "********* Changing Product Schema Passowrd ********* \n";
FNDCPASS apps/${APPS_OLD} 0 Y system/${SYSTEM_NEW} ALLORACLE ${PRODUCT_NEW} >> $LOGFILE
STATUS=$?
if [ $STATUS -gt 0 ];then
 printf "********* Failed to run FNDCPASS apps/${APPS_OLD} 0 Y system/${SYSTEM_NEW} ALLORACLE ${PRODUCT_NEW} ********* \n";
 exit 1;
fi

# ======================================================================
# Change APPS Schema Passowrd
# ======================================================================
printf "********* Changing APPS Schema Passowrd ********* \n";
FNDCPASS apps/${APPS_OLD} 0 Y system/${SYSTEM_NEW} SYSTEM APPLSYS ${APPS_NEW} >> $LOGFILE
STATUS=$?
if [ $STATUS -gt 0 ];then
 printf "********* Failed to run FNDCPASS apps/${APPS_OLD} 0 Y system/${SYSTEM_NEW} SYSTEM APPLSYS ${APPS_NEW} ********* \n";
 exit 1;
fi

# ======================================================================
# Start WLS adadmin console
# ======================================================================
printf "********* Starting WLS adadmin console only ********* \n";
{ echo ${WEBLOGIC_PWD}; echo ${APPS_NEW}; } | sh adadminsrvctl.sh start -nopromptmsg >> $LOGFILE;
STATUS=$?
if [ $STATUS -gt 0 ];then
 printf "********* Failed to startup WLS adadmin console ********* \n";
 exit 1;
fi
sleep 30

# ======================================================================
# Change Datasource password on WLS adadmin console
# ======================================================================
printf "********* Changing Datasource password on WLS adadmin console ********* \n";
rm -f updateDSpwd.py
printf "username = 'weblogic' \n" >> updateDSpwd.py;
printf "password = '${WEBLOGIC_PWD}' \n" >> updateDSpwd.py;
printf "URL='t3://$FULL_HOST_NAME:$PORT' \n" >> updateDSpwd.py;
printf "connect(username,password,URL) \n" >> updateDSpwd.py;
printf "edit() \n" >> updateDSpwd.py;
printf "startEdit() \n" >> updateDSpwd.py;
printf "en = encrypt('${APPS_NEW}','$FMW_HOME/user_projects/domains/EBS_domain_${ORACLE_SID}') \n" >> updateDSpwd.py;
printf "dsName = 'EBSDataSource' \n" >> updateDSpwd.py;
printf "cd('/JDBCSystemResources/'+dsName+'/JDBCResource/'+dsName+'/JDBCDriverParams/'+dsName) \n" >> updateDSpwd.py;
printf "set('PasswordEncrypted',en) \n" >> updateDSpwd.py;
printf "print ('') \n" >> updateDSpwd.py;
printf "print ('') \n" >> updateDSpwd.py;
printf "save() \n" >> updateDSpwd.py;
printf "activate() \n" >> updateDSpwd.py;
java -cp $FMW_HOME/wlserver_10.3/server/lib/weblogic.jar weblogic.WLST updateDSpwd.py >> $LOGFILE;
STATUS=$?
if [ $STATUS -gt 0 ];then
 printf "********* Failed to change Datasource password on WLS adadmin console ********* \n";
 exit 1;
fi

# ======================================================================
# Startup oacore_server1
# ======================================================================
printf "********* Startup-ing oacore_server1 ********* \n";
{ echo ${WEBLOGIC_PWD};} | sh admanagedsrvctl.sh start oacore_server1 -nopromptmsg >> $LOGFILE;
STATUS=$?
if [ $STATUS -gt 0 ];then
 printf "********* Failed to startup oacore_server1 ********* \n";
 exit 1;
fi

# ======================================================================
# Startup ALL services
# ======================================================================
printf "********* Startup-ing ALL services ********* \n";
{ echo apps; echo ${APPS_NEW}; echo ${WEBLOGIC_PWD}; } | sh adstrtal.sh -nopromptmsg >> $LOGFILE;
STATUS=$?
if [ $STATUS -gt 0 ];then
 printf "********* Failed to startup ALL services ********* \n";
 exit 1;
fi

# ======================================================================
# Connect to WLS console to check managed servers statues
# ======================================================================
rm -f serverStateAll.py
printf "username = 'weblogic' \n" >> serverStateAll.py;
printf "password = '${WEBLOGIC_PWD}' \n" >> serverStateAll.py;
printf "URL='t3://$FULL_HOST_NAME:$PORT' \n" >> serverStateAll.py;
printf " \n" >> serverStateAll.py;
printf "connect(username,password,URL) \n" >> serverStateAll.py;
printf "domainConfig() \n" >> serverStateAll.py;
printf "serverList=cmo.getServers(); \n" >> serverStateAll.py;
printf "domainRuntime() \n" >> serverStateAll.py;
printf "cd('/ServerLifeCycleRuntimes/') \n" >> serverStateAll.py;
printf "\n" >> serverStateAll.py;
printf "print 'Servers Status on ' +URL \n" >> serverStateAll.py;
printf "for server in serverList: \n" >> serverStateAll.py;
printf " name=server.getName() \n" >> serverStateAll.py;
printf " cd(name) \n" >> serverStateAll.py;
printf " serverState=cmo.getState() \n" >> serverStateAll.py;
printf " if serverState!='RUNNING': \n" >> serverStateAll.py;
printf " print '**** FoundBadServer ****' \n" >> serverStateAll.py;
printf " print '***Server: '+ name +'-'+serverState \n" >> serverStateAll.py;
printf " break \n" >> serverStateAll.py;
printf " print '***Server: '+ name +'-'+serverState \n" >> serverStateAll.py;
printf " cd('..') \n" >> serverStateAll.py;
wlsstatus=`java -cp $FMW_HOME/wlserver_10.3/server/lib/weblogic.jar weblogic.WLST serverStateAll.py`
printf "$wlsstatus \n" >> $LOGFILE;
if [[ "$wlsstatus" =~ "FoundBadServer" ]]; then
 printf "********* FOUND managed server(s) NOT in RUNNING status ********* \n";
 exit 1;
fi
printf "********* ALL services are UP ********* \n";

printf "********* Done! ********* \n";

One Sample log,

-bash-3.2$ sh change_apps_pwd.sh
Enter your ORACLE_SID (e.g: aux123db): yourdbid (Luohua: here you need to input sid)
********* Changing SYSTEM password *********
********* Running adstpall *********
********* Waiting for mid-tier services come down *********
********* Waiting for mid-tier services come down *********
********* Changing Product Schema Passowrd *********
Log filename : L7595369.log
Report filename : O7595369.out
********* Changing APPS Schema Passowrd *********
Log filename : L7595370.log
Report filename : O7595370.out
********* Starting WLS adadmin console only *********
********* Changing Datasource password on WLS adadmin console *********
********* Startup-ing oacore_server1 *********
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u01/R122_EBS/fs1/inst/apps/yourdbid_yourhostname/logs/appl/rgf/TXK/txkChkEBSDependecies_Sat_Apr_5_17_33_44_2014/txkChkEBSDependecies_Sat_Apr_5_17_33_44_2014.log
********* Startup-ing ALL services *********
********* ALL services are UP *********
********* Done! *********
-bash-3.2$

 

How to sudo to another user to run command in perl

Scnario:

You need to run the script from User ABC and in the script you have to sudo to User DEF to run command.

Net::SSH::Perl enables you to simply and securely execute commands on remote machines, and receive the STDOUT, STDERR, and exit status of that remote command. It contains built-in support for various methods of authenticating with the server (password authentication, RSA challenge-response authentication, etc.). It completely implements the I/O buffering, packet transport, and user authentication layers of the SSH protocol, and makes use of external Perl libraries (in the Crypt:: family of modules) to handle encryption of all data sent across the insecure network

http://search.cpan.org/~schwigon/Net-SSH-Perl-1.35/lib/Net/SSH/Perl.pm

Solution:

## The Script is run with user ABC 
## Here you will sudo to user DEF (login as 'DEF' user)
print "Connecting as DEF user...\n";
my $ssh=Net::SSH::Perl->new($yourhost, debug=>1, protocol=>'1,2');
$ssh->login("DEF",$DEFpasswd);
## as 'DEF' user to provide 777 permission to directories
$ssh->cmd("mkdir -p /u01/def/");
$ssh->cmd("touch "/u01/def/def.txt);
## here it will return back to user ABC

Pls. note, you need to use below directive,

use Net::SSH::Perl;