Build Pipeline Plugin for Jenkins

Today I am going to look into how Jenkins deploy jobs as a pipeline. Build Pipeline Plugin renders upstream and downstream connected jobs that typically form a build pipeline. In addition, it offers the ability to define manual triggers for jobs that require intervention prior to execution, e.g. an approval process outside of Jenkins.

In my earlier post Run JJB to define jobs in Jenkins, I have demonstrated how to deploy jobs automatically by using Jenkins-Jobs-Builder plugin. Now, supposed developers like the jobs you deployed via JJB and want to ‘copy’ the same for their branches builds. By utilizing Build Pipeline Plugin, release engineer can streamline the jobs as pipeline and publish the pipeline definition as pipeline templates.

Before we go further, let’s see how Build Pipeline Plugin works?

1. Install Build Pipeline Plugin.

2.   Renders the upstream and downstream. After that, review the file system jobs/

drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_DeployQA
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_DeployStage
drwxr-xr-x 5 oracle dba 4096 Jul 3 2014 master_clean
drwxr-xr-x 4 oracle dba 4096 Jul 3 2014 master_build
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_code-analysis
drwxr-xr-x 4 oracle dba 4096 Jul 3 2014 master_functional-test
drwxr-xr-x 4 oracle dba 4096 Jul 3 2014 master_unit-test
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_package
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_sonar
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_generate-doc

On the dashboard

master1

From Jenkins config.xml


<au.com.centrumsystems.hudson.plugin.buildpipeline.BuildPipelineView plugin="build-pipeline-plugin@1.4.3">
<owner class="hudson" reference="../../.."/>
<name>master</name>
<filterExecutors>false</filterExecutors>
<filterQueue>false</filterQueue>
<properties class="hudson.model.View$PropertyList"/>
<gridBuilder class="au.com.centrumsystems.hudson.plugin.buildpipeline.DownstreamProjectGridBuilder">
<firstJob>master_clean</firstJob>
</gridBuilder>
<noOfDisplayedBuilds>3</noOfDisplayedBuilds>
<buildViewTitle/>
<consoleOutputLinkStyle>Lightbox</consoleOutputLinkStyle>
<cssUrl/>
<triggerOnlyLatestJob>false</triggerOnlyLatestJob>
<alwaysAllowManualTrigger>false</alwaysAllowManualTrigger>
<showPipelineParameters>false</showPipelineParameters>
<showPipelineParametersInHeaders>false</showPipelineParametersInHeaders>
<startsWithParameters>false</startsWithParameters>
<refreshFrequency>3</refreshFrequency>
<showPipelineDefinitionHeader>false</showPipelineDefinitionHeader>
</au.com.centrumsystems.hudson.plugin.buildpipeline.BuildPipelineView>

Ok. Let’s go to templatize the pipeline.

The templatize steps should be:

1. Get names of branches from remote GIT repository.

git clone ${GIT_URL}

`git branch -r | tail -n +2 | sed 's/^\s*origin\///' | sed 's/\s*$//' | grep -v master`

2.  foreach branch name, have a copy of the master_* directories under jobs/

3. for each new copied job, update the job config.xml file

4. Create the new pipeline in jenkins’ config.ml

The script:

#.jenkins/war/WEB-INF
FULL_HOST_NAME=`hostname -f`
HTTP_PORT="8080"
JENKINS_WEB="$HOME/.jenkins"
GIT_URL="https://github.com/velniukas/helloworld.git"

#shutdown jenkins peacefully
cd ${JENKINS_WEB}/war/WEB-INF
java -jar jenkins-cli.jar -s http://${FULL_HOST_NAME}:${HTTP_PORT}/ safe-shutdown

cd ${JENKINS_WEB}/jobs/
#git config --global http.proxy http://www-proxy.xxx.xxx.com:80
#first clone the repo if it hasn't yet
git clone ${GIT_URL}
#foreach branch
for b in `git branch -r | tail -n +2 | sed 's/^\s*origin\///' | sed 's/\s*$//' | grep -v master`; do
 #foreach jobs/master_*
 for d in `ls master* | egrep ':$' | sed 's/://'`; do
 z=`echo $d | sed "s/master/$b/"`
 cp -r $d $z
 #reset nextBuildNumber
 echo 1 > $z/nextBuildNumber
 #remove historic builds log in 'master' pipeline - E.g, jobs/xxx/builds/2014-07-03_02-04-05
 rm -fr $z/builds/*-*-*_*
 #reset lastFailedBuild, lastStableBuild, lastSuccessfulBuild, lastUnstableBuild, lastUnsuccessfulBuild
 for f in `find $z/builds/ -type f`; do
 echo -1 > $f
 done
 #re-configure config.xml file for the new job
 sed -i "s/master/$b/g" $z/config.xml
 done
 #backup jenkins config.xml
 cp ${JENKINS_WEB}/config.xml ${JENKINS_WEB}/"config.xml_$(date +%F_%R)"
 #get the first line number of 'master' pipeline view
 fln=`nl ${JENKINS_WEB}/config.xml | grep "<au.com.centrumsystems.hudson.plugin.buildpipeline.BuildPipelineView" | awk '{print $1}' | head -1`
 #get the last line number of 'master' pipeline view
 lln=`nl ${JENKINS_WEB}/config.xml | grep "</au.com.centrumsystems.hudson.plugin.buildpipeline.BuildPipelineView" | awk '{print $1}' | head -1`
 let n=$lln+1
 sed -n "1,${lln}p" ${JENKINS_WEB}/config.xml > ${JENKINS_WEB}/config.tmp.xml
 #extract the configuration of 'master' pipeline view
 sed -n "${fln},${lln}p" ${JENKINS_WEB}/config.xml | sed "s/master/$b/g" >> ${JENKINS_WEB}/config.tmp.xml
 tail -n +$n ${JENKINS_WEB}/config.xml >> ${JENKINS_WEB}/config.tmp.xml
 mv ${JENKINS_WEB}/config.tmp.xml ${JENKINS_WEB}/config.xml
done

#restart jenkins server to reload the new pipeline
cd ${JENKINS_HOME}
java -jar jenkins.jar

Run it.

In file system:

drwxr-xr-x 4 oracle dba 4096 Jul 3 00:41 develop_unit-test
drwxr-xr-x 3 oracle dba 4096 Jul 3 00:41 develop_package
drwxr-xr-x 3 oracle dba 4096 Jul 3 00:41 develop_generate-doc
drwxr-xr-x 4 oracle dba 4096 Jul 3 00:41 develop_functional-test
drwxr-xr-x 3 oracle dba 4096 Jul 3 00:41 develop_DeployStage
drwxr-xr-x 3 oracle dba 4096 Jul 3 00:41 develop_DeployQA
drwxr-xr-x 3 oracle dba 4096 Jul 3 00:41 develop_code-analysis
drwxr-xr-x 4 oracle dba 4096 Jul 3 00:41 develop_build
drwxr-xr-x 3 oracle dba 4096 Jul 3 00:43 develop_sonar
drwxr-xr-x 5 oracle dba 4096 Jul 3 00:43 develop_clean
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_DeployQA
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_DeployStage
drwxr-xr-x 5 oracle dba 4096 Jul 3 2014 master_clean
drwxr-xr-x 4 oracle dba 4096 Jul 3 2014 master_build
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_code-analysis
drwxr-xr-x 4 oracle dba 4096 Jul 3 2014 master_functional-test
drwxr-xr-x 4 oracle dba 4096 Jul 3 2014 master_unit-test
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_package
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_sonar
drwxr-xr-x 3 oracle dba 4096 Jul 3 2014 master_generate-doc

Review the change in Jenkins’ config.xml:

master2

You can see the script just copy the pipeline view configuration from ‘master’ but replacing its’ name with the name of branch.

Go to dashboard. Here it encounters a weird issue, the ‘develop’ pipeline was empty initially!!! I have not yet figured out why. But it pops up after I built its root job ‘develop_clean’ for the first time. 🙂

Here is the snapshot,

master3