Quick and Dirty tutorial on Jenkins and Git plugin

Jenkins is a popular open source Continuous Integration tool which is widely used for project development, deployment, and automation. Continuous integration is a process in which all development work is integrated as early as possible. The resulting artifacts are automatically created and tested. This process should identify errors as very early in the process. Jenkins is one open source tool to perform continuous integration and build automation. The basic functionality of Jenkins is to execute a predefined list of steps. The trigger for this execution can be time or event based. For example, every half an hour or after a new commit in a Git repository. Jenkins also monitors the execution of the steps and allows to stop the process if one of the steps fails. Jenkins can also send out notification about the build success or failure.Jenkins can be extended by additional plug-ins, e.g., for supporting Git version control system, provisioning with docker or integration with the puppet.

In this post, we’ll discuss Installation of Jenkins and how to use Jenkins to integrate with GitHub in such a way that a git commit can trigger a Jenkins build. Jenkins can be started via the command line or can run in a web application server. In Linux, you can also install Jenkins as a system service. Almost all the Linux platforms there is a native package of Jenkins available. Please check the Jenkins home page for more details on this.

All the demonstrations in this post are shown using and Amazon EC2 RHEL 7  instance. We’re using an Amazon instance because we need to establish a connection to the system’s public IP via Git web hooks. If you are using any other Linux distribution you can make use of most of the steps described here except the RHEL specific command like yum, systemctl etc. Also, you can find your OS-specific installation details in the Jenkins Wiki.

We will be installing Jenkins using the native package available for the RHEL/Fedora/CentOS. There is another way of installing the Jenkins which includes deploying the Jenkins war file which you can see here

Installing Java:

The prerequisite of installing Jenkins is to setup a Java Virtual Machine on your system. This will be done by installing a latest OpenJDK package. Before that, we will be installing EPEL repository for RHEL 7. You can download the epel package from fedora project URL using wget command and if in case wget is not installed on your machine install it using yum.

# yum install wget

# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

# yum install epel-release-latest-7.noarch.rpm

You can do a yum search OpenJDK to know the available JDK packages.

# yum install java-1.8.0-openjdk-devel.x86_64

Verify the java installation as below,

# java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

Next step is to set the environment variables namely JAVA_HOME and JRE_HOME. This is used by the Java applications to identify the Java virtual machine path. This is accomplished by adding these environment variables to the file /etc/profile.

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME=/usr/lib/jvm/jre

Once this is added you can reload the environment variables from /etc/profile by executing,

# source /etc/profile

Installing Apache Ant and Apache Maven:

The next step is to install the apache foundation siblings Ant and Maven. These siblings are used while building Java based application in Jenkins. Both these packages can be downloaded from the respective project site as a tarball.

Downloading

# wget http://redrockdigimark.com/apachemirror//ant/binaries/apache-ant-1.10.0-bin.tar.gz

# wget http://redrockdigimark.com/apachemirror/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

Installing

The command provided below will extract your downloaded tar ball and put it in /opt. Then we will go to the /opt directory and create a symbolic link.

# tar -zxvf apache-maven-3.3.9-bin.tar.gz -C /opt/
# tar -zxvf apache-ant-1.10.0-bin.tar.gz -C /opt/

# ln -s apache-maven-3.3.9/ maven
# ln -s apache-ant-1.10.0/ ant

The final listing of the /opt will show something like seen below,

# ls /opt
ant apache-ant-1.10.0 apache-maven-3.3.9 maven

Environment Variables

Now we will setup environment variables for this apache sibling and verify the installation. These variables are set by creating files maven.sh and ant.sh under /etc/profile.d and reload the environment variables.

# cat > /etc/profile.d/ant.sh
export ANT_HOME=/opt/ant
export PATH=${ANT_HOME}/bin:${PATH}

# cat > /etc/profile.d/maven.sh
export M2_HOME=/opt/maven
export PATH=${M2_HOME}/bin:${PATH}

# source /etc/profile

At this stage, if everything goes well you'll see and output similar to the one below,

# echo $JAVA_HOME;echo $JRE_HOME
/usr/lib/jvm/jre-1.8.0-openjdk
/usr/lib/jvm/jre

# ant -version
Apache Ant(TM) version 1.10.0 compiled on December 27 2016

# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /opt/maven
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-514.el7.x86_64", arch: "amd64", family: "unix"

All our prerequisites for Jenkins installation is ready and we’ll now proceed to install Jenkins. We will add an RHEL 7 specific Jenkins repository and install with the yum command.

# wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
# rpm --import http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key
# yum install jenkins -y

Once the installation is complete, we will enable the service and start it.

# systemctl enable jenkins.service
# systemctl start jenkins.service
# systemctl status jenkins.service

If you have followed the instalation so far you find Jenkins running under the following URL:

http://<ip of your ec2 instance>:8080/

Note: if there are any connection issues or page not loading in the browser check the firewalls inside the system and flush it if any with the command iptables -F. Also check the security-groups of your EC2 instance to allow traffic on port 8080.

You’ll see a Jenkins page similar to the one above and you can use the password as described there and setup a user to create Jenkins builds.

#  cat /var/lib/jenkins/secrets/initialAdminPassword

Now we will install the git package in the system and Jenkins GitHub plugin to create a build job.

# yum install git

Create a repository in GitHub by importing the repository https://github.com/ajoybharath/hello-world.git

You can clone your newly created repository to your system and make it ready to accept push from your repo to GitHub. You can see more on git installation and setup here

Once all the steps as provided in the above screen shots are done we’ll be ready to create our first build. You can follow the sequence of images below and setup a Jenkins build job. We’re using the following git repository in this demonstration.

https://github.com/ajoybharath/hello-world.git

Apply and Save the job and go ahead and build the project.

If in case you’ll find any issue in the build similar to the one below,

FATAL: command execution failed
java.io.IOException: Cannot run program "mvn" (in directory

Please add the following lines to the Jenkins config:

# vi /etc/sysconfig/jenkins and add the line to "source /etc/profile" the end of the file and stop and start Jenkins.

Try building the job again. Also, we will be adding the GitHub webhook to enable the build trigger if a change is pushed to the repository.

In the Jenkins home page guide to Manage Jenkins=>Configure Systems and find the GIT

Click on the Advanced setting there and check on the “Specify another hook URL for GitHub configuration” and we need to copy the URL specified there.

This URL is the web hook to our Jenkins and we’ll be adding this to our GitHub repository. Navigate to your repository and click on settings tab as shown below,

Also, we have to go to our Jenkins dashboard and select the job we created and click on configure and navigate to Build Trigger and click on the check box GitHub hook trigger for GITScm polling

If everything goes fine, we can do some change in the readme file of the repository and push it which will trigger an automatic build in the Jenkins.

 

January 11, 2017 DevOps No Comments on Quick and Dirty tutorial on Jenkins and Git plugin

Jenkins – An Introduction

Jenkins is a Continuous Integration, Continous Deployment (CI/CD)tool. Jenkins is written in Java and released as open source product. It’s used heavily in Continuous Integration as it allows code to be build deployed and tested automatically. For software development, you can hook it up with most code repo’s and there are loads of plugins that you can integrate with various software tools for better technical governance. Think it as a very advanced task scheduler which can do workflow or as a middle man between your code repo and your build server which triggers a build for every change made in the source code repository.

Jenkins offers the following major features out of the box, and much more can be added through plugins:

  1. Easy installation: Just run java -jar jenkins.war, deploy it in a servlet container or install with any native package manager. No additional install, no database.
  2. Easy configuration: Jenkins can be configured entirely from its user-friendly web GUI.
  3. Rich plugin ecosystem: Jenkins integrates with virtually every SCM or build tool that exists.
  4. Extensibility: Most parts of Jenkins can be extended and modified, and it’s easy to create new Jenkins plugins.
  5. Distributed builds: Jenkins can distribute build/test loads to multiple computers with different operating systems.

Jenkins is pretty much understood after you’ll understand the terms Continous Integration and Continous Deployment (Delivery).

The relevant terms here are “Continuous Integration” and “Continuous Deployment”, often used together and abbreviated as CI/CD. Originally Continuous Integration means that you run your “integration tests” at every code change while Continuous Delivery means that you automatically deploy every change that passes your tests. However recently the term CI/CD is being used in a more general way to represent everything related to automation of the pipeline from where a developer adds his change to a central repository until that code ends up in production. This post is based on the latter meaning of the terms CI/CD.

CI system gathers all your code from different developers and makes sure it compiles and build fine. Once the code is built it deploys it on the test server for testing. Once you’re made sure the build compiles fine then, Jenkins can be tuned to deploy the application on the production server satisfying the goal of CD. There are many different ways in which Jenkins can be set up to drive a CI/CD pipeline. This details provided below describes a simple yet powerful configuration:

The pipeline consists of 4 steps: “Build”, “Unit”, “Integration”, “System” and “Deploy”

Each step is implemented as a Jenkins job of type “free-style software project”.

The first step (“Build”) is connected to a version control system. This step can either poll the repository, get notified via a webhook, or run on a schedule or on demand. Subsequent steps have a build trigger that starts the job when the previous step finished.
Artifacts are copied between the different jobs via the “Copy Artifact” plugin. This requires the “Archive Artifacts” setting to be enabled. Deploy triggers from “Copy Artifact” to a Configuration Manager which will trigger a Container service to deploy the new application.

As told earlier in this post we can install Jenkins using the native package available for the respective Operating System. This is detailed in another post which you can read here. There is another way of installing the Jenkins which we’ll be discussing here and consist of 2 steps,

  1. Install the latest version of Apache tomcat
  2. Download and Deploy Jenkins war file

As a prerequisite, we will be installing Java which can be installed with yum command.

# yum install java-1.8.0-openjdk-devel.x86_64

[root@devopsnode1 ~]# java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
[root@devopsnode1 ~]#

After Java is installed we’ll set the Java path as below:

[root@devopsnode1 ~]# cp /etc/profile /etc/profile_orig
[root@devopsnode1 ~]# echo "export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk" | tee -a /etc/profile
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
[root@devopsnode1 ~]# echo "export JRE_HOME=/usr/lib/jvm/jre" | tee -a /etc/profile
export JRE_HOME=/usr/lib/jvm/jre
[root@devopsnode1 ~]# source /etc/profile
[root@devopsnode1 ~]# echo $JAVA_HOME;echo $JRE_HOME
/usr/lib/jvm/jre-1.8.0-openjdk
/usr/lib/jvm/jre
[root@devopsnode1 ~]#

Now we’ll be installing the latest version of Apache tomcat. We’ll be using the wget command to download the latest Apache tomcat tarball from the website http://tomcat.apache.org/download-90.cgi

[ajoy@devopsnode1 ~]$ wget http://www-us.apache.org/dist/tomcat/tomcat-9/v9.0.0.M17/bin/apache-tomcat-9.0.0.M17.tar.gz

After the tarball is downloaded we’ll extract it and rename the directory to tomcat9, though the renaming is not mandatory step we’ll do it for having a simple tomcat paths 🙂

[ajoy@devopsnode1 ~]$ tar zxf apache-tomcat-9.0.0.M17.tar.gz
[ajoy@devopsnode1 ~]$ mv apache-tomcat-9.0.0.M17 tomcat9
[ajoy@devopsnode1 ~]$ ls
apache-tomcat-9.0.0.M17.tar.gz tomcat9

We’ll have to set roles and users in the tomcat configuration and then start the tomcat server:

[ajoy@devopsnode1 ~]$ cp /home/ajoy/tomcat9/conf/tomcat-users.xml /home/ajoy/tomcat-users.xml_orig
[ajoy@devopsnode1 ~]$ vi tomcat9/conf/tomcat-users.xml

<tomcat-users xmlns=”http://tomcat.apache.org/xml”
xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xsi:schemaLocation=”http://tomcat.apache.org/xml tomcat-users.xsd”
version=”1.0″>
<role rolename=”manager-gui”/>
<role rolename=”manager-script”/>
<role rolename=”manager-jmx”/>
<role rolename=”manager-status”/>
<role rolename=”admin-gui”/>
<role rolename=”admin-script”/>
<user username=”ajoy” password=”ajoy” roles=”manager-gui,manager-script,manager-jmx,manager-status,admin-gui,admin-script”/>
</tomcat-users>

Starting Apache tomcat server:

[ajoy@devopsnode1 tomcat9]$ cd bin/
[ajoy@devopsnode1 bin]$ ./startup.sh
Using CATALINA_BASE: /home/ajoy/tomcat9
Using CATALINA_HOME: /home/ajoy/tomcat9
Using CATALINA_TMPDIR: /home/ajoy/tomcat9/temp
Using JRE_HOME: /usr/lib/jvm/jre
Using CLASSPATH: /home/ajoy/tomcat9/bin/bootstrap.jar:/home/ajoy/tomcat9/bin/tomcat-juli.jar
Tomcat started.
[ajoy@devopsnode1 bin]$

If everything is fine you’ll be able to browse to http://localhost:8080 which will open the default tomcat web page.

Shutting down Apache tomcat server:

[ajoy@devopsnode1 tomcat9]$ cd bin/
[ajoy@devopsnode1 bin]$ ./shutdown.sh
Using CATALINA_BASE: /home/ajoy/tomcat9
Using CATALINA_HOME: /home/ajoy/tomcat9
Using CATALINA_TMPDIR: /home/ajoy/tomcat9/temp
Using JRE_HOME: /usr/lib/jvm/jre
Using CLASSPATH: /home/ajoy/tomcat9/bin/bootstrap.jar:/home/ajoy/tomcat9/bin/tomcat-juli.jar
[ajoy@devopsnode1 bin]$

Now our tomcat server is up and running we’ll now download the Jenkins war file and deploy it in tomcat.

[ajoy@devopsnode1 ~]$ wget http://ftp.yz.yamagata-u.ac.jp/pub/misc/jenkins/war/2.44/jenkins.war

Click on the Manager App button. It will ask for username and password which we have put in the “tomcat-users.xml”.

We’ll provide a Context Path and specify our Jenkins war file with an absolute path and then click on Deploy.

Once it’s deployed you can access the Jenkins as shown above.

Using the cat command open the file and to copy the password and paste it on the Unlock screen.

[ajoy@devopsnode1 ~]$ cat /home/ajoy/.jenkins/secrets/initialAdminPassword

Follow the below screenshot sequences to complete the Jenkins setup:

That’s all folks, you’re ready to build your project with award-winning, cross-platform, continuous integration and continuous delivery application that increases your productivity. You can read here another post describing the build steps.

Data Source Courtesy: Jenkins , JenkinsWiki, Quora,

January 7, 2017 DevOps No Comments on Jenkins – An Introduction