Site Reliability: SLI, SLO & SLA

Service Level Indicator(SLI), Service Level Object(SLO) & Service Level Agreement(SLA) are parameters with which reliability, availability and performance of the service are measured. The SLA, SLO, and SLI are related concepts though they’re different concepts.

It’s easy to get lost in a fog of acronyms, so before we dig in, here is a quick and easy definition:

  • SLA or Service Level Agreement is a contract that the service provider promises customers on service availability, performance, etc.
  • SLO or Service Level Objective is a goal that service provider wants to reach.
  • SLI or Service Level Indicator is a measurement the service provider uses for the goal.

Service Level Indicator
SLI are the parameters which indicates the successful transactions, requests served by the service over the predefined intervals of time. These parameters allows to measure much required performance and availability of the service. Measuring these parameters also enables to improve them gradually.

Key Examples are:

  • Availability/Uptime of the service.
  • Number of successful transactions/requests.
  • Consistency and durability of the data.

Service Level Objective
SLO defines the acceptable downtime of the service. For multiple components of the service, there can be different parameters which defines the acceptable downtime. It is common pattern to start with low SLO and gradually increase it.

Key Examples are:

  • Durability of disks should be 99.9%.
  • Availability of service should be 99.95%
  • Service should successfully serve 99.999% requests/transactions.

Service Level Agreement
SLA defines the penalty that service provider should pay in an event of service unavailability for pre-defined period of time. Service provider should clearly define the failure factors for which they will be accountable(Domain of responsibility). It is common pattern to have loose SLA than SLO, for instance: SLA is 99% and SLO is 99.5%. If the service is overly available, then SLA/SLO can be used as error budget to deploy complex releases to production.

Key Examples of Penalty are:

  • Partial refund of service subscription fee.
  • Additional subscription time added for free.

So here is the relationship. The service provider needs to collect metrics based on SLI, define thresholds of metrics based on SLO, and monitor the thresholds of metrics so that it won’t break SLA. In practical, the SLIs are the metrics in the monitoring system; the SLOs are alerting rules, and the SLAs are the numbers of the monitoring metrics applying to the SLOs.

Usually the SLO and the SLA are similar while the SLO is tighter than the SLA. The SLOs are generally used for internal only, and the SLAs are for external. If a service availability violates the SLO, operations need to react quickly to avoid it breaking SLA, otherwise, the company might need to refund some money to customers.

The SLA, SLO, and SLI are based on such assumption that is the service will not be available 100%. Instead, we guarantee that the system will be available greater than a certain number, for example, 99.5%.

When we apply this definition to availability, for example, SLIs are the key measurements of the availability of a system; SLOs are goals we set for how much availability we expect out of a system; and SLAs are the legal contracts that explains what happens if our system doesn’t meet its SLO.

SLIs exist to help engineering teams make better decisions. Your SLO performance is critical information to have when you’re making decisions about how hard and fast you can push your systems. SLOs are also important data points for other engineers when they’re making assumptions about their dependencies on your service or system. Lastly, your larger organization should use your SLIs and SLOs to make informed decisions about investment levels and about balancing reliability work against engineering velocity.

Note this abstract is taken from SRE Fundamentals, CRE and the book Site Reliability Engineering: How Google Runs Production Systems

Unikernel: Another paradigm for the cloud

In this cloud era it is hard to imagine a world without access to services in the cloud. From contacting someone through mail, to storing work-related documents on an online drive and accessing it across devices, there are lot of services we use on a daily basis that is in the cloud.

To reduce the cost of compute power, Virtualization has been adapted towards offering more services with less hardware. And then came the concept of containers where you deploy the application in isolated containers with light weight images which has few binaries and libraries to run your application, But still we need the underlying VMs to deploy such solutions. All these VMs comes with a cost. While large data-centers are offering services in the cloud, they are also hungry for electric power, which is becoming a growing concern as our planet is being drained of its resources. So what we need now is less power-hungry solutions.

What if, instead of virtualization of an entire operating system, you were to load an application with only the required components from the operating system? Effectively reducing the size of the virtual machine to its bare minimum resource footprint? This is where unikernels come into play.

Unikernel

Unikernel is a relatively new concept that was first introduced around 2013 by Anil Madhavapeddy in a paper titled “Unikernels: Library Operating Systems for the Cloud” (Madhavapeddy, et al., 2013).

You can find more details on Unilernel by searching the scholarly articles in Google.

Unikernels are defined by the community at Unikernel.org as follows.

“Unikernels are specialized, single-address-space machine images constructed by using library operating systems.”

For more detailed reading about the concepts behind Unikernel, please follow this link,

A Unikernel is an application that has been boiled down to a small, secure, light-weight virtual machine which eliminates general purpose operating systems such as Linux or Windows. Unikernels aims to be a much more secure system than Linux. It does this through several thrusts. Not having the notion of users, running a single process per VM, and limiting the amount of code that is incorporated into each VM. This means that there are no users and no shell to login to and, more importantly, you can’t run more than the one program you want to run inside. Despite their relatively young age, unikernels borrow from age-old concepts rooted in the dawn of the computer era: microkernels and library operating systems. This means that a unikernel holds a single application. Single-address space means that in its core, the unikernel does not have separate user and kernel address space. Library operating systems are the core of unikernel systems. Unikernels are provisioned directly on the hypervisor without a traditional system like Linux. So it can run 1000X more vms/per server.

You can have a look in here for details about Microkernel, Monolithic & Library Operating Systems

Virtual Machines VS Linux Containers VS Unikernel

Virtualization of services can be implemented in various ways. One of the most widespread methods today is through virtual machine, hosted on hypervisors such as VMware’s ESXi or Linux Foundation’s Xen Project.

Hypervisors allow hosting multiple guest operating systems on a single physical machine. These guest operating systems are executed in what is called virtual machines. The widespread use of hypervisors is due to their ability to better distribute and optimize the workload on the physical servers as opposed to legacy infrastructures of one operating system per physical server.

Containers are another method of virtualization, which differentiates from hypervisors by creating virtualized environments and sharing the host’s kernel. This provides a lighter approach to hypervisors which requires each guest to have their copy of the operating system kernel, making a hypervisor-virtualized environment resource heavy in contrast to containers which share parts of the existing operating system.

As aforementioned, unikernels leverage the abstraction of hypervisors in addition to using library operating systems to only include the required kernel routines alongside the application to present the lightest of all three solutions.

The figure above shows the major difference between the three virtualization technologies. Here we can clearly see that virtual machines present a much larger load on the infrastructure as opposed to containers and unikernels.

Additionally, unikernels are in direct “competition” with containers. By providing services in the form of reduced virtual machines, unikernels improve on the container model by its increased security. By sharing the host kernel, containerized applications share the same vulnerabilities as the host operating system. Furthermore, containers do not possess the same level of host/guest isolation as hypervisors/virtual machines, potentially making container breaches more damaging than both virtual machines and unikernels.

TechnologyProsCons
Virtual Machines– Allows deploying different operating systems on a single host
– Complete isolation from host
– Orchestration solutions available
– Requires compute power proportional to number of instances
– Requires large infrastructures
– Each instance loads an entire operating system
Linux Containers– Lightweight virtualization
– Fast boot times
– Ochestration solutions
– Dynamic resource allocation
– Reduced isolation between host and guest due to shared kernel
– Less flexible (i.e.: dependent on host kernel)
– Network is less flexible
Unikernels– Lightweight images
– Specialized application
– Complete isolation from host
– Higher security against absent functionalities (e.g.: remote command execution)
– Not mature enough yet for production
– Requires developing applications from the grounds up
– Limited deployment possibilities
– Lack of complete IDE support
– Static resource allocation
– Lack of orchestration tools
A Comparison of solutions

Docker and containerization technology and the container orchestra-tors like Kubernetes, OpenShift are 2 steps forward for the world of DevOps and that the principles it promotes are forward-thinking and largely on-target for the future of a more secure, performance oriented, and easy-to-manage cloud future. However, an alternative approach leveraging unikernels and immutable servers will result in smaller, easier to manage, more secure containers that will be simpler to adopt by existing enterprises. As DevOps matures, the shortcomings of cloud application deployment and management are becoming clear. Virtual machine image bloat, large attack surfaces, legacy executable, base-OS fragmentation, and unclear division of responsibilities between development and IT for cloud deployments are all causing significant friction (and opportunities for the future).

For Example: It remains virtually impossible to create a Ruby or Python web server virtual machine image that DOESN’T include build tools (gcc), ssh, and multiple latent shell executable. All of these components are detrimental for production systems as they increase image size, increase attack surface, and increase maintenance overhead.

Compared to VMs running Operating systems like Windows and Linux, the unikernel has only a tenth of 1% of the attack surface. So in the case of a unikernel — sysdig, tcpdump, and mysql-client are not installed and you can’t just “apt-get install” them either. You have to bring that with your exploit. To take it further even a simple cat /etc/hosts or grep of /var/log/nginx/access.log simply won’t work — once again they are separate processes.
So unikernels are highly resistant to remote code execution attacks, more specifically shell code exploits.

Immutable Servers & Unikernels

Immutable Servers are a deployment model that mandates that no application updates, security patches, or configuration changes happen on production systems. If any of these layers needs to be modified, a new image is constructed, pushed and cycled into production. Heroku is a great example of immutable servers in action: every change to your application requires a ‘git push’ to overwrite the existing version. The advantages of this approach include higher confidence in the code that is running in production, integration of testing into deployment workflows, easy to verify that systems have not been compromised.

Once you become a believer in the concept of immutable servers, then speed of deployment and minimizing vulnerability surface area become objectives. Containers promote the idea of single-service-per-container (microservices), and unikernels take this idea even further.

Unikernels allow you to compile and link your application code all the way down to and include the operating system. For example, if your application doesn’t require persistent disk access, no device drivers or OS facilities for disk access would even be included in final production images. Since unikernels are designed to run on hypervisors such as Xen, they only need interfaces to standardized resources such as networking and persistence. Device drivers for thousands of displays, disks, network cards are completely unnecessary. Production systems become minimalist — only requiring the application code, the runtime environment, and the OS facilities required by the applications. The net effect is smaller VM images with less surface area that can be deployed faster and maintained more easily.

Traditional Operating Systems (Linux, Windows) will become extinct on servers. They will be replaced with single-user, bare metal hypervisors optimized for the specific hardware, taking decades of multi-user, hardware-agnostic code cruft with them. More mature build-deploy-manage tool set based on these technologies will be truly game changing for hosted and enterprise clouds alike.

UnikernelLanguageTargetsFunctions
ClickOSC++XenNetwork Function Virtualization
HalVMHaskellXen
IncludeOSC++KVM, VirtualBox, ESXi, Google Cloud, OpenStackOrchestration tool available
MirageOSOCamlKVM, Xen, RTOS/MCU
Nanos UnikernelC, C++, Go, Java, Node.js, Python, Rust, Ruby, PHP, etcQEMU/KVMOrchestration tool available
OSvJava, C, C++, Node, RubyVirtualBox, ESXi, KVM, Amazon EC2, Google CloudCloud and IoT (ARM)
RumprunC, C++, Erlan, Go, Java, JavaScript, Node.js, Python, Ruby, RustXen, KVM
UnikGo, Node.js, Java, C, C++, Python, OCamlVirtualBox, ESXi, KVM, XEN, Amazon EC2, Google Cloud, OpenStack, PhotonControllerUnikernel compiler toolbox with orchestration possible through Kubernetes and Cloud Foundry
ToroKernelFreePascalVirtualBox, KVM, XEN, HyperVUnikernel dedicated to run microservices
Comparing few Unikernel solutions from active projects

Out of the various existing projects, some standout due to their wide range of supported languages. Out of the active projects, the above table describes the language they support, the hypervisors they can run on and remarks concerning their functionality.

Currently experimenting with the Unikernel in the AWS and Google Cloud Platform and will update you with another post on that soon.

Source: Medium, github, containerjournal, linuxjournal

Helm 3 – Sans tiller – Really?

Helm has recently announced it’s much-awaited version 3. The surprise factor in this release is that the server component added during Helm 2 release is missing. Yeah, you got it right – Tiller – is missing in Helm 3 which means we have a server-less Helm. Let us check out in this post why it was missing and how it matters?

As an introduction, Helm, the package manager for Kubernetes, is a useful tool for: installing, upgrading and managing applications on a Kubernetes cluster. Helm has two parts: a client (helm) and a server (tiller). Tiller runs inside of your Kubernetes cluster as a pod in the kube-system namespace onto current context specified in your .kubeconfig (can be manipulated using –kube-context flag). Tiller manages both, the releases (installations) and revisions (versions) of charts deployed on the cluster. When you run helm commands, your local Helm client sends instructions to tiller in the cluster that in turn make the requested changes. So Helm is our package manager for Kubernetes and our client tool. We use the helm cli to do all of our commands. Tiller is the service that actually communicates with the Kubernetes API to manage our Helm packages.

Helm packages are called charts. Charts are the Helm’s deploy-able artifacts. Charts are always versioned using semantic versioning, and come either packed, in versioned .tgz files, or in a flat directory structure. They are abstractions describing how to install packages onto a Kubernetes cluster. When a chart is deployed, it works as a templating engine to populate multiple yaml files for package dependencies with the required variables, and then runs kubectl apply to apply the configuration to the resource and install the package.

As we already mentioned Tiller is the tool used by Helm to deploy almost any Kubernetes resource. Helm takes the maximum permission to make changes in Kubernetes in order to do this. Because of this, anyone who can talk to the Tiller can deploy or modify any resources on the Kubernetes cluster, just like a system-admin (Think as ‘root’ user in a Linux host). This can cause security issues in the cluster if Helm has not been properly deployed, following certain security measures. Also, authentication is not enabled in Tiller by default, so if any of the pod has been compromised and has permission to talk to the Tiller, then the complete cluster in which tiller is running has been compromised. This stands for the main reason for the removal of Tiller.

The Tiller method:

Tiller was used as an in-cluster operator by the helm to maintain the state of a helm release. It’s also used to save the release information of all the releases done by the tiller — it uses config-map to save the release information in the same namespace in which Tiller is deployed. This release information was required by the helm when it updated or when there were state changes in any of the releases. So whenever a helm update command was used, Tiller used to compare the new manifest with the old manifest of the release and made changes accordingly. Thus Helm was dependent on the Tiller to provide the previous state of the release.

The Tiller less method:

The main need of Tiller was to store release information, for which helm is now using secrets and saving it in the same namespace as the release. Whenever Helm needs the release information it gets it from the namespace of the release. To make a change Helm now just fetches information from the Kubernetes API server, makes the changes on the client-side, and stores a record of the installation in Kubernetes. The benefit of tiller less Helm is that since now Helm make changes in the cluster from client-side, it can only make those changes that the client has been granted permission.

Tiller was a good addition in helm 2, but to run it in production it should be properly secured, which would add additional learning steps for the DevOps and SRE. With Helm 3 learning steps have been reduced and security management is been left in hands of Kubernetes to maintain. Helm can now focus on package-management.

Data courtesy : Medium, codecentric, Helm

Bare-Metal K8s Cluster with Raspberry Pi – Part 3

This is a continuation from the post series Bare-metal K8s cluster with Raspberry Pi – Part 1 & Part 2 here

Another option of running bare-metal K8s cluster in the Raspberry Pi I tried and tested was with Micro K8s which we discuss in this post.

Micro K8s are Lightweight upstream K8s. They are smallest, simplest, pure production K8s. For clusters, laptops, IoT and Edge, on Intel and ARM.

MicroK8s is a CNCF certified upstream Kubernetes deployment that runs entirely on your workstation or edge device. Being a snap it runs all Kubernetes services natively (i.e. no virtual machines) while packing the entire set of libraries and binaries needed. Installation is limited by how fast you can download a couple of hundred megabytes and the removal of MicroK8s leaves nothing behind.

And to give a context on snap, Snaps are app packages for desktop, cloud and IoT that are easy to install, secure, cross‐platform and dependency‐free. Snaps are discover able and install able from the Snap Store, the app store for Linux with an audience of millions.

A snap is a bundle of an app and its dependencies that works without modification across Linux distributions.

We are going to use the same components list as described in the Part 1 of this series.

Each Pi is going to need an Ubuntu server image and you’ll need to be able to SSH into them. Please follow this link here will help us to reach to this stage

Kubernetes Cluster Preparation with SSH connection to the Pi from your terminal

Installing MicroK8s
Follow this section for each of your Pis. Once completed you will have MicroK8s installed and running everywhere.

SSH to your first Pi and install the MicroK8s snap:

sudo snap install microk8s --classic

As MicroK8s is a snap and as such it will be automatically updated to newer releases of the package, which is following closely upstream Kubernetes releases, so we don’t need to worry about the K8s version we’re installing

sudo snap install microk8s --classic --channel=1.15/stable
Channels are made up of a track (or series) and an expected level of stability, based on MicroK8s releases (Stable, Candidate, Beta, Edge). For more information about which releases are available, run:

snap info microk8s

Cheat Sheet for MicroK8s
Before going further here is a quick intro to the MicroK8s command line:

  • microk8s.start – start all enabled Kubernetes services
  • microk8s.inspect – status of services
  • microk8s.stop – stop all Kubernetes services
  • microk8s.enable dns – enable Kubernetes add-ons,“kubedns”
  • microk8s.kubectl cluster-info – status of the cluster:

MicroK8s is easy to use and comes with plenty of Kubernetes add-ons you can enable or disable.

Master node and leaf nodes
Now that you have MicroK8s installed on all boards, pick one which has to be the master node of your cluster.

On the chosen Master node, run the following command:

sudo microk8s.add-node
This command will generate a connection string in the form of :/.

Adding a node
Now, you need to run the join command from another Pi you want to add to the cluster:

microk8s.join 10.55.60.14:25000/JHpbBYMIevZSAMnmjMHmFwanrOYCWZLu
You should be able to see the new node in a few seconds on the master with the following command:

microk8s.kubectl get node

For each new node, you need to run the microk8s.add-node command on the master, copy the output, then run microk8s.join on the leaf.

Removing nodes
To remove a node, run the following command on the master:

sudo microk8s remove-node
The name of nodes are available on the master by running the microk8s.kubectl get node

Alternatively, you can leave the cluster from a leaf node by running:

sudo microk8s.leave

Once Pis are setup with MicroK8s, adding and removing nodes is easy and you can scale up or down as you go.

Voila..!!
If you follow this series, you are now in control of your Kubernetes cluster. One with native kubernetes and docker and the other one with more easier to install and manage MicroK8s

This completes the 3 part series for K8s in Raspberry Pi. In a new follow-up blog post we can see how we can use kubectl & helm charts to deploy a Nginx service, Prometheus and few other services from the DevOps tools set to the cluster.

Bare-Metal K8s Cluster with Raspberry Pi – Part 2

This is a continuation from the post series Bare-metal K8s cluster with Raspberry Pi

As we have 1 master node and 3 nodes setup we continue to install Kubernetes.

Install Kubernetes

We are using version 1.15.3. There shouldn’t be any errors, however during my installation the repos were down and I had to retry in a few times.

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | \
sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q

sudo apt-get install -qy kubelet=1.15.3-00 kubectl=1.15.3-00 kubeadm=1.15.3-00

Repeat steps for all of the Raspberry Pis.

Kubernetes Master Node Configuration
Note: You only need to do this for the master node (in this deployment I recommend only 1 master node). Each Raspberry Pi is a node.

Initiate Master Node

sudo kubeadm init
Enable Connections to Port 8080
Without this Kubernetes services won’t work

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Add Container Network Interface (CNI)
I’ve chosen to use Weaver, however you can get others working such as Flannel (I’ve verified this works with this cluster)

Get Join Command
This will be used in the next section to join the worker nodes to the cluster. It will return something like:

kubeadm join 192.168.0.101:6443 --token X.Y --discovery-token-ca-cert-hash sha256:XYZ
kubeadm token create --print-join-command

Kubernetes Worker Node Configuration
Note: You only need to do this for the worker nodes (in this deployment I recommend 3 worker node).

Join Cluster
Use the join command provided at the end of the previous section
sudo kubeadm join 192.168.0.101:6443 --token X.Y --discovery-token-ca-cert-hash sha256:XYZ

Verify Node Added Successfully (SSH on Master Node)
Should have status ready after ~30 seconds
kubectl get componentstatuses

Another option of running bare-metal K8s cluster in the Raspberry Pi I tried and tested was with Micro K8s will posted in the Part 3 of this series.

A sneak peak into the K8s cluster in Raspberry pi

Bare-Metal K8s Cluster with Raspberry Pi – Part 1

There are multiple ways we can use a Kubernetes cluster to deploy our applications. Most of us opt to use any Kubernetes service from a public cloud provider. GKE, EKS, AKS are the most prominent ones. Deploying a Kubernetes cluster on a public cloud provider is relatively easy, but what if you want a private bare-metal K8s cluster. Being worked extensively in the data center and started my career as a Sys Admin, I personally prefer a piece of tangible hardware to get the feel of building it. This blog post walk you through the steps I took in order to have a bare-metal K8s cluster to play with.

K8s is an open source container orchestration platform that helps manage distributed, containerized applications at a massive scale. Born at Google as Borg, version 1.0 was released in July 2015. It has continued to evolve and mature and is now offered as a PaaS service by all of the major cloud vendors.

Google has been running containerized workloads in production for more than a decade. Whether it’s service jobs like web front-ends and stateful servers, infrastructure systems like Bigtable and Spanner, or batch frameworks like MapReduce and Millwheel, virtually everything at Google runs as a container.

You can find the paper here.

Kubernetes traces its lineage directly from Borg. Many of the developers at Google working on Kubernetes were formerly developers on the Borg project. We’ve incorporated the best ideas from Borg in Kubernetes, and have tried to address some pain points that users identified with Borg over the years.

More than just enabling a containerized application to scale, Kubernetes has release-management features that enable updates with near-zero downtime, version rollback, and clusters that can ‘self-heal’ when there is a problem. Load balancing, auto-scaling and SSL can easily be implemented. Helm, a plugin for Kubernetes, has revolutionized the world of server management by making multi-node data stores like Redis and MongoDB incredibly easy to deploy. Kubernetes enables you to have the flexibility to move your workload where it is best suited. This compliments the hybrid cloud story and in my career it has become more apparent that my customers see this as well to help them resolve issues like; cost, availability and compliance. In parallel software vendors are starting to embrace containers as a standard deployment model leading to a recent increase in requests for container solutions.

As you can see in the workflow comparison below, there is greater room for error when deploying on-premises. Public clouds provide the automation and reduces the risk of error as less steps are required. But as mentioned above, private cloud provides you more options when you have unique requirements.

Pros:

  • Using Kubernetes and its huge ecosystem can improve your productivity
  • Kubernetes and a cloud-native tech stack attracts talent
  • Kubernetes is a future proof solution
  • Kubernetes helps to make your application run more stable
  • Kubernetes can be cheaper than its alternatives

Cons:

  • Kubernetes can be an overkill for simple applications
  • Kubernetes is very complex and can reduce productivity
  • The transition to Kubernetes can be cumbersome
  • Kubernetes can be more expensive than its alternatives

Pre-requisites:

Compute:

3 x Raspberry Pi 4 Model B with 2 GB RAM
1 x Raspberry Pi 3 Model B+ with 1 GB RAM

Storage:

4 x 16GB High Speed Sand-disk Micro-SD Cards

Network:

1 x Network Switch – for local LAN for k8s internal connectivity
1 x Network Router – for Wifi (My default ISP router was used here) only master node had internet connectivity once completed the setup
4 x Ethernet Cables
1 x Keyboard, HDMI, Mouse (for initial setup only)

Initial Raspberry Pi Configuration:

Flash Raspbian to the Micro-SD Cards

Download image from the below link,

Raspbian OS

I have used BalenaEtcher to flash image onto micro-SD card

Perform Initial Setup on Boot on startup screen, we need to connect keyboard, monitor and mouse for this setup.

Choose Country, Language, Timezone
Define new password for user ‘pi’
Connect to WiFi or skip if using ethernet
Skip update software (We will perform this activity manually later).
Choose restart later

Configure Additional Settings Click the Raspberry Pi icon (top left of screen) > Preferences > Raspberry Pi Configuration

System

Configure Hostname
Boot: To CLI

Interfaces
SSH: Enable

Choose restart later

Configure Static Network Perform one of the following:
Define Static IP on Raspberry Pi: Right Click the arrow logo top right of screen and select ‘Wireless & Wired Network Settings’

Define Static IP on DHCP Server: Configure your DHCP server to define a static IP on the Raspberry Pi Mac Address.

Reboot and Test SSH
Username: pi

Password: Defined in step 2 above

From the terminal ssh pi@[IP Address]

Repeat steps for all of the Raspberry Pis.

Kubernetes Cluster Preparation with SSH connection to the Pi from your terminal. I am using a 12 years old Lenovo laptop running MX Linux. Open a terminal and establish ssh connection to the Pi

Perform Updates

apt-get update: Updates the package indexes
apt-get upgrade: Performs the upgrades

Configure Net.IP4.IP configuration Edit sudo vi /etc/sysctl.conf, uncomment net.ipv4.ip_forward = 1 and add net.ipv4.ip_nonlocal_bind=1.
Note: This is required to allow for traffic forwarding, for example Node Ports from containers to/from non-cluster devices.

Install Docker

curl -sSL get.docker.com | sh

Grant privilege for user pi to execute docker commands

sudo usermod pi -aG docker

Disable Swap

sudo systemctl disable dphys-swapfile.service
sudo reboot

We can verify this with the top command, on the top left corner next to MiB Swap should be 0.0.

As we completed the initial steps to create our K8s bare-metal cluster, we can see how we build the cluster in Part 2 of this blog post

Quick and Dirty tutorial on Jenkins and Git plugin

Jenkins is a popular open source Continuous Integration tool which is widely used for project development, deployment, and automation. Continuous integration is a process in which all development work is integrated as early as possible. The resulting artifacts are automatically created and tested. This process should identify errors as very early in the process. Jenkins is one open source tool to perform continuous integration and build automation. The basic functionality of Jenkins is to execute a predefined list of steps. The trigger for this execution can be time or event based. For example, every half an hour or after a new commit in a Git repository. Jenkins also monitors the execution of the steps and allows to stop the process if one of the steps fails. Jenkins can also send out notification about the build success or failure.Jenkins can be extended by additional plug-ins, e.g., for supporting Git version control system, provisioning with docker or integration with the puppet.

In this post, we’ll discuss Installation of Jenkins and how to use Jenkins to integrate with GitHub in such a way that a git commit can trigger a Jenkins build. Jenkins can be started via the command line or can run in a web application server. In Linux, you can also install Jenkins as a system service. Almost all the Linux platforms there is a native package of Jenkins available. Please check the Jenkins home page for more details on this.

All the demonstrations in this post are shown using and Amazon EC2 RHEL 7  instance. We’re using an Amazon instance because we need to establish a connection to the system’s public IP via Git web hooks. If you are using any other Linux distribution you can make use of most of the steps described here except the RHEL specific command like yum, systemctl etc. Also, you can find your OS-specific installation details in the Jenkins Wiki.

We will be installing Jenkins using the native package available for the RHEL/Fedora/CentOS. There is another way of installing the Jenkins which includes deploying the Jenkins war file which you can see here

Installing Java:

The prerequisite of installing Jenkins is to setup a Java Virtual Machine on your system. This will be done by installing a latest OpenJDK package. Before that, we will be installing EPEL repository for RHEL 7. You can download the epel package from fedora project URL using wget command and if in case wget is not installed on your machine install it using yum.

# yum install wget

# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

# yum install epel-release-latest-7.noarch.rpm

You can do a yum search OpenJDK to know the available JDK packages.

# yum install java-1.8.0-openjdk-devel.x86_64

Verify the java installation as below,

# java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

Next step is to set the environment variables namely JAVA_HOME and JRE_HOME. This is used by the Java applications to identify the Java virtual machine path. This is accomplished by adding these environment variables to the file /etc/profile.

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME=/usr/lib/jvm/jre

Once this is added you can reload the environment variables from /etc/profile by executing,

# source /etc/profile

Installing Apache Ant and Apache Maven:

The next step is to install the apache foundation siblings Ant and Maven. These siblings are used while building Java based application in Jenkins. Both these packages can be downloaded from the respective project site as a tarball.

Downloading

# wget http://redrockdigimark.com/apachemirror//ant/binaries/apache-ant-1.10.0-bin.tar.gz

# wget http://redrockdigimark.com/apachemirror/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

Installing

The command provided below will extract your downloaded tar ball and put it in /opt. Then we will go to the /opt directory and create a symbolic link.

# tar -zxvf apache-maven-3.3.9-bin.tar.gz -C /opt/
# tar -zxvf apache-ant-1.10.0-bin.tar.gz -C /opt/

# ln -s apache-maven-3.3.9/ maven
# ln -s apache-ant-1.10.0/ ant

The final listing of the /opt will show something like seen below,

# ls /opt
ant apache-ant-1.10.0 apache-maven-3.3.9 maven

Environment Variables

Now we will setup environment variables for this apache sibling and verify the installation. These variables are set by creating files maven.sh and ant.sh under /etc/profile.d and reload the environment variables.

# cat > /etc/profile.d/ant.sh
export ANT_HOME=/opt/ant
export PATH=${ANT_HOME}/bin:${PATH}

# cat > /etc/profile.d/maven.sh
export M2_HOME=/opt/maven
export PATH=${M2_HOME}/bin:${PATH}

# source /etc/profile

At this stage, if everything goes well you'll see and output similar to the one below,

# echo $JAVA_HOME;echo $JRE_HOME
/usr/lib/jvm/jre-1.8.0-openjdk
/usr/lib/jvm/jre

# ant -version
Apache Ant(TM) version 1.10.0 compiled on December 27 2016

# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /opt/maven
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-514.el7.x86_64", arch: "amd64", family: "unix"

All our prerequisites for Jenkins installation is ready and we’ll now proceed to install Jenkins. We will add an RHEL 7 specific Jenkins repository and install with the yum command.

# wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
# rpm --import http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key
# yum install jenkins -y

Once the installation is complete, we will enable the service and start it.

# systemctl enable jenkins.service
# systemctl start jenkins.service
# systemctl status jenkins.service

If you have followed the instalation so far you find Jenkins running under the following URL:

http://<ip of your ec2 instance>:8080/

Note: if there are any connection issues or page not loading in the browser check the firewalls inside the system and flush it if any with the command iptables -F. Also check the security-groups of your EC2 instance to allow traffic on port 8080.

You’ll see a Jenkins page similar to the one above and you can use the password as described there and setup a user to create Jenkins builds.

#  cat /var/lib/jenkins/secrets/initialAdminPassword

Now we will install the git package in the system and Jenkins GitHub plugin to create a build job.

# yum install git

Create a repository in GitHub by importing the repository https://github.com/ajoybharath/hello-world.git

You can clone your newly created repository to your system and make it ready to accept push from your repo to GitHub. You can see more on git installation and setup here

Once all the steps as provided in the above screen shots are done we’ll be ready to create our first build. You can follow the sequence of images below and setup a Jenkins build job. We’re using the following git repository in this demonstration.

https://github.com/ajoybharath/hello-world.git

Apply and Save the job and go ahead and build the project.

If in case you’ll find any issue in the build similar to the one below,

FATAL: command execution failed
java.io.IOException: Cannot run program "mvn" (in directory

Please add the following lines to the Jenkins config:

# vi /etc/sysconfig/jenkins and add the line to "source /etc/profile" the end of the file and stop and start Jenkins.

Try building the job again. Also, we will be adding the GitHub webhook to enable the build trigger if a change is pushed to the repository.

In the Jenkins home page guide to Manage Jenkins=>Configure Systems and find the GIT

Click on the Advanced setting there and check on the “Specify another hook URL for GitHub configuration” and we need to copy the URL specified there.

This URL is the web hook to our Jenkins and we’ll be adding this to our GitHub repository. Navigate to your repository and click on settings tab as shown below,

Also, we have to go to our Jenkins dashboard and select the job we created and click on configure and navigate to Build Trigger and click on the check box GitHub hook trigger for GITScm polling

If everything goes fine, we can do some change in the readme file of the repository and push it which will trigger an automatic build in the Jenkins.

 

Jenkins – An Introduction

Jenkins is a Continuous Integration, Continous Deployment (CI/CD)tool. Jenkins is written in Java and released as open source product. It’s used heavily in Continuous Integration as it allows code to be build deployed and tested automatically. For software development, you can hook it up with most code repo’s and there are loads of plugins that you can integrate with various software tools for better technical governance. Think it as a very advanced task scheduler which can do workflow or as a middle man between your code repo and your build server which triggers a build for every change made in the source code repository.

Jenkins offers the following major features out of the box, and much more can be added through plugins:

  1. Easy installation: Just run java -jar jenkins.war, deploy it in a servlet container or install with any native package manager. No additional install, no database.
  2. Easy configuration: Jenkins can be configured entirely from its user-friendly web GUI.
  3. Rich plugin ecosystem: Jenkins integrates with virtually every SCM or build tool that exists.
  4. Extensibility: Most parts of Jenkins can be extended and modified, and it’s easy to create new Jenkins plugins.
  5. Distributed builds: Jenkins can distribute build/test loads to multiple computers with different operating systems.

Jenkins is pretty much understood after you’ll understand the terms Continous Integration and Continous Deployment (Delivery).

The relevant terms here are “Continuous Integration” and “Continuous Deployment”, often used together and abbreviated as CI/CD. Originally Continuous Integration means that you run your “integration tests” at every code change while Continuous Delivery means that you automatically deploy every change that passes your tests. However recently the term CI/CD is being used in a more general way to represent everything related to automation of the pipeline from where a developer adds his change to a central repository until that code ends up in production. This post is based on the latter meaning of the terms CI/CD.

CI system gathers all your code from different developers and makes sure it compiles and build fine. Once the code is built it deploys it on the test server for testing. Once you’re made sure the build compiles fine then, Jenkins can be tuned to deploy the application on the production server satisfying the goal of CD. There are many different ways in which Jenkins can be set up to drive a CI/CD pipeline. This details provided below describes a simple yet powerful configuration:

The pipeline consists of 4 steps: “Build”, “Unit”, “Integration”, “System” and “Deploy”

Each step is implemented as a Jenkins job of type “free-style software project”.

The first step (“Build”) is connected to a version control system. This step can either poll the repository, get notified via a webhook, or run on a schedule or on demand. Subsequent steps have a build trigger that starts the job when the previous step finished.
Artifacts are copied between the different jobs via the “Copy Artifact” plugin. This requires the “Archive Artifacts” setting to be enabled. Deploy triggers from “Copy Artifact” to a Configuration Manager which will trigger a Container service to deploy the new application.

As told earlier in this post we can install Jenkins using the native package available for the respective Operating System. This is detailed in another post which you can read here. There is another way of installing the Jenkins which we’ll be discussing here and consist of 2 steps,

  1. Install the latest version of Apache tomcat
  2. Download and Deploy Jenkins war file

As a prerequisite, we will be installing Java which can be installed with yum command.

# yum install java-1.8.0-openjdk-devel.x86_64

[root@devopsnode1 ~]# java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
[root@devopsnode1 ~]#

After Java is installed we’ll set the Java path as below:

[root@devopsnode1 ~]# cp /etc/profile /etc/profile_orig
[root@devopsnode1 ~]# echo "export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk" | tee -a /etc/profile
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
[root@devopsnode1 ~]# echo "export JRE_HOME=/usr/lib/jvm/jre" | tee -a /etc/profile
export JRE_HOME=/usr/lib/jvm/jre
[root@devopsnode1 ~]# source /etc/profile
[root@devopsnode1 ~]# echo $JAVA_HOME;echo $JRE_HOME
/usr/lib/jvm/jre-1.8.0-openjdk
/usr/lib/jvm/jre
[root@devopsnode1 ~]#

Now we’ll be installing the latest version of Apache tomcat. We’ll be using the wget command to download the latest Apache tomcat tarball from the website http://tomcat.apache.org/download-90.cgi

[ajoy@devopsnode1 ~]$ wget http://www-us.apache.org/dist/tomcat/tomcat-9/v9.0.0.M17/bin/apache-tomcat-9.0.0.M17.tar.gz

After the tarball is downloaded we’ll extract it and rename the directory to tomcat9, though the renaming is not mandatory step we’ll do it for having a simple tomcat paths 🙂

[ajoy@devopsnode1 ~]$ tar zxf apache-tomcat-9.0.0.M17.tar.gz
[ajoy@devopsnode1 ~]$ mv apache-tomcat-9.0.0.M17 tomcat9
[ajoy@devopsnode1 ~]$ ls
apache-tomcat-9.0.0.M17.tar.gz tomcat9

We’ll have to set roles and users in the tomcat configuration and then start the tomcat server:

[ajoy@devopsnode1 ~]$ cp /home/ajoy/tomcat9/conf/tomcat-users.xml /home/ajoy/tomcat-users.xml_orig
[ajoy@devopsnode1 ~]$ vi tomcat9/conf/tomcat-users.xml

<tomcat-users xmlns=”http://tomcat.apache.org/xml”
xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xsi:schemaLocation=”http://tomcat.apache.org/xml tomcat-users.xsd”
version=”1.0″>
<role rolename=”manager-gui”/>
<role rolename=”manager-script”/>
<role rolename=”manager-jmx”/>
<role rolename=”manager-status”/>
<role rolename=”admin-gui”/>
<role rolename=”admin-script”/>
<user username=”ajoy” password=”ajoy” roles=”manager-gui,manager-script,manager-jmx,manager-status,admin-gui,admin-script”/>
</tomcat-users>

Starting Apache tomcat server:

[ajoy@devopsnode1 tomcat9]$ cd bin/
[ajoy@devopsnode1 bin]$ ./startup.sh
Using CATALINA_BASE: /home/ajoy/tomcat9
Using CATALINA_HOME: /home/ajoy/tomcat9
Using CATALINA_TMPDIR: /home/ajoy/tomcat9/temp
Using JRE_HOME: /usr/lib/jvm/jre
Using CLASSPATH: /home/ajoy/tomcat9/bin/bootstrap.jar:/home/ajoy/tomcat9/bin/tomcat-juli.jar
Tomcat started.
[ajoy@devopsnode1 bin]$

If everything is fine you’ll be able to browse to http://localhost:8080 which will open the default tomcat web page.

Shutting down Apache tomcat server:

[ajoy@devopsnode1 tomcat9]$ cd bin/
[ajoy@devopsnode1 bin]$ ./shutdown.sh
Using CATALINA_BASE: /home/ajoy/tomcat9
Using CATALINA_HOME: /home/ajoy/tomcat9
Using CATALINA_TMPDIR: /home/ajoy/tomcat9/temp
Using JRE_HOME: /usr/lib/jvm/jre
Using CLASSPATH: /home/ajoy/tomcat9/bin/bootstrap.jar:/home/ajoy/tomcat9/bin/tomcat-juli.jar
[ajoy@devopsnode1 bin]$

Now our tomcat server is up and running we’ll now download the Jenkins war file and deploy it in tomcat.

[ajoy@devopsnode1 ~]$ wget http://ftp.yz.yamagata-u.ac.jp/pub/misc/jenkins/war/2.44/jenkins.war

Click on the Manager App button. It will ask for username and password which we have put in the “tomcat-users.xml”.

We’ll provide a Context Path and specify our Jenkins war file with an absolute path and then click on Deploy.

Once it’s deployed you can access the Jenkins as shown above.

Using the cat command open the file and to copy the password and paste it on the Unlock screen.

[ajoy@devopsnode1 ~]$ cat /home/ajoy/.jenkins/secrets/initialAdminPassword

Follow the below screenshot sequences to complete the Jenkins setup:

That’s all folks, you’re ready to build your project with award-winning, cross-platform, continuous integration and continuous delivery application that increases your productivity. You can read here another post describing the build steps.

Data Source Courtesy: Jenkins JenkinsWiki, Quora,

Git Cheat Sheet

This post is a Git cheat sheet, which git users can bookmark so that you’ll have most basic Git commands handy. A more detailed post about git installation and commit can be seen here

General Git Commands

$ git –version ==> Displays the version fo the git installed

$ git config –global alias.st status ==> Create an alias for git status

$ git help ==> Displays help on the git command

$ git init ==> Initialize git

$ git add . ==> Make everything in CWD ready to commit

$ git add index.html ==> Make one file ready to commit

$ git commit -m “Message” ==> Commit changes

$ git commit -am “Message” ==> Add file and Commit in one commad

$ git rm index.html ==> Remove files from git

$ git add -u ==> Update all changes

$ git rm –cached index.html ==> Remove & not to track a file any more

$ git mv index.html dir/index_new.html ==> Move or Rename files

$ git checkout — index.html ==> Restore file from a latest commit

$ git checkout 6eb715d — index.html ==> Restore file from a custom commit of current branch

$ git clean -n ==> Delete dry run of untracked files

$ git clean -f ==> Delete untracked files

$ git reset HEAD index.html ==> Undo adds

$ git commit –amend -m “Message” ==> Commit to most recent commit

$ git commit –amend -m “New Message” ==> Update most recent commit message

Tagging and Branching and Merging

$ git tag ==> Show all released version

$ git tag -l -n1 ==> Show all released versions with comments

$ git tag v1.0.0 ==> Create release version

$ git tag -a v1.0.0 -m ‘Message’ ==> Create release version with comment

$ git checkout v1.0.0 ==> Checkout a specific release version

$ git branch ==> Show branches

$ git branch branchname ==> Create branch

$ git checkout branchname ==> Change to a branch

$ git checkout -b branchname ==> Create and change to a branch

$ git branch -m branchname new_branchname

or ==> Rename a branch

$ git branch –move branchname new_branchname

$ git branch –merged ==> List all completely merged branch with current one

$ git branch -d branchname

or ==> Delete merged branch

$ git branch –delete branchname

$ git branch -D branch_to_delete ==> Delete not merged branch

$ git merge branchname ==> True Merge

$ git merge –ff-only branchname ==> Merge to Master

$ git merge –no-ff branchname ==> Merge to master/force a new commit

$ git merge –abort ==> Stop merge

Working with remote Git/GitHub

$ git remote ==> Show remote

$ git remote -v ==> Show remote details

$ git remote add origin https://github.com/user/project.git ==> Add remote origin from GitHub project

$ git remote add origin ssh://root@123.123.123.123/path/to/repository/.git ==> Ad remote origin from existing empty project

$ git remote rm origin ==> Remove origin

$ git branch -r ==> Show remote branches

$ git branch -a ==> Show all branches

$ git diff origin/master..master ==> Compare

$ git push -u origin master ==> Default push/set default with -u

$ git push origin master ==> Push to default

$ git fetch origin ==> Fetch

$ git pull ==> Pull

$ git pull origin branchname ==> Pull specific branch

$ git merge origin/master ==> Merge fetched commits

$ git clone https://github.com/user/project.git or: git clone ssh://user@domain.com/~/dir/.git ==> Clone to local system

$ git clone https://github.com/user/project.git ~/dir/folder ==> Clone to a local folder

$ git clone -b branchname https://github.com/user/project.git ==> Clone specific branch to local host

$ git push origin :branchname or: git push origin –delete branchname ==> Delete remote branch/Push nothing

Git Logs

$ git log ==> Show commits

$ git log –oneline ==> Show oneline summary of commits

$ git log –oneline -3 ==> Show oneline summary of last three commits

$ git log –author=”Sven” git log –grep=”Message” git log –until=2013-01-01 git log –since=2013-01-01 ==> Show only custom commits

$ git log -p ==> Show changes

$ git log –stat –summary ==> Show status and summary of commits

$ git log –graph ==> Show history of commits as graph

$ git log –oneline –graph –all –decorate ==> Show history of commits as graph summary

Cherry pick

git cherry-pick <first_commit>..<nth commit>
E.g.
$ git cherry-pick 6e30dac520faef4a1..fd7d0d7873461a26
On branch develop
You are currently cherry-picking commit d856905.

$ git cherry-pick <initial_commit_hash>^..<terminal_commit_hash>
$ git cherry-pick --continue

Archiving

$ git archive –format zip –output filename.zip master ==> Create a zip archive

$ git log –author=sven –all > log.txt ==> Write custom logs to a file

And finally a useful template for creating .gitignore,

.gitignore Templates

 

Git and GitHub An Introduction

Git

Git is a distributed version control system. It is primarily used for software development. It can also be used literally to keep track of changes to any files. Git was created by Linus Torvalds in 2005 for the development of the Linux kernel, with other kernel developers contributing to its initial development. Linus was not satisfied with any of the available free systems to replace BitKeeper. Consider it as a series of snapshots (commits) of your code. You see a path of these snapshots, in which order they were created. You can make branches to experiment and come back to snapshots you took. Currently, it is used by many popular open source projects, such as the Android or the Eclipse developer teams, as well as many commercial organizations.

GitHub

GitHub is a web-based Git repository hosting service, founded in 2008 which offers all of the distributed revision control and source code management (SCM) functionality of Git as well as adding its own features. It actually hosts remote repositories and allows code collaboration with anyone who has an access to the GitHub. It adds enhanced features like UI, bug tracking etc on top of Git. It’s free to use to host public repositories and has an Enterprise version like most open source applications.

This post discusses git essentials like repositories, branches, commits, pull, fetch and clone. You’ll create your own repository and learn git’s workflow, a popular way to create and review code.

Initial Steps:

The two things that you’ll be doing in order to play around with git and GitHub is to install git on your operating system first and then proceed to create a free account in the GitHub. The steps provided below assumes that we are using a Linux machine, though I’ll be providing quick links for installation in other operating systems.

Installing git:

Linux: Debian and its derivatives – #sudo apt-get install git

Linux: Fedora, RH and its derivatives – yum install git or dnf install git

Mac OS – https://git-scm.com/download/mac

Windows – https://git-scm.com/download/win

We’ll use a Ubuntu machine to install git and the steps are as follows,

ajoy@testserver:~$ sudo apt-get update
ajoy@testserver:~$sudo apt-get install git
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
git
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 0 B/2,586 kB of archives.
After this operation, 20.5 MB of additional disk space will be used.
(Reading database ... 62961 files and directories currently installed.)
Preparing to unpack .../git_1%3a1.9.1-1ubuntu0.3_amd64.deb ...
Unpacking git (1:1.9.1-1ubuntu0.3) ...
Setting up git (1:1.9.1-1ubuntu0.3) ...
ajoy@testserver:~$

A more flexible way of installing git is to compile the software from the source. This procedure anyhow is time-consuming and will not be maintained by aptitude. The main advantage of doing so is that it will allow you to download the latest release and also gives you some control over the installation if you wish to customize your git installation. Before proceeding to compile from source you need to install a couple of dependency packages which allows you to compile git.

Another way of installing the latest version is to download the git source and to package it using an easy packaging tool called FPM with all your customizations in the new package. This process will provide you a package in .deb, .rpm or any other format which will be unique with your customization. This is a little more time-consuming way than the previous one.

Setting up git:

ajoy@testserver:~$ git --version
git version 1.9.1
ajoy@testserver:~$

ajoy@testserver:~$ git config --global user.name "ajoy"
ajoy@testserver:~$ git config --global user.email "ajoy.XXXX@XX.com"
ajoy@testserver:~$ git config --list
core.editor=/bin/vim
user.name=ajoy
user.email=ajoy.XXXX@XX.com
ajoy@testserver:~$

These details are stored in a file called .gitconfig which you can edit manually to add different configurations and also make sure you’re following the syntax and formatting properly.

GitHub account:

Next step is to go to the GitHub site and signup. GitHub accounts are free for setting up public repositories but there will be a nominal charge for setting up private repositories.

Creating a local repository:

Now we’ll be creating a local repository on our Linux machine.

ajoy@testserver:~$ mkdir mytestrepo
ajoy@testserver:~$ cd mytestrepo/
ajoy@testserver:~/mytestrepo$

Then we’ll be initializing this directory as a repository. We will execute git init command in the root of the new directory to do this.

ajoy@testserver:~/mytestrepo$ git init
Initialized empty Git repository in /home/ajoy/mytestrepo/.git/
ajoy@testserver:~/mytestrepo$

We’ll now create a file in the directory with touch command or with any of your favorite editors,

ajoy@testserver:~/mytestrepo$ touch file1
ajoy@testserver:~/mytestrepo$ ls
file1
ajoy@testserver:~/mytestrepo$

Once we add a file or change any existing file of a directory which holds a git repo, git will be aware of the changes happening But, git won’t keep a track of it unless until you tell it to do so. This process is called committing, which is a crucial task while using git.

In this point of time, you can use a command called git status to know how git sees your file,

ajoy@testserver:~/mytestrepo$ git status
On branch master

Initial commit

Untracked files:
(use "git add <file>..." to include in what will be committed)

file1

nothing added to commit but untracked files present (use "git add" to track)
ajoy@testserver:~/mytestrepo$

This actually means that git has noticed the creation of a new file called file1 but unless you execute git add command git is not going to do anything with it.

One of the most confusing parts while learning git is the concept of the staging environment and how it relates to a commit. A commit is a record of what files you have changed since the last time you made a commit. Essentially, you make changes to your repo (for example, adding a file or modifying one) and then tell git to put those files into a commit. Commits make up the essence of your project and allow you to go back to the state of a project at any point.

So, how do you tell git which files to put into a commit? This is where the staging environment or index come in. As seen in above, when you make changes to your repo, git notices that a file has changed but won’t do anything with it (like adding it in a commit).

To add a file to a commit, you first need to add it to the staging environment. To do this, you can use the git add <filename> command.
Once you’ve executed the command git add to add all the files you want to the staging environment, you can then tell git to package them into a commit using the git commit command.

ajoy@testserver:~/mytestrepo$ git add file1
ajoy@testserver:~/mytestrepo$ git status
On branch master

Initial commit

Changes to be committed:
(use "git rm --cached <file>..." to unstage)

new file: file1

ajoy@testserver:~/mytestrepo$

Now the file is added to the staging environment and ready to commit.

Note: The staging environment, also called ‘staging’, is the new preferred term for this, but you can also see it referred to as the ‘index’.

Our First Commit:

It’s now time to go for our first commit. Commit is made with git commit -m “<message about the commit>”

ajoy@testserver:~/mytestrepo$ git commit -m "My First Commit"
[master (root-commit) e198044] My First Commit
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 file1
ajoy@testserver:~/mytestrepo$

The message at the end of the commit should be something related to what the commit contains – maybe it’s a new feature, maybe it’s a bug fix, maybe it’s just fixing a typo. Don’t put a message like “asdfadsf” or “1234” or any such junk. It should be something meaningful describing our commit so that other contributors or users who see this will understand what this commit is all about.

Pushing our repository to the GitHub:

Now we have our repository created locally in our system. Our next step is to push this repository to the GitHub so that it will be available for public and users can see our code, pull it to their system modify it and push it back. If we need to keep track of this codes locally, we don’t need to push it to GitHub. GitHub push is recommended if we need to work collaboratively on the code. The pictures provided below shows a demo of creating a repository in the GitHub

When you’re done filling out the information, press the ‘Create repository’ button to make your new repo. GitHub will ask if you want to create a new repo from scratch or if you want to add a repo you have created locally. In this case, since we’ve already created a new repo locally, we want to push that onto GitHub so follow the ‘….or push an existing repository from the command line’ section. But we will first see what that means and then will do our push.

In order to push the local repository to the GitHub, we need to set the remote URL of the newly created GitHub repository to our local repository. The command git remote -v will show whether our local repo is linked to any remote URL.

ajoy@testserver:~/mytestrepo$ git remote -v
ajoy@testserver:~/mytestrepo$ git remote add origin https://github.com/ajoybharath/mytestrepo.git
ajoy@testserver:~/mytestrepo$ git remote -v
origin https://github.com/ajoybharath/mytestrepo.git (fetch)
origin https://github.com/ajoybharath/mytestrepo.git (push)
ajoy@testserver:~/mytestrepo$

The command git remote add origin <url> will set the remote repo url to our local repository. This can be verified by executing git remote -v again.

Now we’ll be pushing our repo to remote GitHub,

ajoy@testserver:~/mytestrepo$ git push -u origin master
Username for 'https://github.com': <username>
Password for 'https://<username>@github.com':
Counting objects: 3, done.
Writing objects: 100% (3/3), 211 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To https://github.com/ajoybharath/mytestrepo.git
* [new branch] master -> master
Branch master set up to track remote branch master from origin.
ajoy@testserver:~/mytestrepo$

The user name and the password is the one we created during the GitHub signup process.

We had done a push to the GitHub now let’s try something a little more advanced.

Assume that you want to make a new feature but are worried about making changes to the main project while developing the feature. This is where git branches are applicable.

Branches allow you to move back and forth between ‘states’ of a project. For instance, if you want to add a new page to your website you can create a new branch just for that page without affecting the main part of the project. Once you’re done with the page, you can merge your changes from your branch into the master branch. When you create a new branch, Git keeps track of which commit your branch ‘branched’ off of, so it knows the history behind all the files. Let’s say you are on the master branch and want to create a new branch to develop your web page. Here’s what you’ll do: execute git checkout -b <my branch name>. This command will automatically create a new branch and then ‘check you out’ on it, meaning git will move you to that branch, off of the master branch. After running the above command, you can use the git branch command to confirm that your branch was created.

ajoy@testserver:~/mytestrepo$ git checkout -b devel
Switched to a new branch 'devel'
ajoy@testserver:~/mytestrepo$ git branch
* devel
master
ajoy@testserver:~/mytestrepo$

The branch name with the asterisk next to it indicates which branch you’re pointed to at that given time.

Now, if you switch back to the master branch and make some more commits, your new branch won’t see any of those changes until you merge those changes onto your new branch.

The next step is to push the devel branch we created to the remote repository. This allows other people to see the changes you’ve made. If they’re approved by the repository’s owner, the changes can then be merged into the master branch. To push changes to a new branch on GitHub, you’ll want to run git push origin <yourbranchname>. GitHub will automatically create the branch for you on the remote repository.

ajoy@testserver:~/mytestrepo$ git push origin devel
Username for 'https://github.com': <username>
Password for 'https://<username>@github.com':
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/ajoybharath/mytestrepo.git
* [new branch] devel -> devel
ajoy@testserver:~/mytestrepo$

Now let’s discuss the word “origin” in the command above. When you clone a remote repository to your local machine, git creates an alias for you. In nearly all cases this alias is called “origin.” It’s essentially shorthand for the remote repository’s URL. So, to push your changes to the remote repository, you could’ve used either the command: git push git@github.com:git/git.git <your branch name> or git push origin <your branch name>

In order to get the most recent changes that you or others have merged on GitHub, use the git pull origin master command (when working on the master branch).

ajoy@testserver:~/mytestrepo$ git pull origin master
From https://github.com/ajoybharath/mytestrepo
* branch master -> FETCH_HEAD
Already up-to-date.
ajoy@testserver:~/mytestrepo$

Now we can use the git log command again to see all new commits.

ajoy@testserver:~/mytestrepo$ git log
commit e19804462d4b87336e4607d26e467f2ca14a5d3a
Author: ajoy <email>
Date: Wed Feb 1 21:10:23 2017 +0530

My First Commit
ajoy@testserver:~/mytestrepo$

Finally, three more basic git commands, git pull git fetch and git clone:

Git pull – Git pull will pull down from a remote whatever you ask (so, whatever trunk you’re asking for) and instantly merge it into the branch you’re in when you make the request. Pull is a high-level request that runs ‘fetch’ then a ‘merge’ by default, or a rebase with ‘–rebase’. You could do without it, it’s just a convenience.

Git fetch – Git fetch is similar to pull, except it won’t do any merging. So, the fetch will have pulled down the remoteBranch and put it into a local branch called “remoteBranch”. creates a local copy of a remote branch which you shouldn’t manipulate directly; instead, create a proper local branch and work on that. ‘git checkout’ has a confusing feature, though. If you ‘checkout’ a local copy of a remote branch, it creates a local copy and sets up a merge to it by default.

Git clone – Git clone will clone a repo into a newly created directory. It’s useful for when you’re setting up your local repo. It additionally creates a remote called ‘origin’ for the repo cloned from, sets up a local branch based on the remote’s active branch (generally master), and creates remote-tracking branches for all the branches in the repo

You can find more usage and command options in this post