Understanding API

API stands for Application Programming Interface. An API is a software intermediary that allows two applications to talk to each other. In other words, an API is the messenger that delivers your request to the provider that you’re requesting it from and then delivers the response back to you. We can say that an API is a set of programming instructions and standards for accessing a Web-based software application or Web tool.

As we understood API is a software-to-software interface, not a user interface. The most important part of this name is “interface,” because an API essentially talks to a program for you. You still need to know the language to communicate with the program, but without an API, you won’t get far. With APIs, applications talk to each other without any user knowledge or intervention. When programmers decide to make some of their data available to the public, they “expose endpoints,” meaning they publish a portion of the language they’ve used to build their program. Other programmers can then pull data from the application by building URLs or using HTTP clients (special programs that build the URLs for you) to request data from those endpoints.

Endpoints return text that’s meant for computers to read, so it won’t make complete sense if you don’t understand the computer code used to write it. A software company releases its API to the public so that other software developers can design products that are powered by its service.

Examples:

When bloggers put their Twitter handle on their blog’s sidebar, WordPress enables this by using Twitter’s API.

Amazon.com released its API so that Web site developers could more easily access Amazon’s product information. Using the Amazon API, a third party Web site can post direct links to Amazon products with updated prices and an option to “buy now.”

When you buy movie tickets online and enter your credit card information, the movie ticket Web site uses an API to send your credit card information to a remote application that verifies whether your information is correct. Once payment is confirmed, the remote application sends a response back to the movie ticket Web site saying it’s OK to issue the tickets. As a user, you only see one interface — the movie ticket Web site — but behind the scenes, many applications are working together using APIs. This type of integration is called seamless, since the user never notices when software functions are handed from one application to another

Docker engine comes with an API. Docker provides an API for interacting with the Docker daemon (called the Docker Engine API), as well as SDKs for Go and Python. The SDKs allow you to build and scale Docker apps and solutions quickly and easily. If Go or Python don’t work for you, you can use the Docker Engine API directly. The Docker Engine API is a RESTful API accessed by an HTTP client such as wget or curl , or the HTTP library which is part of most modern programming languages.

Types of APIs

There are four main types of APIs:

Open APIs: Also known as Public API, there are no restrictions to access these types of APIs because they are publicly available.
Partner APIs: A developer needs specific rights or licenses in order to access this type of API because they are not available to the public.
Internal APIs: Also known as Private APIs, only internal systems expose this type of API. These are usually designed for internal use within a company. The company uses this type of API among the different internal teams to be able to improve its products and services.
Composite APIs: This type of API combines different data and service APIs. It is a sequence of tasks that run synchronously as a result of the execution, and not at the request of a task. Its main uses are to speed up the process of execution and improve the performance of the listeners in the web interfaces.

API architecture types:

APIs can vary by architecture type but are generally used for one of three purposes:

System APIs access and maintain data. These types of APIs are responsible for managing all of the configurations within a system. To use an example, a system API unlocks data from a company’s billing database.

Process APIs take the data accessed with system APIs and synthesize it to create a new way to view or act on data across systems. To continue the example, a process API would take the billing information and combine it with inventory information and other data to fulfill an order.

Experience APIs add context to system and process APIs. These types of APIs make the information collected by system and process APIs understandable to a specified audience. Following the same example, an experience API could translate the data from the process and system APIs into an order status tracker that displays information about when the order was placed and when the customer should expect to receive it.

Apart from the main web APIs, there are also web service APIs:

The following are the most common types of web service APIs:

SOAP (Simple Object Access Protocol): This is a protocol that uses XML as a format to transfer data. Its main function is to define the structure of the messages and methods of communication. It also uses WSDL, or Web Services Definition Language, in a machine-readable document to publish a definition of its interface.

XML-RPC: This is a protocol that uses a specific XML format to transfer data compared to SOAP that uses a proprietary XML format. It is also older than SOAP. XML-RPC uses minimum bandwidth and is much simpler than SOAP. Example – YUM command in Linux uses XML-RPC calls

JSON-RPC: JavaScript Object Notation, this protocol is similar to XML-RPC but instead of using XML format to transfer data it uses JSON.

REST (Representational State Transfer): REST is not a protocol like the other web services, instead, it is a set of architectural principles. The REST service needs to have certain characteristics, including simple interfaces, which are resources identified easily within the request and manipulation of resources using the interface.

SOAPREST
It has strict rules and advanced security to follow.There are loose guidelines to follow allowing developers to make recommendations easily
It is driven by FunctionIt is driven by Data
It requires more BandwidthIt requires minimum Bandwidth
SOAP vs REST
JSONXML
Supports only text and numbers.Supports various types of data for example text, numbers, images, graphs, charts etc.
Focuses mainly on DataFocuses mainly on Document
It has low securityIt has more security
JSON vs XML

The web service APIs honor all the http methods like POST, GET, PUT, PATCH, DELETE. if we compare these with the CRUD operations,

HTTP MethodsCRUD
POSTCreate
GETRead
PUTUpdate/Replace
PATCHUpdate/Modify
DELETEDelete
HTTP methods vs CRUD operations

POST – The POST verb is most-often utilized to create new resources. It will return HTTP status 201 on success and returning a Location header with a link to the newly-created resource with the 201 HTTP status. POST is neither safe nor idempotent.

GET – The HTTP GET method is used to read (or retrieve) a representation of a resource GET returns a representation in XML or JSON and an HTTP response code of 200 (OK). GET is idempotent

PUT – PUT is most-often utilized for update capabilities. PUT is not a safe operation, in that it modifies (or creates) state on the server, but it is idempotent.

PATCH – PATCH is used for modify capabilities. The PATCH request only needs to contain the changes to the resource, not the complete resource. PATCH is neither safe nor idempotent. However, a PATCH request can be issued in such a way as to be idempotent,

DELETE – DELETE is pretty easy to understand. It is used to delete a resource identified by a URI. There is a caveat about DELETE idempotence as calling DELETE on a resource a second time will often return a 404 (NOT FOUND) since it was already removed and therefore is no longer available.

We will now just try creating a RESTFul API with Golang. For those who haven’t tried your hands with Golang can click here to follow the basics of Golang.

We are starting the go program which creates an API. The source code for this is available in this Git repository. A discussion on the implementation of API is out of scope for this blog post. We can discuss that in another post.

Here we concentrate only on the API and the requests and responses we receive and retrieve from the API end point. So, we can start our dummy API interface.

executing the code

Now we can check whether our API is accessible and it’s giving responses for our requests. We are doing it in command line with simple curl request. our application is listening on port 8001 which can be modified as per your wish in the code. We are now hitting the end point ‘/’

curl request

Next we try hitting the end point /events to get all the events in the dummy database created with slice and strut in the main.go file.

curl request to get all events

Now we will simulate the requests with an opensource Firefox extension RESTer. This is available for Chrome too. So for hitting the endpoint “/events” we are using the GET method. the Response 200 states that the request was successful.

GET method

In this simulation, we are using the POST method to create another event in the dummy database inside the application programmatically. Response code 201 is provided for successful creation of the event.

POST method to create new event

Now we can try hitting the endpoint “/events/{id}” which is the endpoint to retrieve one event with a GET method which will display the newly added event with “/events/2”

GET method on /events/2

We will now hit the “/events” endpoint and see whether both the evets are in the response.

GET method on /events endpoint

In the next simulation we are using the PATCH method to modify/update an existing event. In this case we are hitting the endpoint “/events/2” to modify the event id 2.

PATCH method to modify event id 2

Try to GET the expected results from the endpoint “/events” to verify our PATCH request.

GET method to verify PATCH request

In the final simulation we are hitting the endpoint “/events/1” with a DELETE method so that event id 1 will be removed/deleted.

DELETE Method to delete event id 1

Voila..!! We just created an API and tested it with dummy data. I believe we got a quick overview of what an API is and how we can use different HTTP methods to retrieve data or modify data using an API.

Site Reliability: SLI, SLO & SLA

Service Level Indicator(SLI), Service Level Object(SLO) & Service Level Agreement(SLA) are parameters with which reliability, availability and performance of the service are measured. The SLA, SLO, and SLI are related concepts though they’re different concepts.

It’s easy to get lost in a fog of acronyms, so before we dig in, here is a quick and easy definition:

  • SLA or Service Level Agreement is a contract that the service provider promises customers on service availability, performance, etc.
  • SLO or Service Level Objective is a goal that service provider wants to reach.
  • SLI or Service Level Indicator is a measurement the service provider uses for the goal.

Service Level Indicator
SLI are the parameters which indicates the successful transactions, requests served by the service over the predefined intervals of time. These parameters allows to measure much required performance and availability of the service. Measuring these parameters also enables to improve them gradually.

Key Examples are:

  • Availability/Uptime of the service.
  • Number of successful transactions/requests.
  • Consistency and durability of the data.

Service Level Objective
SLO defines the acceptable downtime of the service. For multiple components of the service, there can be different parameters which defines the acceptable downtime. It is common pattern to start with low SLO and gradually increase it.

Key Examples are:

  • Durability of disks should be 99.9%.
  • Availability of service should be 99.95%
  • Service should successfully serve 99.999% requests/transactions.

Service Level Agreement
SLA defines the penalty that service provider should pay in an event of service unavailability for pre-defined period of time. Service provider should clearly define the failure factors for which they will be accountable(Domain of responsibility). It is common pattern to have loose SLA than SLO, for instance: SLA is 99% and SLO is 99.5%. If the service is overly available, then SLA/SLO can be used as error budget to deploy complex releases to production.

Key Examples of Penalty are:

  • Partial refund of service subscription fee.
  • Additional subscription time added for free.

So here is the relationship. The service provider needs to collect metrics based on SLI, define thresholds of metrics based on SLO, and monitor the thresholds of metrics so that it won’t break SLA. In practical, the SLIs are the metrics in the monitoring system; the SLOs are alerting rules, and the SLAs are the numbers of the monitoring metrics applying to the SLOs.

Usually the SLO and the SLA are similar while the SLO is tighter than the SLA. The SLOs are generally used for internal only, and the SLAs are for external. If a service availability violates the SLO, operations need to react quickly to avoid it breaking SLA, otherwise, the company might need to refund some money to customers.

The SLA, SLO, and SLI are based on such assumption that is the service will not be available 100%. Instead, we guarantee that the system will be available greater than a certain number, for example, 99.5%.

When we apply this definition to availability, for example, SLIs are the key measurements of the availability of a system; SLOs are goals we set for how much availability we expect out of a system; and SLAs are the legal contracts that explains what happens if our system doesn’t meet its SLO.

SLIs exist to help engineering teams make better decisions. Your SLO performance is critical information to have when you’re making decisions about how hard and fast you can push your systems. SLOs are also important data points for other engineers when they’re making assumptions about their dependencies on your service or system. Lastly, your larger organization should use your SLIs and SLOs to make informed decisions about investment levels and about balancing reliability work against engineering velocity.

Note this abstract is taken from SRE Fundamentals, CRE and the book Site Reliability Engineering: How Google Runs Production Systems

Unikernel: Another paradigm for the cloud

In this cloud era it is hard to imagine a world without access to services in the cloud. From contacting someone through mail, to storing work-related documents on an online drive and accessing it across devices, there are lot of services we use on a daily basis that is in the cloud.

To reduce the cost of compute power, Virtualization has been adapted towards offering more services with less hardware. And then came the concept of containers where you deploy the application in isolated containers with light weight images which has few binaries and libraries to run your application, But still we need the underlying VMs to deploy such solutions. All these VMs comes with a cost. While large data-centers are offering services in the cloud, they are also hungry for electric power, which is becoming a growing concern as our planet is being drained of its resources. So what we need now is less power-hungry solutions.

What if, instead of virtualization of an entire operating system, you were to load an application with only the required components from the operating system? Effectively reducing the size of the virtual machine to its bare minimum resource footprint? This is where unikernels come into play.

Unikernel

Unikernel is a relatively new concept that was first introduced around 2013 by Anil Madhavapeddy in a paper titled “Unikernels: Library Operating Systems for the Cloud” (Madhavapeddy, et al., 2013).

You can find more details on Unilernel by searching the scholarly articles in Google.

Unikernels are defined by the community at Unikernel.org as follows.

“Unikernels are specialized, single-address-space machine images constructed by using library operating systems.”

For more detailed reading about the concepts behind Unikernel, please follow this link,

A Unikernel is an application that has been boiled down to a small, secure, light-weight virtual machine which eliminates general purpose operating systems such as Linux or Windows. Unikernels aims to be a much more secure system than Linux. It does this through several thrusts. Not having the notion of users, running a single process per VM, and limiting the amount of code that is incorporated into each VM. This means that there are no users and no shell to login to and, more importantly, you can’t run more than the one program you want to run inside. Despite their relatively young age, unikernels borrow from age-old concepts rooted in the dawn of the computer era: microkernels and library operating systems. This means that a unikernel holds a single application. Single-address space means that in its core, the unikernel does not have separate user and kernel address space. Library operating systems are the core of unikernel systems. Unikernels are provisioned directly on the hypervisor without a traditional system like Linux. So it can run 1000X more vms/per server.

You can have a look in here for details about Microkernel, Monolithic & Library Operating Systems

Virtual Machines VS Linux Containers VS Unikernel

Virtualization of services can be implemented in various ways. One of the most widespread methods today is through virtual machine, hosted on hypervisors such as VMware’s ESXi or Linux Foundation’s Xen Project.

Hypervisors allow hosting multiple guest operating systems on a single physical machine. These guest operating systems are executed in what is called virtual machines. The widespread use of hypervisors is due to their ability to better distribute and optimize the workload on the physical servers as opposed to legacy infrastructures of one operating system per physical server.

Containers are another method of virtualization, which differentiates from hypervisors by creating virtualized environments and sharing the host’s kernel. This provides a lighter approach to hypervisors which requires each guest to have their copy of the operating system kernel, making a hypervisor-virtualized environment resource heavy in contrast to containers which share parts of the existing operating system.

As aforementioned, unikernels leverage the abstraction of hypervisors in addition to using library operating systems to only include the required kernel routines alongside the application to present the lightest of all three solutions.

The figure above shows the major difference between the three virtualization technologies. Here we can clearly see that virtual machines present a much larger load on the infrastructure as opposed to containers and unikernels.

Additionally, unikernels are in direct “competition” with containers. By providing services in the form of reduced virtual machines, unikernels improve on the container model by its increased security. By sharing the host kernel, containerized applications share the same vulnerabilities as the host operating system. Furthermore, containers do not possess the same level of host/guest isolation as hypervisors/virtual machines, potentially making container breaches more damaging than both virtual machines and unikernels.

TechnologyProsCons
Virtual Machines– Allows deploying different operating systems on a single host
– Complete isolation from host
– Orchestration solutions available
– Requires compute power proportional to number of instances
– Requires large infrastructures
– Each instance loads an entire operating system
Linux Containers– Lightweight virtualization
– Fast boot times
– Ochestration solutions
– Dynamic resource allocation
– Reduced isolation between host and guest due to shared kernel
– Less flexible (i.e.: dependent on host kernel)
– Network is less flexible
Unikernels– Lightweight images
– Specialized application
– Complete isolation from host
– Higher security against absent functionalities (e.g.: remote command execution)
– Not mature enough yet for production
– Requires developing applications from the grounds up
– Limited deployment possibilities
– Lack of complete IDE support
– Static resource allocation
– Lack of orchestration tools
A Comparison of solutions

Docker and containerization technology and the container orchestra-tors like Kubernetes, OpenShift are 2 steps forward for the world of DevOps and that the principles it promotes are forward-thinking and largely on-target for the future of a more secure, performance oriented, and easy-to-manage cloud future. However, an alternative approach leveraging unikernels and immutable servers will result in smaller, easier to manage, more secure containers that will be simpler to adopt by existing enterprises. As DevOps matures, the shortcomings of cloud application deployment and management are becoming clear. Virtual machine image bloat, large attack surfaces, legacy executable, base-OS fragmentation, and unclear division of responsibilities between development and IT for cloud deployments are all causing significant friction (and opportunities for the future).

For Example: It remains virtually impossible to create a Ruby or Python web server virtual machine image that DOESN’T include build tools (gcc), ssh, and multiple latent shell executable. All of these components are detrimental for production systems as they increase image size, increase attack surface, and increase maintenance overhead.

Compared to VMs running Operating systems like Windows and Linux, the unikernel has only a tenth of 1% of the attack surface. So in the case of a unikernel — sysdig, tcpdump, and mysql-client are not installed and you can’t just “apt-get install” them either. You have to bring that with your exploit. To take it further even a simple cat /etc/hosts or grep of /var/log/nginx/access.log simply won’t work — once again they are separate processes.
So unikernels are highly resistant to remote code execution attacks, more specifically shell code exploits.

Immutable Servers & Unikernels

Immutable Servers are a deployment model that mandates that no application updates, security patches, or configuration changes happen on production systems. If any of these layers needs to be modified, a new image is constructed, pushed and cycled into production. Heroku is a great example of immutable servers in action: every change to your application requires a ‘git push’ to overwrite the existing version. The advantages of this approach include higher confidence in the code that is running in production, integration of testing into deployment workflows, easy to verify that systems have not been compromised.

Once you become a believer in the concept of immutable servers, then speed of deployment and minimizing vulnerability surface area become objectives. Containers promote the idea of single-service-per-container (microservices), and unikernels take this idea even further.

Unikernels allow you to compile and link your application code all the way down to and include the operating system. For example, if your application doesn’t require persistent disk access, no device drivers or OS facilities for disk access would even be included in final production images. Since unikernels are designed to run on hypervisors such as Xen, they only need interfaces to standardized resources such as networking and persistence. Device drivers for thousands of displays, disks, network cards are completely unnecessary. Production systems become minimalist — only requiring the application code, the runtime environment, and the OS facilities required by the applications. The net effect is smaller VM images with less surface area that can be deployed faster and maintained more easily.

Traditional Operating Systems (Linux, Windows) will become extinct on servers. They will be replaced with single-user, bare metal hypervisors optimized for the specific hardware, taking decades of multi-user, hardware-agnostic code cruft with them. More mature build-deploy-manage tool set based on these technologies will be truly game changing for hosted and enterprise clouds alike.

UnikernelLanguageTargetsFunctions
ClickOSC++XenNetwork Function Virtualization
HalVMHaskellXen
IncludeOSC++KVM, VirtualBox, ESXi, Google Cloud, OpenStackOrchestration tool available
MirageOSOCamlKVM, Xen, RTOS/MCU
Nanos UnikernelC, C++, Go, Java, Node.js, Python, Rust, Ruby, PHP, etcQEMU/KVMOrchestration tool available
OSvJava, C, C++, Node, RubyVirtualBox, ESXi, KVM, Amazon EC2, Google CloudCloud and IoT (ARM)
RumprunC, C++, Erlan, Go, Java, JavaScript, Node.js, Python, Ruby, RustXen, KVM
UnikGo, Node.js, Java, C, C++, Python, OCamlVirtualBox, ESXi, KVM, XEN, Amazon EC2, Google Cloud, OpenStack, PhotonControllerUnikernel compiler toolbox with orchestration possible through Kubernetes and Cloud Foundry
ToroKernelFreePascalVirtualBox, KVM, XEN, HyperVUnikernel dedicated to run microservices
Comparing few Unikernel solutions from active projects

Out of the various existing projects, some standout due to their wide range of supported languages. Out of the active projects, the above table describes the language they support, the hypervisors they can run on and remarks concerning their functionality.

Currently experimenting with the Unikernel in the AWS and Google Cloud Platform and will update you with another post on that soon.

Source: Medium, github, containerjournal, linuxjournal

Bare-Metal K8s Cluster with Raspberry Pi – Part 3

This is a continuation from the post series Bare-metal K8s cluster with Raspberry Pi – Part 1 & Part 2 here

Another option of running bare-metal K8s cluster in the Raspberry Pi I tried and tested was with Micro K8s which we discuss in this post.

Micro K8s are Lightweight upstream K8s. They are smallest, simplest, pure production K8s. For clusters, laptops, IoT and Edge, on Intel and ARM.

MicroK8s is a CNCF certified upstream Kubernetes deployment that runs entirely on your workstation or edge device. Being a snap it runs all Kubernetes services natively (i.e. no virtual machines) while packing the entire set of libraries and binaries needed. Installation is limited by how fast you can download a couple of hundred megabytes and the removal of MicroK8s leaves nothing behind.

And to give a context on snap, Snaps are app packages for desktop, cloud and IoT that are easy to install, secure, cross‐platform and dependency‐free. Snaps are discover able and install able from the Snap Store, the app store for Linux with an audience of millions.

A snap is a bundle of an app and its dependencies that works without modification across Linux distributions.

We are going to use the same components list as described in the Part 1 of this series.

Each Pi is going to need an Ubuntu server image and you’ll need to be able to SSH into them. Please follow this link here will help us to reach to this stage

Kubernetes Cluster Preparation with SSH connection to the Pi from your terminal

Installing MicroK8s
Follow this section for each of your Pis. Once completed you will have MicroK8s installed and running everywhere.

SSH to your first Pi and install the MicroK8s snap:

sudo snap install microk8s --classic

As MicroK8s is a snap and as such it will be automatically updated to newer releases of the package, which is following closely upstream Kubernetes releases, so we don’t need to worry about the K8s version we’re installing

sudo snap install microk8s --classic --channel=1.15/stable
Channels are made up of a track (or series) and an expected level of stability, based on MicroK8s releases (Stable, Candidate, Beta, Edge). For more information about which releases are available, run:

snap info microk8s

Cheat Sheet for MicroK8s
Before going further here is a quick intro to the MicroK8s command line:

  • microk8s.start – start all enabled Kubernetes services
  • microk8s.inspect – status of services
  • microk8s.stop – stop all Kubernetes services
  • microk8s.enable dns – enable Kubernetes add-ons,“kubedns”
  • microk8s.kubectl cluster-info – status of the cluster:

MicroK8s is easy to use and comes with plenty of Kubernetes add-ons you can enable or disable.

Master node and leaf nodes
Now that you have MicroK8s installed on all boards, pick one which has to be the master node of your cluster.

On the chosen Master node, run the following command:

sudo microk8s.add-node
This command will generate a connection string in the form of :/.

Adding a node
Now, you need to run the join command from another Pi you want to add to the cluster:

microk8s.join 10.55.60.14:25000/JHpbBYMIevZSAMnmjMHmFwanrOYCWZLu
You should be able to see the new node in a few seconds on the master with the following command:

microk8s.kubectl get node

For each new node, you need to run the microk8s.add-node command on the master, copy the output, then run microk8s.join on the leaf.

Removing nodes
To remove a node, run the following command on the master:

sudo microk8s remove-node
The name of nodes are available on the master by running the microk8s.kubectl get node

Alternatively, you can leave the cluster from a leaf node by running:

sudo microk8s.leave

Once Pis are setup with MicroK8s, adding and removing nodes is easy and you can scale up or down as you go.

Voila..!!
If you follow this series, you are now in control of your Kubernetes cluster. One with native kubernetes and docker and the other one with more easier to install and manage MicroK8s

This completes the 3 part series for K8s in Raspberry Pi. In a new follow-up blog post we can see how we can use kubectl & helm charts to deploy a Nginx service, Prometheus and few other services from the DevOps tools set to the cluster.

Bare-Metal K8s Cluster with Raspberry Pi – Part 2

This is a continuation from the post series Bare-metal K8s cluster with Raspberry Pi

As we have 1 master node and 3 nodes setup we continue to install Kubernetes.

Install Kubernetes

We are using version 1.15.3. There shouldn’t be any errors, however during my installation the repos were down and I had to retry in a few times.

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
sudo apt-key add - && echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | \
sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update -q

sudo apt-get install -qy kubelet=1.15.3-00 kubectl=1.15.3-00 kubeadm=1.15.3-00

Repeat steps for all of the Raspberry Pis.

Kubernetes Master Node Configuration
Note: You only need to do this for the master node (in this deployment I recommend only 1 master node). Each Raspberry Pi is a node.

Initiate Master Node

sudo kubeadm init
Enable Connections to Port 8080
Without this Kubernetes services won’t work

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Add Container Network Interface (CNI)
I’ve chosen to use Weaver, however you can get others working such as Flannel (I’ve verified this works with this cluster)

Get Join Command
This will be used in the next section to join the worker nodes to the cluster. It will return something like:

kubeadm join 192.168.0.101:6443 --token X.Y --discovery-token-ca-cert-hash sha256:XYZ
kubeadm token create --print-join-command

Kubernetes Worker Node Configuration
Note: You only need to do this for the worker nodes (in this deployment I recommend 3 worker node).

Join Cluster
Use the join command provided at the end of the previous section
sudo kubeadm join 192.168.0.101:6443 --token X.Y --discovery-token-ca-cert-hash sha256:XYZ

Verify Node Added Successfully (SSH on Master Node)
Should have status ready after ~30 seconds
kubectl get componentstatuses

Another option of running bare-metal K8s cluster in the Raspberry Pi I tried and tested was with Micro K8s will posted in the Part 3 of this series.

A sneak peak into the K8s cluster in Raspberry pi

Bare-Metal K8s Cluster with Raspberry Pi – Part 1

There are multiple ways we can use a Kubernetes cluster to deploy our applications. Most of us opt to use any Kubernetes service from a public cloud provider. GKE, EKS, AKS are the most prominent ones. Deploying a Kubernetes cluster on a public cloud provider is relatively easy, but what if you want a private bare-metal K8s cluster. Being worked extensively in the data center and started my career as a Sys Admin, I personally prefer a piece of tangible hardware to get the feel of building it. This blog post walk you through the steps I took in order to have a bare-metal K8s cluster to play with.

K8s is an open source container orchestration platform that helps manage distributed, containerized applications at a massive scale. Born at Google as Borg, version 1.0 was released in July 2015. It has continued to evolve and mature and is now offered as a PaaS service by all of the major cloud vendors.

Google has been running containerized workloads in production for more than a decade. Whether it’s service jobs like web front-ends and stateful servers, infrastructure systems like Bigtable and Spanner, or batch frameworks like MapReduce and Millwheel, virtually everything at Google runs as a container.

You can find the paper here.

Kubernetes traces its lineage directly from Borg. Many of the developers at Google working on Kubernetes were formerly developers on the Borg project. We’ve incorporated the best ideas from Borg in Kubernetes, and have tried to address some pain points that users identified with Borg over the years.

More than just enabling a containerized application to scale, Kubernetes has release-management features that enable updates with near-zero downtime, version rollback, and clusters that can ‘self-heal’ when there is a problem. Load balancing, auto-scaling and SSL can easily be implemented. Helm, a plugin for Kubernetes, has revolutionized the world of server management by making multi-node data stores like Redis and MongoDB incredibly easy to deploy. Kubernetes enables you to have the flexibility to move your workload where it is best suited. This compliments the hybrid cloud story and in my career it has become more apparent that my customers see this as well to help them resolve issues like; cost, availability and compliance. In parallel software vendors are starting to embrace containers as a standard deployment model leading to a recent increase in requests for container solutions.

As you can see in the workflow comparison below, there is greater room for error when deploying on-premises. Public clouds provide the automation and reduces the risk of error as less steps are required. But as mentioned above, private cloud provides you more options when you have unique requirements.

Pros:

  • Using Kubernetes and its huge ecosystem can improve your productivity
  • Kubernetes and a cloud-native tech stack attracts talent
  • Kubernetes is a future proof solution
  • Kubernetes helps to make your application run more stable
  • Kubernetes can be cheaper than its alternatives

Cons:

  • Kubernetes can be an overkill for simple applications
  • Kubernetes is very complex and can reduce productivity
  • The transition to Kubernetes can be cumbersome
  • Kubernetes can be more expensive than its alternatives

Pre-requisites:

Compute:

3 x Raspberry Pi 4 Model B with 2 GB RAM
1 x Raspberry Pi 3 Model B+ with 1 GB RAM

Storage:

4 x 16GB High Speed Sand-disk Micro-SD Cards

Network:

1 x Network Switch – for local LAN for k8s internal connectivity
1 x Network Router – for Wifi (My default ISP router was used here) only master node had internet connectivity once completed the setup
4 x Ethernet Cables
1 x Keyboard, HDMI, Mouse (for initial setup only)

Initial Raspberry Pi Configuration:

Flash Raspbian to the Micro-SD Cards

Download image from the below link,

Raspbian OS

I have used BalenaEtcher to flash image onto micro-SD card

Perform Initial Setup on Boot on startup screen, we need to connect keyboard, monitor and mouse for this setup.

Choose Country, Language, Timezone
Define new password for user ‘pi’
Connect to WiFi or skip if using ethernet
Skip update software (We will perform this activity manually later).
Choose restart later

Configure Additional Settings Click the Raspberry Pi icon (top left of screen) > Preferences > Raspberry Pi Configuration

System

Configure Hostname
Boot: To CLI

Interfaces
SSH: Enable

Choose restart later

Configure Static Network Perform one of the following:
Define Static IP on Raspberry Pi: Right Click the arrow logo top right of screen and select ‘Wireless & Wired Network Settings’

Define Static IP on DHCP Server: Configure your DHCP server to define a static IP on the Raspberry Pi Mac Address.

Reboot and Test SSH
Username: pi

Password: Defined in step 2 above

From the terminal ssh pi@[IP Address]

Repeat steps for all of the Raspberry Pis.

Kubernetes Cluster Preparation with SSH connection to the Pi from your terminal. I am using a 12 years old Lenovo laptop running MX Linux. Open a terminal and establish ssh connection to the Pi

Perform Updates

apt-get update: Updates the package indexes
apt-get upgrade: Performs the upgrades

Configure Net.IP4.IP configuration Edit sudo vi /etc/sysctl.conf, uncomment net.ipv4.ip_forward = 1 and add net.ipv4.ip_nonlocal_bind=1.
Note: This is required to allow for traffic forwarding, for example Node Ports from containers to/from non-cluster devices.

Install Docker

curl -sSL get.docker.com | sh

Grant privilege for user pi to execute docker commands

sudo usermod pi -aG docker

Disable Swap

sudo systemctl disable dphys-swapfile.service
sudo reboot

We can verify this with the top command, on the top left corner next to MiB Swap should be 0.0.

As we completed the initial steps to create our K8s bare-metal cluster, we can see how we build the cluster in Part 2 of this blog post

Linux Inside Win 10

I am a zealous fan of Linux and FOSS. I have been using Linux and it’s TUI with bash shell for more than seventeen years. When I moved to my new role I find it bit difficult when I had a Windows 10 laptop and was literally fumbling with the powershell and cmd line when I tried working with tools like terraform, git etc. But luckily I figured out a solution for the old school *NIX users like me who are forced to use a Windows laptop, and that solution is WSL.

Windows Subsystem for Linux a.k.a WSL is an environment in Windows 10 for running unmodified Linux binaries in a way similar to Linux Containers. Please go through my earlier post on LinuxContainers for more details on it. WSL runs Linux binaries by implementing a Linux API compatibility layer partly in the Windows kernel when it was introduced first. The second iteration of it, “WSL 2” uses the Linux kernel itself in a lightweight VM to provide better compatibility with native Linux installations.

To use WSL in Win 10, you have to enable wsl feature from the Windows optional features. Being a aficionado of command line than the GUI, I’ll now list out step by step commands in order to enable the WSL and install your favourite Linux distribution and how to use it.

  1. Open Powershell as administrator and execute the command below Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
    Note: Restart is required when prompted
  2. Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1804 -OutFile Ubuntu.appx -UseBasicParsing

In the above command you can use all the Linux distros available for WSL. For example if you want to use Kali Linux instead of Ubuntu, you can edit the distro url in the step 2 command as “https://aka.ms/ wsl-kali-linux”. Please refer to this guide for all available distros.

Once the download is completed, you need to add that package to the Windows 10 application list which can be done using the command below.

  1. Add-AppxPackage .\Ubuntu.appx

Once all these steps are completed, in the Win 10 search (magnifying glass in the bottom left corner near to the windows logo) type Ubuntu and you can see your Ubuntu (you need to search which ever wsl distro you have added). To complete the initialization of your newly installed distro, launch a new instance which is Ubuntu in our case by selecting and running Ubuntu from search as seen in the screen shot below.

This will start the installation of your chosen Linux distros binaries and libraries along with the kernel. It will take some time to complete the installation and configuration (approximately 5 to 10 minutes depending on your laptop/desktop configuration).

Once installation is complete, you will be prompted to create a new user account (and its password).

Note: You can choose any username and password you wish – they have no bearing on your Windows username.

If everything goes well, we’ll have Ubuntu installed as a sub system. When you open a new distro instance, you won’t be prompted for your password,
but if you elevate your privileges using sudo, you will need to enter your password.

Next step is to updating your package catalog, and upgrading your installed packages with the Ubuntu package manager apt. To do so, execute the below command in the prompt.

$ sudo apt update && sudo apt upgrade

Now we will proceed to install Git.

$ sudo apt install git

$ git --version

And finally test out git installation with the above command. You can also install other tools, packages available in the repository. I have installed git, terraform, aws-cli, azure-cli and ansible. You can install python, ruby, go programming environment as well. Python pip and ruby gem installations are also supported. You can use this sub system as an alternative for your day to day Linux operations and as an alternative terminal for your Powershell if you are using Sublime Text or Atom or Visual studio code

Arch installation – Part 1

This is a continuation for my earlier post on Arch Linux. In this blog we’ll try to install and tame Arch Linux in an easier and quickest way.

The default installation covers only a minimal base system and expects the end user to configure and use it. Based on the KISS – Keep It Simple, Stupid! principle, Arch Linux focus on elegance, code correctness, minimalist system and simplicity.

Arch Linux supports the Rolling release model and has its own package manager – pacman. With the aim to provide a cutting-edge operating system, Arch never misses out to have an up-to-date repository. The fact that it provides a minimal base system gives you a choice to install it even on low-end hardware and then install only the required packages over it.

Also, its one of the most popular OS for learning Linux from scratch. If you like to experiment with a DIY attitude, you should give Arch Linux a try. It’s what many Linux users consider a core Linux experience.

You have been warned…!!! The method we are going to discuss here wipes out existing operating system(s) from your computer and install Arch Linux on it. So if you are going to follow this tutorial, make sure that you have backed up your files or else you’ll lose all of it or else your are installing in a fresh VM (You can deploy a VM easily with a Virtual Box, VMWare Workstation, KVM, Xen etc). Deploying a VM with any of those afore mentioned tools is completely out of scope for this post. again I would reccomend you to try this installation in any of your old laptop or a new VM

Requirements for installing Arch Linux:

  • A x86_64 (i.e. 64 bit) compatible machine
  • Minimum 512 MB of RAM (recommended 2 GB)
  • At least 1 GB of free disk space (recommended 20 GB for basic usage)
  • An active internet connection
  • A USB drive with minimum 2 GB of storage capacity
  • Familiarity with Linux command line

You have been warned..!!! If you are a person addicted to GUI and doesn’t want to do a DIY kind of Linux installation, IMHO, please discontinue reading this and go ahead with Arch derivatives like Manjaro.

Step 1: Download the ISO

You can download the ISO from the official website.
I prefer using the magnet link and downloading the ISO with a torrent client.

Step 2: Create a live USB of Arch Linux

We will have to create a live USB of Arch Linux from the ISO you just downloaded.

If you are on Linux, you can use dd command to create a live USB. I recommend you to try Balena Etcher for creating your usb. This is my favourite way of doing it. It is a portable piece of software and can be used on Windows/Mac OS/Linux etc

You can download it from the link here

Once you have created a live USB for Arch Linux, shut down your PC. Plugin your USB and boot your system. While booting keep pressing ESC, F2, F10 or F12 (depending upon your system) to go into boot settings. In here, select to boot from USB or removable disk. You will be presented with a Linux terminal which will be automatically logged in as root user.

As the initial steps are completed, we’ll discuss the installation part in my next post

Demystifying Arch

There is always a lot of buzz around the Arch Linux distribution in the Linux community. Those who are switching to Linux as a desktop operating system are very reluctant to go the Arch way and those who have experienced with Linux as well. IMHO it’s only because of the mystery prevailing around this distro. Here in this blog we’ll try to address some of those and help to embrace everyone to Arch world with out any concerns.

Arch principles are: Simplicity, Modernity, Pragmatism, User Centrality and Versatility.

Arch Linux is an independently developed, x86-64 general-purpose GNU/Linux distribution that strives to provide the latest stable versions of most software by following a rolling-release model. The default installation is a minimal base system, configured by the user to only add what is purposely required. In other words Arch is a DIY Linux instalaltion where the user is having full control on each and every component getting installed and configured.

The original creator of Arch Linux, Judd Vinet was in total love with the simplicity and elegance of Slackware, BSD, etc. He wanted a distribution that could be DIY, you make it yourself from ground up and it only has what you want it to have.

The solution was to use a shell based manual installation instead of the automatic ncurses or graphical based installations that lead you to learn almost nothing and you know nothing about what exactly was installed on the system. The ZSH based installer of Arch Linux means that you do everything using commands. All this means that you know exactly what is happening, since you are the one doing it. You can customize everything right from the pre-installation steps. You don’t install the OS in Arch. You just download the base system and install it in the root partition/directory. Then you install other required packages needed by you by either chrooting into the system or booting into the new system and it will be a terminal, as it is only base system install the you’ve done. Install and configure each package, you must know what are your required packages and how they are installed and configured. That is how Arch Linux haelps you install your on custom Linux (DIY) and help you to learn by doing, breaking & repeating.

Arch Linux doesn’t care about being easy for Ubuntu/Debian/Redhat/Fedora style to set up. But, there do exist easy-to-install variants if anyone wants to have a touch and feel os Arch Linux; Antergos & Manjaro is essentially Arch with a graphical installer.

Typically in a more simpler steps Arch installation will be like,

  • Download ISO Image
  • Burn the image to a DVD/USB
  • Boot Arch from the media
  • create disk partitions
  • Setup network with/without DHCP, including wired or wireless network
  • Optimize gcc for the specific CPU
  • Config/compile Linux kernel & modules
  • Base packages
  • Environmental configuration
  • Necessary softwares/tools/applications
  • setup X server and GUI

In one word, the installation bundle in those distributions like Debian and CentOS does all things above in a more user-friendly way or even automatically, which you will have to do manually step by step in Arch Linux. But it is not at all a hard to do thing as it sounds. We’ll walk through the installation in the next post.

Linux Containers

The Linux Containers project or LinuX Containers (LXC) is an open source container platform that provides a set of tools, templates, libraries, and language bindings. LXC has a simple command line interface that improves the user experience when starting containers. LXC is a user-space interface for the Linux kernel containment features. Through a powerful API and simple tools, it lets Linux users easily create and manage system or application containers.

LXC offers an operating-system level virtualization environment that is available to be installed on many Linux-based systems. Your Linux distribution may have it available through its package repository. LXC is a free software, most of the code is released under the terms of the GNU LGPLv2.1+ license, some Android compatibility bits are released under a standard 2-clause BSD license and some binaries and templates are released under the GNU GPLv2 license.

LXC is a OS-level virtualization technology that allows creation and running of multiple isolated Linux virtual environments (VE) on a single control host. These isolation levels or containers can be used to either sandbox specific applications, or to emulate an entirely new host. LXC uses Linux’s cgroups functionality, which was introduced in kernel version 2.6.24 to allow the host CPU to better partition memory allocation into isolation levels called namespaces. Note that a VE is distinct from a virtual machine (VM).

The idea of what we now call container technology first appeared in 2000 as FreeBSD jails, a technology that allows the partitioning of a FreeBSD system into multiple subsystems, or jails. Jails were developed as safe environments that a system administrator could share with multiple users inside or outside of an organization.

In 2001, an implementation of an isolated environment made its way into Linux, by way of Jacques Gélinas’ VServer project. Once this foundation was set for multiple controlled userspaces in Linux, pieces began to fall into place to form what is today’s Linux container.

Very quickly, more technologies combined to make this isolated approach a reality. Control groups (cgroups) is a kernel feature that controls and limits resource usage for a process or groups of processes. And systemd, an initialization system that sets up the userspace and manages their processes, is used by cgroups to provide greater control over these isolated processes. Both of these technologies, while adding overall control for Linux, were the framework for how environments could be successful in staying separated.

Current LXC uses the following kernel features to contain processes:

  • Kernel namespaces (ipc, uts, mount, pid, network and user)
  • Apparmor and SELinux profiles
  • Seccomp policies
  • Chroots (using pivot_root)
  • Kernel capabilities
  • CGroups (control groups)

LXC containers are often considered as something in the middle between a chroot and a full fledged virtual machine. The goal of LXC is to create an environment as close as possible to a standard Linux installation but without the need for a separate kernel. LXC is currently made of a few separate components:

  • The liblxc library
  • Several language bindings for the API:
  • python3 (in-tree, long term support in 2.0.x)
  • lua (in tree, long term support in 2.0.x)
  • Go
  • ruby
  • python2
  • Haskell
  • A set of standard tools to control the containers
  • Distribution container templates

Now we will quickly explore the LXC containers.

For most modern Linux distributions, the kernel is enabled with cgroups, but you most likely will need to install the LXC utilities.

If you’re using Red Hat or CentOS, you’ll need to install the EPEL repositories first. For other distributions, such as Ubuntu or Debian, simply type:

$ sudo apt-get install lxc

Create the ~/.config/lxc directory if it doesn’t already exist, and copy the /etc/lxc/default.conf configuration file to ~/.config/lxc/default.conf. Append the following two lines to the end of the file:

lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536

Append the following to the /etc/lxc/lxc-usernet file (replace the first column with your user name):

akbharat veth lxcbr0 10

The quickest way for these settings to take effect is either to reboot the host or log the user out and then log back in.

Once logged back in, verify that the veth networking driver is currently loaded:

$ lsmod|grep veth
veth 16384 0

$ sudo modprobe veth

If in case you couldn’t see the module ‘veth’ not loaded you can use the command above to load it to the kernel.

Next, download a container image and name it “my-container”. When you type the following command, you’ll see a long list of supported containers under many Linux distributions and versions:

$ sudo lxc-create -t download -n my-container

You’ll be given three prompts to pick the distribution, release and architecture. I chose the following:

Distribution: ubuntu
Release: xenial
Architecture: amd64

Once you press Enter, the rootfs will be downloaded locally and configured. For security reasons, each container does not ship with an OpenSSH server or user accounts. A default root password also is not provided. In order to change the root password and log in, you must run either an lxc-attach or chroot into the container directory path (after it has been started). We will use the command below to start the container

$ sudo lxc-start -n my-container -d

The -d option dæmonizes the container, and it will run in the background. If you want to observe the boot process, replace the -d with -F, and it will run in the foreground, ending at a login prompt.

Open up a second terminal window and verify the status of the container:

$ sudo lxc-info -n my-container
Name: my-container
State: RUNNING
PID: 1356
IP: 10.0.3.28
CPU use: 0.29 seconds
BlkIO use: 16.80 MiB
Memory use: 29.02 MiB
KMem use: 0 bytes
Link: vethPRK7YU
TX bytes: 1.34 KiB
RX bytes: 2.09 KiB
Total bytes: 3.43 KiB

There is also another way to see a list of all installed containers, which is provided below:

$ sudo lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6
my-container RUNNING 0 - 10.0.3.28 -

But still you will not be able to use it, so for that you can attach the container directly with your LXC tool sets and work with it.

$ sudo lxc-attach -n my-container
root@my-container:/#

If you set a username now from within the container, you can either use console command to connect to the container with your newly created username and password.

$ sudo lxc-console -n my-container

If you want to connect to the container using SSH, you can install the Open SSH server in the container and then figure out the IP address from the container with the commands below and connect to the IP from your linux host.

root@my-container:/# apt-get install openssh-server

root@my-container:/# ip addr show eth0|grep inet

From your Linux host now ssh to the container using the IP we captured from the earlier command.

$ ssh 10.0.3.25

On the host system, and not within the container, it’s interesting to observe which LXC processes are initiated and running after launching a container:

$ ps aux|grep lxc|grep -v grep
root 861 0.0 0.0 234772 1368 ? Ssl 11:01
↪0:00 /usr/bin/lxcfs /var/lib/lxcfs/
lxc-dns+ 1155 0.0 0.1 52868 2908 ? S 11:01
↪0:00 dnsmasq -u lxc-dnsmasq --strict-order
↪--bind-interfaces --pid-file=/run/lxc/dnsmasq.pid
↪--listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254
↪--dhcp-lease-max=253 --dhcp-no-override
↪--except-interface=lo --interface=lxcbr0
↪--dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases
↪--dhcp-authoritative
root 1196 0.0 0.1 54484 3928 ? Ss 11:01
↪0:00 [lxc monitor] /var/lib/lxc my-container
root 1658 0.0 0.1 54780 3960 pts/1 S+ 11:02
↪0:00 sudo lxc-attach -n my-container
root 1660 0.0 0.2 54464 4900 pts/1 S+ 11:02
↪0:00 lxc-attach -n my-container

And finally to stop the container and verify whether the container is stopped we can use the following commnds in sequence,

$ sudo lxc-stop -n my-container

$ sudo lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6
my-container STOPPED 0 - - -

$ sudo lxc-info -n my-container
Name: my-container
State: STOPPED

You might also want to destroy the contianer:

$ sudo lxc-destroy -n my-container
Destroyed container my-container

This is a very basic example of a container capability. Now in modern container era containerization is much more sophisticated and Docker is a significant improvement of LXC’s capabilities. Its obvious advantages are gaining Docker a growing following of adherents. In fact, it starts getting dangerously close to negating the advantage of VM’s over VE’s because of its ability to quickly and easily transfer and replicate any Docker-created packages. We’ll discuss more on Docker in my next post.