Linux Interview Questions

These are some of the Interview questions we were using while conducting L1, L2 & L3 level interviews, I’ll not posting any answers to these questions as it’s open up to you to search and find the answers. It will help you to understand the topic and you’ll remember the answers all the time.

What is the difference between kernel image and initrd image?

Why is root file system mounted read-only initially?

What will happen if it is mounted read-write?

What are the process states in Linux?

Which log files holds the information of currently logged users?

What is called a fork bomb?What is its relevance?

What is the difference between commands last and lastb?

How do you ensure your users have hard-to-guess passwords?

What is log rotate and Why is it used in a Linux system?

How will you increase the priority of a process?

What is the range for the priority of the processes?

I have a file named “/ \+Xy \+\8_/ \” How do I get rid of it?

What does the command pvscan do?

How will you probe for a new SCSI drive?

How will you take a system backup image for recovery?

What will the command chage –E -1 do?

What will the command option –a will do for usermod?

What is the command gpasswd?

When debugging a core in gdb, what does the command `bt` give?

What do you mean by man (1) (5) and (8)?

How many fields in crontab and what are those?

To display a list of all manual pages containing the keyword “date”, which commad can be used?

What do you mean by restricted deletion flag?

What’s the difference between `telnet` and `ssh`? What’s a good use for each?

What will vgcfgbackup and vgcfgrestore do and can you explain with a scenario?

What is the difference between insmod and modprobe?

What to do if the newly built kernel does not boot?

What are TCP wrappers? What is the library responsible for TCP wrappers?

In how many ways you can block the traffic to ssh from a specific host?

How will you run ssh in debugging mode to identify a failure in establishing a connection?

Which file is responsible for kernel parameter changes?

What do you mean by hardening of a Linux server?How will you achieve this?

How will you tune a Linux server for maximum performance?

What is the use of commands clustat clusvcadm?

What is multipathing?Why is it used?

What is black-listing of devices in multipath?

Which command will display the current multipath configuration gathered from sysfs, the device mapper, and all other available components on the system?

What is multipath interactive console?

What is GFS?

How will you create a GFS file system?

What is called NIC bonding or teaming?

How will you repair a GFS file system?



How do you read arguments in a shell program?

How do you send a mail message to somebody within a script?

Which command is used to count lines and/or characters in a file?

How do you start a process in the background?

How will you define a function in a shell script?

How do you get character positions 10-20 from a text file?

What do you mean by getopts?

How do you test for file properties in a shell script?

How will you define an array?

How can you take in key-board inputs to a shell script?

What is meant by traps in a shell scripts?

Is it possible to use alphanumeric in for loop?

Write a Palindrome script?

Write a script to check no of machines up in network ( Provided list of hosts are available in a file)?

if u have f1;f2;f3 words, need to display f3 first then f1 and at last f2 using a script?

Print the numbers of 1 to 1000 in the format 0001,0002,0003………1000?

Magic SysRq key of Linux

The magic SysRq key is a key combination in the Linux kernel which allows the user to perform various low-level commands regardless of the system’s state.

It is often used to recover from freezes or to reboot a computer without corrupting the filesystem. The key combination consists of Alt+SysRq+commandkey. In many systems, the SysRq key is the PrintScreen key.

First, you need to enable the SysRq key, as shown below.

# echo “1” > /proc/sys/kernel/sysrq

List of SysRq Command Keys

Following are the command keys available for Alt+SysRq+commandkey.

‘k’ – Kills all the process running on the current virtual console.
’s’ – This will attempt to sync all the mounted file system.
‘b’ – Immediately reboot the system, without unmounting partitions or syncing.
‘e’ – Sends SIGTERM to all process except init.
‘m’ – Output current memory information to the console.
‘i’ – Send the SIGKILL signal to all processes except init
‘r’ – Switch the keyboard from raw mode (the mode used by programs such as X11), to XLATE mode.
’s’ – sync all mounted file systems.
‘t’ – Output a list of current tasks and their information to the console.
‘u’ – Remount all mounted filesystems in read-only mode.
‘o’ – Shutdown the system immediately.
‘p’ – Print the current registers and flags to the console.
‘0-9′ – Sets the console log level, controlling which kernel messages will be printed to your console.
‘f’ – Will call oom_kill to kill a process which takes more memory.
‘h’ – Used to display the help. But any other keys than the above listed will print help.

We can also do this by echoing the keys to the /proc/sysrq-trigger file. For example, to reboot a system you can perform the following.

# echo “b” > /proc/sysrq-trigger

Perform a Safe reboot of Linux using Magic SysRq Key

To perform a safe reboot of a Linux computer which hangs up, do the following. This will avoid the fsck during the next re-booting. i.e Press Alt+SysRq+letter highlighted below.

unRaw (take control of keyboard back from X11,
tErminate (send SIGTERM to all processes, allowing them to terminate gracefully),
kIll (send SIGILL to all processes, forcing them to terminate immediately),
Sync (flush data to disk),
Unmount (remount all filesystems read-only),

Finding the ‘find’ command

The common problem users or admins run into when first dealing with a Linux machine is how to find the files they are looking for.

We discuss here the GNU/Linux find command which is one of the most important and much-used commands in Linux systems. find is used to search and locate a list of files and directories based on conditions you specify for files that match the arguments. Find can be used in a variety of conditions like you can find files by permissions, users, groups, file type, date, size and other possible criteria.

1. Find files under the current directory or in a specific directory:

ajoy@testserver:~$ find . -name file2.txt 
ajoy@testserver:~$ find /home/ajoy/ -name file5.txt 

2. Find only files or only directories

ajoy@testserver:~$ find /home/ajoy -type f -name file1.txt
ajoy@testserver:~$ find /home/ajoy -type f -name “*.txt”
==result truncated=====

In the above example if we replace -type f to -type d find will look for only directories.

ajoy@testserver:~$ find . -type d -name “test”

3. Find files ignoring the case

ajoy@testserver:~$ find . -iname file2.txt
ajoy@testserver:/tmp$ find /home/ajoy/ -iname file5.txt

4. Find files limitting the directory traversal

ajoy@testserver:~$ find test/ -maxdepth 3 -name “*.py”
ajoy@testserver:~$ find test/ -maxdepth 2 -name “*.py”

5. Find file inverting the match

ajoy@testserver:~$ find test/ -not -name “*.py”

6. Find with multiple search criterias

ajoy@testserver:~$ find test/ -name “*.txt” -o -name “*.py”

7. Find files with certain permissions

ajoy@testserver:~$ find . -type f -perm 0664

ajoy@testserver:~$ sudo find / -maxdepth 2 -perm /u=s 2>/dev/null

The above example shows the files with suid permissions set. The /dev/null bit bucket is used to remove errors related to permission while find traverses through directories

ajoy@testserver:~$ sudo find / -maxdepth 2 -type d -perm /o=t 2>/dev/null

The above example shows the directories  with sticky bit

8. Find files based on users and groups

ajoy@testserver:~$ sudo find /var -user www-data

ajoy@testserver:~$ sudo find /var -group crontab

9. Find files as per access time and modified time and changed time

ajoy@testserver:~$ find / -maxdepth 2 -mtime 50 ==> last modified 50 days back

ajoy@testserver:~$ find / -maxdepth 2 -atime 50 ==> last accessed 50 days back

ajoy@testserver:~$ find /  -mtime +50 -mtime -100 ==> modified between 50 to 100 days ago

ajoy@testserver:~$ find .  -cmin -60 ==> changed in last 60 minutes (1 hour)

ajoy@testserver:~$ find / -mmin -60 ==> modified in last 60 minutes

ajoy@testserver:~$ find / -amin -60 ==> accessed in last 60 minutes

10. Find files based on size

ajoy@testserver:~$ find .  -size 50 ==> all files of 50 MB

ajoy@testserver:~$ find /  -size +50 -size -100 ==> all files greater than 50 MB & less than 100 MB

ajoy@testserver:~$ find /var -type f -empty ==> empty file

ajoy@testserver:~$ find /var -type d -empty ==> empty directory

11. Listing out files found with find

ajoy@testserver:~$ find . -maxdepth 1 -name “*.txt” -exec ls -l {} \;
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file3.txt
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file6.txt
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file1.txt
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file5.txt
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file4.txt
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file2.txt
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file8.txt
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file7.txt
-rw-rw-r– 1 ajoy ajoy 0 Jan 31 17:40 ./file9.txt

The above command will long list the files matching the find criteria and the one given below will remove the files matching the given criteria

ajoy@testserver:~$ find . -maxdepth 1 -name “*.txt” -exec rm -f {} \;
ajoy@testserver:~$ ls test

12. Finding Smallest and Largest file

4 largest files in the current directory and sub dorectories

ajoy@testserver:~$ sudo find /var -type f -exec ls -l {} \; |sort -n -r |head -4
-rwxr-xr-x 1 root root 998 Dec 5 2012 /var/lib/dpkg/info/sgml-base.preinst
-rwxr-xr-x 1 root root 991 Mar 25 2013 /var/lib/dpkg/info/ureadahead.postinst
-rwxr-xr-x 1 root root 980 Sep 23 2014 /var/lib/dpkg/info/man-db.postrm
-rwxr-xr-x 1 root root 972 May 15 2014 /var/lib/dpkg/info/locales.preinst

2 smallest files in the current directory and sub dorectories

ajoy@testserver:~$ sudo find /var -type f -exec ls -l {} \; |sort -n |head -2 
-rw——- 1 daemon daemon 2 Jan 31 16:36 /var/spool/cron/atjobs/.SEQ
-rw——- 1 root ajoy 40 Jan 31 20:16 /var/lib/sudo/ajoy/0

Command Line Tips and Tricks

To find the largest file:

  • Search under /home for files greater than 100MB, where c stands for bytes.

# find /home -size +104857600c

  • This following command will output all the files under /usr/share which is more than 50MB in the format “file name : file size”

# find /usr/share -type f -size +50000k -exec ls –lh {} \; | awk ‘{ print $9 “: ” $5 }’

  • The below script will display every directory under the directory given as an argument to this script with its size sorted by the largest at the bottom.

Add the following lines to a file and give the file execute permission,


du -sm $(find $1 –maxdepth 1 –xdevtype d)|sort -g

Then do a chmod +x <filename> to assign execute permission

  • The following two commands helps you in detecting the empty files.

# find . -size 0c or,

# find . -empty 

To Fix a Corrupt RPM database:

Execute the below commands as root user

# rm /var/lib/rpm/_db*

# rpm -v rebuilddb

Checking for SELinux :

# getenforce

This command will tell whether the selinux is in Enforcing, Permissive or Disabled mode

# ls -Z

This command helps in identifying the files and folders against selinux details.

# echo 0 > /selinux/enforcing

This will helps to change the selinux from enforcing to permissive temporarily (echo 1 > /selinux/enforcing to set it back to enforcing)

To disable the selinux permanently open the following file in your favourite editor and change the SELINUX context to disabled

# vi /etc/selinux/config


Increase the semaphore count in Linux:

# ipcs -a

—— Shared Memory Segments ——–
key shmid owner perms bytes nattch status
0x00000000 0 gdm 600 393216 2 dest
0x00000000 32769 gdm 600 393216 2 dest
0x00000000 65538 gdm 600 393216 2 dest
0x00000000 98307 gdm 600 393216 2 dest

—— Semaphore Arrays ——–
key semid owner perms nsems
0x00000000 0 root 600 1
0x00000000 65537 root 600 1

—— Message Queues ——–
key msqid owner perms used-bytes messages

# sysctl -a |grep sem
kernel.sem = 250 32000 32 128

# sysctl -w kernel.sem=300
kernel.sem = 300

# sysctl -a |grep sem
kernel.sem = 300 32000 32 128

We need to edit the /etc/sysctl.conf file to make this changes persistent across reboot.

The quick way to find the hardware model of your server:

# dmidecode -t 1

  • An example output,

# dmidecode 2.11
SMBIOS 2.4 present.

Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: VMware, Inc.
Product Name: VMware Virtual Platform
Version: None
Serial Number: VMware-56 4d 42 b0 ee 0c 84 de-af 2d 47 d3 95 40 70 ef
UUID: 564D42B0-EE0C-84DE-AF2D-47D3954070EF
Wake-up Type: Power Switch
SKU Number: Not Specified
Family: Not Specified

  • Another example output,

# dmidecode 3.0
Scanning /dev/mem for entry point.
SMBIOS 2.4 present.

Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Xen
Product Name: HVM domU
Serial Number: ec2c4ad4-1afd-abba-aa66-74e921a4a3b7
Wake-up Type: Power Switch
SKU Number: Not Specified
Family: Not Specified

The Product Name will tell you server details like its is VMWare, HP Proliant, Dell Poweregde etc.

To get the details of the IP Address and other N/W aspects:
# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:0C:29:40:70:EF
inet addr: Bcast: Mask:
inet6 addr: fe80::20c:29ff:fe40:70ef/64 Scope:Link
RX packets:594 errors:0 dropped:0 overruns:0 frame:0
TX packets:530 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:54336 (53.0 KiB) TX bytes:127273 (124.2 KiB)

eth1 Link encap:Ethernet HWaddr 00:0C:29:40:70:F9
inet6 addr: fe80::20c:29ff:fe40:70f9/64 Scope:Link
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:2700 (2.6 KiB)

lo Link encap:Local Loopback
inet addr: Mask:
inet6 addr: ::1/128 Scope:Host
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:240 (240.0 b) TX bytes:240 (240.0 b)

the typical ifconfig command gives you info about all the interfaces configured in the system

the short and simple method you can use is the ip command as below,

# ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:40:70:ef brd ff:ff:ff:ff:ff:ff
inet brd scope global eth0
inet6 fe80::20c:29ff:fe40:70ef/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:40:70:f9 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe40:70f9/64 scope link
valid_lft forever preferred_lft forever

# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface U 0 0 0 eth0 U 0 0 0 eth0

This will provode the kernel routing table and so is the one below,

# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
link-local * U 1002 0 0 eth0 * U 0 0 0 eth0

# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

the ethtool provides the details of all your network interface.

# mii-tool eth1
eth1: negotiated 100baseTx-FD, link ok

this command also throw some brief info

Setting a Temporary IP:

#ifconfig eth0 netmask

#route add default gw

This settings will be lost the moment you reboot your system

Run a command repeatedly and display the output:

#watch -d ls -l

by default the program run every two seconds and lists the contents of the directory where the command executed

another way is a simple while loop

#while true; do ls -l; done

Information on a command:

# which ping

the absolute path of a command

# rpm -qf /bin/ping

which pakage provides you this command

Add the words horizontally to a file:

To paste the contents of two files horizontally we can take the help of paste command,

# cat command

# cat ip

# paste -d ” ” command ip > output

-d ” ” will seperate the contents by a single space instead of a tab which is default.

# cat output

Redirecting the top command output to a file:

# top -b -n2 -d5 > /tmp/top.out

This command will run top 2 times and wait 5 seconds between each output.

Create a directory with different permission:

# mkdir /root/mydir -v -m 1777
mkdir: created directory `/root/mydir’

# ls -lhd /root/mydir/
drwxrwxrwt. 2 root root 4.0K Aug 9 20:35 /root/mydir/

This can be used instead of creating a directory and use chmod command to change the permissions

Create multiple files or directories:

#touch myfile{1..5}.txt

# ls my*

myfile1.txt  myfile2.txt  myfile3.txt  myfile4.txt  myfile5.txt

# mkdir dir{1..4}

Commands cat and tac:

# cat file
This is Linux
This is Ubuntu

# tac file
This is Ubuntu
This is Linux
[root@docker ~]#

Linux Process & Process Management


A Linux server, like any other computer you may be familiar with, runs applications. To the computer, these are considered “processes”.

While Linux will handle the low-level, behind-the-scenes management in a process’s lifecycle, you will need a way of interacting with the operating system to manage it from a higher-level.

In this post, we will discuss some simple aspects of process management. Linux provides an abundant collection of tools for this purpose.

How To View Running Processes in Linux

The easiest way to find out what processes are running on your server is to run the top command:

# top

top – 07:45:38 up 9:38, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 89 total, 2 running, 87 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1014976 total, 517484 free, 102656 used, 394836 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 728192 avail Mem

1 root 20 0 128092 6680 3932 S 0.0 0.7 0:02.84 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/0
6 root 20 0 0 0 0 S 0.0 0.0 0:00.32 kworker/u30:0
7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
————- output truncated ——————-

The top chunk of information gives system statistics, such as system load and the total number of tasks.

You can easily see that there is 2 running process, and 87 processes are sleeping (aka idle/not using CPU resources).

The bottom portion has the running processes and their usage statistics.

Though top gives you an interface to view running processes based on ncurses. This tool is not always flexible enough to adequately cover all scenarios. A powerful command called ps is often the answer to these problems.

List processes with ps command

When called without arguments, the output can be a bit lack-luster:

# ps

3125 pts/0 00:00:00 sudo
3126 pts/0 00:00:00 su
3127 pts/0 00:00:00 bash
3150 pts/0 00:00:00 ps

This output shows all of the processes associated with the current user and terminal session. This makes sense because we are only running bash, sudo and ps with this terminal currently.

We can run ps command with different options to get a complete picture of the processes on this system.

BSD style – The options in bsd style syntax are not preceded by a dash.

# ps aux

UNIX/LINUX style – The options in Linux style syntax are preceded by a dash as usual.

# ps -ef

It is okay to mix both the syntax styles on Linux systems. For example “ps au -x”. In this post, we’re using both style syntaxes.

# ps aux

root 2 0.0 0.0 0 0 ? S Jan11 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Jan11 0:00 [ksoftirqd/0]
root 6 0.0 0.0 0 0 ? S Jan11 0:00 [kworker/u30:0]
root 7 0.0 0.0 0 0 ? S Jan11 0:00 [migration/0]
root 8 0.0 0.0 0 0 ? S Jan11 0:00 [rcu_bh]
root 9 0.0 0.0 0 0 ? R Jan11 0:00 [rcu_sched]
root 10 0.0 0.0 0 0 ? S Jan11 0:00

————- output truncated ——————-

These options tell ps to show processes owned by all users (regardless of their terminal association) in a user-friendly format.

To see a tree view, where hierarchal relationships are illustrated, we can run the command with these options:

# ps axjf

0 2 0 0 ? -1 S 0 0:00 [kthreadd]
2 3 0 0 ? -1 S 0 0:00 \_ [ksoftirqd/0]
2 6 0 0 ? -1 S 0 0:00 \_ [kworker/u30:0]
2 7 0 0 ? -1 S 0 0:00 \_ [migration/0]
2 8 0 0 ? -1 S 0 0:00 \_ [rcu_bh]
2 9 0 0 ? -1 R 0 0:00 \_ 1 2024 2024 2024 ? -1 Ss 0 0:00 /usr/sbin/sshd
2024 3100 3100 3100 ? -1 Ss 0 0:00 \_ sshd: ajoy[priv]
3100 3103 3100 3100 ? -1 S 1000 0:00 \_ sshd: ajoy@pts/0
3103 3104 3104 3104 pts/0 3153 Ss 1000 0:00 \_ -bash
3104 3125 3125 3104 pts/0 3153 S 0 0:00 \_ sudo su –
3125 3126 3125 3104 pts/0 3153 S 0 0:00 \_ su –
3126 3127 3127 3104 pts/0 3153 S 0 0:00 \_ -bash
3127 3153 3153 3104 pts/0 3153 R+ 0 0:00 \_ ps axjf
————- output truncated ——————-

As you can see, the process sshd is shown to be a parent of the processes like bash, su, sudo, and ps ajx itself.

List the Process based on the UID and Commands (ps -u, ps -C)

Use -u option to displays the process that belongs to a specific username. When you have multiple usernames, separate them using a comma. The example below displays all the process that are owned by user wwwrun, or postfix.

# ps -f -u wwwrun,postfix

postfix 7457 7435 0 Mar09 ? 00:00:00 qmgr -l -t fifo -u
wwwrun 7495 7491 0 Mar09 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun 7496 7491 0 Mar09 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun 7497 7491 0 Mar09 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun 7498 7491 0 Mar09 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc/apac

The following example shows that all the processes which have in its command execution.

# ps -f -C

root 9576 1 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bin/
root 9577 9576 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bin/
root 9579 9576 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bin/
root 9580 9576 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bin/
root 9581 9576 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bi

The following method is used to get a list of processes with a particular PPID.

#ps -f –ppid 9576

root 9577 9576 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bin/
root 9579 9576 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bin/
root 9580 9576 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bin/
root 9581 9576 0 Mar09 ? 00:00:00 /opt/tata/perl/bin/perl /opt/tata/bin

List Processes in a Hierarchy (ps –forest)

The example below displays the process Id and commands in a hierarchy. –forest is an argument to ps command which displays ASCII art of process tree. From this tree, we can identify which is the parent process and the child processes it forked in a recursive manner.

#ps -e -o pid,args –forest
468 \_ sshd: root@pts/7
514 | \_ -bash
17484 \_ sshd: root@pts/11
17513 | \_ -bash
24004 | \_ vi ./790310__11117/journal
15513 \_ sshd: root@pts/1
15522 | \_ -bash
4280 \_ sshd: root@pts/5
4302 | \_ -bash

List elapsed wall time for processes (ps -o pid,etime=)

If you want the get the elapsed time for the processes which are currently running ps command provides etime which provides the elapsed time since the process was started, in the form [[dd-]hh:]mm: , ss.

The below command displays the elapsed time for the process IDs 1 (init) and process id 29675.

For example “10-22:13:29? in the output represents the process init is running for 10days, 22hours,13 minutes and 29seconds. Since init process starts during the system startup, this time will be same as the output of the ‘uptime’ command.

# ps -p 1,29675 -o pid,etime=
1 10-22:13:29
29675 1-02:58:46

List all threads for a particular process (ps -L)

You can get a list of threads for the processes. When a process hangs, we might need to identify the list of threads running for a particular process as shown below.

# ps -C java -L -o pid,tid,pcpu,state,nlwp,args

16992 16992 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
16992 16993 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
16992 16994 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
16992 16995 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
16992 16996 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
16992 16997 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
16992 16998 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
16992 16999 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
16992 17000 0.0 S 15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_l

Finding memory Leak (ps –sort pmem)

A memory leak, technically, is an ever-increasing usage of memory by an application.

With common desktop applications, this may go unnoticed, because a process typically frees any memory it has used when you close the application.

However, In the client/server model, memory leakage is a serious issue, because applications are expected to be available 24×7. Applications must not continue to increase their memory usage indefinite because this can cause serious issues. To monitor such memory leaks, we can use the following commands.

# ps aux –sort pmem

root 1 0.0 0.0 1520 508 ? S 2005 1:27 init
inst 1309 0.0 0.4 344308 33048 ? S 2005 1:55 agnt (idle)
inst 2919 0.0 0.4 345580 37368 ? S 2005 20:02 agnt (idle)
inst 24594 0.0 0.4 345068 36960 ? S 2005 15:45 agnt (idle)

In the above ps command, –sort option outputs the highest %MEM at the bottom. Just note down the PID for the highest %MEM usage. Then use ps command to view all the details about this process id, and monitor the change over time. You had to manually repeat ir or put it as a cron to a file.

The VSZ number is useless if what you are interested in is memory consumption. VSZ measures how much of the process’s virtual memory space has been marked by the process of memory that should be mapped by the operating system if the process happens to touch it. But it has nothing to do with whether that memory has actually been touched and used. VSZ is an internal detail about how a process does memory allocation — how big a chunk of unused memory it grabs at once. Look at RSS for the count of memory pages it has actually started using


Resident set size = the non-swapped physical memory that a task has used; Resident Set currently in physical memory including Code, Data, Stack


Virtual memory usage of entire process = VmLib + VmExe + VmData + VmStk

In other words,
a) VSZ *includes* RSS
b) “ps -aux” alone isn’t enough to tell you if a process is thrashing (although, if your system *is* thrashing, “ps -aux” will help you identify the processes experiencing the biggest hits).

# ps ev –pid=27645

27645 ? S 3:01 0 25 1231262 1183976 14.4 /TaskServer/bin/./wrapper-linux-x86-32

# ps ev –pid=27645

27645 ? S 3:01 0 25 1231262 1183976 14.4 /TaskServer/bin/./wrapper-linux-x86-32

Note: In the above output, if RSS (resident set size, in KB) increases over time (so would %MEM), it may indicate a memory leak in the application.

The following command displays all the process owned by Linux username: oracle.

# ps U oracle

5014 ? Ss 0:01 /oracle/bin/tnslsnr
7124 ? Ss 0:00 ora_q002_med
8206 ? Ss 0:00 ora_cjq0_med
8852 ? Ss 0:01 ora_pmon_med

Following command displays all the process owned by the current user.

# ps U $USER

10329 ? S 0:00 sshd: ajoy@pts/1,pts/2
10330 pts/1 Ss 0:00 -bash
10354 pts/2 Ss+ 0:00 -bash

The ps command can be configured to show a selected list of columns only. There are a large number of columns to to show and the full list is available in the man pages.

The following command shows only the pid, username, cpu, memory and command columns.

# ps -e -o pid,uname,pcpu,pmem,comm

1 root 0.0 0.6 systemd
2 root 0.0 0.0 kthreadd
3 root 0.0 0.0 ksoftirqd/0
6 root 0.0 0.0 kworker/u30:0
7 root 0.0 0.0 migration/0

The ps command is quite flexible and it is possible to rename the column labels as shown below:

# ps -e -o pid,uname=USERNAME,pcpu=CPU_USAGE,pmem,comm

1 root 0.0 0.6 systemd
2 root 0.0 0.0 kthreadd
3 root 0.0 0.0 ksoftirqd/0
6 root 0.0 0.0 kworker/u30:0
7 root 0.0 0.0 migration/0

Combined with the watch command we can turn ps into a realtime process reporter. Simple example is like this

# watch -n 1 ‘ps -e -o pid,uname,cmd,pmem,pcpu –sort=-pmem,-pcpu | head -15’

Every 1.0s: ps -e -o pid,uname,cmd,pmem,pcpu –… Sun Dec 1 18:16:08 2009

3800 1000 /opt/google/chrome/chrome – 4.6 1.4
7492 1000 /opt/google/chrome/chrome – 2.7 1.4
3150 1000 /opt/google/chrome/chrome 2.7 2.5
3824 1000 /opt/google/chrome/chrome – 2.6 0.6
3936 1000 /opt/google/chrome/chrome – 2.4 1.6
2936 1000 /usr/bin/plasma-desktop 2.3 0.2
9666 1000 /opt/google/chrome/chrome – 2.1 0.8

Process IDs:

In Linux and Unix-like systems, each process is assigned a process ID, or PID. This is how the operating system identifies and keeps track of processes.

# pgrep bash


The first process spawned at boot, called init, is given the PID of “1”.

# pgrep init


This process is then responsible for spawning every other process on the system. The later processes are given larger PID numbers.

A process’s parent is the process that was responsible for spawning it. If a process’s parent is killed, then the child processes also die. The parent process’s PID is referred to as the PPID.

Process States:

Here are the different values that the s, stat and state output specifiers (header “STAT” or “S”) will display to describe the state of a process:


The process is either running (it is the current process in the system) or it is ready to run (it is waiting to be assigned to one of the system’s CPUs).


The process is waiting for an event or for a resource. Linux differentiates between two types of waiting process; interruptible and uninterruptible. Interruptible waiting processes can be interrupted by signals whereas uninterruptible waiting processes are waiting directly on hardware conditions and cannot be interrupted under any circumstances.


The process has been stopped, usually by receiving a signal. A process that is being debugged can be in a stopped state.
This is a halted process which, for some reason, still has a task_struct data structure in the task vector. It is what it sounds like, a dead process.

D uninterruptible sleep (usually IO)
R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
T stopped, either by a job control signal or because it is being traced.
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z defunct (“zombie”) process, terminated but not reaped by its parent.

For BSD formats and when the stat keyword is used, additional characters may be displayed:
< high-priority (not nice to other users)
N low-priority (nice to other users)
L has pages locked into memory (for real-time and custom IO)
s is a session leader
l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
+ is in the foreground process group

Processes Signals:

All processes in Linux respond to signals. Signals are an os-level way of telling programs to terminate or modify their behavior. The most common way of passing signals to a program is with the kill command. The default functionality of this utility is to attempt to kill a process:

# kill <PID of process>

This sends the TERM signal to the process. The TERM signal tells the process to please terminate. This allows the program to perform clean-up operations and exit smoothly.

If the program is misbehaving and does not exit when given the TERM signal, we can escalate the signal by passing the KILL signal:

# kill -KILL <PID of process>

This is a special signal that is not sent to the program.

Instead, it is given to the operating system kernel, which shuts down the process. This is used to bypass programs that ignore the signals sent to them.

Each signal has an associated number that can be passed instead of the name. For instance, You can pass “-15” instead of “-TERM”, and “-9” instead of “-KILL”.

Signals are not only used to shut down programs. They can also be used to perform other actions.

For instance, many daemons will restart when they are given the HUP or hang-up signal. Apache is one program that operates like this.

# kill -HUP <PID of httpd>

The above command will cause Apache to reload its configuration file and resume serving content.

You can list all of the signals that are possible to send with the kill by typing:

# kill -l


The conventional way of sending signals is through the use of PIDs, there are also methods of doing this with regular process names.

The pkill command works in almost exactly the same way as kill, but it operates on a process name instead:

# pkill -9 ping

is equivalent to

# kill -9 `pgrep ping`

If you would like to send a signal to every instance of a certain process, you can use the killall command:

# killall firefox

Process Priorities

Some processes might be considered mission critical for your situation, while others may be executed whenever there might be leftover resources.You will want to adjust which processes are given priority in a server environment. Linux controls priority through a value called niceness.

High priority tasks are considered less nice, because they don’t share resources as well. Low priority processes, on the other hand, are nice because they insist on only taking minimal resources.

When we ran top at the beginning of this blog, there was a column marked “NI”. This is the nice value of the process:

1 root 20 0 128092 6680 3932 S 0.0 0.7 0:02.84 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0

Nice values can range between “-19/-20” (highest priority) and “19/20” (lowest priority) depending on the system.

To run a program with a certain nice value, we can use the nice command:

# nice -n 15 <command>

This only works while executing a new program.

To alter the nice value of a program that is already executing, we use a tool called renice:

# renice 0 <PID of process>

Kernel Modules

What is a kernel module?

Modules are pieces of code that can be loaded and unloaded into the kernel upon demand. They extend the functionality of the kernel without the need to reboot the system.

If you want to add code to a Linux kernel, the most basic way to do that is to add some source files to the kernel source tree and recompile the kernel. In fact, the kernel configuration process consists mainly of choosing which files to include in the kernel to be compiled.

But you can also add code to the Linux kernel while it is running. A chunk of code that you add in this way is called a loadable kernel module. These modules can do lots of things, but they typically are one of three things: 1) device drivers; 2) filesystem drivers; 3) system calls. The kernel isolates certain functions, including these, especially well so they don’t have to be intricately wired into the rest of the kernel.

For example, one type of module is the device driver, which allows the kernel to access hardware connected to the system. Without modules, we would have to build monolithic kernels and add new functionality directly into the kernel image. Besides having larger kernels, this has the disadvantage of requiring us to rebuild and reboot the kernel every time we want new functionality.

Loadable Kernel Module [LKM]

Loadable kernel modules are often called just kernel modules or just modules, but those are rather misleading terms because there are lots of kinds of modules in the world and various pieces built into the base kernel can easily be called modules. We use the term loadable kernel module or LKM for the particular kinds of modules this post is about.

Some people think of LKMs as outside of the kernel. They speak of LKMs communicating with the kernel. This is a mistake; LKMs (when loaded) are very much part of the kernel. The correct term for the part of the kernel that is bound into the image that you boot, i.e. all of the kernel except the LKMs, is “base kernel.” LKMs communicate with the base kernel.

In some other operating systems, the equivalent of a Linux LKM is called a “kernel extension.”

What is “Linux”? Well, first of all, the name is used for two entirely different things, and only one of them is really relevant here:

  1. The kernel and related items distributed as a package by Linus Torvalds.
  2. A class of operating systems that traditionally are based on the Linux kernel.

Only the first of these is really useful in discussing LKMs. But even choosing this definition, people are often confused when it comes to LKMs.

LKMs did not exist in Linux in the beginning. Anything we use as LKM for today was built into the base kernel at kernel build time instead. LKMs have been around at least since Linux 1.2 (1995).

Device drivers and such were always quite modular, though. When LKMs were invented, only a small amount of work was needed on these modules to make them buildable as LKMs. However, it had to be done on each and every one, so it took some time. Since about 2000, virtually everything that makes sense as an LKM has at least had the option of being an LKM.

How Do Modules Get Into The Kernel?

You can see what modules are already loaded into the kernel by running lsmod, which gets its information by reading the file /proc/modules.

When the kernel needs a feature that is not resident in the kernel, the kernel module daemon kmod execs modprobe to load the module in. modprobe is passed a string in one of two forms:

A module name like softdog or ppp.

A more generic identifier like char-major-10-30.

If modprobe is handed a generic identifier, it first looks for that string in the file /etc/modprobe.conf If it finds an alias line like:

alias char-major-10-30 softdog

it knows that the generic identifier refers to the module softdog.ko.

Next, modprobe looks through the file /lib/modules/version/modules.dep, to see if other modules must be loaded before the requested module may be loaded. This file is created by depmod -a and contains module dependencies. For example, msdos.ko requires the fat.ko module to be already loaded into the kernel. The requested module has a dependency on another module if the other module defines symbols (variables or functions) that the requested module uses.

Lastly, modprobe uses insmod to first load any prerequisite modules into the kernel, and then the requested module. modprobe directs insmod to /lib/modules/version/, the standard directory for modules. insmod is intended to be fairly dumb about the location of modules, whereas modprobe is aware of the default location of modules, knows how to figure out the dependencies and load the modules in the right order. So for example, if you wanted to load the msdos module, you’d have to either run:

insmod /lib/modules/2.6.11/kernel/fs/fat/fat.ko
insmod /lib/modules/2.6.11/kernel/fs/msdos/msdos.ko


modprobe msdos

What we’ve seen here is: insmod requires you to pass it the full pathname and to insert the modules in the right order, while modprobe just takes the name, without any extension, and figures out all it needs to know by parsing /lib/modules/version/modules.dep.

Linux distros provide modprobe, insmod and depmod as a package called module-init-tools. In previous versions that package was called modutils. Some distros also set up some wrappers that allow both packages to be installed in parallel and do the right thing in order to be able to deal with 2.4 and 2.6 kernels. Users should not need to care about the details, as long as they’re running recent versions of those tools.

What is modprobe?

The modprobe utility is used to add and remove kernel modules to/from linux kernel. Linux kernel modules have .ko as module name extension. ‘modprobe’ is intelligent enough to load the dependency of a kernel module(if any) first and then loads the actual module.

5 modprobe Examples
1. Basic example to load an LKM

In the first example we will see how to load a LKM (kept at any path in system) using modprobe. Use the following steps to achieve this :

sudo ln -s /path/to/your-kernel-module /lib/modules/`uname -r`
sudo depmod -a
sudo modprobe your-kernel-module

for example, what I did was :

$ sudo ln -s lkm.ko /lib/modules/2.6.32-21-generic/
$ sudo depmod -a
$ sudo modprobe lkm

NOTE: If your module outputs some debug messages then confirmation of the loaded module could be achieved by looking at the logs from dmesg utility. Alternatively, the lsmod utility can be used to view the currently loaded modules
2. Unload a loaded using modprobe

The modprobe can also be used to remove the loaded module. Or in other words we can unload a loaded module through modprobe using the -r option.

$ sudo modprobe -r lkm

The above command unloads a currently loaded module ‘lkm’ from kernel.
3. Have a dry run

Sometimes we face problems like module not being loaded properly etc. In that case it becomes very important to debug and know the level at which the problem exists. It becomes crucial to know whether the problem is before loading or after loading. To facilitate this type of debugging, there exists option -n which if used, forces modprobe to do everything else except the final stage of loading the module.

$ sudo modprobe -vn lkm
insmod /lib/modules/2.6.32-21-generic/kernel/arch/x86/kernel/lkm.ko

I used the -v option along with -n so that some debugging info could be spitted out by modprobe.
4. Suppress the error information

Usually in some error condition, the modprobe utility would try to output some error info. If that kind of info is not needed, then the -q option is used to suppress this kind of info.

$ sudo modprobe lk
FATAL: Module lk not found.
$ sudo modprobe -q lk

So we see that in the above output, when the command was run without the -q option then an error was thrown while when -q was used the error got suppressed.
5. List the modules

If there is a requirement to list all the modules or modules with specific name then modprobe provides -l option to accomplish this

$ modprobe -l crc*
$ modprobe -l rds

Syntax and Options

modprobe [ -v ] [ -V ] [ -C config-file ] [ -n ] [ -i ] [ -q ] [ -b ] [ -o modulename ] [ modulename ] [ module parameters… ]

modprobe [ -r ] [ -v ] [ -n ] [ -i ] [ modulename… ]

modprobe [ -l ] [ -t dirname ] [ wildcard ]

modprobe [ -c ]

modprobe [ –dump-modversions ] [ filename… ]

Short Option Long Option Option Description
-v –verbose Print messages about what the program is doing. Usually modprobe only prints messages if something goes wrong.This option is passed through install or remove commands to other modprobe commands in the MODPROBE_OPTIONS environment variable.
-C –config This option overrides the default configuration directory/file (/etc/modprobe.d or /etc/modprobe.conf). This option is passed through install or remove commands to other modprobe commands in the MODPROBE_OPTIONS environment variable.
-c –showconfig Dump out the effective configuration from the config directory and exit.
-n –dry-run This option does everything but actually insert or delete the modules (or run the install or remove commands). Combined with -v, it is use-ful for debugging problems.
-i –ignore-install –ignore-remove This option causes modprobe to ignore install and remove commands in the configuration file (if any) for the module specified on the command line (any dependent modules are still subject to commands set for them in the configuration file).
-q –quiet Normally modprobe will report an error if you try to remove or insert a module it can’t find (and isn’t an alias or install/remove command). With this flag, modprobe will simply ignore any bogus names (the kernel uses this to opportunistically probe for modules which might exist).
-r –remove This option causes modprobe to remove rather than insert a module. If the modules it depends on are also unused, modprobe will try to remove them too. Unlike insertion, more than one module can be specified on the command line (it does not make sense to specify module parameters when removing modules).There is usually no reason to remove modules, but some buggy modules require it. Your kernel may not support removal of modules.
-f –force Try to strip any versioning information from the module which might otherwise stop it from loading: this is the same as using both –force-vermagic and –force-modversion. Naturally, these checks are there for your protection, so using this option is dangerous.
This applies to any modules inserted: both the module (or alias) on the command line and any modules it on which it depends.

What is insmod?

insmod command is used to insert modules to Linux kernel.

3 insmod Examples
1. Specify module name as an argument

The following command insert the module airo to the Linux kernel.

# insmod kernel/drivers/net/wireless/airo.ko

Once you’ve inserted the module, use lsmod command to verify that the module has been inserted successfully as shown below.

# lsmod | grep airo
Module Size Used by
airo 66291 0

Note: It is recommended that you use modprobe command for all your Kernel module manipulation, including inserting a module.

All the modules that are available to be inserted to the kernel are listed in the modules.dep file as shown below.

# more /lib/modules/`uname -r`/modules.dep


# more /lib/modules/2.6.32-100.28.5.el6.x86_64/modules.dep

2. Insert a module with any arguments

If there are any arguments that needs to be passed for the module, give that as 3rd option as shown below.

The following command insert the module dummy to the Linux kernel. In this example, the dummy module takes two arguments type and debug. You can set the values for type and debug arguments and pass it to dummy module as shown below.

# insmod dummy type=”wpa” debug=1

3. Specify module name interactively

Let us assume that you want to insert a pcmcia module called ‘axnet_cs’.

First, verify to make sure this module is listed in the modules.dep file

# grep axnet_cs /lib/modules/`uname -r`/modules.dep

Following is the full path of this file.

# ls /lib/modules/`uname -r`/kernel/drivers/net/pcmcia/axnet_cs.ko

Use insmod – and enter the module name interactively.

# insmod –

Or, you can use < to give the module name as shown below.

# insmod – < kernel/drivers/net/pcmcia/axnet_cs.ko

Verify to make sure the module got inserted successfully.

# lsmod | grep axnet
axnet_cs 14627 0

Syntax and Options


insmod [ filename ] [ module options… ]

Real, Effective & Saved UID explained

Each Linux/Unix process has 3 UIDs associated with it. Superuser privilege is UID=0.

Real UID

This is the UID of the user/process that created THIS process. It can be changed only if the running process has EUID=0.

Effective UID

This UID is used to evaluate privileges of the process to perform a particular action. EUID can be changed either to RUID, or SUID if EUID!=0. If EUID=0, it can be changed to anything.

Saved UID

If the binary image file, that was launched has a Set-UID bit on, SUID will be the UID of the owner of the file. Otherwise, SUID will be the RUID.

  • What is the idea behind this?

Normal programs, like “ls”, “cat”, “echo” will be run by a normal user, under that users UID. Special programs that allow the user to have controlled access to protected data, can have Set-UID bit to allow the program to be run under privileged UID.

An example of such program is “passwd”. If you list it in full, you will see that it has a Set-UID bit and the owner is “root”. When a normal user, say “ajoy”, runs “passwd”, passwd starts with:

Real-UID = ajoy
Effective-UID = ajoy
Saved-UID = root

The program calls a system call “seteuid( 0 )” and since SUID=0, the call will succeed and the UIDs will be:

Real-UID = ajoy
Effective-UID = root
Saved-UID = root

After that, “passwd” process will be able to access /etc/passwd and change password for user “ajoy”. Note that user “ajoy” cannot write to /etc/passwd on it’s own. Note one other thing, setting a Set-UID on an executable file is not enough to make it run as a privileged process. The program itself must make a system call.

That is the idea.

CPU affinity an overview

When you are using SMP (Symmetric MultiProcessing) you might want to override the kernel’s process scheduling and bind a certain process to a specific CPU(s).

But what is CPU affinity?

CPU affinity is nothing but a scheduler property that “bonds” a process to a given set of CPUs on the SMP system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity:

The scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications. For example, the application such as Oracle (ERP apps) use # of CPUs per instance licensed. You can bound Oracle to specific CPU to avoid license problem. This is a really useful on large server having 4 or 8 CPUS

Setting processor affinity for a certain task or process using “taskset” command:

The command taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. However taskset is not installed by default. You need to install schedutils (Linux scheduler utilities) package.

Install schedutils

Debian Linux:
# apt-get install schedutils

Red Hat Enterprise Linux:
# up2date schedutils
# rpm –ivh schedutils*

Under latest version of Debian / Ubuntu Linux/RHEL etc taskset is installed by default using util-linux package.

# apt-get install util-linux

# yum install util-linux

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. For example:

0x00000001 is processor #0 (1st processor)
0x00000003 is processors #0 and #1
0x00000004 is processors #2 (3rd processor)

To set the processor affinity of process 13545 to processor #0 (1st processor) type following command:

# taskset 0x00000001 -p 13545

If you find a bitmask hard to use, then you can specify a numerical list of processors instead of a bitmask using -c flag:
# taskset -c 1 -p 13545
# taskset -c 3,4 -p 13545


-p : Opera

ps -eo psr,pid

this gives the processor in each the pid is current assigned to.

taskset -p option pid
option 1 means 1st processor
option 2 means 2nd processor
option 0xffffffff means all the processor

List open files lsof command explained

The command lsof stands for list open files, which will list all the open files in the system. The open files include network connections, devices, and directories. The output of the lsof command will have the following columns:

COMMAND process name.
PID process ID
USER Username
FD file descriptor
TYPE node type of the file
DEVICE device number
SIZE file size
NODE node number
NAME full path of the file name.

Simply typing lsof will provide a list of all open files belonging to all active processes.

# lsof

init 1 root cwd DIR 8,1 4096 2 /
init 1 root txt REG 8,1 124704 917562 /sbin/init
init 1 root 0u CHR 1,3 0t0 4369 /dev/null
init 1 root 1u CHR 1,3 0t0 4369 /dev/null
init 1 root 2u CHR 1,3 0t0 4369 /dev/null
init 1 root 3r FIFO 0,8 0t0 6323 pipe

By default, one file per line is displayed. Most of the columns are self-explanatory. We will explain the details about a couple of cryptic columns (FD and TYPE).

FD – Represents the file descriptor. Some of the values of FDs are,

cwd – Current Working Directory
txt – Text file
mem – Memory mapped file
mmap – Memory mapped device
NUMBER – Represent the actual file descriptor. The character after the number i.e ’1u’, represents the mode in which the file is opened. r for read, w for write, u for read and write.

TYPE – Specifies the type of the file. Some of the values of TYPEs are,

REG – Regular File
DIR – Directory
FIFO – First In First Out
CHR – Character special file

The lsof command by itself without may return lot of records as output, which may not be very meaningful except to give you a rough idea about how many files are open in the system at any given point of view as shown below.

# lsof | wc -l


Use lsof –u option to display all the files opened by a specific user.

# lsof –u ajoy

vi 7190 ajoy txt REG 8,1 47


List opened files under a directory

You can list the processes which opened files under a specified directory using ‘+D’ option. +D will recurse the sub directories also. If you don’t want lsof to recurse, then use ‘+d’ option.

# lsof +D /var/log/

rsyslogd 488 syslog 1w REG 8,1 1151 268940 /var/log/syslog
rsyslogd 488 syslog 2w REG 8,1 2405 269616 /var/log/auth.log
console-k 144 root 9w REG 8,1 10871 269369 /var/log/ConsoleKit/history

List opened files based on process names starting with

You can list the files opened by process names starting with a string, using ‘-c’ option. -c followed by the process name will list the files opened by the process starting with that processes name. You can give multiple -c switch on a single command line.

# lsof -c ssh -c init

init 1 root txt REG 8,1 124704 917562 /sbin/init
init 1 root mem REG 8,1 1434180 1442625 /lib/i386-linux-gnu/
init 1 root mem REG 8,1 30684 1442694 /lib/i386-linux-gnu/

ssh-agent 1528 user1 1u CHR 1,3 0t0 4369 /dev/null
ssh-agent 1528 user1 2u CHR 1,3 0t0 4369 /dev/null

List processes using a mount point

Sometime when we try to umount a directory, the system will say “Device or Resource Busy” error. So we need to find out what are all the processes using the mount point and kill those processes to umount the directory. By using lsof we can find those processes.

# lsof /home

The following will also work.

# lsof +D /home/

# lsof -u ^ajoy

rtkit-dae 1380 rtkit 7u 0000 0,9 0 4360 anon_inode
udisks-da 1584 root cwd DIR 8,1 4096 2 /

The above command listed all the files opened by all users, expect user ‘ajoy’.

List all open files by a specific process

You can list all the files opened by a specific process using ‘-p’ option. It will be helpful sometimes to get more information about a specific process.

# lsof -p 1753

bash 1753 user1 cwd DIR 8,1 4096 393571 /home/ajoy/test.txt
bash 1753 user1 rtd DIR 8,1 4096 2 /
bash 1753 user1 255u CHR 136,0 0t0 3 /dev/pts/0

Kill all process that belongs to a particular user

When you want to kill all the processes which has files opened by a specific user, you can use ‘-t’ option to list output only the process id of the process, and pass it to kill as follows

# kill -9 `lsof -t -u ajoy`

The above command will kill all process belonging to user ‘ajoy’, which has files opened.

Similarly you can also use ‘-t’ in many ways. For example, to list process id of a process which opened /var/log/syslog can be done by

# lsof -t /var/log/syslog


Combine more list options using OR/AND

By default when you use more than one list option in lsof, they will be treated as OR. For example,

# lsof -u user1-c init

init 1 root cwd DIR 8,1 4096 2 /
init 1 root txt REG 8,1 124704 917562 /sbin/init
bash 1995 user1 2u CHR 136,2 0t0 5 /dev/pts/2
bash 1995 user1 255u CHR 136,2 0t0 5 /dev/pts/2

The above command uses two list options, ‘-u’ and ‘-c’. So the command will list process belongs to user ‘lakshmanan’ as well as process name starts with ‘init’.

But when you want to list a process belongs to user ‘lakshmanan’ and the process name starts with ‘init’, you can use ‘-a’ option.

# lsof -u user1 -c init -a

List all network connections

You can list all the network connections opened by using ‘-i’ option.

# lsof -i

avahi-dae 515 avahi 13u IPv4 6848 0t0 UDP *:mdns
avahi-dae 515 avahi 16u IPv6 6851 0t0 UDP *:52060
cupsd 1075 root 5u IPv6 22512 0t0 TCP ip6-localhost:ipp (LISTEN)

List all network files in use by a specific process

You can list all the network files which is being used by a process as follows

# lsof -i -a -p 234

You can also use the following

# lsof -i -a -c ssh

List processes which are listening on a particular port

You can list the processes which are listening on a particular port by using ‘-i’ with ‘:’ as follows

# lsofi :25

exim4 2541 Debian-exim 3u IPv4 8677 TCP localhost:smtp (LISTEN)

List all TCP or UDP connections

You can list all the TCP or UDP connections by specifying the protocol using ‘-i’.

# lsof -i tcp; lsof -i udp;

List all Network File System ( NFS ) files

You can list all the NFS files by using ‘-N’ option. The following lsof command will list all NFS files used by user ‘lakshmanan’.

# lsof -N -u user1 -a

4608 475196 /bin/vi

sshd 7163 ajoy 3u IPv6 15088263 TCP dev-db:ssh-> (ESTABLISHED)

A system administrator can use this command to get some idea on what users are executing on the system.

List Users of a particular file

If you like to view all the users who are using a particular file, use lsof as shown below. In this example, it displays all users who are currently using vi.

# lsof /bin/vi

vi 7258 root txt REG 8,1 474608 475196 /bin/vi
vi 7300 ajoy txt REG 8,1 474608 475196 /bin/vi

Semaphores in a DB perspective

Semaphores can be described as counters which are used to provide synchronization between processes or between threads within a process for shared resources like shared memories. System V semaphores support semaphore sets where each one is a counting semaphore. So when an application requests semaphores, the kernel releases them in sets. The number of semaphores per set can be defined through the kernel parameter SEMMSL.

To see all semaphore settings, run:

# ipcs -ls

  • The SEMMSL Parameter

This parameter defines the maximum number of semaphores per semaphore set.

Oracle recommends SEMMSL to be at least 250 for 9i R2 and 10g R1/R2 databases except for 9i R2 on x86 platforms where the minimum value is lower. Since these recommendations are minimum settings, it’s best to set it always to at least 250 for 9i and 10g databases on x86 and x86-64 platforms.

If a database gets thousands of concurrent connections where the ora.init parameter PROCESSES is very large, then SEMMSL should be larger as well. Note what Metalink Note:187405.1 and Note:184821.1 have to say regarding SEMMSL: “The SEMMSL setting should be 10 plus the largest PROCESSES parameter of any Oracle database on the system”. Even though these notes talk about 9i databases this SEMMSL rule also applies to 10g databases. I’ve seen low SEMMSL settings to be an issue for 10g RAC databases where Oracle recommended to increase SEMMSL and to calculate it according to the rule mentioned in these notes. An example for setting semaphores for higher PROCESSES settings can be found at Example for Semaphore Settings.

  • The SEMMNI Parameter

This parameter defines the maximum number of semaphore sets for the entire Linux system.

Oracle recommends SEMMNI to be at least 128 for 9i R2 and 10g R1/R2 databases except for 9i R2 on x86 platforms where the minimum value is lower. Since these recommendations are minimum settings, it’s best to set it always to at least 128 for 9i and 10g databases on x86 and x86-64 platforms.

  • The SEMMNS Parameter

This parameter defines the total number of semaphores (not semaphore sets) for the entire Linux system. A semaphore set can have more than one semaphore, and as the semget(2) man page explains, values greater than SEMMSL * SEMMNI makes it irrelevant. The maximum number of semaphores that can be allocated on a Linux system will be the lesser of: SEMMNS or (SEMMSL * SEMMNI).

Oracle recommends SEMMNS to be at least 32000 for 9i R2 and 10g R1/R2 databases except for 9i R2 on x86 platforms where the minimum value is lower. Setting SEMMNS to 32000 ensures that SEMMSL * SEMMNI (250*128=32000) semaphores can be be used. Therefore it’s recommended to set SEMMNS to at least 32000 for 9i and 10g databases on x86 and x86-64 platforms.

  • The SEMOPM Parameter

This parameter defines the maximum number of semaphore operations that can be performed per semop(2) system call (semaphore call). The semop(2) function provides the ability to do operations for multiple semaphores with one semop(2) system call. Since a semaphore set can have the maximum number of SEMMSL semaphores per semaphore set, it is often recommended to set SEMOPM equal to SEMMSL.

Oracle recommends to set SEMOPM to a minimum value of 100 for 9i R2 and 10g R1/R2 databases on x86 and x86-64 platforms.

  • Setting Semaphore Parameters

To determine the values of the four described semaphore parameters, run:

# cat /proc/sys/kernel/sem
250 32000 32 128

These values represent SEMMSL, SEMMNS, SEMOPM, and SEMMNI.

Alternatively, you can run:

# ipcs -ls

All four described semaphore parameters can be changed in the proc file system without reboot:

# echo 250 32000 100 128 > /proc/sys/kernel/sem

Alternatively, you can use sysctl(8) to change it:

# sysctl -w kernel.sem=”250 32000 100 128″

To make the change permanent, add or change the following line in the file /etc/sysctl.conf. This file is used during the boot process.

# echo “kernel.sem=250 32000 100 128” >> /etc/sysctl.conf

  • Example for Semaphore Settings

On systems where the ora.init parameter PROCESSES is very large, the semaphore settings need to be adjusted accordingly.

As shown at The SEMMSL Parameter the SEMMSL setting should be 10 plus the largest PROCESSES parameter of any Oracle database on the system. So if you have one database instance running on a system where PROCESSES is set to 5000, then SEMMSL should be set to 5010.

As shown at The SEMMNS Parameter the maximum number of semaphores that can be allocated on a Linux system will be the lesser of: SEMMNS or (SEMMSL * SEMMNI). Since SEMMNI can stay at 128, we need to increase SEMMNS to 641280 (5010*128).

As shown at The SEMOPM Parameter a semaphore set can have the maximum number of SEMMSL semaphores per semaphore set and it is recommended to set SEMOPM equal to SEMMSL. Since SEMMSL is set to 5010 the SEMOPM parameter should be set to 5010 as well.

Hence, if the ora.init parameter PROCESSES is set to 5000, then the semaphore settings should be as follows:

# sysctl -w kernel.sem=”5010 641280 5010 128″