Hands-On Linux & DevOps
Real Challenges. Real Infra. Real Skills.

Master Linux & DevOps troubleshooting on live servers.

Fun, real-world challenges for engineers
and powerful assessments for hiring teams.

journalctl

• journalctl -u sshd

  • show only log for sshd

• logger hello

  • write in syslog

• journalctl -f

  • log is live

• nano /etc/systemd/journald.conf

  • Storage=auto

• mkdir /var/log/journal

  • [unclear note in image]

NFS → Network File System

• Share folder

NFS Server

• dnf install nfs-utils libnfsidmap

• systemctl enable rpcbind

• systemctl enable nfs-server

• systemctl start rpcbind

• systemctl start nfs-server

• systemctl start rpc-statd

• systemctl start nfs-idmapd

• mkdir /myshare

• chmod a+rwx /myshare

• nano /etc/exports

/myshare *(rw,sync,no_root_squash)

• exportfs -rv → export NFS file system

NFS Client

• dnf install nfs-utils rpcbind

• service rpcbind start

  • Start package

• ps -ef | egrep "firewall|iptable"

  • Disable firewall in case on server

• showmount -e 192.168.0.100

  • Show mount from NFS server
  • 192.168.0.100 = NFS server IP

• mkdir /mnt/app → create mount point

• mount 192.168.0.100:/myshare /mnt/app

  • Mount NFS file system

• df -h → verify mounted system

• umount /mnt/app

SAMBA

• SMB → Server Message Block

• CIF → Common Internet File System

• Samba protocol

• dnf install samba samba-client samba-common

• firewall-cmd --permanent --zone=public --add-service=samba

• firewall-cmd --reload

  • Add rule to firewall

• mkdir -p /samba/myshare

• chmod a+rwx /samba/myshare

• chown -R nobody:nobody /samba

  • Create Samba share directory

• chcon -t samba_share_t /samba/myshare

  • Change SELinux security context

• nano /etc/samba/smb.conf

  • Add new filesystem shared

[anonymous]
path = /samba/myshare
browsable = yes
writable = yes
guest ok = yes
guest only = yes
read only = no

• testparm

  • Test SMB configuration

• systemctl enable smb

• systemctl start smb

  • Start Samba service

Mount on Linux Client

• dnf -y install cifs-utils samba-client

• mkdir -p /mnt/sambashare

• mount -t cifs //192.168.0.35/anonymous /mnt/sambashare

  • Mount Samba share without password

Secure Samba Server

• useradd larry

• groupadd smbgrp

• usermod -a -G smbgrp larry

• smbpasswd -a larry

  • Set Samba password for larry

• mkdir /samba/secureshare

• chown -R larry:smbgrp /samba/secureshare

• chmod -R 0770 /samba/secureshare

• chcon -t samba_share_t /samba/secureshare

Samba Secure Share

• nano /etc/samba/smb.conf

[secure]
path = /samba/secureshare
valid users = @smbgrp
guest ok = no
writable = yes
browsable = yes

• systemctl restart smb

fsck & xfs_repair → filesystem check

• fsck → ext2, ext3, ext4

• xfs_repair → xfs only

• Made each boot & fix it

• df -T → check

  • Unmount before fsck

fsck error code

• 0 → no error

• 1 → error corrected

• 2 → reboot needed

• 4 → some error left uncorrected

• 8 → operational error

• -f → force

• -y → repair automatically

• fsck /dev/sdb

• umount /bigdata

• xfs_repair /dev/mapper/stratis...

dd → disk cloning

• dd if=<sourcefilename> of=<targetfilename>

• dd if=/dev/sda1 of=/root/sda1.img

  • Backup copy disk partition

• dd if=/root/sda1.img of=/dev/sdb3

  • Restore

Stratis → advanced storage management

• Extend filesystem automatically when needed

• dnf install stratis-cli stratisd

• dnf makecache --refresh

  • Update latest package list for DNF

• systemctl start stratisd

• systemctl enable stratisd

  • Start daemon

• lsblk

  • List block device

• stratis pool create pool1 /dev/sdb

  • Create pool with /dev/sdb

• stratis pool list

  • Show pool list

• stratis pool add-data pool1 /dev/sdc

  • Extend pool1 with /dev/sdc

Stratis Filesystem

• stratis filesystem create pool1 fs1

  • Create filesystem using Stratis

• stratis filesystem list → get UUID

  • Verify filesystem creation

• mkdir /bigdata

• mount /stratis/pool1/fs1 /bigdata

  • Mount Stratis disk to /bigdata

• stratis filesystem snapshot pool1 fs1 fs-snap

  • Make a snapshot

• nano fstab

UUID="a313..."   /bigdata   xfs   defaults,x-systemd.requires=stratis.service   0 0

• Startup mount

RAID (Redundant Array of Independent Disks)

• RAID0 → add physical disk to make big one
• RAID1 → mirror (slow)
• RAID5 → 3 or more disks
  • Read & write a little on every disk

LVM (Logical Volume Management)

• Combine disk together by software

• Add new HDD on the fly to extend disk space

• Set LVM in Linux install

• Desired capacity: all → set last partition to space left

• /boot → 500 MB LVM ⚠ always

• fdisk /dev/sdc

  • n → new partition
  • t → change partition system ID
  • 8e → change to Linux LVM

• pvcreate /dev/sdc1 → create physical volume

• pvdisplay → infos

• vgcreate vg /dev/sdc1 → create volume group

• vgdisplay → show info

• lvcreate -n lv --size 500 vg

• lvdisplay → show info

• mkfs.xfs /dev/vg/lv → format logical volume

Extend LVM → create LVM partition

• pvcreate /dev/sd01
• vgextend vg /dev/sd01
• lvextend -L +100M /dev/mapper/vg-vg
• xfs_growfs /dev/mapper/vg-vg

Add / Extend Swap Space

• System will not boot if set incorrect

• Recommended: 2x size of RAM

• dd if=/dev/zero of=/newswap bs=1M count=1024

  • Extract disk space from HDD to swap

• chmod go-r /newswap

  • Make file non-readable for others

• mkswap /newswap

  • Make swap file

• swapon /newswap

  • Activate swap (add with the rest)

• nano fstab

/newswap swap swap defaults 0 0

xfs_info → display detailed information

• xfs_info /dev/mapper/cs-root
  • Debug info about main XFS partition

Computer Storage

• Local → RAM / HDD / SSD

• DAS (Direct Attached Storage)

  • USB HDD / DVD

• SAN (Storage Area Network)

  • through iSCSI cable or fiber cable
  • PCI SAN cards / HBA cards

• NAS (Network Attached Storage)

  • through network (TCP/IP) Samba, NFS

Disk Partition

• df → disk info

• fdisk → total & partition

• fdisk -l → get info about partition

• fdisk /etc/sdb → mount partition

  • n → new partition
  • w → write

• mkfs.xfs /dev/sdb1

  • create file system

• mkdir /data → create folder to mount partition

• mount /dev/sdb1 /data

  • mount disk

• nano /etc/fstab

  • mount new disk at startup

/dev/sdb1    /data    xfs    defaults    0    0

• unmount /data

  • unmount disk

• mount -a → read fstab and remount disk

Linux Boot Process → Older Version

• BIOS → Basic Input / Output System

  • executes MBR

• MBR → Master Boot Record

  • executes GRUB

• GRUB → Grand Unified Bootloader

  • executes kernel

• KERNEL

  • kernel executes /sbin/init
  • mount the root file system

• INIT

  • init executes run level programs
  • initial RAM disk → contains drivers

• RUN LEVEL

  • run level programs executed from /etc/rc.d/rc*.d/

Linux Boot Process → CentOS 7, 8, 9

• BIOS → MBR → GRUB 2

• /boot/grub2/grub.cfg

• KERNEL

  • load drivers from initrd.img
  • start the first OS process (systemd)

• SYSTEMD = system daemon

  • read /etc/systemd/system/default.target

systemd-analyze → Optimizing Boot Performance

• Understand how long the Linux system takes to boot by:

  • kernel
  • service
  • initrd

• initrd → initialize system hardware

• systemd-analyze blame

  • sort by time all service

Message of the Day → Message at Login

• nano /etc/motd

• Customize:

  • touch /etc/profile.d/motd.sh

• In /etc/ssh/sshd_config

  • PrintMotd no

• systemctl restart sshd.service

System Run Level

• init 0 → shutdown
• init 1 → single user mode → troubleshoot
• init 6 → reboot the system
• init 2 → multiuser without networking
• init 3 → multiuser with networking
• init 5 → multiuser with networking with GUI

who -r

• To know which level

Boot Process

• CPU -> BIOS -> CMOS
• BIOS = Basic Input / Output System
• CMOS = Complementary Metal-Oxide Semiconductor
• ROM = Read Only Memory
• POST = Power On Self Test
• HDD → 1st sector = MBR
• MBR = Master Boot Record
• OS in RAM
• App goes to CPU

Bootstrap

• How the computer is going to power on

1. Kickstart server

2. Make Kickstart available on the network

3. Make installation source available

4. Make boot media available

5. Start Kickstart installation

   cd /root
   anaconda-ks.cfg   # create for 1st install in root folder

6. Steps:

   • cp /root/anaconda-ks.cfg /var/www/html/
   • chmod a+r /var/www/html/anaconda-ks.cfg
   • systemctl stop|disable firewalld

   • Create new VM with CentOS DVD ISO

     • Start VM → set boot to DVD
     • Change network adapter to bridged adapter
   • Boot: linux ks=https://192.168.1.x/anaconda-ks.cfg
   • Start automated installation

Example network config:

ksdevice=eth0 ip=192.168.0.50 \
netmask=255.255.255.0 \
gateway=192.168.0.1

Containers → Podman

• Podman → manage pods and container images
• Buildah → building/pushing/signing container images
• Skopeo → copy/inspect/delete/signing
• Runc → run & build feature to Podman and Buildah
• Crun → optional runtime for rootless containers

Podman Terms

• Images → containers can be created through images

  • converted to images
• Pods → group of containers deployed on the host

Commands

• dnf install podman

• alias docker=podman

  • create alias to use Podman instead of Docker
• podman info → check Podman registry

• podman search httpd

  • search specific repository
  • check stars for the best

Podman & Docker Commands

• podman images → check previously installed images
• podman pull docker.io/library/httpd → download and install image
• podman ps → list running containers
• podman run -dt -p 8080:80/tcp docker.io/library/httpd → run a downloaded podman container
• podman logs -l → check the logs
• podman stop <container_name|id> → stop container
• To run multiple containers, change port: 8081 and 8082
• podman stop/start <container_name|id> → stop/start container
• podman create --name httpd2 docker.io/library/httpd → create new container from downloaded image

Podman Container at Startup

• Generate a unit file:

  podman generate systemd --new --files --name httpd
  cp container-httpd.service /etc/systemd/system/
  systemctl enable container-httpd.service
  systemctl start container-httpd.service

Docker

• Install from: docs.docker.com
• systemctl start docker
• systemctl enable docker
• docker run hello-world → test if Docker is working

Tweaks with Tuned / Podman

Tweaks with Tuned

• systemctl enable tuned

• tuned-adm active

  • check which profile is active

• tuned-adm list

  • list all profiles

• tuned-adm profile profile-name

  • change profile

• tuned-adm recommend

  • recommendation

• tuned-adm off

  • turned off setting daemon
• Web admin → https://myip:9090

Tweaks with Nice and Renice

• top

  • PR priority level
  • NI nice user setting

• ps axo pid,comm,nice,cls --sort=nice

  • check nice with ps

• nice -n # processname (# -20 → +20)

  • set priority

• renice -n # pid

  • change process priority

Enable firewall

• firewall-config → GUI for options

  • add ports
• firewall-cmd → CLI
• cat /etc/sysconfig/iptables-config
• cat /etc/firewalld/firewalld.conf

Firewall config

• iptables = old, firewalld = new (use in all Linux distros)
• Check iptables rules → iptables -L
• Flush rules → iptables -F

Iptables packet filtering

• Tables (process packets) → filter

  • attached to chains (inspect traffic)

  • INPUT → incoming traffic

  • FORWARD → going to router from another device

  • OUTPUT → outgoing traffic

• Target (what happens when a rule is met)

  • ACCEPT
  • REJECT → send reject response
  • DROP → drop without response

Firewalld

• firewall-cmd --list-all

  • check the rule

• firewall-cmd --get-services

  • list service firewall is aware of

• firewall-cmd --reload

  • reload for new rules

• firewall-cmd --get-zones

  • get a list of all zones
  • public zone is used most of the time

• firewall-cmd --get-actives-zones

  • get active zones

• firewall-cmd --zone=public --list-all

  • get firewall rule for public zone

Add extra service:

• cp /usr/lib/firewalld/services/allservices.xml

  • change service and port number
• sap.xml → firewall-cmd --add-service=sap

• firewall-cmd --add-service=http

  • add a service (http)
  • --permanent

• firewall-cmd --remove-service=http

  • remove a service (http)
  • --permanent

• firewall-cmd --reload

  • reload configuration

• firewall-cmd --add-port=1110/tcp

  • add a port
  • --permanent

• firewall-cmd --remove-port=1110/tcp

  • remove a port
  • --permanent

• firewall-cmd --add-rich-rule='rule family="ipv4" source address="192.168.8.1" drop'

  • block certain IP address

• firewall-cmd --add-icmp-block=ipv6-redirect

  • block ping

• firewall-cmd --direct --add-rule ipv4 filter OUTPUT 0 -d 8.9.10.11 -j DROP

  • block website (host → www.website.com)

Check on listening port

• netstat -tunlp → check open/listening port

Securing SSH config

• more /etc/ssh/sshd_config

  • change port 22
  • PermitRootLogin no

Enable SELinux (SE → Security Enhanced)

• sestatus → check if active
• cat /etc/sysconfig/selinux
• stat myfile → extra context/access
• chcon myfile → change context

OpenLDAP → to manage users from another server

• dnf install openldap* → install everything
• systemctl start slapd
• systemctl enable slapd
• ps -ef | grep slapd → check if running
• cat /etc/openldap/slapd.d/ldap.conf

• cat /etc/nsswitch.conf

  • replace passwd: files sss → ldap

Traceroute → trace network traffic

• traceroute www.mywebsite.com

Secure SSH

Set idle timeout

• nano /etc/ssh/sshd_config

  • ClientAliveInterval 600
  • ClientAliveCountMax 0

Disable root login

• PermitRootLogin no
• PermitEmptyPassword no
• AllowUsers user1 user2
• Port change: 22 → 22224 (not usable port)

Access remote server with SSH keys

• For automation through script

• ssh-keygen

  • leave default folder
  • no passphrase
  • key created

• ssh-copy-id root@192.168.0.53

  • give password of 192.168.0.53

• ssh root@192.168.0.53

  • no password needed!

Apache Web Server

• dnf install httpd
• nano /etc/httpd/conf/httpd.conf
• nano /var/www/html/index.html
• systemctl restart httpd
• systemctl stop firewalld

NGINX

→ Web proxy → MTM between client backend servers
→ Web server

• dnf install nginx -y
• systemctl start nginx
• systemctl enable nginx
• lsof -i :80 → check port 80
• systemctl stop firewalld
• systemctl disable firewalld

• nano /etc/nginx/nginx.conf

  • server { ... }

• nano /etc/nginx/conf.d/myconf.conf

  • server { ... }
• mkdir -p /var/www/mywebserver/html
• nano index.html
• nginx -t → check is everything ok
• systemctl restart nginx

NGINX Reverse Proxy

On server → setup nginx as before

• 403 error → sestatus (selinux status tool)

  • current mode → enforcing
  • fix: chcon -R -t httpd_sys_content_t /var/html/ → fix permission

On client

• nano /etc/nginx/conf.d/myconf.conf

  server {
  location / {
    proxy_pass http://192.168.0.202; # server ip
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
  }

NTP / Chronyd → Time Synchronisation

• nano /etc/chrony.conf → edit conf
• systemctl start chronyd
• systemctl enable chronyd
• chronyc → interactive cmd

timedatectl → Change date/time

• timedatectl → check status
• timedatectl set-timezone Europe/Berlin
• timedatectl set-time YYYY-MM-DD
• timedatectl set-ntp true → set synchronisation

Mail Server → Postfix & s-nail

• /etc/postfix/main.cf → configuration

• systemctl restart postfix

• dnf install postfix → handle email server

• dnf install s-nail → write & send email

• nano /etc/postfix/main.cf

• mail -s "Mail setup" name@domain.com

  • return
  • body… → Ctrl+D

SSH

• ps -ef | grep sshd → check ssh
• systemctl status sshd

DNS

• PTR record → IP to hostname

• A record → hostname to IP

• CNAME record → hostname to hostname

• /etc/named.conf → conf of DNS

• /var/named/ → file for DNS record

• systemctl restart named

• touch forward.lab reverse.lab

• dig client a in a 192.168.0.240

• systemctl start named

• systemctl enable named

Configuring permission

• chgrp named -R /var/named
• chown -v root:named /etc/named.conf
• restorecon -rv /var/named
• restorecon /etc/named.conf

Test DNS conf

• named-checkconf /etc/named.conf
• named-checkzone lab.local /var/named/forward.lab
• named-checkzone reverse.lab /var/named/reverse.lab

Add DNS info in network file

• nano /etc/sysconfig/network-scripts/ifcfg-enp0s3

• DNS=192.168.0.201

• systemctl restart NetworkManager

• nano /etc/resolv.conf

• nameserver 192.168.0.201

Test DNS server

• dig masterdns.lab.local
• nslookup masterdns.lab.local

Website lookup

• nslookup mywebsite.com

  • get IP of a website

• dig mywebsite.com

  • more info about website

System Updates & Software Install

• dnf (yum) → RedHat → /etc/yum.repos.d

• apt-get → Debian

• rpm → RedHat package management

  • standalone package to install locally
  • hard to install extra dependency

• dnf install mypackage

• rpm -qa | grep mypackage

  • check if package installed

• rpm -ihv /tmp/package.rpm

  • install

• rpm -e packagename

  • remove

• dnf remove package

• dnf update -y

  • update everything (preserves old one)

• dnf upgrade

  • delete and replace with new

Advanced Package Management

• rpm -qa | grep mypackage

  • mypackage-1.0.6

• rpm -qc mypackage-1.0.6

  • configuration files

• which mypackage

  • /usr/bin/mypackage

• rpm -qf /usr/bin/mypackage

  • mypackage-1.0.6

Rollback Update & Patch

• dnf history undo mypackageid
• dnf update → keep the old package
• dnf history → get id