Sysadmin

Diagnosing Linux Server Issues in Production: 5 Essential Tools

Diagnosing Linux Server Issues in Production: 5 Essential Tools

Linux Server Diagnostics: 5 Essential Commands

In modern IT infrastructure, service availability is non-negotiable. Operational continuity relies on your ability to quickly identify and resolve bottlenecks. This article outlines a methodical approach to diagnosing common issues on Linux servers, highlighting five fundamental commands with practical examples and real-world scenarios.

Monitoring Resources with top

The top command allows you to observe CPU and memory usage in real time, highlighting the most resource-hungry processes.

top
%Cpu(s): 75.3 us,  5.0 sy, 18.9 id, 0.5 wa  KiB Mem : 16323056 total, 1524300 free, 12567200 used

A high I/O wait percentage indicates that processes are bottlenecked by slow disk or network operations, rather than CPU constraints.

A web application was sluggish despite low CPU usage. Analyzing the system with top revealed an I/O wait exceeding 20%. The issue was resolved by migrating the database to SSDs and implementing caching.

Checking Disk Space with df

The df command displays the available disk space on mounted partitions.

df -h
Filesystem      Size  Used Avail Use% Mounted on  /dev/sda1        80G   72G  4.5G  95% /

Root partition usage exceeding 90% can cause processes to stall or crash.

A server suddenly stopped accepting SSH connections. df revealed a 100% full root partition. Clearing out obsolete logs and configuring automatic log rotation resolved the outage.

Analyzing Disk Performance with iostat

The iostat command provides detailed metrics on disk I/O performance.

iostat -x 1 3
Device:  r/s  w/s rkB/s wkB/s svctm %util  sda      2.0 45.0 120   3600  5.3   98.0

A %util value approaching 100% indicates a saturated disk device.

A database was experiencing slow query times. iostat showed disk utilization at 98%. The solution was migrating to NVMe drives and offloading logs to a separate disk.

Verifying Active Services with netstat

The netstat command lists the services listening on network ports.

netstat -tulpn | grep LISTEN
tcp6   0   0 :::22     :::*   LISTEN  1234/sshd  tcp6   0   0 :::5432   :::*   LISTEN  2345/postgres

If a service is only listening on 127.0.0.1, it will not be accessible from external networks.

A remote client couldn’t connect to PostgreSQL. Netstat showed the service was bound exclusively to localhost. Updating the PostgreSQL configuration and adjusting firewall rules fixed the connectivity issue.

Reviewing Kernel Logs with dmesg

The dmesg command outputs kernel ring buffer messages, which are crucial for tracking down critical system errors.

dmesg -T | tail -20
[Mon Oct 1 19:33:45 2025] Out of memory: Kill process 3456 (java)

This message indicates that the Linux Out-Of-Memory (OOM) killer terminated a process due to memory exhaustion.

A Java service kept crashing unexpectedly. Analyzing the output with dmesg revealed the OOM killer was stepping in. The resolution involved tuning the JVM heap size and provisioning additional RAM.

Commands like top, df, iostat, netstat and dmesg represent the first line of diagnostic defense for any system administrator. They allow you to gather objective evidence, reduce mean time to resolution (MTTR), and improve the overall resilience of production Linux infrastructures.

Share this article:

Written by

Rosario Giordano

Rosario Giordano is a system administrator and IT consultant specializing in cybersecurity and cloud, with over 20 years of experience managing enterprise Linux infrastructures. His areas of expertise include SSH hardening, Kubernetes platforms, PostgreSQL databases, VMware/ Proxmox virtualization, and compliance with NIS2 and ISO 27001 security frameworks