Question 1

Under load a server starts failing accept() and open() with EMFILE (too many open files), and the count slowly climbs over days. What is the most likely root cause?

Accepted Answer

A file descriptor leak, such as sockets or files not closed on an error path EMFILE means the process hit its RLIMIT_NOFILE descriptor limit, and a count that climbs over days points to descriptors not being closed. Check /proc/PID/fd to see what is leaking. Raising the limit only delays it. A full disk gives ENOSPC. Low memory gives ENOMEM or OOM kills. Threads do not consume descriptors by themselves.

Question 2

Linux with vm.overcommit_memory = 1 (always overcommit). A process allocates 10 GB with malloc on a machine with 4 GB of RAM and no swap, and malloc succeeds. Later the process is killed while writing to that memory. Why?

Accepted Answer

Linux overcommits: physical pages are assigned on first touch, and the OOM killer runs when they cannot be found With overcommit enabled, malloc reserves virtual address space only. Pages are backed on first write, and when RAM runs out the OOM killer picks a victim, often the biggest process. malloc did not reserve RAM. The pointer was valid; the kill is SIGKILL, not SIGSEGV. Swapping slows a process but the kernel does not kill for slowness.

Question 3

A Linux 8 core box shows load average 30 but CPU usage around 20%. What does that suggest?

Accepted Answer

Many tasks are blocked in uninterruptible I/O wait, such as on a slow disk or NFS, which Linux counts in load Linux load average counts runnable tasks plus tasks in uninterruptible sleep (state D), usually disk or network file system I/O. With low CPU, the queue is waiting on I/O, not CPU. Treating load as pure CPU demand is the common mistake. The metric works, it just measures more than CPU. Idle threads sleeping interruptibly are not counted.

Question 4

A service with 2000 threads on 8 cores spends much of its CPU time in the kernel, with very high context switch counts and low throughput. What is the best first fix?

Accepted Answer

Reduce the thread count, using a bounded pool or async I/O Far more runnable threads than cores means the scheduler constantly switches, paying for saving state and losing cache and TLB contents, and lock contention grows. A pool sized to the work removes most of it. More threads makes it worse. Time slice tuning does not help threads that block and wake frequently. You cannot pin 2000 threads to 8 cores usefully.

Question 5

A latency sensitive service on a host near its memory limit shows p99 spikes that line up with a high rate of major page faults. What is happening?

Accepted Answer

Pages are being read back from disk (swap or evicted file pages), so some memory accesses wait on I/O A major fault needs disk I/O to bring a page in, so under memory pressure accesses to evicted pages take milliseconds instead of nanoseconds. Fix by reducing memory use, adding RAM, locking critical memory or tuning swap. TLB misses are handled by a page walk and are not page faults. Thread count does not cause major faults directly. Cache thrashing costs nanoseconds, not spikes visible at p99 in milliseconds.

Question 6

A program calls write() and gets success, then the machine loses power one second later. Is the data guaranteed on disk?

Accepted Answer

No, write usually only copies into the page cache; fsync is needed to force it to stable storage On Linux write returns once data is in the kernel page cache, which is flushed later. Only fsync, fdatasync or O_SYNC wait for the device. O_APPEND controls the offset, not durability. close does not flush to disk either; the man page says so explicitly.

Question 7

You must replace config.json so that after a crash readers see either the old or the new file, and the new file must survive power loss once you report success. Which sequence is correct?

Accepted Answer

Write config.tmp, fsync it, rename it over config.json, then fsync the directory fsync of the temp file makes its contents durable before the name points at it, rename atomically swaps names, and fsync of the directory makes the new name durable. Renaming before fsync can leave a zero length or partial file after a crash on some file systems. Truncate then write has a window with no valid file. rename is atomic but not durable until the directory entry is flushed.

Question 8

fsync returns EIO. The program retries fsync and it returns 0. On Linux, can it assume the data is durable?

Accepted Answer

No, the failed dirty pages may have been dropped and marked clean, so the retry says nothing about that data After a writeback error Linux may discard or mark the affected pages clean and report the error once, so a later fsync returns success with the data gone. PostgreSQL changed to panic and recover from its WAL after this was found. The others all assume the kernel retains failed pages, which it does not promise. Reopening does not resurrect them.

Question 9

Design an LRU cache that survives a process crash and restart without losing acknowledged puts. What do you persist and when?

Accepted Answer

Log: append each put and delete to a write ahead log and fsync (or group commit several puts per fsync) before acknowledging. Snapshot: periodically write the full map to a temp file, fsync, rename over the old snapshot, fsync the directory, then truncate the log. Recovery: load the latest snapshot and replay the log; recency order can be rebuilt approximately from log order, since exact LRU order rarely needs durability. Tradeoffs: fsync per put is slow, so batching trades a small loss window or latency for throughput; process crash alone is covered by the page cache, power loss needs fsync.

OS debugging and file durability interview questions

OS for backend debugging

Files and durability

More operating systems topics

Practise these on your commute