Why "too many open files" is more than a socket error

It starts quietly. Your API returns 503 or connection refused. Logs show os.Open: too many open files or accept4: too many open files. Restarting the service fixes it—for a few hours or days—until it happens again.

The phrase "open files" is misleading. In Linux, a file descriptor is just an integer handle to an open kernel object. That object can be a regular file, a directory, a socket, a pipe, an epoll instance, a timerfd, or an inotify watch. When your process hits its file descriptor limit, it cannot open any of these, not just text files.

How limits work

Linux tracks file descriptors at two levels:

  • Per-process soft/hard limit: checked by ulimit -n (soft) and ulimit -Hn (hard). A process can raise its soft limit up to the hard limit, but only root can raise the hard limit.
  • System-wide limit: /proc/sys/fs/file-max. This is the ceiling for all processes combined.

If your application runs under systemd, the relevant setting is often LimitNOFILE= in the unit file, which overrides inherited shell limits.

bash
# Check current shell limit
ulimit -n

# Check hard limit
ulimit -Hn

# Check a running process
cat /proc/<PID>/limits | grep "Max open files"

# System-wide limit
cat /proc/sys/fs/file-max

Finding what is open

Before raising limits, find out what is consuming descriptors. A leak looks like a steadily growing count in /proc/<PID>/fd.

bash
# Count open descriptors for a process
ls /proc/<PID>/fd | wc -l

# List what is open (needs lsof installed)
lsof -p <PID>

# For network sockets specifically
ss -tanp | grep <PID>

If you see thousands of entries pointing to deleted log files, rotated logs, or sockets in CLOSE_WAIT, you have a leak. If the count is stable and simply too high for your traffic, you need more headroom.

Common leak patterns

Applications rarely leak descriptors intentionally. The usual suspects:

  • HTTP response bodies not closed. In Go, forgetting resp.Body.Close() leaves the underlying TCP connection open until garbage collection reclaims it, which can take much longer than you expect.
  • Database connections or statements left open. Connection pools help, but only if you actually close rows and statements.
  • Log files held open after rotation. If your app opens a log file and never reopens it, logrotate moves the file but the process keeps writing to the old inode. The descriptor stays open, and disk space is not reclaimed.
  • Inotify watches or epoll subscriptions not removed. Each watch consumes a descriptor and kernel memory.
  • Thread or goroutine leaks. Each blocked thread may hold sockets or file handles.

Here is a Go example that leaks descriptors under load:

go
func fetch(url string) error {
    resp, err := http.Get(url)
    if err != nil {
        return err
    }
    // BUG: resp.Body is never closed. The TCP connection stays open.
    _, _ = io.Copy(ioutil.Discard, resp.Body)
    return nil
}

The fix:

go
func fetch(url string) error {
    resp, err := http.Get(url)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    _, _ = io.Copy(io.Discard, resp.Body)
    return nil
}

Raising limits safely

If the application genuinely needs more descriptors—for example, a proxy or a high-connection server—raise the limit rather than working around it.

For systemd services:

ini
[Service]
LimitNOFILE=65535

For PAM-based shell limits (/etc/security/limits.conf):

markdown
* soft nofile 65535
* hard nofile 65535

Check the effective limit after restarting. A common mistake is editing the unit file but forgetting systemctl daemon-reload.

Monitoring and prevention

File descriptor exhaustion is easy to miss until it causes an outage. Track these signals:

  • Current open descriptors per process (/proc/<PID>/fd count).
  • Percentage of limit used.
  • System-wide usage versus /proc/sys/fs/file-max.

Set an alert when a process reaches 70–80% of its limit. That gives you time to investigate before the next deploy fails with too many open files.

Incident response checklist

When you hit the limit in production:

  • Identify the affected process: pgrep or check the service name in the error log.
  • Count its descriptors: ls /proc/<PID>/fd | wc -l.
  • List the top descriptor types: lsof -p <PID> | awk '{print $NF}' | sort | uniq -c | sort -rn | head.
  • If it is a leak, restart the service to recover immediately, then find the code path.
  • If it is genuine load, raise LimitNOFILE and redeploy.
  • Check /proc/sys/fs/file-max if many services are failing at once.

Takeaway

"Too many open files" is usually a resource leak wearing a misleading name. Check /proc/<PID>/fd first, fix unclosed bodies and handles in code, and only raise LimitNOFILE when your legitimate concurrency requires it. Monitor descriptor usage like you monitor CPU and memory.

IFLEON can help you audit and stabilize production Linux workloads.