# Why Are My Oracle Sessions Getting SIGSTOP’d?

Finally, I encountered a case that couldn't be solved using traditional Oracle Database tools. I've been experimenting with eBPF for some time, and this is a real-world scenario from an Exadata environment where I *needed* eBPF to help me truly understand the problem.

## The Problem

Users complained that database sessions were occasionally "frozen." Interestingly, these "frozen" sessions could not be killed. When DBAs used `alter system kill session`, those sessions stayed in the `KILLED` state.

Also, the load average of the machine where this was happening was suspiciously high all the time, even though no one complained about things running "too slow."

## The Investigation

I'll keep this story brief: the sessions couldn't be killed because their dedicated server process was STOPPED. By STOPPED, I mean the Linux process was in a `T` state. You can replicate this by sending a SIGSTOP signal to any Linux process, like this:

```bash
kill -SIGSTOP 12345
```

So, why is Oracle stopping those sessions? Initially, I suspected it might be related to Oracle Resource Manager and `cpu_count` settings because the load was high. However, as far as I know, Oracle doesn't use STOP/CONTINUE signals for resmgr. I even ran a quick test in my lab using resmgr under heavy load and did not see any such signals being sent.

## Observing the Signals

This is where the [`bpftrace`](https://bpftrace.org/) comes into play. We can use it to detect when the signal is sent (or received) and who sent it. And it isn't really a voodoo, all you need is a simple [`bpftrace`](https://bpftrace.org/) script:

```bash
#!/usr/bin/env -S bpftrace 

tracepoint:signal:signal_generate /args->sig == 19/ {
    time("%Y-%m-%d %H:%M:%S");
    printf(" SIGSTOP sent from %d (%s) to %d (%s)\n", pid, comm, args->pid, args->comm);
}
```

I was hoping to let this script run for a week or so and then see which processes were sending SIGSTOP signals. But to my surprise, there were tons of such signals, many of them within the same second.

When I tried to see details of the process that sent the signal (the pid is printed using the script above), I noticed that such a process no longer existed. Even when I added printing info about the pid into the `bpftrace` script, like this:

```c
system("cat /proc/%d/stack", pid);
```

I noticed that most of the time the process did not exist anymore when my script reached this call. So, they were very *short-lived* processes. Using a similar approach (reading `PPid` from `/proc/pid/status`), I found the parent process that was spawning them. And lo and behold, the parent process was the Oracle database session process itself!

Turns out, the Oracle database process was spawning a child process every second or so to send a STOP signal, do something, then send a CONTINUE signal again.

Why would Oracle do that?!

## The Why

To understand why, I tried profiling a database session, which was waiting on `SQL*Net message from client` for a while now but was still receiving STOP/CONTINUE signals.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1749143976659/b5b615d0-9d36-4962-b3bb-b7a9ebe4e0b7.png align="center")

I’ve blurred out functions that were named like `Something::doSomething`. Notice how it looks like C++ code? If I remember correctly, most of Oracle's core is written in C. Could this really be Oracle code?!

Turns out that it is *not*. Inspecting shared libraries used by this process, like this:

```sql
cat /proc/12345/maps
```

proved, that Oracle process was using `.so` libraries from `/opt/some-other-product`.

So, it's not even an Oracle code; it's code from a product, that hooked Oracle database sessions in order to extract session-related info in a consistent manner.