A critical vulnerability in the Linux kernel, identified as CVE-2026-98432, threatens Ubuntu systems through a networking memory flaw that could cause crashes and service disruptions. Ubuntu released an advisory on October 15, 2026, and provided an update on October 29. The vulnerability has been assigned a CVSS severity score of 9.8, indicating a high priority […]
EMBRACE affects AI coding tools Cursor, GitHub Copilot, Codex, VS Code and Claude Desktop, enabling code execution, file theft and browser address-bar spoofing.
Three malicious Firefox extensions impersonating Phantom, Jupiter, and Bitget to steal cryptocurrency wallet recovery phrases and private keys. The campaign combines deceptive branding, bundled credential stealers, and remotely delivered JavaScript, with stolen information routed through infrastructure under bonto[.]io. All three listings were removed before Socket published its findings on October 27, 2026. Phantom’s impersonator advertised […]
Recently a new project was released by Praetorian. Praetorian is open-sourcing the offensive tools their red team actually uses on engagements. They call it 12 Caesars and their team is planning to release one tool a week for twelve weeks. Every tool is written in Go and ships as a single binary with zero external dependencies.
Each tool is assigned a Caesar name and a blog post, where you can learn more about these tools.
We’ll look at four of these tools, including the basic ones, to make this article suitable for everyone. Before we proceed, it’s important to remind you that you can only test systems you own or have explicit written authorization to test. We will use our lab for it.
Titus
Our first tool is Titus. Titus finds credentials, API keys, and tokens in source code, files, git history, container images, and more. The tool has 487 detection rules covering AWS, GCP, Azure, GitHub, Slack, databases and hundreds of other services. Similar to TruffleHog, it can validate credentials as it finds them by calling the provider APIs.
It’s also available as a Burp Suite extension and a Chrome extension.
kali > curl -sL -o titus https://github.com/praetorian-inc/titus/releases/latest/download/titus-linux-amd64
kali > chmod 777 titus
kali > ./titus --help
Let’s start by pointing it at test keys released by Truffle Security.
It found all the keys in the repo. Companies often publish their secrets by accident. Some live in the repositories for years, so it’s good to do regular checks to prevent unnecessary leakage.
Nerva
Nerva is a fingerprinter supporting 170 protocols across TCP, UDP, and SCTP, plus dozens of HTTP technology fingerprinters. You can feed it with open ports from naabu, nmap or other tools and it’ll tell you what’s actually running with its version.
kali > curl -sL -o nerva.tar.gz https://github.com/praetorian-inc/nerva/releases/download/v1.70.3/nerva_1.70.3_linux_amd64.tar.gz
kali > tar --no-same-owner -xzf nerva.tar.gz
kali > chmod 777 nerva
kali > ./nerva --help
Now let’s use our Ubuntu to show a basic test.
kali > ./nerva -t 192.168.131.9:22,11434
It will be way more useful when you work with big lists of corporate assets during a pentest.
Brutus
Brutus is a modern Hydra alternative. It supports more than 40 protocols including SSH, RDP, MySQL, PostgreSQL and a lot more. Brutus was built to work with Nerva or naabu output.
kali > curl -sL -o brutus.tar.gz https://github.com/praetorian-inc/brutus/releases/latest/download/brutus-linux-amd64.tar.gz
kali > tar --no-same-owner -xzf brutus.tar.gz
kali > chmod 777 brutus
kali > ./brutus --help
Each command has its own help menu to make it easy to work with.
kali > ./brutus creds --help
Now once again lets use our Ubuntu to spay passwords.
kali > ./brutus creds --target 127.0.0.1:22 --protocol ssh “admin:admin”
It was easy. As mentioned before, this tool chains with Nerva, and that’s how it becomes useful to test the entire infrastructure against common password and detection.
Julius
With AI services proliferating inside corporate environments, we want to know what LLM infrastructure is running on our hosts. Julius will fingerprint more than 60 AI services including Ollama, vLLM, LiteLLM, LocalAI, Hugging Face TGI, AWS Bedrock and AnythingLLM. It can also list available models from endpoints.
kali > curl -sL -o julius.tar.gz https://github.com/praetorian-inc/julius/releases/download/v1.5.1/julius_1.5.1_linux_amd64.tar.gz
kali > tar --no-same-owner -xzf julius.tar.gz
kali > ./julius --help
You can list all the AI services Julious can fingerprint:
kali > ./julius list
We will take a look at our machine.
kali > ./julius probe http://192.168.131.7:11434
Ollama sits on port 11434, and it’s important not to leave it exposed publicly, because everyone can use your models without authentication.
Summary
There are more tools available in this project. Most of them are meant to be chained together and work well this way. The company says its team is actively using these same tools during their engagements. While you won’t need everything at once, experiment with those tools and see what fits you best.
If you like red teaming, we have our Red Team Operator training, where we cover more tools and techniques to help you emulate real APT work, so you can give a company a realistic stress test and help make it secure.
Microsoft releases emergency .NET updates to fix a Brotli memory flaw and Linux privilege escalation, while correcting two incomplete October security patches.
Kali Linux 2026.3 has been officially released, bringing six new security tools, updated image-building scripts, KDE Plasma 6.7, and major NetHunter improvements. Announced on October 26, 2026, this third release of the year includes 126 package updates and upgrades the Linux kernel from version 6.19 to 7.1.5.kali. This release focuses on customization, making it easier […]
Just finished the Checkpoint machine on Hack The Box (Medium difficulty) and it was a really enjoyable Active Directory box with some modern and creative attack paths.
I started with the provided credentials for `alex.turner`. After some enumeration I discovered that a previously deleted account (`mark.davies`) could be restored and that this account had write access to a share called DevDrop. This share was used by the organisation to stage VS Code extensions.
By creating a malicious VSIX extension targeting VS Code engine version 1.118.0 and dropping it on the share, a scheduled task automatically installed the extension and gave me a reverse shell as `ryan.brooks`. This was a really clean example of abusing enterprise software deployment processes.
From the `ryan.brooks` foothold I used Rubeus to obtain a TGT, then leveraged bloodyAD to create a malicious dMSA under a writable OU and impersonated a higher-privileged account. This eventually granted access to a VMBackups share containing a Windows Server memory dump. Using Volatility I extracted the local Administrator NTLM hash from the dump, authenticated with Evil-WinRM, and captured the root flag.
Overall, Checkpoint does a great job of chaining together several current techniques: writable shares for extension abuse, deleted object restoration, dMSA abuse, and memory forensics. It’s a solid Medium box that feels realistic and rewards careful enumeration.
Would recommend it to anyone working on modern Windows / Active Directory attack paths.
In October, several unpatched Linux vulnerabilities were disclosed on public mailing lists, revealing technical details before fixes were widely available. These disclosures included two critical flaws in KSMBD that could let remote attackers execute code, along with reports of privilege escalation and information disclosure. Trend Micro’s Zero Day Initiative (ZDI) submitted the reports, prompting Linux […]
Hackers used apparently AI-generated scripts to compromise government Linux infrastructure, harvest credentials and pursue sensitive databases and official mailboxes in a campaign tracked as CL-UNK-1211. The investigation, published October 15, linked the operation to government entities and state-affiliated infrastructure. Attribution remains unresolved: under Unit 42’s attribution framework, the UNK designation means researchers have not established […]
October 2026 brought a clear demonstration that hardware wallets, while far safer than software alternatives, are not immune to sophisticated physical attacks.
Beginning on 9 October, hundreds of users who had purchased Ledger devices through the Southeast Asian reseller CryptoBilis saw their funds drained across Bitcoin, Ethereum, TRON and other chains. On-chain researchers estimated losses between roughly $86 million and more than $100 million. Ledger confirmed that at least one affected device contained an unauthorized hardware implant and stated that every confirmed loss in the incident was linked to a device sold by CryptoBilis. The company has emphasized that its own infrastructure, firmware and direct sales channels were not compromised.
The Supply-Chain Attack
The attack was physical. Someone in the distribution chain opened genuine Ledger Nano X devices, installed a small circuit board and resealed them. The implant sits inside the enclosure under or behind the screen where foam padding normally rests, and taps the SPI bus that carries data from the Secure Element to the display. When the owner sets up the device and the screen shows the 24-word recovery phrase, the implant’s microcontroller recognizes those screens, reconstructs the words and sends them out over LTE using a provisioned eSIM and a small antenna. Because the original Secure Element remains untouched, the device continues to pass Ledger’s Genuine Check. Mark Karpelès and Tibane Labs have documented multiple generations of the implant, progressing from early versions with visible wires and a smaller battery to later flex-circuit designs that are harder to notice and sometimes keep the original battery.
Source: Mark Karpelès
Mark Karpelès publicly posted a photo of a unit that arrived from Malaysia with flawless packaging. The extra hardware only became visible when he opened the case.
Source: Mark Karpelès
Similar implants have been reported in devices obtained through other channels, including Japanese marketplaces and at least one European store, although Ledger has so far tied every confirmed fund loss specifically to CryptoBilis purchases.
It was confirmed that the Ledger supply chain is compromised in Europe as well. Source: @0xJohannes_
The company has asked the reseller to halt sales and has advised recent buyers not to initialize unused devices. Those who already completed setup are told to move their assets to a new signer purchased directly from Ledger and initialized with a fresh seed phrase.
How to Buy a Hardware Wallet Safely
The single most effective protection is to purchase directly from the manufacturer’s own online store.
Authorized resellers are listed on the manufacturer’s website, yet CryptoBilis was also a partner that introduced modified units into the channel. Marketplace listings of third-party shops carry a much higher risk of both counterfeits and implants.
Once the package arrives, you should confirm that a new device requires you to generate a recovery phrase yourself on first use. Any recovery phrase that’s already written on a card or printed in documentation belongs to someone else and should never be used. The phrase must appear only on the device’s own screen and you must write it down yourself on paper or metal. Never type it into a computer or phone, don’t use websites or applications with it.
Below you can see a photo of the hardwallet owner signing a transaction inside the faraday cage to avoid any interference and possible theft.
Source: @oren_z0
Check Your Wallet
The first question is the source of the device. Anyone who bought a Ledger from CryptoBilis or another untrusted reseller in recent months should treat the unit as unsafe. Genuine Check is still worth running because it confirms that the Secure Element is original and the firmware is approved, yet it can’t detect added circuitry that merely listens to the screen.
You can also visually inspect the case for extra wires or antennas that can reveal tampering, but modern implants are designed to be inconspicuous and reliable detection usually requires opening the device. After a wallet is set up, compare receiving addresses shown on the device screen against those displayed in the companion application and review transaction history on block explorers for unexpected outgoing transfers. The recovery phrase must be generated by the device under your direct observation and written down only by you.
Mark Karpelès also has a few videos on opening and closing Ledger wallets that you might find helpful.
What to Do If You Suspect Compromise
If a device was purchased from CryptoBilis or any other questionable source and has already been initialized, the existing recovery phrase should be treated as compromised. Move your assets to a new wallet whose seed was generated on a trusted device bought directly from the manufacturer. Never reuse the old seed for any purpose.
More Recommendations
A BIP-39 passphrase adds a meaningful layer of protection because it isn’t displayed during ordinary setup and must be known in addition to the seed. It’s also known as passphrase (the extra “25th word”). It’s optional and you may choose it later. It is not generated or displayed by the device during ordinary setup. Adding it creates a hidden wallet, which means you have completely different keys and addresses.
For larger holdings use a multisignature arrangement to reduce reliance on any single device. Always check the address where you send your money to, as it can often be replaced by malware or bots making small transactions with similar addresses to mislead you into sending money to this similar address.
Hardware wallets still provide strong protection against the majority of remote thefts precisely because private keys never leave the Secure Element and every transaction requires physical confirmation on the device itself.
Summary
Hardware wallets can pass authenticity tests and still contain an implant capable of capturing the recovery phrase the moment it appears on screen. The best thing you can do is to buy devices only from the manufacturer, generate the seed yourself on the device and write it down offline. For anyone who recently purchased a device from a third party, the cost of generating a new seed and moving funds is far lower than the cost of discovering the compromise after the fact.
After switching my Fedora laptop from GNOME to Hyprland, I started paying a bit more attention to what actually runs when I log in. Under GNOME, a lot of this was easy to forget about. With my new setup, I wanted to know which applications were starting with the session, which services were enabled,
Just finished the Insane difficulty Eloquia machine on Hack The Box which one of the more layered and creative Windows boxes I’ve done in a while.
The path starts with a custom Django publishing platform that integrates with a Qoogle OAuth service. After registering and creating articles, you discover that the application trusts certain meta-refresh tags and can be leveraged for redirects. Combined with OAuth authorization flows and a bit of CSRF, this opens the door to higher-privilege access inside the web application.
From the admin interface, SQL Explorer becomes available. Because the backend is SQLite, `load_extension` can be abused to load a malicious DLL. That gives the first reverse shell as the `eloquia\web` user and the user flag.
Privilege escalation requires deeper enumeration. Edge browser credentials stored for the Olivia.KAT account can be extracted and decrypted offline. WinRM then provides a stable shell as that user. From there, Seatbelt and process enumeration reveal an interesting custom service called Failure2Ban that runs with high privileges and periodically restarts. By racing a binary overwrite of `Failure2Ban.exe` with a reverse shell, you land as `NT AUTHORITY\SYSTEM` and grab the root flag.
What made this machine feel genuinely Insane was the number of different skill sets required: web application logic, OAuth/CSRF, SQLite extension loading, Windows credential dumping, service abuse under AppLocker and PowerShell restrictions, and careful timing for the final binary overwrite.
numbat monitors Claude Code, Codex and other AI coding agents and rebuilds their sessions. Tested on a staged session; its blocking is opt-in and narrow.
Completed another Hack The Box machine: Devhub (Medium).
Devhub was a great exercise in exploring modern development tooling, service misconfigurations, and Linux privilege escalation. The journey began with web enumeration and identifying an exposed MCP Inspector service. Further investigation revealed a command injection vulnerability that provided initial access to the machine.
From there, internal service enumeration uncovered a localhost-only Jupyter instance. Accessing it through SSH port forwarding and investigating its authentication mechanism opened the path to another user. The final stage involved examining an internal operations service, understanding its exposed functionality, and identifying a weakness that ultimately led to root access.
Devhub highlights how development tools, internal services, and poorly secured administrative functionality can create unexpected attack paths when combined. Another valuable machine for practising enumeration, lateral movement between local users, and privilege escalation.
As most of you know, we have played a key role in the defense of Ukraine. Among our many activities in defense of Ukraine is the hacking of IP cameras throughout the country. In this way, we can spy on Russian activities and war crimes. We did this at the request of the Ukraine Army starting in April 2022.
To hack these cameras we used multiple methods and techniques. In hacking, we often need to explore multiple methods to be successful. Persistence is a key hacker characteristic.
As hackers, of course, it is important to take a strategic approach to any target. Always use the simplest methods first before progressing to more advanced and time-consuming attack methods.
In our first step, we identified the unprotected cameras using such sites as Shodan, Google, and Censys. Next, we tried default credentials. These default credentials vary by camera and manufacturer, so make certain to check our list of default credentials for nearly every camera and manufacturer. That technique yielded a few cameras.
Next, we tried to hack the cameras with weak passwords. This yielded significant results! The primary tool we used in that effort was cameradar.
In this tutorial, I will show you how to use this tool for IP camera hacking just like we did in the Ukraine war!
RTSP
Before we begin to hack IP cameras, you need a bit of background in RTSP. RTSP is the protocol that most of these IP cameras use. Not all of the cameras use RTSP, but the vast majority do. Before we go any farther, we need to say that those cameras using proprietary or other protocols will not be exploitable by cameradar.
RTSP is an application-layer protocol used for commanding streaming media servers via pause and play capabilities. It thereby facilitates real-time control of the streaming media by communicating with the server — without actually transmitting the data itself.
Rather, RTSP servers often leverage the Real-Time Transport Protocol (RTP) in conjunction with the Real-Time Control Protocol (RTCP) to move the actual streaming data.
Most IP camera use the Real-Time Streaming Protocol (RTSP) to establish and control video and audio streams. The content is delivered using Real-time Transport Protocol (RTP). RSTP does not provide any configuration of the device. That must be done using the URI and IP address. Any configuration changes must be done via the web interface.
Most systems support RTSP as a fallback even if they are using a different protocol such a PSIA or ONVIF
When a user initiates a video stream from an IP camera using RTSP, the device sends an RTSP request to the streaming server. This jump starts the setup process.
Subsequently, the video and audio data can then be transmitted using RTP.
You can think of RTSP in terms of a television remote control for media streaming, with RTP acting as the broadcast itself.
While similar in some ways to HTTP, RTSP defines control sequences useful in controlling multimedia playback.
While HTTP is stateless, RTSP has state; an identifier is used when needed to track concurrent sessions
Like HTTP, RTSP uses TCP to maintain an end-to-end connection and, while most RTSP control messages are sent by the client to the server, some commands travel in the other direction (i.e. from server to client).
RTSP uses the following commands, typically sent from the client to the server, when negotiating and controlling media transmissions:
Options: This request determines what other types of requests the media server will accept.
Describe: A describe request identifies the URL and type of data.
Announce: The announce method describes the presentation when sent from the client to the server and updates the description when sent from server to client.
Setup: Setup requests specify how a media stream must be transported before a play request is sent.
Play: A play request starts the media transmission by telling the server to start sending the data.
Pause: Pause requests temporarily halt the stream delivery.
Record: A record request initiates a media recording.
Teardown: This request terminates the session entirely and stops all media streams.
Redirect: Redirect requests inform the client that it must connect to another server by providing a new URL for the client to issue requests to.
Other types of RTSP requests include ‘get parameter,’ ‘set parameter ,’ and ’embedded (interleaved) binary data,’
Now that you have a little background in RTSP, you are ready to start cracking IP camera credentials!
Step #1: Download and Install cameradar
Although cameradar can be run natively in Linux, I find that it works best in a docker container.
Now, you are ready to begin to brute-force IP cameras!
Step #2: Run the RTSP Credential Brute-forcer
Now that you have docker and cameradar installed, you only need to point cameradar at the IP address of the camera that you want to brute-force!
For instance, to brute force a camera at 192.168.1.101 (obviously, not an IP address of a real camera), we would simply enter;
kali > sudo docker run ullaakut/cameradar -t 192.168.1.101
Cameradar will now attempt to find a RTSP stream at one of the default RTSP ports namely 554, 5554 and 8554. If you suspect there may be other ports with RTSP streams (you may want to run an nmap scan first), you can add them with the -p switch, such as
kali > sudo docker run ullaakut/cameradar -t 192.168.1.101 -p 9554
Step #3: Using Custom Username and Password Lists
By default, cameradar uses a small username and password list of the most common usernames and passwords. It’s good strategy to use these first but if they are unsuccessful, it’s time to bring out the big guns!
In this context, big guns means larger and more appropriate username and password lists. From my experience hacking cameras in Ukraine and Russia, the usernames usually are simple such as admin, root, admin1, admin3, etc. This means that you can probably use the default username list but passwords vary quite a bit. That’s why you should use a good password list that is appropriate for your environment (for instance, using a Spanish list in a Spanish speaking nation).
First, the password list must be json format. There are several websites that can covert your text file to json such as https://anyconv.com/txt-to-json-converter/. Your .txt file will then be appended with a json extension. So, if we were using the seclist’s password list;
Password Cracking of IP camera credentials is very similar to other remote password cracking once you become familiar with the RTSP protocol. In fact, in many ways it is easier, as it is rare to find a lockout (limiting how many attempts you can make) on the RTSP protocol. By using a tool like cameradar, we were able to successfully access a large percentage of IP cameras with weak passwords.
My websites kept going down for two or three minutes at a time, and then fixing themselves.
No pattern. No warning. By the time I logged in over SSH, the server looked perfectly healthy. Load was back to normal, PHP workers had exited, and there was nothing left to look at. Just a few angry messages from visitors and an uptime monitor saying the site was down at 3:41 AM.
This post is the story of how I chased that problem, the tool I ended up writing, and what it can do today. If your server randomly slows down and you can never catch it in the act, I think you will recognize a lot of this.
The first real clue I got was a load average of over 20 on a 4-core box, while the CPU was only about 3% busy.
That combination makes no sense if you think load means “CPU is busy”. It doesn’t. Load counts processes that are waiting as well as running. So my server was not working hard. It was stuck, with dozens of processes queued up behind something.
MariaDB looked fine. Apache was responsive. Memory was okay. The only odd thing was a pile of lsphp workers and a very high iowait number.
And every single time, by the time I got there, the evidence had already disappeared.
Why monitoring was not enough
I already ran Netdata, and I want to be clear: Netdata is great.
But monitoring answered a question I already knew the answer to. It told me that my server had a problem, at 3:41 AM, with a nice red graph. What I needed to know was why — which process was stuck, what it was waiting for, and who or what started it.
That’s a different job. Monitoring samples numbers for dashboards. I needed something that would grab deep evidence at the exact moment things went bad, and keep it for me to read in the morning.
So I thought about it like an aeroplane. A plane doesn’t just have gauges in the cockpit. It has a black box that records everything, so that after something goes wrong, you can go back and find out what happened.
That was the idea: a black box for my web server.
The first version was simple (and not very smart)
The plan was basic, and I kept it basic on purpose:
Every minute, take a cheap sample — load, CPU, memory, swap, PHP worker count, MariaDB threads, mail queue, TCP states, stuck processes. Almost all of it read from /proc, so it costs nothing.
If a number crosses a threshold, go into panic mode and capture the expensive stuff: full process tree, iostat, vmstat, open files, disk stats, kernel messages, MySQL process list.
Keep capturing until the server recovers, then write a summary and stop.
I had one hard rule from day one: the tool must never make an outage worse. A forensic recorder that hangs the server while trying to investigate it is useless. So everything runs with timeouts, the heavy commands only run during an incident, and package managers like dnf are detected but never invoked — calling one during a disk stall could block forever.
That first version worked. It caught incidents. And then it started confidently telling me things that were completely wrong.
Where it got interesting: every wrong answer taught me something
This is the part I actually want to share, because the tool only became useful after it was wrong a few times. I keep a file in the repo called docs/decisions.md listing every approach I had to throw away. Here are the ones worth telling.
Mistake 1: it blamed whatever was running
My early version scored causes by looking for process names. If it saw backup or security or maintenance processes in the list, it gave points to “maintenance activity”.
One incident got reported as “Maintenance (84%)”. It was nonsense. My server runs Imunify360, and those daemons are running 24 hours a day. Their presence during an incident tells you absolutely nothing. Worse, the tool had also matched its owntimeout wrapper and counted that as evidence.
What I learned: presence is not evidence. Now a process being there scores almost nothing. It only becomes real evidence if that same process also shows up as the measured top consumer of CPU or disk. If nothing is actually consuming anything, the report says so out loud.
Mistake 2: it named a process using 9% of the CPU
Another incident had the CPU at 93% on a 4-core box. The tool proudly named the biggest process… which was using 9%.
The real answer was fourteen short-lived PHP workers at about 26% each. Ranking by single process can never express “PHP as a whole took the box”.
What I learned: group by program, not by process ID. Now reports combine workers into one row, with the count, the peak, and the average. And if total CPU is high while no single process is over 15%, the report prints a warning that the cause is spread out.
Workers grouped by program, so a pool of short-lived PHP processes reads as one line instead of disappearing.
Mistake 3: it was always measuring the recovery, not the event
This one was the most annoying, and the most important.
Load average over one minute is a lagging number. It peaks after the real work is done. In one incident the load went from 0.63 to 22.46 between two samples, 69 seconds apart. The trigger fired, panic mode started, and the first measurement landed exactly as the workers were exiting.
In other words, the design was structurally guaranteed to miss any burst shorter than the sampling interval. Making the trigger slightly faster would not have fixed it. The ordering itself was wrong.
What I learned: if the evidence expires, you cannot wait for a trigger to go and fetch it. So I added a ring buffer that samples continuously in the background and keeps a rolling window of recent history. When an incident fires, the tool already has the minutes before the trigger. That’s the --runup command: what was building up before everything went bad.
Mistake 4: the sampler was 29 seconds late
For a while my reports kept saying “wait channels: unavailable”. I assumed the capture was broken.
It wasn’t. It was late. Panic mode collected things in a nice readable order — general system state first, then the detailed stuck-process forensics. But those early commands took time: one of them (ss -antp) took 17 seconds by itself. By the time the important capture ran, 29 seconds after the trigger, every stuck process had cleared.
Eight attempts across four incidents. Zero stuck tasks captured. The tool worked perfectly and recorded nothing, which is the worst kind of bug, because it looks like a missing feature instead of a measuring error.
What I learned: collect in order of how fast the evidence disappears, not in the order that reads nicely. Sub-second kernel reads first, slow sampling commands last. And anything truly perishable moved into the always-running ring buffer.
Mistake 5: one incident stayed open for three and a half hours
A crash in the panic script meant nothing closed one incident. It stayed open for 3 hours 27 minutes, until a later unrelated event got pulled into it.
So the report described a disk stall at 18:32, but its only snapshot was taken at 21:59 during an unrelated nightly update. The verdict? “Package manager (38%)” — with “corroborating evidence” that the three-hour gap had invented all by itself.
What I learned: evidence has an age, and age matters. Now incidents get closed as soon as a healthy sample arrives, every capture records how far it was from the trigger, and a stale capture can no longer be used to name a cause.
The payoff: the day it finally proved the real cause
After all that, the tool caught what I had been chasing for months. Not guessed — proved, from the kernel’s own records.
The stuck processes were all sitting in the same place:
analysis: The automatic analysis: load 8.73 at 5% CPU, and the kernel wait channel naming exactly what the stuck processes were waiting on.
In plain English: every process trying to safely save a file to disk was queued behind the filesystem journal, and the journal could not commit because the virtual disk was not completing its writes. One stuck write freezes everything that touches the filesystem. That is exactly how you get load 20 at 3% CPU.
Then the disk numbers backed it up. During two incidents the recorder caught:
the disk at 100% utilisation while moving roughly zero data
service time up to 1001 ms per request
one read that took 7849 ms
15 requests stuck in flight, never completing
compared to a normal all-day average of about 11% utilisation
A disk reporting “completely busy” while transferring nothing is not a disk being overworked by my websites. It’s a disk that isn’t being served by the host underneath it. This is a Namecheap VPS, and that is the conclusion the evidence points to: the guest was asking politely and the host wasn’t answering.
And the best part: the processes doing all that saving were not my websites at all. It was server housekeeping — cPanel tailwatchd, quota checks, security file-integrity scans, the mail queue runner, cron — all calling fsync at the same time on a disk that could not absorb it. My biggest actual disk user was moving a pathetic 96 to 232 KB per second.
I sent Namecheap the timestamps and the figures — two exact incidents, nine consecutive one-second disk samples each, with the service times and the stuck request count. That was only possible because something had been recording while I was asleep. Without it, the whole conversation would have been me saying “my server feels slow sometimes” and them saying “we see no problem”.
I’ll be honest about scope, because that honesty is the whole point of the tool: the dramatic zero-throughput numbers appeared in 2 of 9 incidents. In the other 7 the stall was still proven, just by a different kernel measurement (PSI), because the disk sampler happened to miss the worst second. The reports say which is which rather than pretending every incident is the same.
What Server Forensics Recorder can do today
So that’s how Server Forensics Recorder ended up the way it is. It’s on GitHub, free and MIT licensed, because this problem is clearly not unique to my server. Everything is plain Bash — there’s no daemon, no web interface and no database, and every report is text you can read over SSH.
It records continuously, cheaply. One lightweight sample per minute, plus a background ring buffer. Normal overhead is tiny because almost everything comes from /proc.
It captures deep evidence only during an incident — process trees, disk statistics, open files, kernel messages, pressure stall information, MySQL state — ordered so the most perishable evidence is grabbed first.
It writes its own root-cause analysis. When an incident ends, you get a report that separates observed facts (measurements, no opinion) from inference (reasoning) from proven (direct kernel evidence). It ranks the possible causes with a confidence score, and here’s the bit I’m most pleased with: the confidence is capped when the decisive evidence is missing, and the report tells you it was capped and why. It never claims 100%. If nothing clears the noise floor, it says Inconclusive instead of inventing a cause.
It names the URL and the visitor, not just the layer. “PHP was busy” is useless. At incident close, the recorder reads the domain access logs for the incident window and reports the actual pages hit per domain, the top client IPs, and counts for commonly abused WordPress endpoints like xmlrpc.php, wp-login.php and admin-ajax.php. It’s Cloudflare-aware, so it points you at the real visitor IP instead of blaming the CDN, and it labels the server’s own loopback requests so wp-cron never looks like an attacker.
That feature found a genuine attack. A single IP — 96.126.176.53 — was responsible for 22,542 of 25,802 requests, about 87% of everything in the window, flooding two WordPress admin endpoints behind a fake Chrome user agent. It came in through Cloudflare, which is exactly why a server-level firewall ban would have been useless: the connection my server sees is a Cloudflare edge, not the attacker. I blocked it at Cloudflare instead, and it stopped in minutes.
Request attribution: the real pages behind index.php, plus the client IPs driving them.
It’s just as useful in reverse. Another night I was sure I was under attack — and the report showed 11 requests in the whole window, from different IPs, on normal mobile browsers. Not an attack. Just my slow disk again. That saved me from blocking innocent visitors and adding firewall rules for nothing.
It records CPU steal. That’s time my server was ready to run but the host gave the physical core to somebody else. It’s the one number in a hosting dispute that cannot be blamed on my own workload.
The commands I use most:
Command
What it tells me
server-forensics --incidents
Every recorded incident, newest first, with the trigger reason
server-forensics --last-analysis
The automatic root-cause verdict for the latest incident
server-forensics --offenders
Which program actually caused the spike, grouped properly
server-forensics --io
Full disk evidence — latency, utilisation, per-process I/O
server-forensics --requests
Which URLs and client IPs drove the load
server-forensics --runup
What was building up before the trigger fired
server-forensics --health
Is the recorder itself running properly
Who this is for
I wrote this for my own cPanel server, but it will help you if:
Your site freezes for a minute or two and then recovers, and you can never catch it live. This is the exact case it was built for.
You see high load with low CPU and nothing you read online explains it.
You’re on a VPS and suspect the host. Suspecting is cheap. Having timestamps, latency figures and kernel stack traces to send to support is what actually moves a ticket forward.
You run WordPress sites and need to know if it’s an attack or not. The request report answers that in seconds, so you don’t block real customers out of panic.
You manage servers for clients. Instead of “it was probably a traffic spike”, you can show what was stuck and what it was waiting for.
You’re learning Linux performance. Honestly, reading these reports taught me more about load average, iowait, the ext4 journal and fsync than any article did — because it was my own server, with real numbers.
It is not a replacement for monitoring. Keep Netdata or Prometheus for live graphs and alerts. This sits next to them and answers the question they can’t.
Install it
It’s built for AlmaLinux 8 with cPanel, CloudLinux, Apache with mod_lsapi, MariaDB and Exim, but the collector is generic enough for most Linux web servers.
git clone https://github.com/Dr-Hack/Server-Forensics-Recorder.git
cd Server-Forensics-Recorder
sudo bash install.sh
It installs a systemd timer that runs once a minute. Reports land in /var/log/server-forensics/. Set your thresholds using your own server’s normal numbers — start conservative and tighten later.
The morning-after view: one command, and the last outage explains itself.
MIT licensed. Issues and pull requests are welcome.
FAQ
Why is my server load high when CPU usage is low?
Because load average counts processes that are waiting, not just processes that are running. If load is high while the CPU is idle, your processes are blocked on something — usually disk I/O, sometimes memory pressure or a lock. High iowait alongside high load points at storage.
Does this replace Netdata, Prometheus or Grafana?
No. Those tell you when something happened. This tells you why. Run both.
Will it slow down my server?
Normal operation is one cheap sample a minute, mostly /proc reads, plus a light background sampler. Expensive commands only run during an incident, each with a timeout and an output limit, so the recorder can’t fill the disk it’s already waiting on.
Does it work without cPanel?
The per-minute collector and the analysis work on most Linux web servers. The deeper snapshots are tuned for cPanel-style stacks, and the request attribution expects cPanel-style domain log paths, though you can point it elsewhere.
Can I use its output to raise a ticket with my host?
That’s exactly why the disk latency and CPU steal numbers are recorded. Quote the timestamps and the measurements. It’s much harder to wave away than “my site feels slow”.
Is it safe to run during an outage?
That was the first design rule. Timeouts everywhere, bounded output, and package managers are detected but never run, because calling one during a disk stall could block indefinitely.
Have you hit the high-load-low-CPU problem? Tell me what it turned out to be on your server — I’m collecting these cases, and the wait-channel maps in the tool get better every time I see a new one.
During a pentest you might land on a Linux machine and make your way to root. Then comes cracking passwords, which is quite a challenging thing, unless you have good GPU. This can take a significant time with no promise it will work. Linux hashes, unlike NTLM hashes you have on Windows can’t be reused in lateral movement. Thankfully, there is a technique called PAM patch that can help you get those passwords when users log in via SSH.
Recon with Known Hosts
Before we start patching PAM, we need to understand the network a bit to see what hosts users connect to from this machine. SSH clients store previously accessed servers in a known_hosts file under each user’s .ssh directory. Those files have a list of hosts.
For example, /home/username/.ssh/known_hosts might have a git host. That can be a pivot point. If the compromised machine is in a restricted environment, that host may sit in another subnet or behind access controls you couldn’t otherwise reach. If the user has the same credentials on that host, it will be a pivot point.
Preparing the Host
Before implementing a credential capture mechanism, it’s important to ensure the host accepts password-based logins. SSHD can be configured to forbid password authentication entirely, relying solely on key-based access. To enable credential capture, the following must be set in /etc/ssh/sshd_config.
root > vim /etc/ssh/sshd_configPasswordAuthentication yes
Once this change is in place, the groundwork is set.
Creating a Logging Script
The next step is creating a small script that will record login attempts. With root privileges, create a new file at /usr/local/bin/logc.sh. We then make it executable and create a log file to store credentials.
Now we need to edit the PAM authentication mechanism. PAM configurations vary slightly between distributions, but for SSH specifically, you will need to edit is /etc/pam.d/sshd. For broader system-wide coverage, other files such as /etc/pam.d/common-auth (Debian/Ubuntu) or /etc/pam.d/password-auth (CentOS) could be patched instead.
Once restarted, test the patch by logging in with valid credentials.
Now check the log file.
root > cat /var/log/.authc.log
Each entry has the username, the password they entered, the host they connected from and the date of the attempt. Over time, this log will have a lot of credentials from user sessions, helping you a lot during a pentest.
Summary
There is a great method of harvesting SSH credentials on Linux by modifying the PAM. After finding potential lateral movement hosts, SSH is set up to allow password authentication. A custom logging script is created to save usernames and their passwords. Then the script is integrated into PAM by editing /etc/pam.d/sshd. Restarting SSH activates this change and future connections will be stored with usable credentials.
If you’re interested in red teaming and want to build the skills required to be a pentester, we offer our Red Team Operator training program.
Login
