Why I built it
Every company laptop was supposed to run a monitoring agent. On most of them it did not. It crashed on Windows, failed at startup on every recent Ubuntu release, and had been that way for three years.
They walked me through the agent in my first week and I understood almost none of it. I had never used Linux, and the words people used about the thing were not words I knew. The requirements amounted to "make it work". The company gave me a Windows laptop and an Ubuntu laptop, so I installed the agent on both and used it until it failed, writing down each failure. To see the system from the user's perspective, I made fresh user accounts to test under, and I asked colleagues to run it on theirs. Two weeks later I had a list of more than fifteen bugs and vulnerabilities, and that was what I worked from.
Halfway through the summer I picked up a second project on the business side. Sales procurement ran on email and paper across six departments, and one cycle took over a week.
What it does
I built three things over the three months: a monitoring agent that now runs on 400 Windows and Ubuntu laptops, a release pipeline that pushes an update to all of them in about five minutes, and an enterprise resource planning system for sales procurement used by more than 300 people.
I led the first two and was primary developer on the third, on a team of five.
How it works
How an update reaches 400 laptops
- 1
I build a new release on a dashboard on my machine.
One click builds the Windows version and the Linux version of the agent.
FastAPIC++
- 2
The dashboard signs the release and publishes it.
It goes to the company server, behind an nginx reverse proxy. Only someone with the private signing key can publish.
SHA-256nginx
- 3
The agent on every laptop checks the server for new releases.
300 Windows laptops and 100 Ubuntu laptops.
C++WindowsUbuntu
- 4
Each laptop verifies the release before installing it.
It checks the signature and the SHA-256 checksum. If either fails, nothing is installed.
signing keySHA-256
- 5
The updated agent sends its telemetry back.
The fields the company asked for, including the active browser URL.
CassandraMongoDB
On Windows I kept the parts of the existing agent that worked and replaced the parts behind the crashes. The Linux side I wrote from scratch. I hand-wrote every package the agent needed, so it compiles to one binary with no outside dependencies. It reads the active browser URL through Chrome's accessibility tree. Before it went anywhere near the fleet I ran it on seven pilot machines and watched the logs come back through ngrok.
I followed the team's existing workflow. The code was on GitLab. The developer who handed me the project reviewed each change before Jenkins built and shipped it, and the tests checked that every telemetry field came back with the right type.
The ERP uses Python and FastAPI with a PostgreSQL database and a React front end. Each person logs in with a role and sees only the pages that role allows. Jenkins and GitLab handle the deploys.
The hard parts
Three years of crashes, and nobody knew why
The agent crashed on most Windows machines without logging anything useful. The build pulled in a 300MB package, and on a laptop with ordinary memory the process ran out of room and the OS killed it.
I stopped depending on that package, hand-wrote the core logic around the parts of the agent that worked, and compiled to a binary. 300MB became 20MB, and every Windows machine where it had been crashing came up and stayed up. Finding the cause took two weeks of reading the existing code.
The Linux build was written against X11
With Windows working I turned to the hundred Ubuntu machines, where the agent did not start at all. The original targeted X11, and current Ubuntu ships Wayland, so those calls failed at startup.
I rewrote the Linux side too. It came out under 1MB and reached all 100 machines.
When the agent stopped, nothing started it again
The agent now worked, and it could still stop on a machine, from memory exhaustion or from something outside the process ending it. Nothing restarted it, so that laptop reported nothing until someone noticed, which biases the data toward the machines where it kept running.
Nobody asked me to fix this. I built a watchdog that checks process health and restarts the agent within seconds of it stopping. If both processes go down at once, the agent writes one last telemetry log inside the grace period the OS gives during shutdown, recording why it stopped so the shutdown can be debugged later. Windows Task Scheduler is the last fallback. Downtime incidents dropped 60%.
Logs were lost when a laptop went offline
Testing turned up a case I had not covered. With the laptop off the network, telemetry had nowhere to go, and the logs for that stretch were lost. I went back and gave the agent a local store, so it writes logs to the laptop and flushes them once the network is back.
That left a file sitting on the machine between the writing and the sending, and the person using the laptop could edit it in that window. The agent now records when the file was first written and checks it against the file's last edited time at flush. When the two disagree, the log is flagged as tampered with.
Six department heads described the workflow six different ways
I had no business background, so I sat down with the head of every department and asked how sales procurement actually worked. They gave me six answers. The heads of customer solutions and of sales gave me different sequences for what happens between a customer booking a demo and devices being installed.
I had no way to judge which was right, so I built a dependency graph of all the requirements and worked out which steps were physically impossible until another step finished. That ruled out most of the orderings and left two that both worked. I built both, put them side by side, and let the two heads argue it out in front of a running demo until they agreed.
The cycle went from over a week to two or three days.