Facebook Google Plus Twitter LinkedIn YouTube RSS Menu Search Resource - BlogResource - WebinarResource - ReportResource - Eventicons_066 icons_067icons_068icons_069icons_070

Tenable Blog

  • Twitter
  • Facebook
  • LinkedIn

Logs of Our Fathers

At USENIX in Anaheim, back in 2005, George Dyson treated us to a fantastic keynote speech about the early history of computing. You can catch a videotaped reprise of it here, on the TED site. I highly recommend it - there's lots of interesting and quirky stuff. I managed to talk him into giving me a copy of his powerpoint file, and subsequently tracked him down and am re-posting this material with his permission.

November, 1951

Machine Log #1

This is the original syslog. It's one of those thread-bound kids' class notebooks, and it contains the run-time log of John Von Neumann's programmable computer from Princeton's Institute for Advanced Study. For those of us who are used to dealing with system log data, it's not a whole lot different from what we're dealing with today.

"Machine ran fine, code isn't"

Von Neumann's system logs were, like today's logs, time-ordered and free-form. Date/time on the left hand side, then details. Then, as today, there are no "standards" in the reporting format. It's just information. When I think of logging standards like RFC 3195 I now wonder "how many notebooks would that take, with all the XML and useless overhead?"

The log message at the bottom of the sheet above reads "Machine ran fine, code isn't. Difficulty indescribable." There are a couple things here that are interesting. To this day, we refer to programs as "code" - if you've ever wondered where the term comes from, now you know. It is, literally, from the dawn of computing. Another thing this log entry shows is what may be the first case of finger-pointing between the hardware guys and the software guys. I think that every coder - at some point or other - hits a subtle problem they can't understand, and says "the compiler must be screwy!" or that there's a hardware problem. Here we see the origin of the tradition.

"This is now the 3rd different output"

Nowadays we can be pretty confident that, when we tell our computers to work on huge numbers, that they will mostly do it. There are stories, of course, of flaws in hardware. I remember back in the early 90s when there was a bug in the Intel -486 processor's math unit that made it sometimes give wrong answers. One of our old-school hardware engineers, when I worked at DEC, told me a funny story about a VAX system that had a floating point math accellerator card; the card wasn't seated in the backplane correctly and, as it slowly fell out from fan vibration, the least significant bits of the return values turned to zeroes because the contacts were no longer touching.

In Von Neumann's day, computers' competition was mechanical calculators - things like the Marchant Calculator that Richard Feynman tells stories about in his wonderful "Los Alamos From Below" talk - machines that were fairly slow. Some of the calculations Feynman's team completed for the Manhattan Project (specifically problems regarding expected bomb yield) sometimes took days to complete, with teams of calculator-grinders passing stacks of results from one station to another. Feynman used the word "program" to describe the order in which the sub-calculations of a larger calculation were performed.

How do you make sure the machine is right? Nowadays, we take it for granted thanks to the idea of regression testing: if you run a calculation on the same machine 5 times, you should get the same results - and if you run the same calculation on a different machine, you should still get the same results. With modern hardware, engineers can let the regression test run overnight and cross-test billions of different problems. With Von Neumann's machine, it was a different situation: there was only one computer and if it didn't give the same answer from run to run, it might take a week for a human to figure out what the right answer actually was. The log entry above: "this now is the 3rd different output - I know when I am licked" is some poor operator who had run the same program three times - and gotten three different answers. Ouch!

"I have now duplicated BOTH RESULTS"

Our poor operator from the previous slide is now trying to regression-test a result in which he got two different results. "Since there is no way to verify against PO#1 without doing it over again, I will invest that much time." ("PO" means "Program Output") How do you resolve that? You run it again and see which result you start getting over and over. Unfortunately, he got two different results repeatedly.

I've done syslog analysis of one sort or another since the late 1980s, and there have been a few times when I've found log entries that reek of desperation. This one tops them all. The operator must have been one truly frustrated human being.

A high speed run

Nowadays, overclockers have it easy. You can buy a motherboard that comes with software drivers that allow you to fiddle with the CPU clock speed, or memory bus speed, etc. In Von Neumann's lab, the "clock" was a mechanical device and the speed of calculation depended on physical properties of the machine: go too fast and contacts might not touch correctly, or a belt might fly off. One way of regression testing the system was to do a calculation with the clock speed on "low" and see if it gave the same results.

Bugs and Mice

"Mouse climbed into blower behind regulator rack..."

Grace Hopper's "bug" was a moth that got caught in the mechanisms of a Mark II computing machine in 1947. Von Neumann's lab had a "mouse." "Mouse climbed into the blower behind regulator rack, set blower to vibrating: result no more mouse and a !!! of a racket." This shows you the importance of timing on terminology - a mere 6 years' difference and Hawley and Englebart might have been making "bugs" for desktop input, while programmers were trying to thrash all the "mice" out of their code. Actually, the term "bug" is much older than Grace Hopper's usage - she was just joking. Thomas Edison was known to use "bug" in the 1870s; it may have been a term used by telegraphers.

"putting a tar-like substance.."

Can you imagine finding a syslog message like:
kernel: /dev/rst0a emitting tar-like substance
In case you can't read it, it says: "IBM machine putting a tar-like substance on cards."

V-belts smell like ... victory

I think "The smell of burning v-belts is in the air" speaks for itself; it certainly evokes a ruined afternoon.

When I was a newly-minted programmer/system administrator at Gould in 1987, we suffered an air conditioning failure on a friday night and I came in early on monday to discover my computer room was about 115 degrees F inside. The power supply on our big Gould PowerNode had wisely shut itself off, but one of our Sun-2 fileservers baked to death. After that, I got an air conditioning thermostat and wired it up across an RS-232 loopback cable, with a process that would look for that serial port coming ready and sending warnings into syslog. I'm glad I didn't have to figure out how to monitor a V-belt, though.

The Hell With It

"M/c off"

I can't think of any applications that leave syslog messages like this one.

"..high speed to duplicate and verify"

"code error, machine not guilty"

I have tremendous sympathy for the poor operators of the early machines. Someone might make a programming error that had you spending an hour running around checking for current levels in your system.

my computer runs like an old, grey, mare...

State of the Art SIM

Circa 1950, apparently, the way logs were aggregated was - well - they stack fairly neatly.

Von Neumann's computer ran until midnight, July 15, 1958. This was also reflected in the log: a final scrawled signature by the operator.

shutdown -h now


What's the point of all this? I suppose it could best be summarized by the French expression "Plus ca change, plus c'est la meme chose" - the more things change, the more they stay the same. Von Neumann's computer had a tiny fraction of the power and memory of the desktop supercomputers we're used to, nowadays. Our available storage is orders of magnitude greater - but the syslogs? There's really not much difference, is there?

Special thanks to The Archives of The Institue for Advanced Study, and George Dyson.

Related Articles

Are You Vulnerable to the Latest Exploits?

Enter your email to receive the latest cyber exposure alerts in your inbox.

Try for Free Buy Now
Tenable.io FREE FOR 30 DAYS

Enjoy full access to a modern, cloud-based vulnerability management platform that enables you to see and track all of your assets with unmatched accuracy. Sign up now.

Tenable.io BUY

Enjoy full access to a modern, cloud-based vulnerability management platform that enables you to see and track all of your assets with unmatched accuracy. Purchase your annual subscription today.

65 assets

Choose Your Subscription Option:

Buy Now
Try for Free Buy Now

Try Nessus Professional Free


Nessus® is the most comprehensive vulnerability scanner on the market today. Nessus Professional will help automate the vulnerability scanning process, save time in your compliance cycles and allow you to engage your IT team.

Buy Nessus Professional

Nessus® is the most comprehensive vulnerability scanner on the market today. Nessus Professional will help automate the vulnerability scanning process, save time in your compliance cycles and allow you to engage your IT team.

Buy a multi-year license and save. Add Advanced Support for access to phone, community and chat support 24 hours a day, 365 days a year. Full details here.

Try for Free Buy Now

Try Tenable.io Web Application Scanning


Enjoy full access to our latest web application scanning offering designed for modern applications as part of the Tenable.io platform. Safely scan your entire online portfolio for vulnerabilities with a high degree of accuracy without heavy manual effort or disruption to critical web applications. Sign up now.

Buy Tenable.io Web Application Scanning

Enjoy full access to a modern, cloud-based vulnerability management platform that enables you to see and track all of your assets with unmatched accuracy. Purchase your annual subscription today.



Buy Now

Try for Free Contact Sales

Try Tenable.io Container Security


Enjoy full access to the only container security offering integrated into a vulnerability management platform. Monitor container images for vulnerabilities, malware and policy violations. Integrate with continuous integration and continuous deployment (CI/CD) systems to support DevOps practices, strengthen security and support enterprise policy compliance.

Buy Tenable.io Container Security

Tenable.io Container Security seamlessly and securely enables DevOps processes by providing visibility into the security of container images – including vulnerabilities, malware and policy violations – through integration with the build process.

Try for Free Contact Sales

Try Tenable Lumin


Visualize and explore your Cyber Exposure, track risk reduction over time and benchmark against your peers with Tenable Lumin.

Buy Tenable Lumin

Contact a Sales Representative to see how Lumin can help you gain insight across your entire organization and manage cyber risk.