Who me? Hold your nose, dear reader, as we dive headlong into the bucket of ice water that is the start of the work week. But fear not, because The Reg is here to warm your insides with a dose of Who, Me? – our weekly story about tech tricks gone wrong.
This week our hero is a reader we’ll regomize as ‘Emily’, who worked at a stock trading company as an applications engineer – but, as Emily points out, when she started there, the company was small enough that the engineering team needed to know how to have to do just about everything. So the job title is more or less symbolic.
The company where Emily worked acquired a worldwide unlimited license for its trading software, which meant it could run the code on as many servers as it wanted.
The trick was that each license had to be associated with a hostname and the hostnames had to be registered with the vendor. And while the license was unlimited, it was not perpetual. Incidental innovations were therefore necessary.
And that got messy when renewals were requested for the wrong hostname, causing emails to bounce back and forth to the vendor in an attempt to activate the right license on the right host.
The situation became so annoying that someone came up with the bright idea of assigning one name to all servers. And the name chosen was “server”.
That may sound crazy, but it’s worth emphasizing at this point that this was a stock trading company, so time was of the essence. A day when a server was down due to an expired license cost money. Although it seemed dangerous to call every server “server,” there was some logic at work.
The company’s technical team also believed that it could distinguish each box thanks to information in the internal DNS table. And the vendor didn’t care what its customers called their servers.
Emily joined soon after the shift to all hostnames being called “server”.
Part of her job was to refine the trading software. Do you know the old saying: “There’s time, money and quality, choose two”? Well, that’s pretty much how this software worked. On the one hand, time was of the essence mentioned above – so the software could be tuned to be very responsive. But that came at a price in terms of stability.
And we’ve already mentioned what happens when there is downtime.
So Emily was tasked with tweaking instances of the trading software, looking for the sweet spot between speed and stability. Usually she said to Who, me? she would have as many as 50 instances open at a time in the test environment – each instance easily recognizable to the trained observer, but all referred to as “server.”
And so during one of these tweaking sessions she made an adjustment and then restarted the server to test the change.
The silence that followed was chilling. A lot of yelling ensued, which confirmed Emily’s worst fear: the server she had just restarted was not in the test environment.
The CEO even came running in to find out what the heck had just happened to stop the trading. And here the story may not be quite a case of “Who, me?” – because the circle of angry traders around the red-faced application engineer on the IT desk has made it terribly clearly whose fault it all was.
What’s worse, Emily could do absolutely nothing to speed up the restart process. As she waited for the ‘ping’ to signal success (the longest few minutes she had experienced before or since), it seemed to all the world (and the CEO) as if she was doing absolutely nothing about the crisis she had created.
Fortunately, a lesson has been learned from all this. Emily’s next task was to ensure that each server’s prompt and background clearly identified it at a glance, including whether it was in production or testing.
We’re always looking for stories like Emily’s. Whether you got away with your joke or not, we want to know. Click here to send an email to Who, Me? and we can immortalize your pranks on a Monday morning. ®