Taking Hits

Without the lows, you can’t appreciate the highs. Or so they say. Such wholesome thoughts were far from my mind yesterday when I got call from dad: Something was very wrong with our network.

A huge lightning storm had just crossed over the island. In its wake, it left our gizmos reeling. The Public Internet was down all along on the foreshore. Dad and Brenden barely had a connection on their devices on the farm.

I’m used to the system shitting its guts. Usually, however, I’m on the island when it happens and I’m the one causing the Rapid Unscheduled Disassemblies. This time it was nature messing about, and I wasn’t on the island to diagnose the problem.

Whatever hit us was big, but one part of the system had survived. The portion of the network serving the northern gardens was still operational. That proved critical in maintaining communication between dad and Brenden on the island, and myself in Sydney.

Dad and Brenden leapt into damage control mode. They’re seasoned fixers, having repeatedly ripped Matilda, our 40+ year old Land Rover, out from the jaws of death. Compared to a British-engineered 4×4, this techno-babble stuff is child’s play. I peppered them with questions, ‘is there anywhere the signal is strong? Are lights flashing on this and that gizmo?’ And so forth.

Meanwhile, in Sydney, I raced to restore remote access to the network. We use Virtual Private Network (VPN) technology to allow access from anywhere in the world, but for whatever reason, our VPN was also failing.

I had command line access, but I couldn’t access the various graphical interfaces that allow us to observe total system health. Instead of pretty graphs, I had lines of text coming through one at a time, each one painfully slow as it made the journey to Sydney via satellite. Getting information about the health of system components was like sucking a golf-ball through a garden hose.

Translation: It’s cooked mate.

Eventually, I got through, and was met with a sea of red lights. Combined with the information dad and Brenden passed through, the problem became apparent: Lightning had destroyed our main radio mast. Without that mast operational, our point-to-point links, fibre lines, and access points were dead in the water.

We had to find a way to get the mast back online. Customers had paid us for internet access, and we ourselves depended on the network in all sorts of ways. We couldn’t just order new parts: The isolation of Lord Howe Island is no joke. Either we fixed this with what we had on hand, or the network wasn’t going to wake up.

As we formulated a plan, dad remarked that the situation resembled that of the crew of the Enterprise trying to restore power after an attack by the Klingons.

In true Scotty form, we decided to rip parts off non-essential bits of the ship to get critical systems back on-line. Gear was pulled off the boatshed and mounted on the mast. Dad and Brenden raced back and forth testing that the boatshed radios weren’t also damaged by lightning. Then, we discovered that the problems ran deeper than the mast itself.

Despite the layers of Electro Static Discharge (ESD) protection we had built into the mast, the cable linking the mast to the server room appeared dead. We had to repurpose a secondary mast cable. That cable wasn’t set up to carry the 24V power required by the repurposed boatshed radio. While dad and Brenden worked to replace the gear, I played keyboard warrior, reconfiguring the power output in software.

Dad delivers a Trekkie response to news that the system was coming back online

Finally, we got a signal. One by one, lights on the dashboard started flicking from red to green.  We had power, but we had taken damage. The ProDive end of the Public Internet network was non-operational, its parts cannibalised to save the main mast.

Since launch we’ve served over 120 customers using the Public Internet, a take-up rate we didn’t dream was possible. On my last two trips to the island, I didn’t need to panic about any system failures, a state of being I hadn’t experienced in years.

After all that success, we were probably due to hit a speed bump. We’re already formulating plans to ensure this doesn’t happen again, including even more ESD shielding, and replacement of critical wireless links with more optical fibre.

When she got word of the situation, Jessica called me straight away from London. She was having none of my dejection. She said: ‘Hugh, it’s a blip in a grand scheme. A bump in a long road.’ She is right. We’ll recover from this, and come back stronger.