Lightning Strikes Twice

Earlier this year, lightning knocked out our fancy-pants network. The response? Replace fried parts, increase electrostatic-disharge (ESD) protection. Problem solved? Not quite.

Last week a massive lightning storm hit the island. One strike was so big that everyone on the island thought it had landed outside their house: Every room across the settlement was filled with light.

On the mainland, my phone rang: ‘Hugh, it’s Brenden. She’d dead mate.’

Conduit running through palm forest

Clearly, we needed a new approach. The damage was severe. Our network is shaped like a tree: A trunk runs from our servers, east across the farm, out to Lagoon Beach, then north towards the Far Rocks. Along the way, branches poke out and serve delicious internets to customers.

Both lightning storms had cut the tree at the base of the trunk, where the connection is provided by point-to-point wireless radios. Every branch above the cut dies.

We took the opportunity to upgrade some of our servers. More cooling capacity and more oomph

We decided to go for the nuclear option: Replacing the wireless link with underground optical fibre. Fibre is great: It’s ultra-high bandwidth, immune to ESD, uses almost no power, and doesn’t corrode.

The link between the farm and our boatshed on Lagoon Beach is already fibre. This new link would mean that almost the entire network trunk would be underground fibre.

Dammit Brenden, that’s not where that goes!

Problem: Fibre is expensive and delicate. And unlike between house and boatshed, there was no underground conduit sitting idle ready to be filled with photons.

So we got to digging. And wrangling conduit. And crying a little when the cable tangled. In the end, we did it: The fibre is online and the old wireless link has been relegated to backup duties.

All that work for the tiny fibre in the lower right. Contrast with the (already thin) copper cables at left

Just in time for the summer peak season, our network has received a massive upgrade.

Taking Hits

Without the lows, you can’t appreciate the highs. Or so they say. Such wholesome thoughts were far from my mind yesterday when I got call from dad: Something was very wrong with our network.

A huge lightning storm had just crossed over the island. In its wake, it left our gizmos reeling. The Public Internet was down all along on the foreshore. Dad and Brenden barely had a connection on their devices on the farm.

I’m used to the system shitting its guts. Usually, however, I’m on the island when it happens and I’m the one causing the Rapid Unscheduled Disassemblies. This time it was nature messing about, and I wasn’t on the island to diagnose the problem.

Whatever hit us was big, but one part of the system had survived. The portion of the network serving the northern gardens was still operational. That proved critical in maintaining communication between dad and Brenden on the island, and myself in Sydney.

Dad and Brenden leapt into damage control mode. They’re seasoned fixers, having repeatedly ripped Matilda, our 40+ year old Land Rover, out from the jaws of death. Compared to a British-engineered 4×4, this techno-babble stuff is child’s play. I peppered them with questions, ‘is there anywhere the signal is strong? Are lights flashing on this and that gizmo?’ And so forth.

Meanwhile, in Sydney, I raced to restore remote access to the network. We use Virtual Private Network (VPN) technology to allow access from anywhere in the world, but for whatever reason, our VPN was also failing.

I had command line access, but I couldn’t access the various graphical interfaces that allow us to observe total system health. Instead of pretty graphs, I had lines of text coming through one at a time, each one painfully slow as it made the journey to Sydney via satellite. Getting information about the health of system components was like sucking a golf-ball through a garden hose.

Translation: It’s cooked mate.

Eventually, I got through, and was met with a sea of red lights. Combined with the information dad and Brenden passed through, the problem became apparent: Lightning had destroyed our main radio mast. Without that mast operational, our point-to-point links, fibre lines, and access points were dead in the water.

We had to find a way to get the mast back online. Customers had paid us for internet access, and we ourselves depended on the network in all sorts of ways. We couldn’t just order new parts: The isolation of Lord Howe Island is no joke. Either we fixed this with what we had on hand, or the network wasn’t going to wake up.

As we formulated a plan, dad remarked that the situation resembled that of the crew of the Enterprise trying to restore power after an attack by the Klingons.

In true Scotty form, we decided to rip parts off non-essential bits of the ship to get critical systems back on-line. Gear was pulled off the boatshed and mounted on the mast. Dad and Brenden raced back and forth testing that the boatshed radios weren’t also damaged by lightning. Then, we discovered that the problems ran deeper than the mast itself.

Despite the layers of Electro Static Discharge (ESD) protection we had built into the mast, the cable linking the mast to the server room appeared dead. We had to repurpose a secondary mast cable. That cable wasn’t set up to carry the 24V power required by the repurposed boatshed radio. While dad and Brenden worked to replace the gear, I played keyboard warrior, reconfiguring the power output in software.

Dad delivers a Trekkie response to news that the system was coming back online

Finally, we got a signal. One by one, lights on the dashboard started flicking from red to green.  We had power, but we had taken damage. The ProDive end of the Public Internet network was non-operational, its parts cannibalised to save the main mast.

Since launch we’ve served over 120 customers using the Public Internet, a take-up rate we didn’t dream was possible. On my last two trips to the island, I didn’t need to panic about any system failures, a state of being I hadn’t experienced in years.

After all that success, we were probably due to hit a speed bump. We’re already formulating plans to ensure this doesn’t happen again, including even more ESD shielding, and replacement of critical wireless links with more optical fibre.

When she got word of the situation, Jessica called me straight away from London. She was having none of my dejection. She said: ‘Hugh, it’s a blip in a grand scheme. A bump in a long road.’ She is right. We’ll recover from this, and come back stronger.