Doctor Digital MY BUSINESS STOPPED when the internet cable to Tasmania was cut. What can I do for my business so that never happens again?
Doctor Digital Says
There is nothing more heart-stopping than the moment where all of your business resources, communication, connectivity and capacity stop dead in its tracks and there is nothing you can do about it. That was the situation many Tasmanian businesses (organisations and people) found themselves in when both the Internet cables servicing the state were uncannily severed in two places on the same day. It's a one in a million moment of risk management - or is it?
While Tasmania was digitally cut off from the world, parts of Australia were inundated with water. A couple of years ago it was devastating fires. These businesses are having similar issues but different circumstances. As our businesses become more enmeshed and dependent on single sources of failure such as Internet connectivity, it is worth remembering that this is a modern evolution of risk and dependency, and it is something for businesses to plan for and manage as you would for natural disasters. You may want to take a look at Business Tasmania's Emergency Management for Natural Disasters Toolkit which has some helpful templates and guides.
Risk management and business continuity planning at its simplest is about fully understanding where your business is vulnerable, and strategically creating contingencies if one of those vulnerabilities happen. You have insurance for things like floods, fires, theft - this is risk planning to help fund your business if it needs to recover from any of these things. If you don't already have one, it's time to make a digital backup plan for your business for the next time there is a significant Internet outage.
Internet or power outages that will really disrupt business seldom have any lead time to prepare, so a backup plan for this type of event is assuming that the outage is happening and there may be no guidance from the carriers or utilities involved as to when services will be restored. As in the case with the March 2022 Tasmanian outage, as there was no Internet and in some places no phone service, it was very difficult to find out what was going on, as the usual sources of information - ie the Internet, social media, news feeds etc, were all down too.
Let's look at how your Internet can fail, some actions to take, how to create your plan, and mitigate as much of the risk as possible in a situation where there are many variables you are unable to control. But before that, some simple context about Tasmania's Internet.
Tasmania and Internet connectivity
National, and therefore, international network connectivity to Tasmania is currently available from two optical fibre cable service providers. One is Telstra, which operates two cables across Bass Strait, and the other is Basslink, which operates a single cable across Bass Strait. Bass Strait has been the traditional and only telecommunications route to Tasmania for cable (copper and fibre) which makes Tasmania an end point in the network and not a connection point (network node).
What this means is that we rely on network nodes to provide services that are hosted in a distributed manner across the global Internet network. So when our cables get damaged or cut, Tasmania's connection to the internet is effectively cut off. That is why banks, cloud servers like Amazon and Google, video conferencing, online streaming services and complex powerful network capability are not hosted here. When our network went down, so did our access.
For Tasmanian businesses, on-island internet access is either via NBN or via Mobile Carrier broadband that uses 4G and/or 5G services
- NBN offers fixed, wireless and satellite connections, depending on physical location – ie an address
- NBN fixed services are FTTP, FTTN and FTTC – fibre-to-the Premises/area Node/ or street Curb
- NBN fixed wireless is a tower to premises microwave radio link that also depends on a physical location - ie an address, usually regional or remote
- NBN satellite is a whole-of-Australia satellite coverage primarily for remote/regional premises
All of these only offer a single link without a separate backhaul connection (redundancy).
For business continuity planning, the issue is not the NBN connection to an individual premises, the issue is the whole-of-state capacity to offer an uninterrupted connection to the off-island Internet nodes. As we have seen from this incident, the likelihood of the connection being interrupted is relatively low but the impact is definitely serious.
Some points of failure
Before we get into the depths of backup Internet plans, let’s walk through the different ways the Internet can disconnect or fail completely. First steps if there is a significant outage are to establish that it is not a point of failure within your system, check your connections, turn devices on and off, check power, check your service provider for notifications of any issues and if possible contact other businesses to see if they are having an outage. Once you have established it is an external issue, you can narrow things down.
Generally speaking, there are four common faults that lead to broken Internet:
1. Internal ISP issue: These problems occur within the network of your Internet Service Provider (ISP), and are usually quick and easy to fix. You can expect a downtime of a couple of hours.
2. Major ISP issue: Sometimes faults within your ISP’s network aren’t straightforward. For example, they may experience the shutdown of critical hardware, or a disconnect in their operations centre or remote Exchange which can take time to find and fix. Downtime can be 24 to 48 hours or more.
3. Accounts issue: Your ISP will shut down your Internet connection if they think you haven’t paid your bill (oops). You will usually get lots of notice before things escalate to this point, and all services will resume immediately once the accounts dispute has been resolved.
4. Infrastructure failure: Critical infrastructure may be interrupted by extended power outages and physical disturbance (for example, if someone digging up some infrastructure hits a cable). Depending on the extent of the damage and the remoteness of access, there could be a couple of days of downtime.
What are your reliances, risks and vulnerabilities?
It’s a good idea to consider how an Internet failure would impact your business. What does your business depend on to deliver your product or service - banking, EFTPOS, online bookings, email, data storage, inventory management, logistics, CRMs, SMS or messaging for clients, website carts and fulfillment, payroll, invoicing?
If any of the above weren’t available for a couple of days, how would your business cope? Could staff still be productive? Could you still bring in revenue? Can you handle, process, store and invoice for cash? Can you realistically afford to risk an Internet outage for more than a couple of hours, how many, for how long?
Make a list of the dependencies your business has based on the suggestions above, and map the severity of impacts that it would have on your business - this will help you to prioritise how you manage risk, what you can financially tolerate, and what the cost of that will be so you can think about what the investment to mitigate it might look like.
One of the solutions to consider is having a redundancy or failover system. Redundant internet is a connection that kicks in automatically when your primary internet connection goes down, which means you never lose connectivity. It's like having a backup generator that can keep a home running after a storm knocks out the neighbourhood's power.
There are a few types of redundancies to consider when creating a back up plan, some you can do yourself, others that can be purchased from organisations that specialise in Redundancy as a Service (RaaS).
- Electronic redundancy: This is partial protection achieved by routing your internet traffic through a variety of alternate network paths via standby routers and switches.
- Physical redundancy: Another form of partial protection, it involves deploying two separate paths between two locations, with each one ending in a different piece of equipment - say having both satellite and fiber connections.
- Wireless option: This kind of redundancy employs 4G LTE wireless connections. If an outage happens, the network seamlessly shifts to a battery-backed wireless connection, where computers can connect to a hotspot.
- Using a different backup carrier: Some companies have a second internet service provider, or ISP, at the ready that can come online if the main connection is severed.
The solution you come up with for your business will be specific to where your greatest risks and dependencies are.
Creating a business contingency
Business Tasmania has created a suite of business contingency planning tools including a comprehensive template to complete to help you plan for future risks and how to recover from them. One of the main risks to businesses is that they fail to plan for issues such as significant Internet outage (less than 2% have active plans for this), which costs millions of dollars a year in lost revenue. Planning and communicating that plan to your team is an easy thing to put off in the daily busyness of business, but can be a costly omission when required.
Let's look at a simple scenario: Many businesses now have all their data in the cloud - because it's safe and secure, until you can't access it of course. For a services business, your database and contacts are critical. If your database is cloud-based, you may want to have a locally stored/desktop version so you can contact customers by phone when a lack of email means there is no way to email or message. Perhaps your new protocol will be to download details for the customers you are dealing with that day so you have a phone contact available, and you will make this policy and communicate it to your staff as part of your business contingency planning.
The Internet outage in Tasmania was a very unlikely confluence of adverse events on the mainland and Tasmania, but even if it was very unlikely, it happened and closed many businesses for over 5 hours of trading. Low risk doesn't mean no risk, and the strong message is that you need to work out your critical dependencies, plan for them to be unavailable for a period of time, and then invest in the mitigations you feel comfortable with to ensure you are able to continue to support your customers and staff until access and capacity is restored.
Being able to communicate to staff and customers what is happening is important, as is being able to transact if you don't have access to electronic funds or credit cards for a period of time. Focus on your time-critical issues and your revenue, while also remembering that sometimes there isn't an easy or plannable solution, and you simply have to be able to be agile and creative with an ever changing situation.