How Leaders Are Adapting Traditional Corporate Network Design To Remote Work

Most leaders have now learned the hard way that mandating an entire organization pivot to remote work requires more than an executive order and positive thinking. 

Hardware and software need to be reconfigured, meaning IT budgets have to be rethought (and no, not reduced...) while resource allocation for training - a traditional corporate afterthought - has become a necessity in order to keep productivity up. As with so many aspects of change, once the hortatory, chest-thumping rhetoric about embracing it has been forgotten, the reality is that if performance in a challenging new environment is to be optimized, attention must be paid to the boring but essential mechanics of how processes actually impact the people doing the work. JL

Jim Salter reports in ars technica:

In The Before Times, telework was just a fancy term for 'working from not in the office.' The most important thing a business should do to adapt to increasing telework is to plan for it coherently instead of buying something and hoping it works. The most obvious problem that businesses continue to face with a greatly multiplied number of remote workers is the size of the company's Internet connection. If you need a half, or three quarters of your workforce to remote in to work, you need enough bandwidth to accommodate them. Software, configurations, and expectations for users mean more change and more impact (on) budgets for IT support and training.We're 10 months into 2020, and businesses are still making adjustments to the new realities of large-scale telework (which, if you're not in the IT biz, is just a fancy term for "working from not in the office"). In the Before Times, telework was an interesting idea that tech companies were just starting to seriously flirt with as a normal way of doing business—whereas now, most businesses large or small have a hefty fraction of their workforce staying home to work. 

Unfortunately, making such a sweeping change to office workflow doesn't just disrupt policies and expectations—it requires important changes to the technical infrastructure as well. Six months ago, we talked about the changes the people who work from home frequently need to make to accommodate telework; today, we're going to look at the ongoing changes the businesses themselves need to make.

We’re going to need a bigger boat pipe

The most obvious problem that businesses have faced—and are continuing to face—with a greatly multiplied number of remote workers is the size of the company's Internet connection. If you need a quarter—or half, or three quarters—of your workforce to remote in to work every day, you need enough bandwidth to accommodate them.

(Remember, when working from home, most people connect back to the office via VPN. That means all of the bits you're moving around on your laptop while you're at work are transiting through your office's Internet connection via that VPN. So even though you're at home, you're almost certainly leaning on your work's Internet.)

Smaller businesses are generally facing the worst of this particular problem. In most places, small businesses are still using residential-style asymmetrical Internet connections, typically with a 10:1 upload/download bandwidth ratio. When almost all of your workers are in the office, a connection with 10 times the downstream bandwidth makes sense, for the same reasons it does at most people's homes—the majority of the content lives in the cloud, and the majority of the network throughput is downloaded, not uploaded.

This changes dramatically once you have a substantial fraction of the workplace working remotely. Now, the office itself—and its domain, file, and application servers—are "the cloud" from the perspective of your workforce, and while their home Internet connections still make sense—10:1 biased toward download—the office is badly out of whack. A 200 x 20Mbps connection looks a lot worse when you're bottlenecking on the 20Mbps side of it for a half-hour at a time.

The problem is even worse than it appears at first glance.

With roughly a third of the office's workers working from home, the download utilization is dwarfed by the upload—despite the upload side of the pipe being a tenth the width of the download.

For about a half-hour in the afternoon, the 20Mbps upload is completely saturated, which brings everything to a halt—even the underutilized download pipe feels constrained, because it takes longer to get HTTPS requests and DNS requests out through the saturated upload side, before the downloads themselves can begin.

The problem is even worse than it appears at first glance: the reason that most of the day seems to plateau at closer to 10Mbps than 20Mbps isn't lower demand from the workforce but, rather, lower supply from the ISP. During workdays, the ISP's supposedly "20Mbps" upload pipe tends to saturate at only 10-12Mbps.

The only real fix for this problem is upgrading to a symmetrical Internet connection—even a relatively basic 100Mbps fiber connection would offer five to 10 times better throughput than the coaxial-cable network connection this and many other businesses are currently limited to. However, such a connection is often considerably more expensive.

Absent (or in addition to) the fiber, retraining employees to remote control office PCs instead of moving files back and forth across the VPN is one of the best ways to conserve bandwidth. An employee actively using a full-screen RDP session will typically consume between 10MiB and 25MiB per hour connected and working; the same employee downloading even a small CAD project or a few images could consume five to 10 times that much bandwidth in just a few minutes.

Traffic shaping—i.e., prioritizing packets based on what protocol they're using, or where they're going—can also help somewhat. But this tends to be more band-aid than cure. If 10 employees are each trying to download a 100MiB asset over a 20Mbps connection, you're in for plenty of pain no matter how cleverly you've optimized your network flow.

The actual network equipment also frequently needs upgrades, as businesses shift their workforce more heavily to full-time remote work. More VPN connections mean more compute workload on the device providing those connections—and this tends to impact both smaller and larger businesses.

For a smaller business, this may mean upgrading from a high-end consumer router—such as Netgear's Nighthawk series, which for many years made a fine small-business gateway—to a more professional solution. Those solutions include options like pfSense or OPNSense running on x86 hardware. These commercial-grade but SMB-friendly solutions may be purchased directly from vendors like Netgate as turnkey appliances, or they can be downloaded and installed on generic industrial mini-PCs like the Qotom shown above.

Even relatively low-end x86 CPUs, such as an Intel Celeron J1900, are far more powerful than ARM-based offerings. They can handle more connections, more total bandwidth, and more VPN users. For businesses with more employees and larger bandwidth caps, it's easy—and still relatively inexpensive—to choose designs with Core i3, i5, i7, or even Xeon CPUs.

On the midsize-to-enterprise side of things, the answer tends to be simpler but more expensive: replace cheaper Cisco or Juniper routers and VPN endpoints with more expensive ones. Although this might easily mean dropping $5,000 on a single device instead of $500, the upgrade generally doesn't involve significant reconfiguration—existing configs can frequently be imported directly to new gear with little hassle. (SMBs who already had pfSense or OPNSense routers also have this option—it's easy to buy a bigger, meaner *Sense router and import your old configs.)

The most important thing a business should do to adapt to the increasing presence of telework is to plan for it coherently.

As usual, where the impact falls more heavily on small businesses isn't the equipment cost—it's the hassle and complexity. Many small businesses have no formal IT support, and they may have managed to install and configure their own consumer router—but most will not be able to handle the additional complexity of a commercial-grade router and will need to hire consultants to install and manage the newer, more capable gear.

The good news for small businesses is that the added complexity comes with additional benefits—commercial-grade router distributions also offer advanced VLAN management, traffic shaping, and prioritization. These technologies are usually considered must-have prerequisites for inexpensive VOIP phone systems, which can save business considerable money and offer greater flexibility.

VPN technology, policies, and user training

The last leg of the remote-work triangle isn't necessarily about the hardware—it's about software, configurations, and expectations for users themselves. As usual, this tends to mean more change and more impact for small businesses, which often have less availability and lower budgets for IT support and training.

Larger businesses, as we've discussed previously, generally have long-standing policies and training available for remote work—and although they may have upgraded their equipment, they probably haven't needed to make major changes that will impact the user experience. For a midsize business or enterprise, the biggest challenge here tends to involve a decision between globally routed VPNs versus split-tunnel VPNs.

A globally routed VPN requires a connected home user to send all traffic across the VPN, where it can be inspected by corporate network gear. The idea here is that this makes it more difficult for users to exfiltrate proprietary data—since all network data must flow through the company network before reaching the Internet. This setup also, at least in theory, makes it easier for the midsize business or enterprise to detect malware issues on remote users' PCs, since malware can't reach its command-and-control servers without also going through the corporate network.In practice, very few of the businesses implementing global-route VPNs are effectively analyzing data to recognize either malware or exfiltration problems in the first place. Without advanced real-time analysis—which requires expensive software—the global-route VPN accomplishes only two things, and neither of them is good. A global route massively increases bandwidth consumption over the company network, and it frustrates users who need to access local resources, like their home printers.

Moving back to the small business side of things, it's common to see much wider changes—and user confusion—surrounding the technology needed to get and stay connected. Small businesses which had gotten by on consumer routers will frequently need to change from IPSEC VPNs to OpenVPN or, in some cases, from IPSEC to proprietary SSL/TLS VPNs.Some of the more agile and leading-edge companies are also shifting from older solutions to WireGuard, a newer, faster VPN technology that can offer lower experienced latencies, faster connections, and better network roaming. We expect to start seeing more Nebula mesh adoptions soon, as that solution continues to mature.

Although these technology shifts are occurring for good reason and offer superior experiences to remote employees once adopted, the common pain point here is that they are shifts. Many employees don't handle them well. Confusion about "the old VPN" versus "the new VPN" is extremely common, and even relatively simple changes can frequently throw less IT-savvy employees for a loop. I've also seen employees get confused about the difference between their company's VPN client versus a partner company's VPN client—connecting to the partner's Cisco VPN will, of course, absolutely not grant you access to your own employer's office file server.

The confusion we're describing here might sound minor to someone who's never done professional IT support—but at any kind of scale, it adds up very quickly. A single, badly confused employee can require several hours to sort out—particularly when the support they actually need may very well be for a home computer, which hasn't been set up for corporate access and control in the first place.

We'll also note again that there are frequently different ways to accomplish the same task remotely, and they can have very different impacts on limited network resources. An architect or engineer working on a 40GiB Revit project by remote-controlling his or her office PC using Windows' built-in Remote Desktop will only consume a few MiB in an hour—whereas the same person, working on the same project, could consume a thousand times as much bandwidth if attempting to work directly on their laptop itself.

Conclusions

Although the current situation won't be with us forever—at least, not as impactfully as it has been this year remote work itself looks like it is here to stay. The progression from an office building full of employees to a network of home and otherwise remote workers was almost certainly inevitable—things may have accelerated from where they were, but we were going to arrive eventually.

Although businesses—and employees—are frequently still struggling to adapt to the new reality, it has its benefits as well as its challenges. For every employee who yearns for the days of water cooler chats, you can find three more who are reasonably happy to have the same chat over Zoom, with a cat in their lap or a dog at their feet.

The most important thing a business should do to adapt to the increasing presence of telework is to plan for it coherently instead of simply buying something and hoping it works. If the necessary expertise isn't available in-house and on salary, bringing it in on contract is a good idea. Nearly a year in to the pandemic, most full-time IT professionals and MSPs have significant experience with making this transition, and they can offer advice and strategy to C-levels based on real-world experience.