The secret world of microwave networks
The secret world of microwave networks

            
  
  
  
    
      
    
  
  
    Would definitely melt a chocolate bar
  

    

    

    
      Has anyone spanned the Atlantic with floating microwave masts yet? We're not sure...
    

    

    
            
              
            
          

    
  



  

  
      
    
    
                      
                      
          
Stretching between London and Frankfurt there is a private, mysterious network that is twice as fast as the normal Internet. The connection, provided by a series of microwave dishes on masts, was once completely secret: only one very rich company was allowed to use it, and no one else knew about it.
A couple of years later though, a competitor completed its own microwave link between the two cities—and thus the first company, not wanting to lose out on potential business, revealed that it too had a link between the cities. If a competitor had never emerged, that first link would probably still be shrouded in secrecy today.
Similar stories can be found all over the world, but because these networks are privately owned, and because they're often used by financial groups trying to find an edge on the stock market and eke out a few extra billions, you have to dig deep to find them.
For example, back in 2013, just as the algorithmic high-frequency trading (HFT) bubble was starting to pop, researchers in California looked at a large corpus of market trades and found that, starting in March 2011, there had been a sudden latency drop of 2.5 milliseconds between Chicago and New York. Prior to that, the previous fastest trading latency was about 7.5 milliseconds, so a reduction to 5ms was significant. The researchers then used the FCC and other records to deduce that, at the time of their 2013 study, there were 15 (!) networks licensed to operate microwave links between the two cities. The latency drop was probably caused by a new low-latency microwave network coming online.

    
              
                A 1949 newspaper clipping showing AT&T's "just 34 hops" microwave link from New York to Chicago.
                              
            
                  
          
    
    
      A 1949 newspaper clipping showing AT&T's "just 34 hops" microwave link from New York to Chicago.

          
  
        
            

But what about the British weather?
Microwave networks themselves are incredibly old hat. Way back in 1949, New York and Chicago (712 miles apart as the eagle flies) were connected by a 34-hop line-of-sight microwave network operated by AT&T. In the UK, from the mid-1950s until the 80s, the nation's trunk communications network was fashioned out of microwave radio links; it carried everything from television and telephone to national defence data.

          
                      
                  
                    
        
          
    
    
          
          

    
              
                The UK's proposed microwave backbone in 1956.
                              
            
                  
          
    
    
      The UK's proposed microwave backbone in 1956.

          
  
        
            

Deploying a microwave network was cheaper than running cables, and the link capacity was greater than other copper-wire technologies at the time. Working against microwave networks, however, is the fact that their high-frequency signals (usually between 6GHz and 30GHz) are highly directional and require line of sight between each base station. Furthermore, while there is tons of bandwidth available at the top end of the microwave spectrum, high-frequency signals suffer from severe attenuation when travelling through rain, cloud, or just about anything that isn't clear air—an effect usually referred to as "rain fade" (older satellite TV users probably know all about that one).
Rain fade can be mitigated by boosting the transmission power, by using larger dishes and hydrophobic coatings, or a fail-over protocol where the link drops down to a lower frequency that can better penetrate the inclement atmosphere when necessary. Another option, starting in the 1980s, was to replace or augment microwave links with optical fibre, which was more reliable and provided a seemingly bottomless cup of bandwidth.
Fast forward to today and fibre has mostly superseded microwave for trunk network connections. Microwave networks are nevertheless still widely used, and the underlying technologies are being actively developed. Nowadays they're mostly used to connect geographically remote areas to the Internet—I have a friend in Norway whose town is on the wrong side of a mountain, so their connection is bounced via a microwave tower at its summit—or for specialist applications, such as private financial networks. Up until a few years ago, almost every cell tower was backhauled via microwave, but many of those links are being replaced with fibre to keep up with the throughput requirements of busy LTE and LTE-Advanced cells.

    
                A map of some of the microwave networks in southern England and continental Europe, mapped out by Alexandre Laumonier. The microwave links down to Cornwall are for the transatlantic submarine cables that land there. This map is by no means exhaustive; it was compiled by a third party.
                                  
                              
                  
          
    
    
      A map of some of the microwave networks in southern England and continental Europe, mapped out by Alexandre Laumonier. The microwave links down to Cornwall are for the transatlantic submarine cables that land there. This map is by no means exhaustive; it was compiled by a third party.

              
          Credit:

                      
          
          Alexandre Laumonier

                      
                  
          
  
        
            

How low can you go?
When the world is already blanketed in a dense mesh of high-speed fibre-optic cabling, the obvious question is: why use your own microwave network?

          
                  
                    
        
          
    
    
          
          
The first reason is somewhat obvious; if you have your own network connection, it's usually easier to guarantee things like security, quality of service, bandwidth, and other factors that businesses value highly.
The second reason, as we've already alluded to, is that microwave networks—somewhat surprisingly—can have lower latency than fibre. With some advanced networks, that latency is only a few microseconds slower than the speed of light. Fibre can be pretty quick over short stretches, but it soon starts lagging over longer distances, such as between two stock exchanges or a multinational's offices.
Fibre networks are hamstrung by the intertwining forces of money and geography. Laying a fibre network is incredibly expensive: you have to dig a trench that's hundreds (or thousands) of miles long, or lease access to ducts that have already been laid by infrastructure companies such as BT Openreach. You also have to respect the geography of the land; when faced with a mountain or river, do you go straight across at great expense, or do you make a diversion to the nearest bridge or tunnel? Combine these two factors and you'll understand why most of the world's terrestrial fibre networks slink alongside existing roads and railways—it's just the most sensible option.
Every time a network architect makes one of these sensible decisions, there's a small increase to the end-to-end latency. Add them up, and you end up with a few extra milliseconds—which is when the low-latency microwave networks swoop in to pick up some business.

    

    A map of terrestrial and submarine European backbone links, circa 2001. Finding modern maps of terrestrial cabling is tough—but for the most part, similar routes are used today—just with upgraded hardware.
                
  

Let's take London-Frankfurt as a real-world example. As the raven flies, the two cities are 396 miles apart. A radio signal travelling through air at just under the speed of light (299,700km per second) would cover that distance in 2.126 milliseconds. Through a glass or plastic fibre, where light has to bounce along the refractive index rather than travel in a straight line, the speed of light is reduced to around 200,000km/s, resulting in a theoretical minimum latency of 3.186 milliseconds.

          
                  
                    
        
          
    
    
          
          
In reality though, the fibre network between London and Frankfurt isn't just a single straight piece of fibre. For a start, depending on where the London server is, the packet of data might bounce around a few times until it gets to the right router for its journey across to Europe. Along the way, there are other routers and repeaters. And once the packet arrives in Frankfurt, it's the same deal as in London: the destination server is probably a few hops away.
Add geography and infrastructure to the mix, a submarine cable crossing (probably via Calais in France or Ostend in Belgium), plus the fact that a router in London might decide to send the packet via Paris instead, and the average latency between two servers in London and Frankfurt is actually closer to 17 milliseconds.
Now, at long last, for the punchline: a private microwave network between London and Frankfurt has a latency of about 4.2 milliseconds. I say "about," because the newest connections are still secret and their exact latencies are unknown. That's why businesses and financial institutions which absolutely must have the fastest connection opt for microwave links instead of fibre.
Building a point-to-point network
Microwave networks have two key advantages: radio signals travel through air about 50 percent faster than light moves down fibre, and you can (usually) build microwave links in a straight line between the two end points. The latter aspect means that the total physical distance travelled by a packet can be significantly reduced, plus you have the option of building the microwave network so that it actually terminates near the user, meaning packets have to traverse fewer routers.

    
              
                The BridgeWave FP80-3000, capable of up to 3Gbps over 80GHz. For scale, the antenna is 30cm across.
                              
            
                  
          
    
    
      The BridgeWave FP80-3000, capable of up to 3Gbps over 80GHz. For scale, the antenna is 30cm across.

          
  
        
            

Infrastructure-wise, microwave networks are essentially a bunch of masts with two transceivers at the top, each pointing in opposite directions—kinda like a glorified semaphore system, really. The distance between the masts will vary, depending on geography, but the average distance between masts is about 25 to 40 miles. The actual maximum distance is determined by each transceiver's height above ground level, radio frequency licensing restrictions, and the lay of the land; if you put a tall mast on top of a hill and have a powerful enough transmitter, you can comfortably achieve 50+ miles before you hit the radio horizon.

          
                  
                    
        
          
    
    
          
          
Cost-wise, you're looking at about £10,000 to £20,000 per transceiver for something like the BridgeWave unit (pictured), plus about £100,000-£200,000 for each mast itself (including access, construction, backup power, etc).
Going back to the London-Frankfurt example, the network probably consists of about 20 masts—so, somewhere between £2.5 and £5 million to physically build the network. That's not counting staff, ongoing operational costs (power, leases, support), and numerous other factors. The networking companies we spoke to for this story were loath to give us exact figures, but one London-Frankfurt microwave network operator gave us a ballpark figure of "between 10 and 20 million euros."


Finally, just to bring us full circle, it's interesting to consider how much it would cost to run a (theoretical) straight-line fibre network from London to Frankfurt. If BT Openreach had a 400-mile duct running between London and Frankfurt, it would cost about £1 per metre per year to lease space for a fibre-optic loop—and 400 miles is 644,000 metres. The price of fibre-optic cabling varies, depending on the number of cores and the shielding, but somewhere between 50p and £1 per metre is about right for humdrum terrestrial cable. And then, similar to microwave, you need signal amplifiers—which are expensive, but cheaper than microwave transceivers—every 20 or 30 miles.
Laying a new run of fibre is cheaper than rolling out your own private microwave network, then—but that only works if there's already a duct between the two points that you're trying to connect. If you have to build your own 400-mile duct... good luck!

Fast networks need fast software
Slicing a few milliseconds off the physical latency between a trader and the stock market is impressive, but it's meaningless unless you have some software that can capitalise on it.
For the most part, financial institutions use these microwave networks for algorithmic high-frequency trading (HFT), which is exactly what it sounds like: a computer algorithm that ingests stock market data, and quickly takes stock positions (sell, buy, long, short) before other traders can react. The first algorithms in the early 2000s battled human traders—but of course those humans soon stepped aside and developed their own algorithms. Nowadays it's just a grand ol' algorithmic slugfest.

          
                  
                    
        
          
    
    
          
          
The internal details of these algorithms are, as you can imagine, rather mysterious and highly proprietary, so no one will really talk about them in great detail—but we can talk about one interesting case where some algorithmic trading software went badly wrong in 2012.

    

    A graph showing the erratic behaviour of the Knight Capital algorithmic trading software. You can see that it was fixed by 10am. There's a dip at 9:52am where presumably someone intervened, but then the software kicked in again.
                    
                      Nanex
                  
          
  

On the morning of August 1, 2012, Knight Capital, which at the time one of the top HFT companies, rolled out a new piece of trading software. Starting at around 9:30am the new software went completely nuts, accidentally buying and selling shares worth $7 billion (£5.69 billion) in 45 minutes. The company did eventually unwind those trades at a cost of about $460 million (£374 million)—about half the company's value before the incident. Apparently traders still refer to the incident as "the Knightmare."
One proposed method of stopping incidents like the Knightmare, and generally to limit HFT from wading too far into ethically grey areas, is by leveraging blockchain tech. For example, front-running—sneaking in trades a few microseconds early based on advance knowledge of pending changes to the market—could be stymied if the market adopted a slightly slower, transactional, blockchain-style ledger.
Beyond microwaves
Over the last couple of years, some networking companies have also been experimenting with lasers (i.e. free space optics) rather than microwaves for point-to-point links.
The current crop of laser-based networks aren't necessarily much faster—latency can be reduced slightly, bandwidth is about the same)—and the super-high-frequency (~200,000GHz) signals are severely attenuated by rain and cloud. The attenuation can be somewhat counteracted with adaptive optics (where the receiver attempts to deform its shape to compensate for distortions in the signal), but you still need a fail-over link—a backup microwave, millimetre wave, or fibre cable—in case of bad weather.
AOptix, which initially developed ground-to-aircraft laser network links for the US military and then moved into terrestrial laser networks, closed down in February 2016. AOptix gear had been used in 2014 to build links between stock exchanges in the US, and in Europe, but it isn't clear whether laser tech has been widely adopted by other network builders since then.

          
                  
                    
        
          
    
    

        
        
          
          

    
    
      A video demonstrating AOptix's laser network gear.

          
  

Point-to-point millimetre wave networks, which use the block of spectrum above microwaves (~30GHz to 300GHz), are growing in popularity—but in many cases they're just a drop-in alternative with slightly higher throughput and worse bad-weather performance than microwave. The millimetre wave spectrum is also being mooted for next-gen (5G) cellular networks, and is already being used by a small number of 802.11ad (WiGig) devices, but those applications are very different from directional point-to-point networks
What's next for private networks?

    
              
                One of BT's many microwave towers, on top of a hill somewhere.
                              
            
                  
          
    
    
      One of BT's many microwave towers, on top of a hill somewhere.

          
  
        
            
While researching this story we pushed a number of network-building companies to tell us about their current or future projects, but almost without exception they kept shtum. There's no edge to be had if your secret network isn't secret, after all.
I was told about one particularly wild idea, though: building a microwave network across the Atlantic. Theoretically you would place the microwave transceivers on tethered barges (like a small oil rig, essentially), or alternatively use tethered balloons. Like a terrestrial microwave network, the main advantage would be a massive reduction in latency. Currently it takes about 25 milliseconds for a squirt of light to travel the 3,100 miles from the west coast of England to the east side of Long Island, New York; via microwave, it would be closer to 16 milliseconds.
Nine milliseconds might not sound like a lot, but if you then factor in a microwave network from London to Cornwall (which already exists), and Long Island to New York City (which also already exists), you could get the complete London-NYC latency down to about 25 milliseconds—a damn sight faster than the current by-fibre latency of about 60 milliseconds.
But, alas, as you've probably guessed, no one has yet come forward to finance the construction of 80 floating microwave masts that span the Atlantic. That we know about, anyway...


          
                  

                  
          


  




  
  
          
  

  
    

    
      Sebastian is the editor of Ars Technica UK. He usually writes about low-level hardware, software, and transport, but it is emerging science and the future of technology that really get him excited.
    
  



  
    
    
    139 Comments
  
      





  
              
  


  
    
      
      
    
    
              
                      
              
            
                    
        
                    
                    
        
                    
                    
        
                    
                    
        
                    
                    
        
                  



  

  