One common source of confusion amongst those new to the world of datacenter networking involves the use of the terms "layer 2 switching" and "layer 3 switching."   Some of you will no doubt be familiar with the seven layer OSI model, which divides computer networking up into seven conceptual layers, based on function.  For those of you who aren't, our discussion must begin with an overview of this model.  First, some terminology:

 

Host - An individual computer, connected to a network (or the Internet).

Network Interface Card - The ethernet card or wifi adapter on your computer.

Address - a unique identifier for another computer on a network.

 

The physical layer consists of the ethernet and fiber optic cables that link computers together, as well as the wireless signals in wifi networks; it also consists of the raw hardware that transmits and receives data on each end.  An example of a layer 1 device would be a hub; hubs, though now mostly replaced by switches, were at one time commonly used to connect multiple servers or workstations into a network; they are pure layer 1 devices in that they take an input signal and repeat and amplify it through all of the connected interfaces on the device.   This introduces performance problems; only one attached host can transmit data at any given time.  When two hosts try to transmit simultaneously, the result is a collision, and each host will then use an algorithm to wait a random amount of time before retransmitting (this is called 'back-off').   The more hosts you have connected to a hub, the more collisions that occur.  Since the time wasted on collisions, and on waiting to retransmit, is time not spent sending information, the result is that the network becomes slower and slower the more hosts you connect to it, until finally it reaches an manageable state.   

 

This layer is the most minimal logical layer, above the physical layer.  At layer 2, devices communicate via MAC address, a value that is generally hardcoded into the Network Interface Card.   Each host on a network has a unique MAC address; when one host wishes to send a message to another, it, at the lowest level, sends an Ethernet frame to that specific MAC address.   The classic layer 2 device is the common Ethernet switch.  A switch, instead of sending all received traffic out all of its interfaces (which is how hubs operate), will instead learn which hosts are behind specific interfaces, and send traffic addressed to those hosts through that interface only.   These devices are what are commonly known as "layer 2 switches."   They are extremely fast, but have one weakness, which we will explore next.

 

 

This layer deals in packets instead of frames, and IP addresses instead of MAC addresses.    Whereas MAC addresses are hardwired unique identifiers built into network interface cards, that are only accessible on a local network, Layer 3 IP addresses are globally routable addresses that allow communication between any two computers connected to the Internet, anywhere in the world.  When a computer needs to transmit IP traffic to a specific address, it has to look up the MAC address on its local network that corresponds to this IP address.  When you configure an IP address on a computer, you have to set three values: the address, the subnet mask, and the default gateway.   When your computer goes to send traffic, first, it determines whether or not the IP address is that of a host on the local network, or on a remote network.  It does this using the subnet mask; this value, commonly 255.255.255.0 on residential networks (allowing 256 IP addresses), but frequently other values on production or datacenter networks, helps the computer to demarcate the bounds of the IP network it is immediately connected to.  If the IP address it is sending traffic to is on the local network, it sends out a special broadcast ethernet message called an ARP (address resolution protocol) request.  This request tells all hosts on the local network that the sending host is trying to learn the MAC address associated with the given recipient IP address so it can send traffic to it.   Whichever host actually has that IP address configured on it will then respond with an affirmative answer, allowing the sending host to learn its MAC address.  The sending host then proceeds to transmit layer 2 ethernet traffic to that specific MAC address.    

 

This of course exposes the main problem with layer 2 switches.  Over time, on extremely large networks, the amount of broadcast traffic due to ARP requests and other related activity will become excessive.  This results in a condition not dissimilar to that of an oversaturated layer 1 hub; hosts connected to the network cannot send regular traffic as frequently as they would otherwise, because the network is blocked while the layer 2 switch handles the broadcast traffic.  This eventually reaches a point where the layer 2 network's performance is seriously impacted to an adverse degree.   Many network engineers prefer to not allow more than 256 hosts on the same layer 2 network, or broadcast domain.  256 hosts is of course a miniscule fraction of the billions of systems connected to the Internet.

 

Thus, to allow networks to scale, and to let the Internet happen, we have to do something called routing.  When you configure an IP address on your computer, the default gateway that you set is the IP address of your router.   This router allows your local host to talk to other hosts on the Internet that aren't on your local area network.   This works as follows: when your computer wants to send IP traffic to a host, recall that it first sees if the host is on its local network.  If it is on the local network, it then broadcasts an ARP request to all the hosts on the network to find the corresponding MAC address.  If its not on the local network, your computer will instead broadcast an ARP request, to obtain the MAC address of your router.  Your computer will then send the traffic to that specific MAC address.

When the router receives the incoming traffic, it will note that the destination IP address on it is not the same as its local address.   It will then forward the traffic to the next router that has a route to that specific address.  The way this usually works is that routers are situated so that one of their network interfaces is on a downstream local network, such as the wifi network within your house, and the other interface is configured on an upstream "wide area network", which is just another ethernet network that connects that router to upstream routers, and the rest of the Internet beyond.   Routers on end-user networks usually just have a default gateway assigned to them, just like the computers behind them.   Routers that connect to multiple upstream networks, like routers near the core of the internet, will use a dynamic routing protocol instead of a statically assigned default gateway; this routing protocol lets multiple routers communicate and exchange information on which router has the best route to any given network.  If one router should fail, the other routers communicating using this dynamic routing protocol can automatically bypass it and send traffic via different route. 

 

Routers have one disadvantage: they tend to be much slower than ethernet switches.   The processing required to do IP routing was historically implemented in software on a typical router.  Each router that separates any two IP addresses on the internet tends to introduce a performance penalty; the more hops in the route, the more latency in the connection, as a general rule.   The most commonly used dynamic routing protocol in the Internet backbone, BGP, consequently uses hop counts as its metric for selecting the best route between any two destinations.

 

So, with this behind us, we now know what layers 1, 2 and 3 of the OSI stack are, we know what a layer 2 switch is, and we know what a router is.

 

Basically, a layer 3 switch is a combination of a layer 2 switch and a router.  Layer 3 switches are fairly expensive machines; a high quality new layer 3 switch will generally cost upwards of $2,000.   Layer 3 switches use fast hardware rather than slow software to perform IP routing; they are designed so that the added overhead of routing is kept to a minimum.   Frequently, two hosts on different layer 2 networks, connected to the same high-end layer 3 switch, and using it as their router, will be able to communicate with no more latency than they would experience if they were on the same layer 2 switch.   Thus, layer 3 switches are hybrid devices let us solve the problem inherent in layer 2, that of excessive broadcast traffic slowing down the network, without introducing the increased latency and performance overhead of a conventional layer 3 router.  

 

It is oft-said that there is no such thing in life as a free lunch; this flexibility comes at an increased price, as the hardware required for layer 3 switching is more powerful and expensive than that required for mere layer 2 switching, and layer 3 switches have some limitations.  Most layer 3 switches cannot run the dynamic routing protocol that links together the backbone of the internet: BGP.  They can run faster, simpler protocols designed to handle smaller networks, but they are generally unable to run full BGP.  As a result, there is still a great demand for high-end routers, with specialized hardware, to run at the internet backbone and link together high-traffic IP networks.  Also, layer 3 switches are overkill for most end-user networks, like residential DSL connections or small business offices.  For these users, the overhead introduced by conventional routers is not a problem, as the router is still generally faster than the speed of the Internet uplink to which it is attached.  Layer 3 switches are therefore found most frequently in datacenters, where they connect large clusters of hundreds or thousands servers across high-traffic production networks, that need to rapidly exchange data and information as quickly as possible, with minimal overhead.   

 

At Corporate Colocation, we use layer 2 switches at our access layers.  Individual customers connect to ports on layer 2 switches on our network.  These in turn our uplinked to layer 3 switches, which act as the default gateway for the majority of our customers.  These layer 3 switches ensure fast traffic across all of the many thousands of servers in our datacenters, and let us flexibly allocate IP addresses to customers in small ranges.  A customer can get four IP addresses from us on one network in one of our datacenters, and eight IP addresses from us on another network in an adjacent datacenter, connecting one server to each network, and via our layer 3 switches, the two servers will be able to exchange data with practically no increase in speed or latency vs. if they were right next to each other. Finally, because the core of our network lies at boundaries of Internet backbone providers, we use very high end layer 3 routers, with full BGP support, to connect our customers to the outside world.  The layer 3 routers we use are effectively as fast, if not faster, than most layer 3 switches; they are special routers, however, and not switches, optimized specifically for handling BGP traffic linking together multiple networks at the very heart of the Internet.   They are also extremely expensive; whereas a low end software router can be purchased at any electronics store for around $100, and a layer 2 switch costs between $50 and $2,000, and a good layer 3 switch costs upwards of $2,000, a high end layer 3 router for Internet backbone applications typically starts at around $50,000 and can cost several million dollars, depending on its size, configuration and featureset.