Way back in 2007, as the IEEE 802.3 Higher Speed Study Group was contemplating the next speed of Ethernet, it came to the realization that the bandwidth requirements of networking and computing applications were growing at different paces. The bandwidth requirements of networking applications were doubling approximately every 18 months, while the bandwidth capabilities of computing applications were doubling approximately every 24 months. The impact of this difference in bandwidth growth is illustrated in Figure 1. From these observations it was obvious that the mentality of a “one size fits all” solution space was no longer the optimal solution when determining the next speed of Ethernet.

Ultimately, the group chose 2 speeds. Thus the IEEE Std 802.3ba^TM-2010 introduced 40 Gigabit Ethernet (40 GbE) and 100 Gigabit Ethernet (100 GbE).

In addition, as part of this standard, several physical layer specifications, summarized in Figure 2, were introduced.

Figure 1. Bandwidth Projections

Figure 2. IEEE Std 802.3ba-2010 Physical Layer Specifications

While Figure 1 and Figure 2 are useful reference information, the implications from that information offer greater clarity to the challenges facing the industry. Figure 1, which helped to justify the need for 40 Gigabit Ethernet and 100 Gigabit Ethernet, clearly illustrates that when it comes to bandwidth, time is not on our side -- despite what the Rolling Stones sang.

The IEEE 802.3 Ethernet Bandwidth Ad hoc was formed to do an industry assessment on the future bandwidth needs of Ethernet. This ad hoc has already heard multiple presentations pointing out how bandwidth is growing throughout the entire Ethernet Eco-system.¹ This is also evident from Figure 2, which illustrates the number of physical layer specifications that were developed as part of IEEE Std 802.3ba-2010. This project offered the largest number of new physical layer specifications ever, as compared to prior Ethernet projects that targeted speed bumps. The number of physical layer specifications selected was seen as addressing the various computing and networking applications that drove the need for higher speeds. However, the sheer number of new physical layer specifications at the time of developing the standard is an indication of the industry expectations of Ethernet.

Looking at Figure 2, the reliance on technology used to support 10 Gigabit Ethernet is also evident. Examples include:

• 10GBASE-KR is the basis for 40GBASE-KR4, 40GBASE-CR4, and 100GBASE-CR10.
• 10 Gigabit VSCEL technology as the basis for the 40G / 100G MMF physical layer specifications.
• Implementations of the optical specifications leverage electrical interfaces based on “n” lanes of 10 Gigabit per second (for 40 GbE: XLAUI / XLPPI, for 100GbE: CAUI / CPPI).

While this technology lends itself to implementations supporting 40 Gigabit Ethernet and 100 Gigabit Ethernet, these implementations are not optimized in terms of cost, power, or density.

However, there is no free lunch, and sometimes one aspect of the solution must be sub-optimal in order to have a solution at all. This leads one to another observation regarding Figure 2: There is no backplane specification for 100 Gigabit Ethernet. The most obvious explanation is that a x10 solution for backplanes would not be considered a viable solution at all.

The Clock Is Ticking
Current implementations of I/O for 10 Gigabit Ethernet, 40 Gigabit Ethernet, and 100 Gigabit Ethernet also highlight the need for a 100 Gigabit Ethernet backplane solution. Using industry standard MSA form factors (SFP+, WSFP, CFP, and CSP), it is easy to conceive front panel capacities ranging anywhere from 400 Gigabit per second to 3.2 Terabit per second. Running line rate across these ports would necessitate a backplane to support anywhere from 3.2 terabit per second to 44.8 terabit per second.²

While developing the systems are always left to the implementers, when one considers different types of fabrics (A+B and N+1) to support such front panel capacities, the impact of 10 Gigabit per second signaling versus 25 Gigabit signaling becomes readily apparent, as the target capacities can not be addressed in a feasible manner. (See Figure 3.³)

Figure 3.

This issue was recognized by the industry, and in September 2011, the IEEE formed the IEEE P802.3bj 100 Gb/s Backplane and Copper Cable Task Force. The objectives for this project that the Task Force must address are:

• Support full-duplex operation only.
• Preserve the 802.3 / Ethernet frame format utilizing the 802.3 MAC.
• Preserve minimum and maximum FrameSize of current 802.3 standard.
• Support a BER of better than or equal to 10^-12 at the MAC/PLS service interface.
• Define a 4-lane 100 Gb/s backplane PHY for operation over links consistent with copper traces on “improved FR-4” (as defined by IEEE P802.3ap or better materials to be defined by the Task Force) with lengths up to at least 1m.
• Define a 4-lane 100 Gb/s PHY for operation over links consistent with copper twin-axial cables with lengths up to at least 5m.

The last 2 objectives are truly the key ones, as they describe that the physical layer specifications will be 4 lane solutions, implying that each of the individual lanes must support approximately 25 Gigabit per second. While a 2.5x increase in the bit rate is clearly a challenge, there are a number of tools that can be used to help solve the challenge -- but each comes at a cost that must be considered.

The Channel -- Improving the quality of the channel always helps to make life easier from a signaling perspective. The use of better materials or cabling, connectors, design, and fabrication techniques can help to reduce the amount of attenuation, improve the return loss, and lower crosstalk. These improvements in channel performance can simplify the required circuitry, and potentially reduce power and latency. Before just implementing these solutions, however, their costs must be considered. The other issue is the lack of an upgrade path for legacy systems in the field that may ultimately require a forklift upgrade.

Equalization -- There are a number of types and implementations of various types of equalization that may be employed at either the transmitter or receiver. There is a relationship between the equalization and the target channel. However, the more complex the equalization, the more one will need to worry about power, latency, and die size implementations.

Forward Error Correction (FEC) -- FEC can improve the overall performance of a link, but it comes at the expense of added cost and complexity. Furthermore, the interaction between any proposed FEC scheme with the IEEE Std 802.3ba^TM-2010 architecture must be considered.

Modulation -- The debate of NRZ versus PAM-4 signaling is back, but has also slightly changed. The debate isn’t one or the other, but should NRZ or both be done. It is anticipated that an NRZ scheme would support a channel insertion loss of approximately 25 to 30 dB insertion loss at 12.9 GHz without the use of any forward error correction (FEC), while a PAM-4 specification would target the support of channels with a channel loss of approximately 30dB insertion loss at approximately 7 GHz. NRZ is the incumbent electrical signaling for chip-to-chip, chip-to-module, and board-to-board applications, which also means familiarity and support from existing design and test tools. The proponents of PAM-4 point to the associated cost of NRZ channels and support to legacy 10GBASE-KR backplanes.

Timing Is Everything
The move to 25 gigabit per second electrical signaling is not limited to backplane and copper cables. There is significant work underway in the industry already, such as the OIF CEI-28G-VSR project, which is targeting 28 gigabit per second signaling for chip-to-module retimed applications. In addition, the Next Generation 100 Gb/s Optical Ethernet Study Group was formed in the IEEE 802.3 Working Group to look at the next generation of 100 GbE optical interfaces.⁴ Given that 100GBASE-LR4 and 100GBASE-ER4 are already designed to support 4 lambda’s of 25 Gb/s, it is assumed that the next generation of 100 GbE optical interfaces will be based on a 4x25 gigabit per second electrical interface. The move from a 10x10 electrical interface to a 4x25 based interface will help to reduce the cost, power and size of the associated modules. While reducing cost, power, and size is obviously a good thing, it will only exasperate the need for the 100 gigabit per second backplane and copper cable physical layer specifications.

It is anticipated that the new standard will be ratified in 2014, and will be integral into the design of next-generation networking equipment and blade servers. It would seem safe to assume that the technologies developed as part of this project would also be applicable to the next speed of Ethernet. Furthermore, referring to Figure 1 and the exponential growth in data, there will be increased pressure to solve these problems quickly.

Endnotes
1. For more information on the IEEE 802.3 Ethernet Bandwidth Ad Hoc, see http://www.ieee802.org/3/ad_hoc/bwa/public/index.html <http://www.ieee802.org/3/ad_hoc/bwa/public/index.html> . 
 2. 100GbE Electrical Backplane / Cu Cabling Call-For-Interest, see http://www.ieee802.org/3/100GCU/public/nov10/index.html <http://www.ieee802.org/3/100GCU/public/nov10/index.html> . 
3. 100GbE Electrical Backplane / Cu Cabling Call-For-Interest.  Used with permission from Elizabeth Kochuparambil, Cisco Systems. 
4. See http://www.ieee802.org/3/index.html <http://www.ieee802.org/3/index.html>  for further information.

John D’Ambrosia is Chair of IEEE P802.3bj 100 Gb/s Backplane and Copper Cable Task Force and Chief Ethernet Evangelist, CTO Office, Dell. To learn more about the IEEE P802.3bj 100 Gb/s Backplane and Copper Cable Task Force and details about participation, see http://www.ieee802.org/3/.

What’s your take on this subject? Leave a comment and get the conversation going.