THE CASE FOR CO-PACKAGED OPTICS:  LOWER POWER

As network traffic continues to grow due to higher bandwidth applications, such as AI/ML (Artificial Intelligence/Machine Learning), high-resolution video streaming and virtual reality, the strain placed upon data center networks continues to increase.  These insatiable demands for more bandwidth are resulting in higher speed and higher density optics and ASICs.  The aggregate throughput of switch and optics devices is doubling every two to three years, which ultimately results in doubling the speed and increases in power of the host to pluggable optics electrical interface.  Unfortunately, while Moore's Law continues to allow us to scale our chips and equipment (despite non-trivial technical challenges), its corollary, Dennard Scaling, has been dead for a while.  This means the power required for the new generation of switch and optics semiconductors is higher than the previous one.

For Cisco's Webscale data center customers, this has many implications.  To continue scaling a typical data center built around a fixed electrical power budget, both compute and networking must keep up with these new bandwidth demands, but within the same power envelope as before or face an expensive upgrade.

In the compute space, the requirement to remain within a fixed power budget has forced changes:

• Movement from single core to lower frequency multicore CPUs (central processing units)
• Movement away from general purpose CPUs to focused accelerators GPUs (graphics processing units) for applications such as AI/ML inference and training.

In the networking space, data center topology compromises must occur to remain within the required power envelope, and we must reconsider how we design our equipment.

• Take a "clean sheet" architectural approach. Cisco's Silicon One achieves significant power efficiency improvements by rethinking how networking silicon is built.
• Use the latest silicon process technology to optimize the design
• Innovate thermal design to reduce the power needed to cool the system
• Holistically design our silicon, optics, and systems to optimize power consumption and thermal cooling

However, in our continued quest for innovative designs, we needed to continue to innovate. This is where co-packaged optics (CPO) come in.

THE THREE PILLARS OF CO-PACKAGING OPTICS

Pillar#1 -Removal of a Level of DSPs to Save Power

As switch system speeds and densities have increased, so has the percentage of the system power consumed by front panel pluggable optics. At 25G/lane and faster speeds, the necessity of active DSP-based retimers has driven up system power.

One of the key innovations of co-packaged optics is to move the optics close enough to the Switch ASIC die to allow removal of this additional DSP (see Figure 1).

Pillar#2 -The Remote Light Source

In traditional pluggable optics, all sub-components of the optics reside in the pluggable modules. But as optics move closer to the ASIC, the partitioning of the optical components is a critical decision.  Lasers are highly sensitive to high temperature and experience increased failure rates when placed in hotter environments (e.g. adjacent to a very hot switch ASIC).  Moving the lasers away from the high power ASIC to cooler locations within the system chassis results in several improvements:

1. Lasers can be passively cooled to a lower temperature, enabling them to be more efficient in generating optical power / Watt, lowering system power without active components like a TEC (thermo-electric cooler).
2. Lasers can be replaced from the chassis faceplate. Since the lasers are the least reliable components of the optics subsystem, making the light source accessible from the system front panel to allow easy insertion and removal is important to ensuring CPO systems have similar MTBF (mean time between failure) to legacy systems.
3. The industry can standardize on the form factor and design of the remote light source, which allows for multi-vendor sourcing. [Industry standard MSA for ELSFP (External Laser Small Form Factor Pluggable)] Cisco's demo at OFC is the first system demo to use the industry standard form factor.

Pillar#3 -Production-Proven Silicon Photonics Platform

To place optical components very close to the Switch ASIC silicon die, two orders of magnitude (over 100x) of miniaturization is required over existing pluggable modules.  To do this, many previously separate ICs (TIA, driver, modulator, mux/demux) must be combined together on a single IC.  Cisco's Silicon Photonics technology enables this.

Figure 2 -4x OSFP800 vs. Cisco 3.2T CPO module (>100x volume reduction)

In this era of supply chain challenges, it is important to choose a partner with proven, reliable technology. One of Cisco's advantages in the CPO space is the experience developing, optimizing, and shipping millions of Silicon Photonics-based optical modules.

Figure 3 -Cisco 106.25Gbps/lane Silicon Photonics SISCAP modulator [1] (a) First generation siscap configuration; (b) driving siscap; (c) measured pam4 53gbaud transmit waveform (106.25Gbps) from second generation siscap and driver

CPO SYSTEM BENEFITS SUMMARY

As a result of these innovations, the power required for connecting the Switch ASIC to front panel pluggable optics can be reduced by up to 50%, resulting in a total fixed system power reduction of up to 25-30%.

Figure 4 -51.2T system power reduction from pluggable to CPO

 

CISCO'S OFC DEMONSTRATION OF CO-PACKAGED OPTICS (CPO)

At OFC 2023, Cisco is proud to demonstrate these next steps -a side-by-side comparison of the real power reduction between:

• Cisco 8111-32EH, a conventional 32-port 2x400G 1RU router fully populated with 2x400G-FR4 pluggable optics modules (64x400G FR4) based on the Cisco Silicon One G100 ASIC
• Cisco CPO router populated with a full complement of co-packaged Silicon Photonics-based optical tiles driving 64x400G FR4 also based on the Cisco Silicon One G100 ASIC with CPO substrate

Figure 5- Cisco's OFC 2023 CPO Demo System (in 128x400G FR4 chassis configuration)

KEY ADVANTAGES

Cisco's CPO demonstration at OFC 2023 highlights some of the key advantages of Cisco's technology development.

Integrated Silicon Photonics Mux/Demux for 400G FR4

One of the challenges of co-packaging optics is the requirement to miniaturize the optical components to fit on an ASIC package (over 100x lower volume than a conventional QSFP-DD or OSFP module). This requires optics and packaging innovation.

Any CPO architecture must provide the flexibility to support all data center optics types, including those using parallel single mode fiber, e.g. 4x100G DR4, and CWDM (coarse wave division multiplexing) e.g. 400G FR4.

400G FR4 uses 4 different wavelengths of light on the same fiber, each carrying 100Gbps. This means 4 different wavelengths need to be combined together.  This is often done using an external lens, which takes up significant volume.

Cisco has invented an innovative way to do this mux/demux on the Silicon Photonics IC, which we are demonstrating as part of the OFC demo.

Figure 6 -400G FR4 module block diagram (highlighting mux/demux) [2]

Multiple modules running at once

Integration of optical tiles on the Switch ASIC package requires innovation in package mechanical design (to ensure mechanical reliability), power delivery (to deliver current to both the switch ASIC and the optical tiles in a small area), and thermal cooling (to remove the higher power density).

Cisco's demo has a full complement of optical tiles drawing their full power.

Enhanced thermal design to permit conventional air cooling

Another of the challenges of integrating optics with switch ASICs is while total system power decreases, the thermal density in the center of the system grows, because optics move from the front panel to the ASIC package.

Other vendors use liquid cooling to manage this higher thermal density.

Cisco has worked with key partners to develop advanced heat sink technologies, which allow conventional, reliable air cooling to continue to be used instead of forcing customers to change their infrastructure to support liquid cooling before they want to.

CISCO IS THE RIGHT CHOICE FOR OF CO-PACKAGED OPTICS (CPO)

Cisco recognizes the potential benefits of CPO technology and is investing to ensure we are ready for this inevitable transition.

However, CPO poses extremely complex problems that system vendors and network operators must solve before significant deployments can begin.  For example, it must be reliable, serviceable, deployable, offer significant power savings, and be cost-effective.  This is the reason for our demo at OFC.  Cisco expects trial deployments coincident with the 51.2Tb switch cycle followed by larger scale adoption during the 101.2Tb switch cycle.

We believe it's not a matter of if co-packaged optics will occur, it is a matter of when. Cisco has expertise in systems, ASICs, and optics, which makes it one of the few companies that can successfully implement and deploy co-packaged optics in volume.  We remain dedicated to investing for this inevitable transition, but realistic that it may be still some ways away.

You can visit our Cisco Silicon One website to learn more about our industry-leading routing and switching silicon architecture. You can visit our Cisco Optics website to learn more about our pluggable optics offerings and Silicon Photonics technology.

---

REFERENCES:

[1] Traverso et al, Integrated Silicon Photonics Transmitters, IEEE Xplore, August 17, 2022

[2] Welch et al, Towards 400G BASE-FR4 Baseline Proposal, IEEE March 2019