Welcome to the Fast Lane Blog!

Take a look around. Let us know if you have any questions.

NVMe over FC: Chasing the SCSI I/O Bottleneck

Posted by Stephen Sellars on Feb 25, 2021 11:00:00 AM

In just about every SAN module and every SAN course I teach there are conversations about NVMe over FC. And I like to start those conversations with a quick story about chasing bottlenecks. Every computer system will have a bottleneck. One subsystem will reach max capacity before the others do and will become the limiting resource. As a student it is important to understand where the bottleneck exists. Architectural changes can be made to lessen the impact of the bottleneck. Advances in hardware or software can often be made to improve the overall system capacity, but then a completely new bottleneck can arrive based on the new system. That is what makes IT so challenging, but also why it is so interesting. We must keep up with technology’s advances and always remember that in our world, nothing is ever static.

Data storage is consistently getting faster, delivering improved economics for data centers of all sizes. Flash memory-based storage is a key technology that has significantly increased the performance of storage systems. Solid state disks (SSDs) are now so fast that the SCSI I/O interface has become the bottleneck. This is why the discussion about NVMe over FC is so important, and why students ask so many questions about it in my classes.

NVMe over Fibre Channel is a solution that is defined by two standards: NVMe-oF and FC-NVMe. NVMe-oF is a specification from the NVM Express organization that is transport agnostic, and FC-NVMe is an INCITS T11 standard. These two standards define how NVMe leverages Fibre Channel. NVMe over Fibre Channel was designed to be backward compatible with the existing Fibre Channel technology, supporting both the traditional SCSI protocol and the new NVMe protocol using the same hardware adapters, Fibre Channel switches and Enterprise AFAs. There is no need to ‘rip and replace’ the SAN infrastructure with NVMe over Fibre Channel.

The students in my classes that have already implemented NVMe over FC usually say they saw an increase in IOPS of at least two times by simply moving over to NVMe/FC from the traditional SCSI FCP. They speak of achieving 50% lower latency. Accomplish up to twice as much throughput improvement over traditional SCSI FCP when using NVMe/FC. There are lots of applications such as Oracle, SAP, Microsoft SQL Server and others that can immediately take advantage of NVMe/FC performance benefits. NVMe over Fibre Channel can also enable new SAN workloads. Big data analytics, Internet of Things (IoT) and AI/deep learning all benefit from the faster performance and lower latency of NVMe over FC.

An NVMe all-flash array connected to a storage network will help eliminate the SCSI I/O interface bottleneck and deliver more value back to the business. This scalable storage network will deliver reliable flash that performs at the speed of memory. It has already become a race to see which companies will take advantage of the benefits first. Critical applications will accelerate transactions and lead to better user experiences. Databases will increase the number of queries they support, leading to faster decisions and results. VM farms will be more efficient with higher VM densities per server, reducing infrastructure costs and simplifying IT. More types of workloads can be consolidated on hypervisors due to the improved storage performance.

I just read about one new solution comprising of VMware vSphere 7.0, Broadcom Gen 6 host bus adapter (HBA) and fabric, and NetApp AFF A800 all-flash storage array, a complete nondisruptive upgrade from SCSI to NVMe/FC. Broadcom’s Emulex FC HBAs and Brocade switches support NVMe/FC and SCSI Fibre Channel Protocol (FCP) simultaneously on the same port and fabric. This support simplifies the creation of multipath SANs that enterprises require. Further, Emulex Gen 7 FC HBAs are capable of round-trip latencies in the range of 10 microseconds while providing multimillion IOPS, making these HBAs an excellent fit for NVMe-oF solutions.

To learn more, check out some of Fast Lane Training’s SAN implementation & administration courses here!

Topics: VMware, ONTAP 9.7, SAN, NVMe, Oracle, SAP

Need help choosing a training path that's right for you?

Drop us a line here and one of our education services consultants will reach out!