Buying a new storage array raises a difficult choice: iSCSI or Fibre Channel?
Technical and emotional factors come into play and it can be tough to sort through things like which protocol has better reliability, performance, cost, scalability and flexibility. In this article we'll assess reliability and performance of these technologies. In a later instalment, we'll discuss cost and scalability options.
Fibre Channel and iSCSI SAN history
Fibre Channel technology has been around since the mid-1980s and was a ratified standard in 1994. Fibre Channel is to SCSI what TCP is to IP. It is the transport mechanism used to pass SCSI commands between a host initiator and a storage target. Fibre Channel SAN storage popularity took off in the mid- to late-1990s with the dot.com information explosion. At the time, it was really the only viable way for a data center full of computers to access centralized storage. Fibre Channel SANs required a dedicated fiber optic storage network with storage-specific network switches and host bus adapter (HBA) cards installed in servers. From their onset, Fibre Channel SANs were considered to be expensive and complex.
The cost and complexity of Fibre Channel SANs sent industrious engineers to the drawing board. In 2003, the Internet Engineering Task Force (IETF) ratified iSCSI. This new protocol allowed the ubiquitous SCSI storage commands to be routed over an IP network instead of Fibre Channel. The goal was to lower the cost for centralized storage by consolidating the storage and IP networks. iSCSI commonly connects small or non-production servers to data storage, but don't let the cautious deployment tendencies fool you; iSCSI is in its prime. Major storage players, including EMC, IBM and NetApp offer iSCSI as front-line connectivity option to their arrays.
Fibre Channel SANs are generally more reliable than iSCSI SANs. Fibre Channel networks are relatively small and isolated networks, when compared to their TCP/IP cousins. This isolation means that the security risks, human-induced errors and hardware failures that may plague IP networks can't touch Fibre Channel SAN traffic. Mixing IP and storage traffic could mean that mission-critical storage traffic is routed right alongside an intern's web requests to myspace.com. Perhaps more important to reliability, FCP delivers fixed length frames in the order sent with robust error checking. iSCSI, on the other hand, rides on TCP/IP, which was designed to tolerate unreliable networks. TCP is a byte stream protocol that requires significant overhead to re-assemble out of order data and verify its integrity.
Fibre Channel and iSCSI SAN performance
Fibre Channel also has an advantage over iSCSI in the performance department. Ethernet has long had only 1 GbE as a suitable option for connecting iSCSI. Recently, 10 Gb Ethernet is getting a lot of press. However, it is pretty new and therefore expensive, canceling out a lot of the benefits that "cheaper iSCSI" provides. As of this writing, a 10 Gb Chelsio Communications iSCSI HBA goes for at least $US1,600 and 10 Gb Cisco Systems LAN ports can easily cost $US3,000 each.
Fibre Channel has several connectivity options which include 1 Gbps, 2 Gbps, 4 Gbps and 8 Gbps. Most reasonably priced departmental SAN switches today offer very dense, full line rate ports with non-blocking bandwidth to any other port on the switch. Fibre Channel also requires HBAs to be installed in the hosts to connect to the SAN. These HBAs offload Fibre Channel SAN network processing from the server CPU.
iSCSI HBAs can be implemented either with software drivers using your standard NIC or iSCSI HBA hardware. The software option taxes the server CPU and the iSCSI HBA options aren't much cheaper than Fibre Channel HBAs. If you don't use TOE cards, the iSCSI interfaces --common NICs -- are much cheaper, which have four times the bandwidth.
A final advantage is that Fibre Channel guarantees in-order delivery of data packets. This means that the unlike iSCSI adapters, Fibre Channel HBAs are not burdened with the sorting and reassembly of frames. Out-of-order delivery is a normal aspect of TCP/IP networks and iSCSI adapters have to pick up the slack. As I/O workloads increase, so does the amount of data that must be temporarily written to memory, sorted and then coped again for further processing. There are some interesting proposals like iSER, which promise to solve this iSCSI problem, but one has to wonder if it's too little too late for iSCSI now that Fibre Channel over Ethernet (FCoE) is building steam.