A Storage Area Network (SAN) is a high-speed storage network within a LAN. It enables groups of servers to access data in the form of blocks from storage devices. All servers in a SAN have permissions to access the storage resources. A SAN provides concurrent server access on LANs or WANs to all the storage devices. A SAN ensures scalability, reliability, security, low network traffic, improved data availability, and transparent backups and restores.
This ReferencePoint describes the SAN components and the design considerations in implementing a SAN. It also describes the SAN topology and protocols. Finally, it explains the SAN implementation process.
SAN Components
The following devices and components constitute a SAN:
-
Servers
-
Storage Devices
-
Interfaces
-
Interconnects
-
Applications
The SAN architecture depends on the User Interfaces (UIs), interconnects, and fabrics used on the network. This architecture governs the performance of the SAN.
SAN Servers
A SAN supports a heterogeneous server environment and operates on several server platforms, such as UNIX and its versions, Windows NT, Windows 2000, various versions of Novell NetWare, Linux, HP-UX, and Sun Solaris. One of the limitations of the heterogeneous environment is the inability of the servers to share data among themselves because the file systems are different. This limitation is overcome by using data conversion applications that convert data from one file system to another.
SAN Storage Devices
A typical storage device in a SAN stores large amounts of data to avoid the data scattering over multiple storage devices. SAN storage devices are stored outside the primary network, enabling even distribution of storage devices across networks. The following storage devices are used in a SAN:
-
Just a Bunch of Disks (JBOD)
-
Disk arrays
-
Optical storage libraries
-
Tape libraries
-
Disk storage systems
SAN Interfaces
describes several common SAN interfaces:
SAN Interconnects
SAN interconnects connect the interface topologies with the Fibre Channel fabrics. Several of the SAN interconnects, such as cables, adapters, connectors, hubs, routers, multiplexors, and bridges, together form a Fibre Channel fabric.
Fibre and copper cables are the two types of cables that connect SAN components with storage devices. Copper cables are used for distances of up to 30 meters. The two types of fiber cables follow:
-
Multimode fiber cables: Sets the connection over distances ranging up to 2.56 kilometers.
-
Singlemode fiber cables: Sets the connection over distances ranging up to 68.8 kilometers.
Connectors are used to connect fiber-optic devices with copper devices in a SAN network for data transmission. Media Interface Adapters (MIAs) convert fiber-based to copper-based connections.
Adapters act as an interface between SAN interfaces and host buses to facilitate proper communication between them. Host Bus Adapters (HBAs) and Network Interface Cards (NICs) are examples of adapters, which communicate using TCP/IP, FICON, and ESCON protocols.
A hub is a centralized device in a SAN network that connects various SAN devices. A fiber hub can connect up to 120 nodes. Every port on a hub has a Port Bypass Circuit (PBC) that prevents a failed device from affecting other working devices.
Bridges provide communication across networks using various protocols. They help in connecting SCSI and Fibre Channel interfaces. Multiplexors are bridges that enable the transmission of signals from multiple devices through a single transmission media.
Redundant Array of Inexpensive Disks (RAID) provides an array of storage devices that are fault-tolerant, creating a large virtual drive that reduces the need for multiple drives. Routers transfer data across networks using the addressing schemes and transmission media. For example, storage routers use the Fibre Channel Protocol (FCP) to transfer data across networks. Gateways connect individual LANs over WANs. Switches connect SAN devices with each other but unlike hubs, they do not share the network bandwidth. They pass signals to specific ports and reduce the network traffic. Figure 3-1-1 shows the relationship between various SAN interconnects:
SAN Application Components
SAN application components include the following:
-
Applications to configure and manage the SAN fabric using management software such as:
-
Tivoli SANergy from IBM: Allows simultaneous sharing of the same storage, file systems, and the same files among multiple computers connected to a SAN.
-
Tivoli Storage Network Manager from IBM: Discovers, displays, allocates, monitors, automates, and manages various components of SAN Fabric and disk storage resources.
-
SmartClient from BakBone: Allows network administrators to centrally control the attached media devices.
-
SANworks Enterprise Network Storage Manager and SANworks Storage Resource Manager from Hewlett Packard: Reduces the load on the network server and creates procedures and functions to perform specific network storage and management-related tasks.
-
-
Applications to make regular backups and perform restore operations, volume managers for remote mirroring, disk striping, data replication, and other network management software. Examples of these management applications are the Tivoli Storage Manager from IBM and NetVault Dynamically Shared Drives (DSDs).
-
Applications to share files, folders, and file systems. These applications use zoning and Logical Unit Number (LUN) for masking. For example, the SV Zone Manager from Viacom and SANPoint from Veritas provide centralized management and storage network administration.
SAN Design Methodology
Efficiency of any network depends on the design of the existing network, the components, and the storage systems used. To design and optimize a successful SAN setup, you need to perform the following steps:
-
Assess
-
Plan
-
Implement
-
Manage
In the Assessing step, information is gathered about the existing network setup and the requirements for designing a SAN. Site auditing collects all the required information to set up a SAN. It is possible to either change the SAN design to adapt to the existing LAN structure, or the existing LAN structure can be changed to accommodate a SAN.
Site audit also helps in sorting out various business and technical issues and helps check SANs compatibility with other systems on the network. It helps in planning decisions and future tasks for designing a SAN. For example, when choosing storage devices for a SAN, their compatibility with the existing network systems is considered. These phases help decide the topology of the SAN setup.
In the Planning step, a decision is made about the devices, network topologies, and protocols to be used for the SAN design and implementation. This step also includes specifying business and technical goals, the quality of service, and the design infrastructure for successful SAN setup. A complete SAN document is created that specifies all the information pertaining to design.
The Implementing step includes the actual setting up of a SAN by going through the detailed document prepared during the planning step.
The Managing step administers a successful SAN implementation. The step performs all the SAN management and administration tasks that help a SAN function.
SAN Design Considerations
A SAN design is customized based on the needs, requirements, and limitations of an organization. While performing SAN customization, you need to evaluate the following:
-
Business requirements
-
Storage pooling
-
Data availability
-
Heterogeneity
-
Network performance
The priority for implementing any of these considerations varies with different organizations.
SAN Business Requirements
A SAN can enhance data storage and a retrieval solutions for many enterprise activities:
-
Enterprise Resource Planning (ERP): Operates on the enterprise resources, such as finance, manufacturing, and product distribution in an organization. ERP requires high transaction speed because it works on critical organizational data.
-
Online Transaction Processing (OLTP): Responds immediately to client queries. Data-processing speed is important for the success of OLTP.
-
E-business and e-commerce: Occurs between business organizations or between an organization and a customer.
-
Data Warehousing: Separates the static data of an organization from the operational data and serves as a data repository for data mining tools. Extracting and storing data that is specific to a business is called data mining.
Storage Pooling
Storage pooling is an important factor when designing a SAN. There are three types of storage pools - Local, Centralized, and Distributed. A local pool provides one-to-one data connection between a storage device and a server and uses the same fabric switch. Figure 3-1-2 shows a local storage pool:
A centralized storage pool provides a one-to-many data connection between multiple servers and the storage system. Figure 3-1-3 shows a centralized storage pool:
A distributed storage pool provides a many-to-many connection between multiple servers and storage systems. Figure 3-1-4 shows a distributed storage pool:
Note | It is useful to have a local or a centralized storage pool when designing a local or a remote SAN. |
Heterogeneity
SAN supports multi-vendor platforms and operating systems, such as Windows NT, Windows 2000, Linux, Solaris, and Macintosh. The performance of a SAN is judged on how easily SAN integrates with these heterogeneous environments. Before building a SAN, the number of fabric switches and the total number of ports is calculated. When designing a SAN structure, you should keep the future requirements of the organization in mind. The structure should have the flexibility to support new devices without affecting the existing network design and performance.
Network Performance
Performance is an important aspect when designing a SAN. Several factors contribute to the performance of a SAN follow:
-
Data accessibility
-
Device connectivity
-
Network manageability
-
Fault tolerance
The physical structure of an existing network is an important consideration when designing a SAN. Proper placement of storage systems with respect to the SAN servers helps in designing a performance-oriented SAN. The physical layout of the network, such as the distance between the storage device and the servers, helps determine the actual cost for setting up a SAN.
Storage Environment Selection
You need to select an appropriate storage device before designing a SAN. For example, Redundant Array of Inexpensive Disks (RAID) provides an array of drives and storage devices that are fault-tolerant and can be used in a SAN design. In RAID, data is stored in several drives. All the levels of RAID provide various performance and redundancy levels. The basic RAID level is RAID 0 and does not offer any redundancy.
Redundancy in RAID is provided through virtual disks, ensuring data protection from disk failures. Virtual disks provide high-level fault tolerance and allow damaged disk swapping without affecting network performance. These disks are suitable for heavy database transactions, which are generally levels 3 and 5. Table 3-1-2 lists RAID levels, their descriptions, and use:
Level | Description | Use |
---|---|---|
0 | Is the simplest RAID level and stores data on all the drives. | Provides maximum throughput at reduced cost. |
1 | Uses the mirroring concept of storing data and creates a copy of the data on multiple drives. | Provides high-level security to data. It is expensive to have multiple drives in this level. |
3 | Stores data using the parity concept and stores the parity values on a separate drive. | Provides a high data transfer rate and is less expensive than other RAID levels. |
5 | Stores data using the parity concept and stores parity values across multiple drives. | Provides high-level security, can be used with multiple applications, and can withstand single drive failure. |
6 | Stores data using the parity concept and stores parity-related information on stripped drives along with data. | Provides considerable security and high read speed. |
Data Access Using SAN
In a SAN setup, multiple heterogeneous server platforms can access a storage device simultaneously. Several of the ways to access data from storage devices follow:
-
Logical Disk Volume Partitioning
-
Physical Disk Volume Partitioning
-
File Pooling
-
Sharing Data
Logical Disk Volume Partitioning
In logical disk volume partitioning, logical disk volumes are created and assigned to various servers to store data. A logical disk can be stored on a single storage resource or across multiple devices. The storage controller manages the logical volumes and restricts other servers from accessing the disk.
Physical Disk Volume Partitioning
The simplest and most commonly used method to access data from a storage resource in a heterogeneous environment is through the physically partitioned storage disk volumes. Every server is assigned an exclusive set of disk volumes using this method. The assigned disk volumes become inaccessible to other servers after they are assigned.
File Pooling
In the file pooling data access method, a mountable namespace, instead of a disk space, is assigned to a server. Namespace enables a partition set to add or remove file sets even if the server is running. The size of the namespace varies depending on the size of the data file being accessed.
The mountable space is released after the server accesses the files. All other servers can then use this disk space to access files. The namespace has the advantage of enabling various file systems to share the same storage space and add a large number of hard drives.
Sharing Data
The sharing data method enables data access in two ways:
-
Sharing data copy: Allows the server to copy the data. After a file is copied to the server, it is updated at regular intervals whenever the file changes.
-
True data sharing: Allows simultaneous data access to multiple servers. The three ways by which a server can access data are:
-
One-at-a-time access method: Allows sequential data access to servers on a first come-first served basis.
-
Multiple read access method: Allows the requesting servers to read data simultaneously. Updates are performed one at a time using this access method.
-
Multiple read/write method: Allows simultaneous reading and data updates by the servers. The limitation of this access method is that it can lead to data conflicts due to simultaneous updates.
-
SAN Topology and SAN Protocols
A topology depicts the logical arrangement of nodes on a network. A node can be any device attached to the network. When selecting a topology for SAN implementation, consider the present network type, the amount of network span, and the required storage space.
Simple Topology
A simple point-to-point topology establishes a direct connection between a server and a disk array. This is the simplest of all the SAN topologies with a minimal set of options. It provides easy connection and fast access to the disk array. This topology is unreliable because it contains various single points of failure, such as the server, cable, HBA, and disk array controller. Figure 3-1-5 shows a simple topology with a point-to-point connection between a server and a disk array:
A simple point-to-point connection can be extended because disk arrays support two redundant controllers. A new HBA can be added to the server and then connected to the disk array. In this topology, the single point of failure is the server because it contains two HBAs. The server works even when the cable, the disk controller array, or the HBA fails. Figure 3-1-6 shows a two-path HBA point-to-point connection:
Adding another server can extend the point-to-point topology further. In the figure, both the servers are connected individually to the disk controller. Adding another server increases the single point failure, which includes servers, cables, disk arrays, and HBAs. Figure 3-1-7 shows the point-to-point connection between two servers:
Cluster the servers to eliminate any single point failure. Clustering provides access to the disk arrays even if one server fails. Figure 3-1-8 shows a clustered point-to-point connection:
FC-AL Topology
Adding a SAN interconnect to a simple point-to-point connection forms a FC-AL topology. This topology is helpful in a SAN setup where storage devices are placed at long distances from the servers. A hub is used to interconnect servers and disk arrays. A hub contains two connections from each server, while two links from the hub are connected to the disk array controller.
In the FC-AL topology, the hub is the single point of failure. An entire network ceases to function when a hub fails. This topology is ideal in a small organization with only a few servers that can tolerate the downtime. Figure 3-1-9 shows a simple FC-AL topology:
You can extend the simple FC-AL topology by adding another hub to create an error-free structure. This structure can be used in a SAN setup that does not require immediate scalability. Organizations use this topology to build individual SAN structures. You can easily expand the SAN setup using this topology. Figure 3-1-10 shows the fail-proof Fibre Channel topology:
Fabric Topology
Fibre Channel fabric is a switch-based Fibre Channel connection network. Switches transfer information across various ports. Fabric-based topology enables end users to access the data storage devices simultaneously and also connects long distance networks. A switch acts as a SAN interconnect for different systems across long distances.
The advantage of using fabrics over the FC-AL topology is that they provide quicker data access. Fabric-based topology contains a switch or higher SAN interconnect instead of a hub. Figure 3-1-11 shows the simple fabric-based topology:
SAN supports a large number of protocols, such as Fibre Channel over Internet Protocol (FCIP), Internet Fibre Channel Protocol (iFCP), Transmission Control Protocol/Internet Protocol (TCP/IP), and Internetwork Packet Exchange (IPX). It also supports storage protocols, such as Storage IP and Infiniband. FCIP and iFCP are the most commonly used SAN protocols.
Fibre Channel Over Internet Protocol
FCIP enables Fibre Channel data transfer by tunneling data between SAN- and IP-based networks. Tunneling is the technique of transmitting private data of an enterprise network through a public network. FCIP transmits the data in a way that the routing nodes in the public network are unaware that the data of a private network is transmitted. This feature enables data sharing across the enterprise network.
FCIP enables quick data transmission in SAN and depends on IP-based network resources to interconnect SAN over LANs, MANs, and WANs. It encapsulates Fibre Channel frames within TCP/IP to link Fibre Channel SANs over WANs. The primary purpose of an FCIP protocol is to create and administer multiple FCIP links to transfer data from one SAN to another. The basic functions of an FCIP entity are to forward Fibre Channel frames and perform Fibre Channel frame encapsulation.
Internet Fibre Channel Protocol
The iFCP deals with the backup and storage data operations in a SAN. It is a gateway-to-gateway protocol, which provides the services of Fibre Channel fabric to Fibre Channel end devices within a TCP/IP network. iFCP helps extend Fibre Channel SANs over the Internet. It helps pass data to and from Fibre Channel storage devices in SAN using TCP/IP. The TCP component manages congestion in data flow, error checking, and data retrieval services for the Fibre Channel SANs. iFCP supports connectivity over long distances and also supports various data rates, media types, and connectors.
iFCP supports various types of communication within a network environment. The communication devices can vary from various Fibre Channel end-devices to an IP-network to SAN. The types of iFCP communication taking place within a Fibre Channel frame follow:
-
Device-to-device: iFCP communication does not need a supporting Fibre Channel fabric because the data transaction occurs between similar type of devices.
-
Device-to-SAN: iFCP communication needs a supporting Fibre Channel fabric to perform the data transaction. This fabric performs data mapping between the Fibre Channel devices and SAN components.
-
SAN-to-SAN: iFCP communication uses an existing Fibre Channel fabric to perform data transaction by incorporating a TCP/IP network within a SAN.
Implementing SAN
To implement a feasible SAN network, the following are certain organization-specific issues that need to be resolved:
-
Data transfer rate for transferring and accessing data across the network.
-
Number of servers sharing and accessing information across the network.
-
Distance at which the network needs to be spanned.
Constructing a SAN
The construction phase of SAN implementation follows the design consideration, component selection, and the fabric design phases. For this phase, an experienced and skilled team is required with an extensive working knowledge of heterogeneous platforms. After the implementation phase, all the design schemes and implementation considerations are documented for future reference. Some of the key points that the documentation should contain are the following:
-
Requirements: Include all business requirements for implementing SAN as a solution.
-
Design considerations: Include a list of possible limitations that may be encountered during SAN implementation.
-
Logical and physical layout: Include recorded information related to the storage structure design of a SAN. The documentation should contain clearly drawn diagrams that depict the entire SAN setup.
-
Positioning of storage devices: Include recorded information pertaining to the physical location of various storage devices that are used in the SAN network.
-
Infrastructure information: Include the details of the existing infrastructure and information about future additions and updates should be recorded in the documentation.
-
Topology details: Include details of the topology and the cabling system to decrease future debugging efforts.
-
Applications and management tools: Include details of the applications running on various SAN components.
Implementing Backup Solutions
Backup is the process of duplicating data from the primary storage device on to the secondary storage. Organizations back up important data as part of their risk management process and store the data for future reference. The size of the secondary storage device depends on the size of the organization. It is a time-consuming process and generally requires network downtime.
Some of the important features of the SAN backup architecture that help in implementing a robust backup mechanism are the following:
-
Tape mirroring: Provides safe and error-free data recovery. A mirroring controller supports multiple tape units to provide a reliable backup solution.
-
Clustered server: Provides faster recovery of backed up data, using SCSI multiplexers shared by servers for daily backups. Multiplexers perform backup operations for the entire cluster without affecting the network performance.
-
Remote backups: Enables SCSI multiplexers and Fibre Channel bridges to be placed at a remote site to back up data on a tape library. SCSI and Fibre Channel bridges are linked using Fibre Channel. Figure 3-1-12 shows local and remote backups using the mirroring technique:
Figure 3-1-12: Backup from Local and Remote Sites Using the Mirroring Technique
The backup retrieval process involves restoring data from secondary storage devices to the primary storage device. There are two types of backups - Full and Incremental.
A full backup copies all the files to the secondary storage device, which can be done at the end of the day. In an office environment, every day new files are created and existing files are updated. In the full backup process, newly created files in addition to the updated files are backed up to secondary storage devices. In incremental backup, only modifications and new files are copied to the secondary storage file. Full backup is safer, but consumes more time and space.
The Distributed Backup Method
The distributed backup method enables data to be backed up when the backup devices are connected directly to the server. This process works well for organizations with a smaller network because the number of servers on a network determines the speed of the backup operation. In large organizations with multiple servers, the use of distributed technique results in slower backup. To manage backup operations efficiently, the backup administrator needs to store backed up data on disk drives. Figure 3-1-13 shows the architecture for distributed backups:
The Centralized Backup Method
The centralized backup method uses IP networks with a centralized backup repository. This method uses the CPU time of the server and slows down server performance. It also extends the backup time to go beyond the scheduled duration of the server.
An Ethernet LAN is used to transport data to the tape library. There are two servers in this setup and data is passed through the server twice before being copied to the tape library, which increases the network downtime. The centralized backup technique is cost-effective because backups can be scheduled using management tools and would not require the interference of the network administrator. Figure 3-1-14 shows the centralized backup architecture:
The Non-Centralized Backup Methods
In the non-centralized methods, a dedicated storage network is used to carry out the backup operation. Dissimilar to centralized storage, this method provides a higher data transfer rate. It provides all the advantages of the centralized backup method. In addition to providing reliable and efficient backup operations, SAN also provides data storage without using the server, which allows SAN to manage storage devices more easily. Figure 3-1-15 shows the backup method without a server:
In a backup method without a LAN, the LAN is free from data backup and data retrieval processes. To administer data flow from storage and backup devices, an administrator server is used. In server-less backup operations, a server is needed to supervise backup operations. The non-centralized backup methods provide higher bandwidth during the backup process. Figure 3-1-16 shows a Fibre Channel SAN architecture performing LAN-less backups:
In Fibre Channel SAN, the tape drives in a tape library increases the backup speed. As a result, SAN is a cost-effective and time saving solution because it eliminates the load on the server and the LAN.
SAN Virtualization
SAN uses storage devices that contain multiple disks and drives on a network. You may run out of drive letters when you attempt to map these drives. Proper planning is required to use the various drives for proper sharing of the available storage space.
SAN virtualization is the process of effectively using storage devices and drives. This virtualization process resolves all the storage space-related conflicts. The SAN virtualization process integrates the physical disks into a pool. The pooled physical disks are accessible to application servers without being mapped. The techniques for sharing the virtual disk capacity of the SAN are the following:
-
Multiple host arrays
-
Dedicated storage domain servers
-
File system redirectors
-
In-band virtualization engines
-
Logical Unit Number (LUN) masking filters
These techniques vary from each other in terms of cost and application. Selecting a technique depends on the platform being used and the location of SAN.
The multihost storage array scheme provides high-level performance settings. Multiple platform connectivity support is also provided by the multihost storage array. Pooling is done at the storage level along with RAID. At times, the addition of extra pools results in the loss of centralized storage allocation. Figure 3-1-17 shows multiple host arrays with RAID:
Dedicated storage domain servers allocate disk space to multiple hosts and deals with SAN virtualization. The virtualization process is applied on the network storage control layer above the operating system. This process controls the device operations, system networking, and SAN security features. It also enables the centralized administration of SAN and distributes management tasks to a large storage pool.
The number of domain servers required on the network and I/O performance can be optimized based on the design of the network. Optimized network enables server-less and LAN-less backups and data recovery across the storage pool. Figure 3-1-18 shows the dedicated storage domain server with multiple hosts:
The file system redirectors enable files to be accessed across the LAN for input/output across a SAN. Each SAN host requires software to map the addresses with the file names. The interface for the mapping process can be a File System Manager or an external metadata controller. File system redirectors help control disk I/O traffic. Figure 3-1-19 shows file system redirection operation:
Securing a SAN
Security can be implemented in SAN at either the hardware or software levels. Securing physical components of a SAN ensures hardware level security. Software level security is implemented with the help of SAN management applications. Two important SAN components that should be provided maximum securities are the SAN fabric and the storage file system.
The security of the SAN fabric depends on the Fibre Channel switches because all data passes through them. Fabric zoning is the best technique that can be used to secure data in the various storage systems in a SAN. This technique creates logical subsets in storage networks that can be accessed both at the device and port level. Creating logical subsets enables the network administrator to restrict devices or ports of one zone from accessing the devices or ports of other zones.
Port level zoning assigns devices or ports to various zones. Ports of the same zone can always communicate with each other. In device level zoning, an entire device is assigned to a single zone. Zoning enables a device or a port to be a member of more than one zone. This feature also allows specific devices or ports to access data that is stored in other zones. An example of device level zoning is switch-based zoning. Figure 3-1-20 shows switch-based zoning:
Fabric zoning enables the creation of a virtual SAN called SANlets. It is a logical storage network of SAN storage devices. SANlets have limited access to the resources that they share among themselves. The ncontrollers that have appropriate access and administration privileges. Otherwise, data is easily accessible to end users. Physical access to the storage controllers should also be controlled.