More Related Content
What's hot (20)
Similar to Introduction to Enterprise Data Storage, Direct Attached Storage, Storage Area Network, Network Attached Storage, Data Storage Management, File System, Cloud Data Stores, Using Grids for Data Storage (20)
Recently uploaded (20)
Introduction to Enterprise Data Storage, Direct Attached Storage, Storage Area Network, Network Attached Storage, Data Storage Management, File System, Cloud Data Stores, Using Grids for Data Storage
- 1. Syllabus Data Storage: Introduction to Enterprise Data Storage, Direct Attached Storage, Storage Area Network, Network Attached Storage, Data Storage Management, File System, Cloud Data Stores, Using Grids for Data Storage. Cloud Storage: Data Management, Provisioning Cloud storage, Data Intensive Technologies for Cloud Computing. Cloud Storage from LANs to WANs: Cloud Characteristics, Distributed Data Storage.
- 2. • Cloud Storage is a service that allows to save data on offsite storage system managed by third-party and is made accessible by a web services API. Storage Devices : Storage devices can be broadly classified into two categories: 1. Block Storage Devices 2. Object Storage Devices 3. File Storage Devices Data Storage in Cloud Computing
- 3. Block Storage Devices : The block storage devices offer raw storage to the clients. These raw storage are partitioned to create volumes. Data Storage in Cloud Computing
- 4. Block Storage Devices Data Storage in Cloud Computing
- 5. Object Storage Devices Data Storage in Cloud Computing
- 6. File Storage Devices Data Storage in Cloud Computing
- 7. Cloud Storage Classes : Cloud storage can be broadly classified into two categories: 1. Unmanaged Cloud Storage 2. Managed Cloud Storage Cloud Storage Classes
- 8. Managed Cloud Storage • Managed cloud services are services that offer partial or complete management of a client's cloud resources or infrastructure. Management responsibilities can include migration, configuration, optimization, security, and maintenance.
- 11. 1. The cloud storage system stores many copies of data on many servers at the various locations. 2. The data is stored at various locations so that if one system fails it can change the pointer location to where the object is stored. 3. The cloud provider uses the virtualization software to aggregate the storage assets into cloud storage system. This system is called as StorageGRID. 4. Storage GRID creates a virtualization layer which fetches storage from various storage devices into a single management system. 5. It manages the data from CIFS and NFS file system over the Internet. Creating Cloud Storage System
- 12. “Enterprise storage is a centralized repository for business information that provides common data management, protection.” An Enterprise Storage System is a centralized repository for business information. It provides a common resource for data sharing, management and protection via connections to other computer systems. Enterprise storage systems are designed to process heavy workloads of business-critical information. Types of Enterprise Storage : 1. Direct-Attached Storage. 2. Network-Attached Storage. 3. Storage Area Networks. Introduction of Enterprise Data Storage
- 13. DAS stands for Direct Attached Storage. It is a digital storage device connected directly to the server, workstation, or personal computer via the cable. In Direct Attached Storage, applications use the block-level access protocol for accessing the data. There is no need for any network to attach the device to the server or workstation. So, DAS (Direct Attached Storage) is not a part of the storage network. Some examples of this storage device are solid-state drive, hard drives, tape libraries, and optical disk drives. The System of DAS is attached directly to the computer through the HBA (Host Bus Adapter. As compared to NAS devices, its device attaches directly to the server without the network. The modern systems of this storage device include the integrated controllers of a disk array with the advanced features. Direct Attached Storage
- 14. Types of DAS Following are the two types of Direct Attached Storage (DAS): 1. Internal DAS 2. External DAS Internal DAS Internal DAS is a DAS in which the storage device is attached internally to the server or PC by the HBA. In this DAS, HBA is used for high-speed bus connectivity over a short distance. External DAS External DAS is a DAS in which the external storage device is directly connected to the server without any device. In this type of DAS, FCP and SCSI are the protocols which act as an interface between server and the storage device. Direct Attached Storage
- 16. Network-attached Storage (Commonly known as NAS) is a file storage device which is connected to the network and enables multiple users to access data form the centralized disk capacity. The users on a LAN access the shared storage by the ethernet connection. This storage is fast, low-cost and offers all the advantages of a public cloud on the site. It uses file access protocols such as NFS, SMB, NCP, or AFP. NFS is a file-based protocol which is popular on Unix systems. SMB stands for Server Message Block, which is used with the Microsoft Windows systems. AFP is also a file access protocol that is used with the Apple computers. Network Attached Storage
- 17. It is basically designed for those network systems, which may be processing millions of operations per minute. It supports the storage device for the organization, which need a reliable network system. It is more economical than the file servers and more versatile than the external disks. Advantages : 1. The architecture of NAS is easy to install and configure. 2. Every user or client in the network can easily access to Network Attached Storage. 3. A main advantage of NAS is that it is more reliable than the simple hard disks. 4. The performance is good in serving the files. 5. The devices of NAS are scalable and can be easily accessed remotely. Network Attached Storage
- 18. Disadvantages of NAS Following are the disadvantages or limitations of a Network Attached Storage (NAS): 1. The speed of transferring the data is not as fast as DAS. 2. Users also require basic knowledge of computer networks to use the NAS efficiently. 3. Those users or clients who want to back up their data, then they cannot proceed directly. They can do it by using the installed operating system only. Network Attached Storage
- 20. SAN is an abbreviation of the Storage Area Network. Storage Area Network is a dedicated, specialized, and high-speed network which provides block-level data storage. It delivers the shared pool of storage devices to more than one server. The main aim of SAN is to transfer the data between the server and storage device. It also allows for transferring the data between the storage systems. Storage Area networks are mainly used for accessing storage devices such as tape libraries and disk-based devices from the servers. It is a dedicated network which is not accessible through the LAN. It consists of hosts, switches, and storage devices which are interconnected using the topologies, protocols, and technologies. Storage Area Network
- 21. Protocols of SAN Following are the most common protocols of SAN (Storage Area Network): 1. FCP (Fibre Channel Protocol) 2. iSCSI 3. FCoE 4. NVMe Storage Area Network
- 22. FCP (Fibre Channel Protocol) It is the most commonly used protocol of the Storage Area Network. It is a mapping of SCSI command over the Fibre Channel (FC) network. ISCSI It stands for Internet SCSI or Internet Small Computer System Interface. It is the second-largest block or SAN protocol. It puts the SCSI commands inside an ethernet frame and then transports them over an Internet protocol (IP) ethernet. FCoE FCoE stands for "Fibre Channel Over Internet". It is a protocol which is similar to the iSCSI. It puts the Fibre channel inside the ethernet datagram and then transports over an IP Ethernet network. NVMe NVMe stands for Non-Volatile Memory Express. It is also a protocol of SAN, which access the flash storage by the PCI Express bus. Storage Area Network
- 23. Advantages : 1. It is more scalable. 2. Security is also a main advantage of SAN. If users want to secure their data, then SAN is a good option to use. Users can easily implement various security measures on SAN. 3. Storage devices can be easily added or removed from the network. If users need more storage, then they simply add the devices. 4. Another big advantage of using the SAN (Storage Area Network) is better disk utilization. Storage Area Network
- 24. NAS Vs SAN
- 25. Data Storage Management Tools Maintaining storage devices is a tedious job for storage administrators. They adopt some utilities to monitor and manage storage devices. Management level tasks are configuration, migration, provisioning, archiving and storage monitoring/reporting. Storage Resource Management (SRM) tools include configuration tools, provisioning tools and measurement tools. ● Configuration tools handle the set-up of storage resources. These tools help to organize and manage RAID devices by assigning groups, defining levels or assigning spare drives. ● Provisioning tools define and control access to storage resources for preventing a network user from being able to use any other user’s storage. ● Measurement tools analyses performance based on behavioral information about a storage device. An administrator can use that information for future capacity and upgrade planning.
- 26. Data Storage Challenges Massive Data Demand An industry survey estimates the digital world to increase by 80 zettabytes by 2024 Performance Barrier Rapid growth in data has caused a parallel increase in the size of databases. In the traditional storage method, the response time taken for queries is slow and it should be increased. Be it a social networking site, an enterprise database or a web application, all requires faster disk access to read and write data. Power Consumption and Cost Because of increase in storage demands, IT organizations and data centers need larger storage with minimal cost. Performance lags with minimal cost but has other expenses like licensing and maintenance. Apart from this, other factors such as power consumed by storage devices, cooling systems, man power for managing it and space for data centers are to be considered
- 27. Unified Storage A new innovative solution ‘Unified Storage’ is developed and addresses the issues discussed earlier. Basically this type of storage solution is a combination of NAS and SAN and termed as NUS (network unified storage). This type of storage system handles both file and block level accessing and hence storage devices can be accessed by single and multiple hosts. The main advantage of this system is reduced cost and it supports fiber channel and iSCSI.
- 28. FILE SYSTEMS FAT File System • FAT system was first devised in the so-called computer environment in the early years. FAT was planned for systems with very small RAM and small disks. • It required much less system resources compared to other file systems like UNIX. • Essentially, the FAT system has made a come back. Thumb or flash drives have become very common and have smaller size that makes the FAT system useful. The smaller sizes are even formatted in FAT16. NTFS • In the 1990s, Microsoft recognized that DOS based Windows was inadequate because of demands in business and industry. They started working for better software which can suit larger systems. • NTFS is much simpler than FAT. While files are used, the system areas can be customized, enlarged, or moved as required. NTFS has much more security incorporated. NTFS is not apt for small-sized disks.
- 29. Cloud File System In cloud file systems, the considerations are: • It must sustain basic file system functionality. • It should be an open source. • It should be grown-up enough that users will at least think about trusting their data to it. • It should be shared, i.e., available over a network. • It should be paralleling scalable. • It should provide honest data protection. • Due to its low cost, lower management overhead and elasticity, organizations move towards using cloud computing. • A cloud file system should be scalable enough to adopt large organizations file systems under different workloads with good performance requirements. • Cloud file systems should have high throughputs then local file systems. • Cloud file system should have minimal operation latency. • The system should also be scalable to multiple hosts operating in parallel. • Transparency and backwards compatibility is important to facilitate migration to the cloud with less effort.
- 30. Ghost File System Ghost (General hardware-oriented system transfer) cloud file system is used in Amazon Web Services (AWS). It gives high redundant elastic mountable, cost-effective and standards- based file system. A fully featured scalable and stable cloud file systems is provided by ghost cloud file system. GFS (Ghost File System) run over Amazon’s S3, EC2 and SimpleDB web services. When using GFS, user can have complete control of the data and can be accessed as a standard network disk drive. Benefits of Ghost CFS ● Elastic and cost efficient: Pay for what you use from 1 GB to hundreds of terabytes. ● Multi-region redundancy: Aiming to take advantage of AWS’s 99.99% availability ● Highly secure: Uses your own AWS account (ghost cannot access your data). ● No administration: Scales elastically with built in redundancy—no provisioning or backup. ● Anywhere: Mount on a server or client or access files via a web page or from a mobile phone.
- 31. Features of Ghost CFS • Mature elastic file system in the cloud. • All files and metadata duplicated across multiple AWS availability regions. • WebDav for standard mounting on any Linux, Windows or Mac server or client in the world. • FTP access. • Web interface for user management and for file upload/download. • File name search. • Side-loading of files from torrent and from URL. Web Distributed Authoring and Versioning or WebDAV is a protocol whose basic functionality includes enabling users to share, copy, move and edit files through a web server. It can also be used to support collaborative applications with features like file locking and revision tracking
- 32. Gluster File System • GlusterFS is an open source, distributed file system capable of handling multiple clients and large data. • GlusterFS clusters storage devices over network, aggregating disk and memory resources and managing data as a single unit. • GlusterFS is based on a stackable user space design and delivers good performance for even heavier workloads. • GlusterFS supports clients with valid IP address in network. • Users no longer locked with legacy storage platforms which are costly and monolithic. GlusterFS gives users the ability to deploy scale-out, virtualized storage, centrally managed pool of storage. • Attributes of GlusterFS include scalability and performance, high availability, global namespace, elastic hash algorithm, elastic volume manager, gluster console manager, and standards-based.
- 33. Hadoop File System • A distributed file system designed to run on commodity hardware is known as Hadoop Distributed File System (HDFS). • In HDFS, files are stored in blocks ranging from 64 MB to 1024 MB. • The default size is 64 MB. The blocks will be distributed across the cluster and replicated for fault tolerance. XtreemFS: A Distributed and Replicated File System • XtreemFS is a distributed, replicated and open source. XtreemFS allows users to mount and access files via WWW. • Engaging XtreemFS a user can replicate the fi les across data centers to reduce network congestion, latency and increase data availability. • Installing XtreemFS is quite easy, but replicating the files is bit difficult.
- 34. Kosmos File System • Kosmos Distributed File System (KFS) gives high performance with availability and reliability. • For example, search engines, data mining, grid computing, etc. It is deployed in C++ using standard system components such as STL, boost libraries, aio, log4cpp. • KFS is incorporated with Hadoop and Hypertable. CloudFS • CloudFS is a distributed fi le system to solve problems when fi le system is itself provided as a service. • CloudFS is based on GlusterFS, a basic distributed fi le system, and supported by Red Hat and hosted by Fedora.
- 35. CLOUD DATA STORES A data store is a data repository where data are stored as objects. Data store includes data repositories, flat files that can store data. Data stores can be of different types: ● Relational databases (Examples: MySQL, PostgreSQL, Microsoft SQL Server, Oracle Database) ● Object-oriented databases ● Operational data stores ● Schema-less data stores, e.g. Apache Cassandra or Dynamo ● Paper files ● Data files (spread sheets, flat files, etc)
- 36. DATA MANAGEMENT FOR CLOUD STORAGE • For cloud storage, a standard document is placed by Storage Industry Resource Domain Model (SIRDM). • It states the importance of simplicity for cloud storage. Figure shows the SIRDM model which uses CDMI standards. • SIRDM model adopts three metadata: system consisting of storage metadata, data metadata and user metadata. • By using these metadata, cloud storage interface can offer services without adding unnecessary complexity in managing the data
- 37. DATA MANAGEMENT FOR CLOUD STORAGE • User metadata is used by the cloud to find the data objects and containers. • Storage system metadata is used by the cloud to offer basic storage functions like assigning, modifying and access control. • Data system metadata is used by the cloud to offer data as a service based on user requirements and controls the operation based on that data. Cloud Data Management Interface (CDMI) To create, retrieve, update and delete objects in a cloud the cloud data management interface (CDMI) is used. The functions in CDMI are: ● Cloud storage offerings are discovered by clients ● Management of containers and the data ● Sync metadata with containers an objects
- 38. Cloud Storage Requirements Multi-tenancy In a multi-tenancy model, resources provided are pooled, so that it may be shared by multiple customers based on their needs. Due to the elasticity property in cloud computing, shared pool of storage model makes the provider cost effective and billing is made easy. Security Secure cloud storage requires a secure transmission channel and methods. Securing data can be done using encryption, authentication and authorization. ● Encryption is the process of scrambling data in such a manner as to make it unreadable without special information, called a key, to make it readable again. ● Authentication is the process of determining their identity. Authentication can employ passwords, biometrics, identifying tokens and other means. ● Authorization determines access rights on the data and the levels of authorization. Secure Transmission Channel The four primary methods used to secure network communications are as follows: 1. Transport Layer Security (TLS) and Secure Sockets Layer (SSL) 2. Hypertext Transfer Protocol Secure (HTTPS) 3. Private Networks 4. Virtual Private Networks (VPNs)
- 39. Performance Cloud storage performance can be categorized into two: speed and latency. Factors that affect cloud storage performance are: available network bandwidth, types of systems available in provider’s end, method adopted for compression and caching Quality of Service (QoS) Quality of service (QoS) refers to levels of performance and efficiency of the system that they can provide. Data Protection and Availability To ensure that data is protected from loss and theft, providers must take some precautionary measures: ● Physical site security ● Protection against power loss ● Protection against loss of network access ● Data redundancy ● Server redundancy and server fail-over ● Redundant data sites ● Levels of redundancy Metering and Billing Metering and billing in cloud storage are done based on: data uploaded, data downloaded, data stored and depends on requests and types of request.
- 40. DATA-INTENSIVE TECHNOLOGIES FOR CLOUD COMPUTING