Having the right data in the right place continues to be a critical mandate for IT professionals responsible for instrumenting and maintaining application ecosystems. While it’s become easier to have distributed frontends spanning on premises, public and hybrid ecosystems, finding efficient and cost-effective ways to ensure scalability and data locality has only become more challenging. This combined with the fact that generally practiced application architecture is undergoing transformation means that object storage solutions being evaluated by organizations must also support modern protocols and methods for access. These factors have been steadily driving the adoption of object storage as a part of modern enterprise data strategies.
New Data Sources
Depending on who you ask, we’re headed towards somewhere between ~30-40 billion connected mobile devices by end of 2023 leveraged by 3.8 billion users resulting in 1 petabyte of data being managed by each IT professional. By the end of this year alone, as a world population we’re likely to be on a 44 zettabyte per year run rate – 40x more bytes than the stars in the observable universal! In order to manage, scale and extract business value from this phenomenal data explosion, the use of metadata has become increasingly important.
By extracting analytics and business insights, organizations are able to monetize their data, out-maneuver the competition and find ways to become more efficient. Unlike other storage mediums, object stores enable very large amounts of metadata to be stored with any individual object allowing for more detailed, granular index and search as well as higher context analytics.
Cost
Despite all of the benefits that object storage provides, it’s also often significantly more cost effective than other options. Being the same technology that enables public cloud and service providers, it’s gotten the benefit of having to learn to be efficient for hyperscale ecosystems without performance degradation. Various techniques such as erasure coding, versioning, customization, embedded analytics along with the delegation of low-level block management to the local file systems, enabling the object store to only have to worry about high-level details. This further enables file lookup tables to be kept to a minimal size and provide scalability to many hundreds of petabytes while maintaining high performance levels. These efficiencies and cost factors make object storage an ideal candidate for a variety of secondary and long-term storage use cases
DevOps
The current trends we’re seeing in IT of flattening organizations and the adoption of modern principles that bring development and operations practices closer together are all centered on increasing agility, speed and efficiency. A value proposition that object storage provides for this operating environment is the fact that it can be provisioned and scaled quickly, often by the application owner once the ecosystem is put in place. Block storage volumes can only be accessed when they’re attached to an OS while object stores include data and metadata that can be access directly through APIs and over common web protocols.
This means that application developers have direct access to data need to help them speed delivery of services while reducing overhead on the rest of the infrastructure and storage team when key changes are needed. What’s more, since object storage were greatly popularized in public cloud applications like Amazon S3, Rackspace Cloud Files, Azure and Google Cloud Storage, the capabilities baked into today’s solutions also enable customers and service providers to get closer to treating their own on-premises ecosystem like a cloud environment.
The Dell EMC Object Storage Advantage
Dell has over 16 years of experience in helping customers transition to cloud operating models. This heritage has been leveraged and applied to the ECS solution which provides hyper-scalable, next gen object storage for large data set use cases. One primary advantage of the EMC approach is that they’ve put a significant level of engineering into the automation of best practices, which ensures optimal deployment configuration and keeps administrators out of trouble. By providing a multi-use platform, ECS also helps drive cost savings and efficiency buy consolidating backup, secondary, storage, DRaaS backbone and a migration target for legacy NFS stores. Capabilities for interfacing with cloud native apps over Swift serving as an analytics repository for Splunk, Hawq, Spark and Hadoop also enable interoperability with existing environments when ECS is introduced into brownfield deployments.
Continuity, Availability and Resilience for ECS Object Storage
As with any part of an application ecosystem, the components are only useful if their available, resilient and can scale as needs of the organization change and object storge is no different.
As a strategic Dell partner, Kemp developed the ECS Connection Manager to provide object storage high availability, resilience and traffic flow optimization while continuously maintaining enterprise-wide security posture and departmental segmentation of data. The Connection Manager serves as a termination end point for all client application traffic requests and storge responses. This enables a tight coupling to the storage engines, along with performance visibility that drives intelligent decisions about local and cross-cluster traffic distribution. Multi-site single namespace is also enabled which significantly simplifies administration. Additional features such as IPv4/6 translation, SSL optimization, QoS and traffic steering driven by the unique characteristics of individual requests, makes ECS Connection an optimal component to be included in every object storage deployment.
Key Dell EMC ECS + Kemp ECS Connection Manager Capabilities
As a solution, instrumented specifically with ECS in mind, the ECS Connection Manager provides a number of capabilities that optimize and simplify any object storage deployment:
- S3 Storage Awareness & Optimization – Kemp ECS Connection Manager delivers S3-optimized algorithms to ensure the accurate routing of object read/write requests along with even traffic distribution. Automatic detection of the method used by applications to access the ECS storage buckets (S3 path or virtual host addressing) further enables traffic routing efficiency without the need of manual configuration for hybrid environments.
- Storage QoS (Quality of Service) – ECS Connection Manager QoS enables the provision of consistent and predictable levels of service for multi-tenant and multi-application infrastructures. In these shared environments, QoS simplifies the task of ensuring that critical workloads are not impacted by ‘noisy neighbors’ or rogue applications by providing granular rate limiting controls that may be assigned globally or to specific storage objects and paths. Watch the video on QoS and rate limiting for Dell ECS for more information.
- Multi-Network Support – Many customer environments require departmental network segmentation while enabling access to shared backend resources. The ECS Connection Manager simplifies the process of publishing ECS service endpoints across multiple network segments while reducing the requirement for complex routing or the introduction of exceptions that violate Infosec standards.
- High-Speed Network Integration – ECS Connection manager hardware platforms are available with native connectivity at speeds of 25 Gbps, 40 Gbps and 100 Gbps to simplify integration with existing network infrastructure and to support high-speed access to Dell EMC storage platforms.
- Dynamic Global Host Resolution – Name resolution and multi-site global server load balancing (GSLB) are critical to the resilience of ECS deployments. The Kemp ECS Connection Manager natively includes this functionality and supports hybrid environments with applications leveraging multiple methods for accessing the ECS infrastructure. Dynamic Global Host Resolution within this feature set provides flexible, multi-site distribution and availability for ECS deployments leveraging both Path Style and Virtual Host addressing.
- Enhanced Defense and Public Sector Support – Compliance with the FIPS 140-2 security standard means that customers can leverage ECS Connection Manager to secure data in-flight with algorithms and ciphers that are consistent with the standard. Common Access Card (CAC) support also means that access to the Kemp ECS Connection Manager interface and management interfaces of ECS can equally be done according to Department of Defense (DoD) standards. IPv6 interoperability (certified to NIST standards) is further provided for environments that must deploy with this network topology.
Object storage should certainly be considered as a part of any modern data storage strategy. Dell’s solution in ECS is one of the premier options with capabilities that can be used to address a variety of use cases. Combined with Kemp’s ECS Connection Manager, the efficiency, scale and resilience of any object storage deployment can further be greatly improved.
Resources
You can learn more about the solutions discussed in this post at the following links:
- https://kemptechnologies.com/ecs-connection-manager/
- ECS Connection Manager web page
- VIDEO: Quality of Service (QOS) and rate limiting for Dell ECS using ECS Connection Manager
- CASE STUDY: Global pharmaceutical company maintains uninterrupted access to ECS with Kemp
