Set the Speed Dial for Pavilion Data

NVMe SSDs are the most expensive non-volatile storage in today’s data centers. But optimizing your ROI on NVMe can be tricky. Conventional wisdom says the highest performance with the lowest latency is achieved by installing the SSD directly into the server and scaling throughput or IOPS in a parallel fashion by adding more servers, each with its own NVMe SSD. However, increasing IOPS or throughput to match your workload is directly proportional to the amount of NAND Flash and power settings on the SSD. If your workload requires a small amount of storage, as most databases do, but your need…

NVMe is fast flash storage, but big five suppliers take it slowly

Even with the inevitability of NVMe, major storage vendors are slow to take the plunge. While many have upgraded the back end of their arrays to accommodate NVMe drives, implementing a true end-to-end solution is proving to be a daunting technology challenge. For vendors like Pavilion Data, NVMe is so fundamentally different than SATA or SAS, we have chosen to re-architect our array from the ground up for NVMe. By implementing an internal PCIe fabric with 20 controllers, the Pavilion Data array can deliver extremely low latencies in an inconspicuous 4 RU package that is ideal for rack-scale design. Read…

Supercomputing Architecture for Any Data Center

Mapping the Human Genome was just the beginning for the life sciences discipline of bioinformatics.  Now, the goal is to apply every map of every person against a perfectly healthy baseline to identify sequences of DNA that carry diseases like diabetes, asthma, migraine, and schizophrenia.  However, even with the most powerful compute, network and storage resources on our planet, full genomic map comparisons are not possible. The International HapMap Project is a more efficient method to isolate genetic mutations using massively parallel processing with Hadoop® MapReduce and Spark® analytics to obtain statistically significant comparisons.  HapMap would not exist without the…

Disaggregation for Less Aggravation

NVMe SSDs ushered in an era of standards-based access to Solid State storage.  Previous protocols like SCSI and SAS were designed for spinning disk media and carried severe overhead that squelched the performance that can be delivered by NAND.  It’s no coincidence that IDC forecasts NVMe will make up more than 50% of all enterprise SSD shipments by 2020[1]. One step forward, two steps back While significant increases in IOPS and throughput can be delivered with today’s NVMe SSDs.  The deployment model is arcane.  By implementing NVMe inside of the server, we are back to the problems of direct-attached storage…

Modern Applications and Rack Scale Shared Accelerated Storage

Modern cognitive applications are transforming entire industries. From eCommerce to AdTech, to Internet of Things (IoT) to Online Gaming to Security and Fraud prevention. The list goes on. These applications provide real-time, actionable insights that organizations rely on to make better decisions faster. The consumers of these insights are often not human beings; instead, they are other systems and applications that affect the desired outcome. To make better decisions requires access to massive amounts of data. And in the era where the bar for speed is real-time, these decisions need to be made faster and faster, this data needs to…

Are Storage Systems Designed for NVMe?

Pavilion Data is glad to see established vendors such as Dell-EMC and Pure Storage entering the NVMe-based storage bandwagon. There are two categories of storage devices leveraging NVMe: Traditional Shared Storage Shared Accelerated Storage Traditional Shared Storage is used for scale-up workloads and usually a replacement for existing SATA or SAS-based All Flash Arrays (AFA) utilizing Fiber Channel or iSCSI protocols. To leverage NVMe, these systems are retrofitted in two ways:  1) media replacement with NVMe, and 2) replace the internal controller-to-drive interconnect bus from SAS/SATA to NVMe.  No other change in architecture is usually done. This provides a performance…

Pavilion’s perspective as vendors enter the NVMe game

NVMe Demands Storage System Re-Design, Not Retro-Fits We are experiencing exciting times in the storage industry.  During the week of May 1, 2018, Dell announced the NVMe-based PowerMax.   At Pavilion we welcome these kinds of announcements, since it accelerates the customer awareness around the fact that the future of storage is NVMe.   Let the loud and expensive bullhorn sound…its fascinating to watch.   Why?  Because the transition to all-NVMe storage is being validated before our eyes. You know you’re on to something important when the AFA storage incumbents make their retrofit product announcements.  Rather than doing the difficult, yet required work…

The Maslow’s Hierarchy of Storage

In 1943, Abraham Maslow published a paper that described Motivation Theory. In short, he suggests five interdependent levels of basic human needs (motivators) that must be satisfied in a strict sequence, ranging from fundamental level (survival) to the highest level (self-actualization). In the world of storage technologies, we think Maslow’s Hierarchy provides a compelling analogy to frame some of the most exciting and transformative developments occurring today. Prior to digging into Maslow, previous blog posts describe two major concepts: The NVME specification that standardizes high-performance access to direct-attached NVMEs, and The evolution to NVMe-oF which allows to these NVME devices…

Ditch the DAS: The Benefits of Rack-Scale Flash

About a decade ago, new companies formed around building online applications in the areas of SaaS, Social Media, and other verticals that required the ability to scale effortlessly in multiple dimensions to support growth and peaks in demand. These companies and technologists built a new kind of infrastructure to service a rapidly-growing customer base that required real-time information. They relied upon low-latency storage resources directly installed in servers as direct-attached-storage (DAS) in order to put the data as close to the CPU as possible. The scale-out database technology that underpinned these applications could manage data across the cluster, and avoided…

The “P” in Pavilion Actually Stands for Performance – Part 2

This is the second half of the Pavilion blog focusing on 3 important design areas of storage products:  Bandwidth, Latency, and Density.  This is the second entry in this series, focused on Latency and Density. Latency The access latency from the Host to the media is composed of host storage stack latency, network stack latency, IO controller latency and media access latency. The first three components here are fairly standard and we minimize the number of memory copies and limit data touches to keep latency at a minimum. The media access latency is largely governed by the type of media (NVMe…

Why NVMe? Why NVMe over Fabrics? Why Pavilion Data?

Applications are becoming increasingly parallel. What used to be done on a single application server is now spread across a cluster of servers operating in parallel. This allows for scaling in multiple dimensions. Need more bandwidth or compute power for your clustered application? Just add more servers to the cluster. Hyperscale customers have been doing this for nearly a decade now, and the number of customers that embrace this architecture are growing constantly. This scale-out approach also is used to scale storage performance and capacity. Traditional all-flash-arrays cannot offer enough density and/or performance to satisfy these applications. Therefore, the servers…

Don’t be fooled: Off-The-Shelf SSDs make a solid design choice in storage systems

Storage system vendors have chosen to integrate flash in two ways: incorporate standard off-the-shelf SSDs, or design their own flash modules and controllers.  Many of the early all-flash array pioneers, like Violin and TMS, designed their own custom flash modules for what were very sound reasons at the time.  The choice to go in one direction or another in this area revolves around several criteria, most notably Performance, Time-to-Market, and Cost.   I explore all 3 as it relates to this subject below: Performance Many vendors made the design choice to build their own flash controllers before NVMe products were available, due…

The “P” in Pavilion Actually Stands for Performance

Storage system designers constantly make engineering tradeoffs to maximize three key attributes – Throughput, Latency and Density. This blog series will delve into each of these separately and cover the thought process we went through when we designed the Pavilion Storage Platform as it relates to these attributes.    This first blog of the series focuses on Throughput.   Part 1 – Throughput We can generalize the Pavilion Storage Platform as a dataflow system in order to understand the throughput performance characteristics. Such a system can be viewed to have well defined system demarcation boundaries, and Amdahl’s law can be…

The Penrose Triangle

We live in a connected world. Millions of apps, billions of users and trillions of things. Consequently, the amount of data that is generated is exploding. Yet at the same time, we are seeing massive fragmentation of where it originates from, its key structure, where it resides, how it gets accessed and processed and how it is ultimately consumed. New-fangled clustered databases, in-memory caches, exotic massively parallel filesystems, structured to semi-structured to unstructured, modern and disparate query languages, are just a few of the technologies that are causing this significant fragmentation. At the same time, new application architectures are evolving…

Motivation Behind Designing the Pavilion Platform

Over the course of two decades I have been deeply involved in developing products that spanned various domains such as data networking, storage area networking, PCIe interconnects, IO Virtualization, Flash controller, and all flash array systems. My journey took me through a variety of companies such as Cisco/Andiamo, Aprius (acquired by FusionIO) and Violin Memory and I am proud to be part of these teams that developed and shipped innovative products that served the data center. Around the time I was at Aprius the idea of sharing IO on a compute node gained credence when it became evident that every…

Next Generation Storage With No Compromise

Businesses are transforming to be more digital and are constantly in the quest of getting a competitive advantage over their competitors. One of the important and popular means is to do more with the data that they have. We all know that the amount of data as in volume, velocity and variety are continuing to grow unabated. To maintain competitive edge in the digital era implies the ability to take control of this deluge, analyze and deliver insights accurately to better serve business needs. This has turned the focus back to storage software, storage media and systems as it was…

Different Approaches for Delivering Ultra Low Latency Shared Storage

Several new storage systems have come to market with the goal of delivering shared flash resources as a service to high-scale, distributed applications. These products take advantage of some of the following technology developments in the storage and networking space: Standards-based PCIe-Connected SSDs, RDMA-Capable Ethernet Networking up to 100 Gbe, a standard storage protocol designed for PCIe-Connected SSDs (NVMe), and a standard protocol for remotely Accessing NVMe devices (NVMe-Over-Fabrics, or NVMe-oF).   Note that Red Hat 7.4 and Ubuntu 16 both now include NVMe-oF support inbox. These 4 technologies can allow applications to get performance from networked storage similar to SSDs installed directly in…