Data Beyond Storage

Los Altos, Calif.-based Hammerspace has a simple mission: allow for the delivery of unstructured data across the hybrid cloud, without that data tied to storage. By removing the limitations associated with storage silos — and with metadata-driven machine learning attached to the process — the company’s “Data-as-a-Service” concept aims to make accessing and using data a much simpler task for businesses.

David Flynn, CEO of Hammerspace, spoke with the Media & Entertainment Services Alliance (MESA) about the origins of his new company and its unique solution, where the company’s machine learning engine comes into play, and how media and entertainment firms can use Hammerspace to accelerating content production.

MESA: What was the impetus for Hammerspace, how did the company come about?

Flynn: The name Hammerspace is inspired by the extradimensional realm found in animation and video games where tools and inventory are impossibly stored until the hero has an immediate need for something specific. Nobody asks too many questions, but this manipulation of time-and-space is essential for moving the plot forward – and we do this for data.

Hammerspace is the result of six years of R&D, with much of the work done in the open-source community to make standard open protocols like Network File System (NFS) and Server Message Block (SMB) ready for a future defined by the ability to dynamically scale-out workflows across clouds, data centers and containers. Hammerspace believes that the key to accomplishing this is to break free from storage data silos and the unnecessary copying of data that comes with them.

People create data silos because they manage data by managing storage, and the typical approach to working around silos is to copy data to make it available to data engineers and scientists. This copy approach for giving access to data is tedious, slow and expensive. IT is forced to deliver against unknown service-level agreements (SLAs) since the data user often isn’t sure what they need until they have it. The whole exercise doesn’t scale.

Hammerspace takes a data-centric perspective to data, virtualizing data so that users can consume it free from the rigidity of storage silos. Abstracted from the infrastructure, data workers can self-service their data on-demand, anywhere across the hybrid multi-cloud.

MESA: How has Data-as-a-Service evolved in recent years, and what does Hammerspace do differently than others in the space?

Flynn: The ability to offer true, on-demand unstructured file and object Data-as-a-Service to applications and data consumers across hybrid multi-cloud environments is something unique to Hammerspace.

There are other solutions out there who make similar claims, but they use copy data management to awkwardly move whole data sets and volumes between environments, often through backup mechanisms, which they then activate on the other side. Backup and copy approaches don’t work because they fall into the trap of perpetuating data silos and sacrificing data agility while contributing to data sprawl.

Hammerspace virtualizes data through a global namespace managed through powerful extensible metadata shared across any cloud or data center, making data instantly accessible, live and active in all sites, concurrently. Combined with the ability to move live data non-disruptively using standard and open protocols and machine learning driven data management automation, Hammerspace stands out from the field of players who try to market their point solutions as something they are not: a solution for hybrid multi-cloud.

MESA: How are media and entertainment companies making use of Data-as-a-Service?

Flynn: Hammerspace makes it easy for artists to find, access, and manipulate their media on-premises, into the cloud, and across great distances using machine learning driven automation and extensible metadata; while still using their favorite Media Asset Management (MAM) tools.

Extensible metadata changes every aspect of the media workflow, by reshaping the structure and organization of how people manage data. Rich metadata allows system administrators, producers and artists to apply the creative process to solving problems rather than adhering to storage IT concepts.

AI generated metadata harvested from data makes content contextual and tactile while automating the management of data across the infrastructure. To achieve this level of data agility, we must separate the management of metadata from data. Storage systems will never do this, and MAMs (media asset management systems) don’t operationalize hybrid multi-cloud environments; both need access to expansive metadata control so that humans can process it without having to follow limitations dictated by machines.

MESA: Where does machine learning come into play with Hammerspace’s offering?

Flynn: AI is a co-pilot to IT, helping to manage the infrastructure by taking care of tedious and repetitive tasks. A machine learning engine optimizes to balance performance vs. by continuous monitoring of telemetry so that IT can spend less time on mundane, reactive tasks and work on more value-creation work.

Global access to billions of files across the hybrid cloud demands a unique approach to data orchestration. The Hammerspace machine learning engine employs a continuous market economy simulation between real data and available infrastructure resources. The model treats storage services as landlords with resources to lease, and data files as tenants who spend limited currency to meet specific needs.

Hammerspace continuously collects performance telemetry from workloads for each file accessed, in the form of metadata. This monitoring provides a rich understanding of how the infrastructure is performing, so Hammerspace can automatically correct for issues before users notice. Real-time decisions for data placement are fully automated, balancing performance and cost across the hybrid cloud.

MESA: What are some of Hammerspace’s favorite use case examples, where M&E companies have made especially good use of the company’s services?

Flynn: One of our customers, a major animation studio, is making the transition to integrate on-premises file and object infrastructure while streamlining their workflows across four locations and two continents with an enterprise-grade geo-spanning global namespace; replacing home-grown automation scripts and manual processes to improve reliability and scalability. A collection of NAS vendors populates this multi-tier storage environment, so the ability to consolidate existing infrastructure into a global namespace simplifies burst-to-cloud use-cases significantly accelerating content production while keeping costs low.

MESA: What’s next for Hammerspace, what can we look forward to from the company in the future?

Flynn: Hammerspace will continue to work with partners to provide direct and native integration into their tools. You can expect to see more activity from us as we work with various MAM vendors and the high-level analytic services provided by popular public cloud services.