Tuxera – Storage Innovation Cannot Stop at the Hardware

Duncan Beattie, Tuxera

All we ever do in the media industry today is talk about data. How to store it, how to use it, and how to move it. Wrangling all that data is the limiting factor in our creativity and productivity.

If we are being honest, this is an area in which we, as an industry, struggle. Media is the king of unstructured data. We fling huge files around, mixed with image sequences and small audio files. We make impossible demands: an editor hitting the space bar expects a multi-gigabyte stream to start instantly and sustain perfectly.

This is not easy.

At the bottom layer, we have the hardware. On the one hand, this is still improving every day. Processors are getting faster; storage volumes are getting larger. The one limiting factor at the moment is the availability of the critical hardware, so we need to be sure we are making the very best of what we have.

At the top there are the creative tools: the graphics, the editing, the compositing. Users develop close relationships with their favorite platforms, and to get the best talents you have to support their choice of applications.

The challenge lies between the two. How do you get the huge amount of rich data from the commoditized storage into the preferred applications, at a speed and security that meets the demands of media users?

The IT industry has developed a couple of communications protocols which have widespread acceptance: SMB and NFS. SMB originated in the Windows ecosystem and is widely supported across Linux and MacOS; NFS was originally developed for Linux environments. Both are commonly deployed today, using open-source tools like Samba for SMB and Ganesha or NFSD for NFS.

But, as we all know, media is a very special case. We deal in huge file sizes; we need very high sustained data rates; and we frequently need to manipulate growing files, to edit during ingest. The now venerable open-source tools really struggle with these demands.

That is not a criticism of SMB, which is a fantastic protocol. But the means of deployment can make the experience disappointing. For high performance media applications, you often need time-consuming and costly  additional development to work around the limitations and get close to the needed performance. And still we see brakes on productivity like editing and finishing on low res proxies then sitting and waiting while the full resolution version is rendered and output.

In short, the hardware is now really good and can readily support what we need. The communications software is a real bottleneck, and we need it to catch up and keep up.

There is one other really major problem we have to tackle if open source is used as part of the solution build. SMB and NFS do not play nicely with each other. In fact, they do not play at all – they have no knowledge of each other. If your infrastructure needs to use both – to connect high throughput servers to creative workstations, for example – then at best you are going to drown in permissions hell; more likely you are quickly going to get corruption and destruction. Not good.

At Tuxera we have been tackling this issue for more than a decade. Fusion SMB is a tailored, managed implementation of SMB, for all platforms, that delivers incredible speed: users today regularly see up to 10GB/s on MacOS and 11.5GB/s on Windows using a single 100Gbe port, and lab tests have already proved 250 GB/s to a single Linux workstation.

That is now joined by Fusion NFS, which delivers the same blistering performance – up to 22.7 GB/s over 200 gigabit ethernet, which simple arithmetic will tell you is close to the theoretical maximum. It does this in user mode: most speed claims for NFS are in kernel mode which does not include the vital security and resilience elements. User mode (also called user space), is a restricted environment where processes cannot directly access hardware or kernel memory, protecting the server.

The Fusion platform is engineered to allow SMB and NFS to co-exist in a multi-protocol environment, building and maintaining a single coherent, protected data structure for all storage arrays, operating systems and applications. However you build your hardware platform – even if you build your own servers – and your software stack, Fusion will support it, seamlessly and transparently. It is agnostic towards the version of Linux or the file system.

Whether you are designing a product to fit into the creative environment or integrating a complete system, you should be thinking about the detail of the communication protocol, not just the speed of the network.

As I said at the beginning, hardware is getting to be really good. Communication is the real limitation. Take, for example, modern cameras which connect using SMB. You can use a legacy, open source, “free” flavor of SMB, but any network stress will cause the stream to be interrupted and recording to be lost. Or you could use a tailored for media, powerful SMB tool to ensure the bandwidth is there to deliver your critical media at maximum throughput.

The business of creating media is under deep stress, to be better, quicker and cheaper. Developments in CPUs and GPUs, in storage architectures and network switching all have the potential to have a huge impact on those challenges. But they are not the whole answer: the implementation of the communications protocol may not sound like a sexy part of the design decision-making, but it is absolutely the key to making it all work. Workflow should not be restricted by your choice of protocol and Innovation cannot stop at the hardware.