‘Insane’ amounts of data drive new storage technology – BBC News

The year 2039 may seem far away, but Ian Crawford is already making plans for it.

It marks the centenary of the outbreak of the Second World War – a big year for its employer, the Imperial War Museum.

Mr. Crawford is Chief Information Officer at the museum and is overseeing a project to digitize its vast collection of photographs, audio and film.

With a collection of approximately 24,000 hours of film and video and 11 million photos, this is a huge task.

And leading up to 2039, World War II material will be a priority.

Making digital copies of these historical sources is critical because the original copies deteriorate over time and will one day be lost forever.

“If you have the only copy, you want to be confident that your storage system is reliable,” says Ian Crawford.

The amount of data required for such long-term storage is constantly growing, as the latest scanners can capture documents and films in great detail.

“The growth potential is truly enormous,” says Mr Crawford.

“We are now looking at objects themselves and scanning in 3D – that can produce very large files.”

This deluge of data isn’t just hitting museums, it’s pouring down everywhere.

Businesses are buying more space for backup data, hospitals need a place to store data, the government needs a place to store ever-increasing amounts of information.

“We continue to create crazy amounts of data,” said Simon Robinson, chief analyst at research firm Enterprise Strategy Group.

“For most organizations – it varies widely – their data volume doubles every four to five years. And in some sectors it is growing much faster than that,” he says.

Data that needs to be kept for a long time is not stored in traditional data centers, those huge warehouses, with racks of servers and flashing lights. These operations are for data that needs to be accessed and updated regularly.

Instead, the most popular way to preserve data long-term is to record it to tape. In particular, a format known as LTO (Linear Tape Open), the latest version of which is called LTO-9.

The tapes themselves don’t look much like old VHS tapes, but are slightly smaller and squarer.

The cassette contains one kilometer of magnetic tape, which can store 18 terabytes of data.

That’s a lot: just one tape can hold the same amount of data as almost 300 standard smartphones.

The Imperial War Museum in Duxford uses a tape system from Spectra Logic. The machine, about the size of a large wardrobe, can hold up to 1,500 LTO tires.

Such LTO systems dominate the long-term storage market. They have been around for decades and have proven to be reliable.

It’s also quite cheap, which is important because customers generally want to pay as little as possible for long-term storage.

Yet some are convinced that things can be better.

In a former wallpaper factory in Chiswick, west London, a start-up company has developed a long-term storage system that uses lasers to burn tiny holograms into a light-sensitive polymer.

CEO Charlie Gale points out that magnetic tape can only store data on the surface, while holograms can store data in multiple layers.

“You can do things called multiplexing, where you can stack multiple sets of information into one space. That’s really the superpower of what we do. And we believe we can fit more information into less space than ever before,” he says .

HoloMem’s polymer blocks can withstand extreme temperatures without damaging data – between -14C and 160C.

By comparison, magnetic tape must be kept between 16 and 25 degrees Celsius, which entails significant heating and cooling costs, especially in countries with extreme temperatures.

Tape also needs to be replaced after about 15 years, while the polymer lasts at least 50 years.

Mr. Gale notes that because the laser chemically changes the polymer, the data cannot be tampered with once it is written.

Holomem’s prototype system, which can store and retrieve data, will be ready later this year.

Mr Gale says the cost of the system has been kept low by using standard, off-the-shelf components, including the laser. So he is confident that HoloMem can match or even beat the cost of magnetic tape.

HoloMem will have to be competitive, as a formidable competitor is emerging on the market.

Through its research department, Microsoft is developing its own long-term data storage system.

Like HoloMem, it has decided it’s time to move away from magnetic tape, but Microsoft has chosen glass as its storage material.

The system is called Project Silica and uses high-powered lasers to create small structural changes in the glass, called voxels, which can be used to store data. The voxels are incredibly small and can be packed into layers.

Microsoft says a 2mm thick piece of glass, about the size of a DVD, could store more than seven terabytes of data.

The system stores the windows in racks, where they can be accessed by small crab-like robots that zip along rails.

Cheap and durable, glass is an attractive storage medium, says Richard Black, head of Project Silica.

“It is virtually immune to temperature, humidity, particles and electromagnetic fields,” says Mr Black.

It could potentially retain data for hundreds and perhaps thousands of years.

Such a system could one day be integrated into Microsoft’s massive cloud computing business, Azure.

But that is still a long way off, as the system still has years of development ahead of it.

Back in Duxford, the Imperial War Museum, like many other organisations, has been experimenting with artificial intelligence. They recently tested whether AI could identify different models of Spitfire in images from the image catalog.

Mr Crawford thinks AI could be incredibly useful in cataloging his digital library, work that would take humans hundreds of years to do.

AI’s ability to sift through vast amounts of data has made preserving that data even more important; something valuable could be lurking there.

“In the past, companies would archive data in case they needed it. Now there’s a real business reason why they might want to go back and do some analysis,” says Mr Robinson.