LTO Technology & Video Surveillance


Benefits and Best Practices


The video surveillance industry has evolved dramatically over the last 50 years. From the days of whirring VCRs, where individuals were literally, ‘caught on (VHS) tape,’ to the introduc­tion of digital video recorders and spin­ning hard disks – and now, an era of network-based video surveillance and associated IT infrastructure. During this transition, the hard disk drive displaced VHS tape as the typical media for stor­ing video surveillance footage. Howev­er, with the number of higher megapixel cameras on the rise and longer retention requirements, end-users are struggling to deal with the data and cost requirements associated with storing this amount of footage. In systems with long-term data retention, end-users can now use file-based tape storage such as LTO (Linear Tape Open) technology in combination with disk, to effectively meet their re­quirements in a cost effective manner without compromising on the quality of the video stored or the length of time the video footage is retained.

Influential factors driving demand for cameras

Legislation

The level of pertinent legislation and regulation varies both nationally and internationally. Typically in North America and Western Europe there is little legislation relating directly to the general installation or use of video sur­veillance equipment, however, specific industries such as critical national infra­structure or casinos and gaming may be subject to specific regulations.

Compliance with privacy and data protection laws tend to have a large influ­ence on video surveillance installations and their data retention policies. In May 2017, the German government responded to an increase in security threats and the lack of video evidence available in public places including stations and city centers, by introducing new legislation altering current privacy laws to allow greater in­stallation of video surveillance cameras.

Legislation and regulations are par­ticularly influential in the Middle East. These can dictate which organizations must install video surveillance equip­ment, the level of coverage and include equipment specifications such as mini­mum camera resolutions, frame rates, and footage retention times. For example, in Qatar, hotels and shopping malls must store footage for a minimum of 120 days.

Multiple use

Today, video surveillance equipment has multiple uses beyond that of traditional security or loss-prevention applications such as:

  • Monitoring, control and recording of business operations.
  • Health and safety – crowd monitoring and control.
  • The collection of consumer market­ing information, especially in retail sector.
  • Anti-litigation purposes for collect­ing potential video evidence in case of lawsuits.
  • Traffic monitoring and enforcement systems.

The multiple uses of video surveil­lance systems has assisted in the propa­gation of the network surveillance cam­era as another Internet of Things (IoT) sensor which can be integrated into more complex and larger systems.

Decline in camera prices

Decrease in average selling prices of equipment is enabling end-users to obtain a larger return on investment. IHS Markit data shows that since 2012 the average price of a network surveillance camera in the American market has more than halved. Over the last five years, the composition of the competitive environ­ment has changed significantly through the growth of many Chinese video sur­veillance vendors who typically sell cameras and recording devices at prices below their western competitors.

Higher specification of cameras

IHS Markit has measured uplift in the specification of cameras across several different parameters. Currently the most common network camera ship­ping is 1080p resolution but there will be continued growth in higher resolution cameras. Four megapixel cameras are in­creasing in popularity and 4K and even higher resolution cameras are expected to gain market share.

Various compression technologies haveHigher specification of cameras been developed to mitigate some of the effects of an increase in camera im­age quality and the associated demands on network transmission, compute and storage. The typical compression used in the surveillance industry in 2017 is based on the H.264 AVC (MPEG-4 Part 10) standard. Many vendors have released innovative proprietary updates in the last 18 months often marketed as some form of ‘H.264+’ and ‘smart bandwidth man­agement.’ The next generation H.265 (HEVC) compression is still some way off mass market adoption in the video surveillance industry. Despite advances in compression technologies demand for storage continues to increase.

Increased value of data & longer retention times

The vast majority of recorded video surveillance footage is never re­viewed. As much detail as possible should be retained until it can be deter­mined which parts are relevant or valua­ble. Increased value of dataFor security purposes the value usu­ally relates to the recording of a security incident. However, the uncertainty of which parts of recorded video are go­ing to be useful often means trade-offs between the quality at which video is stored and the length of time it is stored.

Typically, extending the retention time for video surveillance data is jus­tified by the perceived increase in the value of video surveillance data. As end-users are realizing the full value of this data, they wish to retain it for longer periods. This may be due to a new ap­plication for video surveillance data, or a perceived security threat has increased. Sometimes, retention does not increase uniformly across a system. Strategi­cally important or mission critical cam­eras may have a different policy to other cameras.

In general, retaining footage for longer increases the size of the available potential usable dataset for analysis.

Requirements specific to storing video surveillance footage

A different approach to traditional IT data storage

Video surveillance storage has some specific requirements and differs from traditional IT data storage in the following ways:

  • Video surveillance storage workloads should be configured heavily towards writing data to storage media. Most stored footage is never accessed and is deleted without it being reviewed. Therefore, generally, the require­ments to read data are much lower than the requirements to write. This is different to traditional IT workloads which are typically more mixed.
  • Video surveillance workloads can be typified as requiring high bandwidth due to the need for multiple concur­rent data streams to be recorded at any one time. The throughput should be designed to cope with these require­ments particularly during periods of high activity which may cause spikes in bandwidth required. It is critical in events such as those where recording must be maintained from all video streams at the defined quality with no dropped frames and no data loss.
  • Often a video surveillance storage system requires a high level of re­dundancy and failover. No system is 100% redundant. However, in the event of a failure and rebuild pro­cess, the system requires enough re­dundancy to continue performing its primary function i.e., the recording of video surveillance footage.
  • Video footage should remain accessi­ble to the video management software regardless of where it is stored. Data retained for long periods of time in separate silos can inhibit users if sep­arate manual interaction is required for access. Keeping latency to a mini­mum for playback and post recording analytics should also be an objective.
  • Where there is a low probability of data access but a need for long-term retention the energy costs of data storage should be a considera­tion. Treating all data with the same policy and storing all data, regardless of age, in the same manner can have ramifications for energy costs.
  • Systems should be designed with the ability to scale and support future expansion in capacity with minimal disruption. Future expansion can come about due to change in: inter­nal policy, legislation/ regulations or the video surveillance front-end (additional cameras or upgraded replacement cameras added to the system). Flexible systems where ad­ditional capacity is easy to add are required to cope with future changes such as these.

Video surveillance data as evidence

In addition to how a video surveillance workload can differ from traditional IT data storage, there are other consid­erations specific to video surveillance data, and that its use as evidence in civil or criminal court proceedings.

When storing video surveillance footage there is potential that the data may be used as evidence in court. How­ever, it is not clear until post recording what data is going to be relevant. There­fore storing the best possible quality footage whilst maintaining a chain of custody should always be a primary con­cern. Reducing quality of footage for budgetary reasons can undermine the purpose of storing it in the first place.

When footage is required as poten­tial evidence there are further considera­tions:

  • Should evidence of a crime be col­lected, a complete chain of custody to ensure the video evidence has not been tampered with or doctored is required. Keeping data in a unified system, meaning the footage can be replayed by the server which record­ed it, can help to keep the chain of custody.
  • Chain of custody in the cloud is pos­sible through the use of Criminal Justice Information Services (CJIS) compliant datacenters. However, data storage here can be subject to premium costs which are often ne­gated in a total cost of ownership (TCO) analysis.
  • Stored video footage should be able to be exported easily whilst main­taining the chain of custody.
  • If a police department has stored evidential video surveillance footage that has been involved in a convic­tion or a serious crime, they are often required to store this evidence for the duration of any custodial sentence. This has the potential to increase the retention period to decades.

LTO technology and video surveillance

No one-size fits all approach to video surveillance storage

A huge range of storage require­ments spans different video sur­veillance end-user groups. Therefore, treating all video surveillance data with a one-size fits all approach can lead to inefficiencies. The value and require­ments for video surveillance data often changes over its lifetime – so configur­ing a storage solution with one type of storage and the same storage policy can be inefficient.

Commonly, when choosing a storage solution, a trade-off between budget and mission ensues. Often to meet a budget the quality of video surveillance footage or duration it is stored for is compro­mised. This does not always need to be the case. Often performance, capac­ity and retention can be balanced whilst minimising sacrifices of redundancy, ac­cess or scalability. Using multiple tiers of active storage can help achieve this.

LTO Technology RoadmapIn general IT and in some video sur­veillance storage systems where a two-tier approach of (active storage and ar­chival) has been implemented, often the visibility and integration between the two tiers through a single file system and application has been problematic. Once data was moved to the archive tier it be­came inactive. For users to gain access IT staff had to be involved to manually retrieve the data. If using a tape archive, the entire contents were manually copied back to disk and it then needed manually searching through.

This approach is outdated as multi­ple tiers of active storage, across differ­ent storage media (disk, tape and flash) can be connected in one system. This means the characteristic of each storage tier is utilized to best fit to the video sur­veillance storage requirements and the entire system remains accessible by the user and any software applications.

File-based magnetic data tape stor­age can provide an important tier for the storage of video surveillance data in an active multi-tiered storage approach.

What is LTO technology and LTFS

Linear tape open (LTO) technology is an open format of magnetic tape data storage jointly developed by HPE, IBM and Quantum. In the non-video surveillance wider IT market, LTO tech­nology is primarily used for data back-up, disaster recovery and long term data retention. It is often used for mak­ing multiple copies of data and for cold storage offsite in vaults. Since the 2010 release of the fifth generation, LTO tech­nology has included the linear tape file system (LTFS). LTFS utilizes partition­ing to create an index which points to the requested file positions on the tape. This means that data can be dragged-and-dropped and viewed in a similar way to hard disk drives.

The LTFS format provides a self-describing cartridge which means ap­plications do not require any additional information to read the data stored. In the event of a disaster where loss of data occurs, data can be recovered from tape.

In a tape library configuration used for long-term retention the time to lo­cate and start reading data is slower than a disk array. However, this is offset by the potential to offer tape as a lowercost, high-capacity alternative to hard disk drive arrays for longer term retention. Tape does not substitute disk as primary storage.

The LTO format has a published technology roadmap with future speci­fications.

What can LTO technology offer the video surveillance user?

When utilized in a multi-tiered storage solution, an LTO tape library can be implemented as a cost effective, connected, long-term, high-capacity storage tier. End-users can manage their storage system and remain within budget as tape can provide a larg­er capacity storage tier and longer term retention for the same cost. This means end-users can compromise less on video quality and retention time by using tape. Every detail can be stored until users can determine later what should be kept.

The utilization of tape to store vid­eo will not be best suited for all video surveillance applications. Multi-tiered storage works well with a high number of camera feeds which require extend­ed storage retention at maximum qual­ity. Data is transferred automatically between storage tiers based on pre-defined policies. The capacity of each tier and policy governing data transfer between them is typically based on the likelihood of requiring the highest performance. For example, the most recent video may be kept on the high­est performing disk tiers in the initial 7-14 days post-recording when access is most likely to be required. After this, it is moved to a tape tier (at which point it is deemed unlikely to be required for review). However, the key point about this configuration is that data remains available through the VMS should re­view be required. In addition, any video flagged up in a security or operational event of interest may have a different policy and may be retained for longer. Where events are required to be kept long term or video evidence is used, a copy may be automatically transferred to a particular tier or even duplicated to offsite storage.

The approach is well suited to sys­tems with large capacity and retention requirements for video surveillance.

Tape storage for high capacity, low use and long-term retention used in combination with disk provides lower total cost than all-disk or disk-cloud hybrids.

Using tape in video surveillance storage - exampleUsing tape in a multi-tiered storage solution for long-term retention can mean that rather than pushing data to the cloud for storage, users can retain full ownership and access to their data for minimal cost. Cloud storage can be used as an effective storage tier, particu­larly in collating data from small cam­era count multiple site installations, or as an offsite backup of a small amount of mission critical camera feeds. How­ever, as a storage tier for high capacities or long-term retention, often there are additional costs to cloud storage which reduce its viability; for example, high bandwidth costs, access and retrieval charges etc. Chain of custody is also a key concern as there are often signifi­cant premiums beyond a list price for the use of a government (for example CJIS) certified secure data center.

Total cost of ownership (TCO) analysis should be used in conjunction with performance analysis to make a purchase decision for storage. A bal­ance between cost and performance is often necessary. In many scenarios the speed of data recall offered by a tape library is not sufficient and it should be discounted. Yet, for some users if they can trade off immediate, for near im­mediate access speeds to their oldest data, tape can bring high capacity, long-term retention storage, at price points they would otherwise have been unable to afford. Once the disk tier is config­ured further disk storage may only be required if additional bandwidth or camera streams are added. When only increasing the length of time data is retained building out capacity on tape can become even more economical as capacities scale.

In one example of a TCO analysis, when maintaining a storage system over a decade the comparison between a LTO tape library, an all-disk and a disk-cloud hybrid solutions found that the TCO for the LTO library solution was nearly sev­en times lower than the all disk solution and over five times lower than a disk-cloud hybrid solution.

As an approximate cost comparison of individual media, a recent LTO tech­nology cost audit found an average re­tail price of $0.02/GB RAW, for a LTO generation 7 cartridge. IHS Markit’s HDD market tracker estimates a typical enterprise grade HDD price at $0.06/ GB.

Access all your recordings without leaving the VMS application. No need to think of tape as a separate archive.

The use of tape storage in the video surveillance industry has suffered from several misconceptions. One of the most prominent ones is that a tape library is disconnected from the rest of the sys­tem. When tapes are used in a modern multi-tier storage configuration the tier is not disconnected and can be accessed through a single file system shared be­tween tiers. The file system is storage media agnostic and operates as ‘one drive.’ The tape tier will typically have Reference architectureslower performance than the disk stor­age tiers and this has caused problems with application access in the past. Some video management software (VMS) pro­grams have search time-outs and if files cannot be found instantly may return er­rors or even move to a database repair mode which can impair performance. However, many current leading VMS programs are now directly able to access and control tape libraries with a plug-in. This enables users, without leaving their VMS application, access to a timeline of footage. A reference architecture and ex­ample workflow for the ‘batch pre-fetch’ plug-in is shown in Figure below.

The batch pre-fetch plugin unlocks the full value of tape storage in a video surveillance solution as it enables the user of the chosen VMS application a simple workflow to tap into a huge data pool within the application. Pro­viding users with all recorded video in one place could make a critical dif­ference in forensic investigations and video analytics. Often, older recordings stored in public cloud solutions and cold store libraries cannot be accessed and controlled directly from a VMS applica­tion. In these cases, to achieve the same level of functionality, a VMS user or IT personnel may manually have to recall data and import to the VMS. The batch pre-fetch plugin alleviates the additional time and cost and retains the metadata to connect to the full files.

Use cases

The use of tape storage in a multi-tiered video surveillance environ­ment has been successfully deployed in city surveillance projects, is popular with police departments to store both fixed video surveillance cameras and body worn camera video and has been deployed in the private sector. The con­cepts described here could be suited to any video surveillance application with extended retention requirements which face budget challenges.

When choosing a video surveil­lance storage solution, analysis should go beyond basic capacity calculations. Integrators and end-users should assess how the usage and value of recorded footage can change over time and how retaining access to all data in a single system can change the TCO analysis. Despite some misconceptions, tape as an element of a modern storage solu­tion can provide some specific cost ben­efits when used to build out capacity for long-term retention.

Conclusion

The evolution of the video surveil­lance industry is causing a video surveillance data explosion. The mul­tiple uses of cameras and lower prices mean there are now a greater number of video surveillance cameras enter­ing the market than ever before. The greater number of cameras and higher specification new models are producing increasing amounts of data. The length of time this data is being retained for is also increasing. These factors have placed greater demands on storage systems and on budgets to be capable of securely retaining large amounts of video surveillance data for extended periods.

The requirements of certain video surveillance environments for high capacity, long-term retention with in­frequent recall of older footage mean file-based tape such as LTO technol­ogy paired with disk storage can pro­vide an excellent value proposition as a video surveillance storage solution. LTO technology can offer users full access and control over their data with long-term retention and high reliabil­ity whilst keeping costs to a minimum. LTFS and the batch pre-fetch plugins, available for many leading VMS solu­tions, enable end-users to simply go back in time further within their pre­ferred video management application, gaining valuable insights which may otherwise have been lost due to com­promises on stored video quality or re­tention times.


By Josh Woodhouse – Senior Analyst, Video Surveillance, IHS Market

 

 

 

By Josh Woodhouse – Senior Analyst, Video Surveillance, IHS Market


Click for Magazine version

Leave a Reply