Benefits and Best Practices
The video surveillance industry has evolved dramatically over the last 50 years. From the days of whirring VCRs, where individuals were literally, ‘caught on (VHS) tape,’ to the introduction of digital video recorders and spinning hard disks – and now, an era of network-based video surveillance and associated IT infrastructure. During this transition, the hard disk drive displaced VHS tape as the typical media for storing video surveillance footage. However, with the number of higher megapixel cameras on the rise and longer retention requirements, end-users are struggling to deal with the data and cost requirements associated with storing this amount of footage. In systems with long-term data retention, end-users can now use file-based tape storage such as LTO (Linear Tape Open) technology in combination with disk, to effectively meet their requirements in a cost effective manner without compromising on the quality of the video stored or the length of time the video footage is retained.
Influential factors driving demand for cameras
Legislation
The level of pertinent legislation and regulation varies both nationally and internationally. Typically in North America and Western Europe there is little legislation relating directly to the general installation or use of video surveillance equipment, however, specific industries such as critical national infrastructure or casinos and gaming may be subject to specific regulations.
Compliance with privacy and data protection laws tend to have a large influence on video surveillance installations and their data retention policies. In May 2017, the German government responded to an increase in security threats and the lack of video evidence available in public places including stations and city centers, by introducing new legislation altering current privacy laws to allow greater installation of video surveillance cameras.
Legislation and regulations are particularly influential in the Middle East. These can dictate which organizations must install video surveillance equipment, the level of coverage and include equipment specifications such as minimum camera resolutions, frame rates, and footage retention times. For example, in Qatar, hotels and shopping malls must store footage for a minimum of 120 days.
Multiple use
Today, video surveillance equipment has multiple uses beyond that of traditional security or loss-prevention applications such as:
- Monitoring, control and recording of business operations.
- Health and safety – crowd monitoring and control.
- The collection of consumer marketing information, especially in retail sector.
- Anti-litigation purposes for collecting potential video evidence in case of lawsuits.
- Traffic monitoring and enforcement systems.
The multiple uses of video surveillance systems has assisted in the propagation of the network surveillance camera as another Internet of Things (IoT) sensor which can be integrated into more complex and larger systems.
Decline in camera prices
Decrease in average selling prices of equipment is enabling end-users to obtain a larger return on investment. IHS Markit data shows that since 2012 the average price of a network surveillance camera in the American market has more than halved. Over the last five years, the composition of the competitive environment has changed significantly through the growth of many Chinese video surveillance vendors who typically sell cameras and recording devices at prices below their western competitors.
Higher specification of cameras
IHS Markit has measured uplift in the specification of cameras across several different parameters. Currently the most common network camera shipping is 1080p resolution but there will be continued growth in higher resolution cameras. Four megapixel cameras are increasing in popularity and 4K and even higher resolution cameras are expected to gain market share.
Various compression technologies have
been developed to mitigate some of the effects of an increase in camera image quality and the associated demands on network transmission, compute and storage. The typical compression used in the surveillance industry in 2017 is based on the H.264 AVC (MPEG-4 Part 10) standard. Many vendors have released innovative proprietary updates in the last 18 months often marketed as some form of ‘H.264+’ and ‘smart bandwidth management.’ The next generation H.265 (HEVC) compression is still some way off mass market adoption in the video surveillance industry. Despite advances in compression technologies demand for storage continues to increase.
Increased value of data & longer retention times
The vast majority of recorded video surveillance footage is never reviewed. As much detail as possible should be retained until it can be determined which parts are relevant or valuable. 
For security purposes the value usually relates to the recording of a security incident. However, the uncertainty of which parts of recorded video are going to be useful often means trade-offs between the quality at which video is stored and the length of time it is stored.
Typically, extending the retention time for video surveillance data is justified by the perceived increase in the value of video surveillance data. As end-users are realizing the full value of this data, they wish to retain it for longer periods. This may be due to a new application for video surveillance data, or a perceived security threat has increased. Sometimes, retention does not increase uniformly across a system. Strategically important or mission critical cameras may have a different policy to other cameras.
In general, retaining footage for longer increases the size of the available potential usable dataset for analysis.
Requirements specific to storing video surveillance footage
A different approach to traditional IT data storage
Video surveillance storage has some specific requirements and differs from traditional IT data storage in the following ways:
- Video surveillance storage workloads should be configured heavily towards writing data to storage media. Most stored footage is never accessed and is deleted without it being reviewed. Therefore, generally, the requirements to read data are much lower than the requirements to write. This is different to traditional IT workloads which are typically more mixed.
- Video surveillance workloads can be typified as requiring high bandwidth due to the need for multiple concurrent data streams to be recorded at any one time. The throughput should be designed to cope with these requirements particularly during periods of high activity which may cause spikes in bandwidth required. It is critical in events such as those where recording must be maintained from all video streams at the defined quality with no dropped frames and no data loss.
- Often a video surveillance storage system requires a high level of redundancy and failover. No system is 100% redundant. However, in the event of a failure and rebuild process, the system requires enough redundancy to continue performing its primary function i.e., the recording of video surveillance footage.
- Video footage should remain accessible to the video management software regardless of where it is stored. Data retained for long periods of time in separate silos can inhibit users if separate manual interaction is required for access. Keeping latency to a minimum for playback and post recording analytics should also be an objective.
- Where there is a low probability of data access but a need for long-term retention the energy costs of data storage should be a consideration. Treating all data with the same policy and storing all data, regardless of age, in the same manner can have ramifications for energy costs.
- Systems should be designed with the ability to scale and support future expansion in capacity with minimal disruption. Future expansion can come about due to change in: internal policy, legislation/ regulations or the video surveillance front-end (additional cameras or upgraded replacement cameras added to the system). Flexible systems where additional capacity is easy to add are required to cope with future changes such as these.
Video surveillance data as evidence
In addition to how a video surveillance workload can differ from traditional IT data storage, there are other considerations specific to video surveillance data, and that its use as evidence in civil or criminal court proceedings.
When storing video surveillance footage there is potential that the data may be used as evidence in court. However, it is not clear until post recording what data is going to be relevant. Therefore storing the best possible quality footage whilst maintaining a chain of custody should always be a primary concern. Reducing quality of footage for budgetary reasons can undermine the purpose of storing it in the first place.
When footage is required as potential evidence there are further considerations:
- Should evidence of a crime be collected, a complete chain of custody to ensure the video evidence has not been tampered with or doctored is required. Keeping data in a unified system, meaning the footage can be replayed by the server which recorded it, can help to keep the chain of custody.
- Chain of custody in the cloud is possible through the use of Criminal Justice Information Services (CJIS) compliant datacenters. However, data storage here can be subject to premium costs which are often negated in a total cost of ownership (TCO) analysis.
- Stored video footage should be able to be exported easily whilst maintaining the chain of custody.
- If a police department has stored evidential video surveillance footage that has been involved in a conviction or a serious crime, they are often required to store this evidence for the duration of any custodial sentence. This has the potential to increase the retention period to decades.
LTO technology and video surveillance
No one-size fits all approach to video surveillance storage
A huge range of storage requirements spans different video surveillance end-user groups. Therefore, treating all video surveillance data with a one-size fits all approach can lead to inefficiencies. The value and requirements for video surveillance data often changes over its lifetime – so configuring a storage solution with one type of storage and the same storage policy can be inefficient.
Commonly, when choosing a storage solution, a trade-off between budget and mission ensues. Often to meet a budget the quality of video surveillance footage or duration it is stored for is compromised. This does not always need to be the case. Often performance, capacity and retention can be balanced whilst minimising sacrifices of redundancy, access or scalability. Using multiple tiers of active storage can help achieve this.
In general IT and in some video surveillance storage systems where a two-tier approach of (active storage and archival) has been implemented, often the visibility and integration between the two tiers through a single file system and application has been problematic. Once data was moved to the archive tier it became inactive. For users to gain access IT staff had to be involved to manually retrieve the data. If using a tape archive, the entire contents were manually copied back to disk and it then needed manually searching through.
This approach is outdated as multiple tiers of active storage, across different storage media (disk, tape and flash) can be connected in one system. This means the characteristic of each storage tier is utilized to best fit to the video surveillance storage requirements and the entire system remains accessible by the user and any software applications.
File-based magnetic data tape storage can provide an important tier for the storage of video surveillance data in an active multi-tiered storage approach.
What is LTO technology and LTFS
Linear tape open (LTO) technology is an open format of magnetic tape data storage jointly developed by HPE, IBM and Quantum. In the non-video surveillance wider IT market, LTO technology is primarily used for data back-up, disaster recovery and long term data retention. It is often used for making multiple copies of data and for cold storage offsite in vaults. Since the 2010 release of the fifth generation, LTO technology has included the linear tape file system (LTFS). LTFS utilizes partitioning to create an index which points to the requested file positions on the tape. This means that data can be dragged-and-dropped and viewed in a similar way to hard disk drives.
The LTFS format provides a self-describing cartridge which means applications do not require any additional information to read the data stored. In the event of a disaster where loss of data occurs, data can be recovered from tape.
In a tape library configuration used for long-term retention the time to locate and start reading data is slower than a disk array. However, this is offset by the potential to offer tape as a lowercost, high-capacity alternative to hard disk drive arrays for longer term retention. Tape does not substitute disk as primary storage.
The LTO format has a published technology roadmap with future specifications.
What can LTO technology offer the video surveillance user?
When utilized in a multi-tiered storage solution, an LTO tape library can be implemented as a cost effective, connected, long-term, high-capacity storage tier. End-users can manage their storage system and remain within budget as tape can provide a larger capacity storage tier and longer term retention for the same cost. This means end-users can compromise less on video quality and retention time by using tape. Every detail can be stored until users can determine later what should be kept.
The utilization of tape to store video will not be best suited for all video surveillance applications. Multi-tiered storage works well with a high number of camera feeds which require extended storage retention at maximum quality. Data is transferred automatically between storage tiers based on pre-defined policies. The capacity of each tier and policy governing data transfer between them is typically based on the likelihood of requiring the highest performance. For example, the most recent video may be kept on the highest performing disk tiers in the initial 7-14 days post-recording when access is most likely to be required. After this, it is moved to a tape tier (at which point it is deemed unlikely to be required for review). However, the key point about this configuration is that data remains available through the VMS should review be required. In addition, any video flagged up in a security or operational event of interest may have a different policy and may be retained for longer. Where events are required to be kept long term or video evidence is used, a copy may be automatically transferred to a particular tier or even duplicated to offsite storage.
The approach is well suited to systems with large capacity and retention requirements for video surveillance.
Tape storage for high capacity, low use and long-term retention used in combination with disk provides lower total cost than all-disk or disk-cloud hybrids.
Using tape in a multi-tiered storage solution for long-term retention can mean that rather than pushing data to the cloud for storage, users can retain full ownership and access to their data for minimal cost. Cloud storage can be used as an effective storage tier, particularly in collating data from small camera count multiple site installations, or as an offsite backup of a small amount of mission critical camera feeds. However, as a storage tier for high capacities or long-term retention, often there are additional costs to cloud storage which reduce its viability; for example, high bandwidth costs, access and retrieval charges etc. Chain of custody is also a key concern as there are often significant premiums beyond a list price for the use of a government (for example CJIS) certified secure data center.
Total cost of ownership (TCO) analysis should be used in conjunction with performance analysis to make a purchase decision for storage. A balance between cost and performance is often necessary. In many scenarios the speed of data recall offered by a tape library is not sufficient and it should be discounted. Yet, for some users if they can trade off immediate, for near immediate access speeds to their oldest data, tape can bring high capacity, long-term retention storage, at price points they would otherwise have been unable to afford. Once the disk tier is configured further disk storage may only be required if additional bandwidth or camera streams are added. When only increasing the length of time data is retained building out capacity on tape can become even more economical as capacities scale.
In one example of a TCO analysis, when maintaining a storage system over a decade the comparison between a LTO tape library, an all-disk and a disk-cloud hybrid solutions found that the TCO for the LTO library solution was nearly seven times lower than the all disk solution and over five times lower than a disk-cloud hybrid solution.
As an approximate cost comparison of individual media, a recent LTO technology cost audit found an average retail price of $0.02/GB RAW, for a LTO generation 7 cartridge. IHS Markit’s HDD market tracker estimates a typical enterprise grade HDD price at $0.06/ GB.
Access all your recordings without leaving the VMS application. No need to think of tape as a separate archive.
The use of tape storage in the video surveillance industry has suffered from several misconceptions. One of the most prominent ones is that a tape library is disconnected from the rest of the system. When tapes are used in a modern multi-tier storage configuration the tier is not disconnected and can be accessed through a single file system shared between tiers. The file system is storage media agnostic and operates as ‘one drive.’ The tape tier will typically have 
slower performance than the disk storage tiers and this has caused problems with application access in the past. Some video management software (VMS) programs have search time-outs and if files cannot be found instantly may return errors or even move to a database repair mode which can impair performance. However, many current leading VMS programs are now directly able to access and control tape libraries with a plug-in. This enables users, without leaving their VMS application, access to a timeline of footage. A reference architecture and example workflow for the ‘batch pre-fetch’ plug-in is shown in Figure below.
The batch pre-fetch plugin unlocks the full value of tape storage in a video surveillance solution as it enables the user of the chosen VMS application a simple workflow to tap into a huge data pool within the application. Providing users with all recorded video in one place could make a critical difference in forensic investigations and video analytics. Often, older recordings stored in public cloud solutions and cold store libraries cannot be accessed and controlled directly from a VMS application. In these cases, to achieve the same level of functionality, a VMS user or IT personnel may manually have to recall data and import to the VMS. The batch pre-fetch plugin alleviates the additional time and cost and retains the metadata to connect to the full files.
Use cases
The use of tape storage in a multi-tiered video surveillance environment has been successfully deployed in city surveillance projects, is popular with police departments to store both fixed video surveillance cameras and body worn camera video and has been deployed in the private sector. The concepts described here could be suited to any video surveillance application with extended retention requirements which face budget challenges.
When choosing a video surveillance storage solution, analysis should go beyond basic capacity calculations. Integrators and end-users should assess how the usage and value of recorded footage can change over time and how retaining access to all data in a single system can change the TCO analysis. Despite some misconceptions, tape as an element of a modern storage solution can provide some specific cost benefits when used to build out capacity for long-term retention.
Conclusion
The evolution of the video surveillance industry is causing a video surveillance data explosion. The multiple uses of cameras and lower prices mean there are now a greater number of video surveillance cameras entering the market than ever before. The greater number of cameras and higher specification new models are producing increasing amounts of data. The length of time this data is being retained for is also increasing. These factors have placed greater demands on storage systems and on budgets to be capable of securely retaining large amounts of video surveillance data for extended periods.
The requirements of certain video surveillance environments for high capacity, long-term retention with infrequent recall of older footage mean file-based tape such as LTO technology paired with disk storage can provide an excellent value proposition as a video surveillance storage solution. LTO technology can offer users full access and control over their data with long-term retention and high reliability whilst keeping costs to a minimum. LTFS and the batch pre-fetch plugins, available for many leading VMS solutions, enable end-users to simply go back in time further within their preferred video management application, gaining valuable insights which may otherwise have been lost due to compromises on stored video quality or retention times.
By Josh Woodhouse – Senior Analyst, Video Surveillance, IHS Market
