Should I use data deduplication to manage unstructured data?

Absolutely; there are two areas where data deduplication, or single-instance storage, can help. Deduplication is fairly common today in backups

Requires Free Membership to View

using a variety of appliances or software. Since this is backup, however, it doesn't address the immediate problem of unstructured data growth at the source. There are deduplication products that can help at the source, but they're not as popular today because the added workload in identifying similar blocks or byte sequences, or files, depending on the level of deduplication you choose. From a source or production storage perspective, deduplication may impact performance, making the technology less appealing. This is why we see a lot of deduplication deployed at the backup level. See the article Data Deduplication Explained for more information.

@34164 Still, specific products, like email archiving tools, can help reduce the amount of data being stored, keeping just what you need from a policy perspective. The deduplication element of this approach allows you to shrink storage requirements. Deduplication is also appearing at the WAN level to reduce data volumes transferred between locations -- particularly when replicating data. Deduplication certainly isn't limited to unstructured data, but that's where deduplication can really shine.

Listen to the Unstructured data FAQ audiocast.

This was first published in March 2007

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.