The Good, The Bad, and The Ugly of Metadata

The Good

Back in 2001, I was leading a project on building a knowledge management system called SenseViewer for the Russian national parliament. The members of parliament (MPs) were working on a new criminal procedural code, and SenseViewer allowed them to manage this huge amount of information.

There were 3 competing legislative bills submitted by different groups of MPs. Each bill was accompanied by tons of associated information: opinion and recommendations of experts from Russia and European Parliament, court rulings in Russia, Europe, and USA, similar legislation in other countries, etc. SenseViewer allowed users to find in each bill of the code, for every article (or even a part of the article), all semantically categorized information associated with this article. Moreover, they could track how exactly that information was associated with the proposed piece of legislation.

For example, you could pick up an article in the code and view all expert opinions on that article accompanied by categorized and labeled pro and contra arguments, references to the relevant legislation in other countries, contradictions, relevant court rulings, and so on. You could also see how biased an expert was by accessing his or her background and navigating through other comments that the expert provided for other articles or bills.

SenseViewer was able to automatically identify how different pieces of information relate to each thanks to a semantic mark up (for example, that a certain piece of content contains an argument that supports a certain piece of the proposed legislation). The markup was applied on a very granular level (down to a paragraph or part of a paragraph) in a way that allowed you to smartly navigate through a huge amount of knowledge while differentiating between:

• Facts and opinions
• Arguments that support a certain point of view and the ones that suggested an alternative point of view
• Different types of legal information (legislation, court decisions, court opinions, bills, etc.)
• Content created by an author and information about the author
• …and so on.

This kind of navigation was a must-have for making high quality decisions because unless one can gain a complete knowledge about the domain, the one cannot really make any good decisions.

After the implementation was completed, and MPs began to actually use SenseViewer, one of the users told me that the system represented a danger for those MPs who offered low quality legislative bills. When the information is well-structured, and you can clearly see how different pieces of information relate to each other, you can easily discover contradictions, gaps, and disbalance, so the author’s incompetence could be instantly unveiled. In one instance, it went so far that one of the law enforcement agencies had to withdraw some of their suggestions after realizing (thanks to SenseViewer) how weak their arguments were.

The Bad

The Cambridge Analytica scandal showed that collecting publicly available user’s profile, page likes, and location – that is, in fact, metadata – can be as harmful as getting access to your sensitive personal information.

The problem is that in most countries, including USA, privacy protections against government surveillance are mostly applied to communications content.

In other words, law enforcement agencies need to follow strict procedures to obtain a permission to monitor the contents of your phone conversations.

But there’s also phone call metadata.

It’s the information about the calls you are placing and receiving: time, duration, to whom you called, who called you, etc. By contrast with requesting a permission to monitor the contents of your phone calls, requirements to monitor metadata are mostly left to the discretion of authorities. In the US, a law enforcement officer can request phone calling records merely with a subpoena.

Do you think that nothing sensitive can be retrieved from the metadata? Well, think again. If you combine information you retrieved from the metadata with information you can get about people via social media, it may go far beyond the date and time of your call.

The guys from the Stanford university did a research to investigate the privacy properties of phone metadata. They’ve found that “telephone metadata is densely interconnected, can trivially be reidentified, enables automated location and relationship inferences, and can be used to determine highly sensitive traits.”

For example, just by analyzing calls metadata and information retrieved from public sources, they could identify that one caller had a cardiac arrhythmia and another caller owned an AR rifle (this information has been later confirmed by the callers).

The metadata of the former caller’s calls showed that the caller received a long phone call from the cardiology group at a regional medical center, talked with a medical laboratory, answered several short calls from a local drugstore, and made brief calls to a self-reporting hotline for a cardiac arrhythmia monitoring device.

The call metadata of the latter caller showed that the caller frequently called to a local firearm dealer that prominently advertises a specialty in the AR semiautomatic rifle platform. The caller also placed lengthy calls to the customer support hotline for a major firearm manufacturer; the manufacturer produces a popular AR line of rifles.

And that has been discovered without having to obtain any permissions from the authorities on 100% legal grounds.

The researches have concluded that “telephone metadata is extraordinarily sensitive, especially when paired with a broad array of readily available information. For a randomly selected telephone subscriber, over a short period, drawing these sorts of sensitive inferences may not be feasible. However, over a large sample of telephone subscribers, over a lengthy period, it is inevitable that some individuals will expose deeply sensitive information. It follows that large-scale metadata surveillance programs, like the NSA’s, will necessarily expose highly confidential information about ordinary citizens.”

You can read the full report here.

The Ugly (with solutions, though)

When used right, metadata can increase the value of your content and help you deliver it to the right people at the right time.

There’s a problem, though. Someone has to add metadata to the content. This is where challenges usually begin to come up:

1. Authors may forget to add metadata.

2. Authors may provide incomplete metadata.

3. Authors may provide poor or incorrect metadata.

The last issue is especially critical because if you add poor metadata to good content, you will likely get poor results because the content will become either undiscoverable or irrelevant for a specific user’s context. So no one will appreciate how great your content is if it’s not found when it should be found or if it’s found but doesn’t match the user’s situation.

Partially, these issues can be addressed by introducing a metadata model. Metadata model contains metadata fields and specifies values for each field. It also defines mandatory fields and validation rules. A metadata model associated with content enables you to enforce metadata consistency and validity. If a metadata field isn’t filled out or an invalid value is provided, the author will get a validation error.

(From the technical perspective, if you are in DITA, subject scheme maps are a good way to implement a metadata model.)

However, a metadata model addresses the issue only partially. It can ensure technical validity of the content metadata (that is issues 1 and 2 from the list above), but alone, it cannot guarantee semantic quality. Suppose, for example, that you’ve written a troubleshooting procedure and now need to provide metadata. The metadata model may define that you must specify the product name, the subject, and if the subject is troubleshooting, you also need to define the type of the issue. If any of these metadata is not provided, you’ll get a validation error.

So the metadata model definitely helps here to make your metadata consistent and valid. However, the metadata model alone doesn’t know whether the subject and type of the issue you selected for this troubleshooting procedure adequately reflect the contents. In other words, if you selected a wrong subject or type of the issue, from the perspective of the metadata model, the metadata is still fine because the metadata model only cares about filling out mandatory fields and setting metadata values from a certain range.

This is when automation comes into the play. Natural language processing (NLP) technologies can analyze content written in a natural language and identify subjects and categories the content describes. While the metadata model defines what metadata needs to be specified, the NLP identifies how this metadata should be set.

This could sound good, but there’re a few things that we need to keep in mind.

First, It’s still too risky to blindly rely on the technology and just believe that everything NLP identifies is correct. So the question is how we can verify the semantic correctness of the metadata (which, by the way applies to the case when metadata is manually set by the author). One of the approaches that we are working on right now is to use performance of the content in search results. We have to monitor how often the content is found when it’s supposed to be found vs. how often the content is NOT found when it’s supposed to be found vs. how often the content is found when it’s NOT supposed to be found. The relationship between these factors defines the semantic quality of the metadata.

Second, don’t think of NLP as an almighty technology that can allow you to retrieve good semantics from a poorly written piece of content. The efficiency of the outcome produced by the NLP depends on the quality of the content. Garbage in, garbage out. The quality of the NLP will be far higher when parsing a well-structured, well written content with a good writing style than when analyzing a poorly written content.

Maybe in a distant future, the NLP and AI will be able to analyze any content regardless of how well it’s written, but at this point, you have to help the NLP help you to produce good results.

In one of the next posts, I’ll be sharing some insights on what NLP technologies look at when analyzing texts and how you can improve the quality of the text in order to improve the efficiency of the NLP.