EMERSE NLP Guide

Our NLP system is implemented as part of Solr. This means, when you send a document to Solr, Solr will tokenize the document, and index those tokens, thus making it searchable. Documents in Solr consist of many fields, only one of which is the text of the document, and the others are metadata fields, such as when the document was written, the patient it is for etc. Each such field needs a description of how Solr should process it, in particular, how it should be tokenized. This is what the field’s type is for.

Fields a document can have, plus their types, are all defined in the managed-schema XML file in the core’s conf directory in the index. A declation of a field type look like this:

Once you have the type, you can make zero or more fields actually use that type, like so:

The main work in running NLP in Solr is defining the <analyzer type="index"> element in the field type. The analyzer with type="index", called the index analyzer, is used when indexing documents, which is what we primarily talk about in this document, but the other analyzer is the query analyzer. We will talk about the query analyzer here.

However, the class of the field type is important too: you must use org.emerse.solr.TextWithHeaderFieldType instead of the more normal solr.TextField if your tokenizer has hasHeader="true" set. (See below for more info.)

When you send a header with the document text, you are giving the analyzer additional information it needs for tokenization and indexing. However, that header is not really part of the document, and so shouldn’t be stored and presented as if it were. In order to present the header to the analyzer but not store it, we needed a custom class for the field type.

However, we do actually need to store the header in order to support document updates, where you may update the metadata of the document, but not change the text. This is implemented as a delete-and-insert in Solr, meaning it synthesizes a new document based on the existing document and the update you wish to make to fields of the document. Thus, we need a place to store the header, so we can reconstruct the header-plus-document-text to present it to the analyzer. This place is a field named NLP_HEADER and must have exactly the definition:

Finally, and importantly, if the field type has this custom class, you must always have a header in every document.

For quick reference, a document with a header attached looks like:

The header is the first line. See here for more details. It’s possible to extend the header to include custom NLP tokens, as covered below.

One of the simpler examples of a field type of the document text field is:

In the rest of the document, whenever you see a <analyzer type="index"> element or when talk about an analyzer, analysis chain, or a pipeline, it is this index analyzer element in the field type for the document text’s field that we are talking about.

EMERSE uses a custom Lucene Analyzer to run NLP over the text, and produce tokens for indexing. Ordinarily, tokens are just words of the text, sometimes stemmed, lowercased, etc, depending on how you want search terms to match the text. However in EMERSE, we have multiple tokens per word of text, with different tokens conveying different information about the text. So as to prevent ambiguity, tokens that convey the words of the text itself are prefixed with either TX_ or tx_. Tokens that convey other things are not allowed to start with these prefixes.

Other than tokens that refer to the words of the text itself, there are two broad categories of tokens:

The inclusion of these two types of tokens is the contribution of the NLP system to the search index, and enables all the NLP search features in EMERSE to work.

Each token has an range of characters it refes to in the text. Similarly, each token has a range of "positions" it refers to. These "positions" are conceptually positions of the words of the text, with the first word in the text at position one, second word of the text at position two, etc. Tokens that refer to words of the text itself only span one position, but NLP tokens can span multiple positions.

Positions are determine how search phrases match. If a phrase such as alzheimers disease is to match, the end position of the token tx_alzheimers must be one less than the start position of the token tx_disease.

Words determine what the positions are, and those same words have character ranges as well, so we want any token than spans a certain set of positions to have the corresponding range of characters. However, NLP systems generally only output character ranges, and they don’t necessarily correspond exactly with the way we break up words in our system. For this reason, we reconcile the character offsets of the output of NLP with the tokenization of words in Solr, assigning position ranges based on the character ranges, and clipping character ranges to the character ranges of those corresponding positions. (Not all characters need to be in the character range of a position; whitespace or punctuation is often not part of any "word" and hence not part of character ranges of tokens, unless those tokens span multiple words and those characters are between those words.)

However, in assigning positions to NLP token based on their character ranges, we can make a choice: we can either have a single token in the index associated with multiple positions, or we duplicate that token once for each such position, so that each token spans only a single position. Conventionally, we duplicate the token for annotations, but have the token span many positions for entities.

Here is an example to conceptually illustrate the Token-Aligned Layered Index using a sentence: This is not a teddy bear.

There are four layers in this index. Text tokens are in the first row (the prefix omitted). The following rows contain NLP tokens that are aligned with text tokens. The second row are the annotations, third row has an entity which is a concept from UMLS, and the four row has an entity which is a semantic group. Each concept is assigned one or more semantic groups, making semantic groups a very large category of concepts. More information of semantic group can be found here.

A Lucene analysis chain, or the "NLP pipeline" as we call it, comprises:

CharFilters work mainly at the character level of the text, effectively pre-processing the text so what is seen down the line is in more of a normal form. For instance, it can cause the pipeline to effectively treat special characters as other more normal characters or to ignore parts of the text that correspond to HTML markup.

The tokenizer decides what the words and sentences are in the text.

The token filters do the NLP work, generally adding additional tokens to the stream, though it is possible for them to remove tokens as well. The order of token filters doesn’t matter, unless we otherwise note that it does.

Here is a possible implementation of the EMERSE NLP pipeline:

We’ll go over the options for each of these.

So far, we’ve discussed the index analyzer. What about the query analyzer?

The query analyzer is used when analyzing the text of user’s queries, so that the words users type into term bundles in EMERSE are broken into the same tokens as for indexing, that way they match up and documents can be found. Howveer, the query analyzer doesn’t do NLP, since the user isn’t entering general text, so the analysis chain needs to be different, which is why there are separate analyzers for index and query.

You should use the same char filters and tokenizer as the main index analyzer, but use none of the token filters. Instead, use only the org.emerse.nlp.TokenAdjusterFilterFactory token filter. The token adjuster filter factory appends the TX_ or tx_ prefixes to words, except when it appears the word already has one of the reserved prefixes, as given in the passed attributed.

To facilitate Solr indexing, a mandatory component is the Analyzer chain (pipeline). Similarly, for Solr queries, an analyzer chain must be defined. Since EMERSE offers flexibility to customize the NLP pipeline, it’s crucial to emphasize that the tokens extracted from a query phrase should precisely match those generated by the analyzer during the Solr indexing process.

Here is the query analyzer that EMERSE recommends:

org.emerse.nlp.TokenAdjusterFilterFactory is designed to work with the OpenNLPTokenizer created by solr.OpenNLPTokenizerFactory only. Failing to use the specified tokenizer will lead to unexpected results, which may impact the functionality of EMERSE.

Alternatively:

Prior to version 7, EMERSE includes MappingCharFilter in both indexing pipeline and query pipeline. The mapping file, mapping-delimiters.txt, may contain the removing of dots from various character sets, for example:

This type of mapping should be removed in version 7, as detecting a dot is a common method for NLP to identify sentence boundaries.

Model files are in a subdirectory named models. By default, it includes OpenNLP trained token, sentence and POS models for English. If the documents are in a different language, models may be available on OpenNLP website.

Sentiment detection in EMERSE relies on pattern matching. Newer technologies, such as convolutional neural networks (CNNs), were not adopted primarily due to their impact on indexing performance. To efficiently index large volumes of notes within a reasonable timeframe, pattern matching remains a viable solution.

EMERSE can detect negation, uncertainty, non-patient mentions, and patient history. Some detection types involve two pattern files: one for the detection patterns and another for overriding these patterns to address false positives. For example, negations.txt versus negationIgnore.txt, history.txt versus historyIgnore.txt. Here is a snippet of negation patterns:

The text in each line, i.e. couldn’t be, is a trigger phrase to activate negation detection. The 4 asterisks * represent a placeholder for anything that is being negated. * can be placed either before or after the trigger.

Negation-Ignore file, may include:

For the given negation pattern * couldn’t be, if the sentence is: Pneumonia can not be excluded, negation will not be concluded due to the negation ignore trigger couldn’t be excluded. Be aware, ignore file contains trigger only.

Uncertainty only implements a pattern file probability_resource_patterns.txt and non-patient mention, on the other hand, has 3 pattern files, among which, family_history_resource_patterns_A.txt is a regular pattern file, family_history_resource_patterns_B.txt is a template file and @@@@@ in each line can be replaced by the familial terms defined in family_history_resource_patterns_C.txt

If a mention is identified through a given rule, EMERSE will tag tokens with the corresponding attribute ID before and/or after the token pattern in the rule until the sentence boundary is reached. The direction is defined in conjunction with the rule as either prepending *****, indicating movement towards the beginning of the sentence, or appending *****, indicating movement towards the end of the sentence. Additionally, tokens matching the rule will be tagged with a distinct attribute ID specifying the rule.:

If a negation is provided as not *****, every token after not will be tagged as negation (POL_NEG as the attribute ID) and not itself will be tagged as the "trigger" for this negation (PPOL_NEG as the attribute ID). Tokens This is will not be touched since ***** is after not, which means only the tokens after the trigger will be considered as negated.

From the above example, it is evident that a set of attributes is required to describe the occurrence of a mention.

dictionary subdirectory contains the UMLS dictionary tailed by EMERSE using CTAKES UI. The dictionary file is a SQL file since that’s the default output from CTAKES. EMERSE parses this SQL file and extracts all concepts in conjunction with their groups. Entity recognition through using the xref:[ctakes-token-filter] requires a script file built through CTAKES Dictionary Creator. (As of 2024 this program still required Java 8 to run properly).

The EMERSE team has a default dictionary file to distribute to sites installing EMERSE, or upgrading from prior versions that are not NLP-enabled. Sites simply need to provide to us with a copy of a valid UMLS license first.

For those wishing to make their own dictionary, users can create a custom dictionary and specify the path of the script file for OpenNLPTokenizerFactory using its dictionary attribute.

By default, this dictionary resides at:

At the time of this writing (May 2024), the source of the data to match concepts in the clinical notes comes from the UMLS version 2023AB. A subset of source vocabularies and semantic types are included in the default EMERSE distribution, specified in the tables below. These were selected because they are the mostly likely to have clinical relevance and appear in the EHR notes. Each semantic type is mapped to a Type Unique Identifier (TUI).

This table is for storing NLP Semantic Groups, which has been discussed at: Token-Aligned Layered Index Structure

This table maintains NLP attribute groups, which has been mentioned at: [set-of-attributes]