Administrator Guide

When editing a user, you can change the roles, first name, last name and username (login ID).

EMERSE verifies a user’s password in one of two ways. The password can be stored in EMERSE’s database, in which case you can set the password on this page. The password field is always empty even if this is the method of authentication, but filling it in and saving the user will change their password.

The other way to verify the password is to use LDAP. In this case, the password cannot be changed by EMERSE, and the password field is always not used. To use LDAP, check the External Authentication (LDAP) checkbox and save the user. Using LDAP to authenticate requires that LDAP has been configured in emerse.properties. See the configuration documentation on how to do that.

Finally, you can select which roles the user has. Each role is listed at the end, with its description shown, not the internal code. Roles and privileges are described next. Changing a user’s roles doesn’t take effect until their next login. So, if they are currently logged in and need a new privilege you just assigned, they will have to log out, then log back in to get it.

To prepare a synonyms file for importing into EMERSE, you must structure the file as a three-column tab-separated values (TSV) file.

Column 1: A key value that links related concepts together. This can be anything but should be unique for each grouping of concepts.

Column 2: The term/phrase

Column 3: The Concept Type, of which there are 4. These 3 Concept Types are used for categorizing how the suggestions are displayed to users and also how the terms will be matched. They are:

Additional metadata can be added to the file which will be displayed to the users. These also must follow the three column TSV format and should appear at the beginning of the file, using the following formatting

Column 1: The phrase: emerse_synonyms_metadata

Column 2: An attribute, which is case-insensitive.

Column 3: The value for the attribute.

Note that you can provide any attribute, but only specific ones are supported by EMERSE. Other attributes not supported will be stored in the database but not used.

Currently supported Attributes are as follows:

The beginning of an example Synonyms import file would look like:

In the example above, there are three groups of concepts (group 1001, group 1021, and group 1041). The group numbers are arbitrary and have no meaning, but must be distinct from one another. Group 1021 has mix of three concept types: seven regular Synonyms (Concept Type = 1), one Related Term (Concept Type = 2), and five Spelling Alternatives (Concept Type = 3). Group 1041 has a CUI (Concept Type = 4) and multiple regular Synonyms (Concept Type = 1). For group 1041 if the CUI ("C0018681") is entered, all of the other terms mapped to that group will be suggested. If any standard term/regular synonym (e.g., "head pained") for group 1041 is entered, then the CUI will be suggested.

When a user enters a term, EMERSE does the following to determine which Synonyms to display:

This organizational scheme of four Concept Types allows for some terms to be suggested based on a user-entered term, but supports the ability to have a one-way match to reduce extraneous matching when desired. For example, if the terms entered is 'adenomatous', the term 'polyps' will be displayed as a Related Term (see figure above). However, since in that grouping (group 1021 in the above example) the term 'polyps' is not a Regular concept type (i.e., it is not Concept Type 1 but rather Concept Type 2) then it means that a user-entered search term of 'polyps' will not match to anything in that 1021 grouping. This can be seen in the figure, below, where none of the terms in group 1021 are displayed to the user when 'polyps' is entered as a search term.

A Synonyms file should be prepared as a 3 column TSV file as described above. To upload the file, login to the Administrator App and choose the Synonyms tab. Then select Upload, and click on the Choose button to select the TSV file. After the file is selected press Upload.

Minimal feedback is provided while the file is loaded and verified. Common problems include a file size too large for the server to handle, an upload time that exceeds the server’s timeout maximum, or a string length that exceeds what the database allows (we recommend no terms > 200 characters). A description of a few problems that might be encountered, and possible solutions, can be found in the Troubleshooting Guide.

When uploading a Synonyms file, note that if the same name is used as an existing dataset (based on the included metadata; and a case-sensitive match), the newly uploaded dataset will replace the existing dataset.

Multiple Synonym datasets can be loaded into EMERSE via the Admin App. After loading, a dataset will be in a Deactivated state, meaning that it will not be available or even visible to users. Clicking the Activate button will do two things: (1) it will make the dataset available to users and (2) it will invoke a background process that counts how many times each term appears in the overall set of documents. The counts resulting from this background process provide an intelligent approach to displaying the synonyms to users because it allows users to rank the terms based on how often they appear, including providing the option to hide terms that never appear in the corpus of all notes in the Solr index.

Counting the terms can take hours, depending in part on how many terms are in the dataset and also on how many documents are indexed. While the count is ongoing the synonyms will be available to users but users will not be able to change options that limit the results by count. Counts will not appear for a dataset until the entire counting process for all of the terms is complete. The counting process is refreshed only at the time a dataset is Activated. To refresh the counts, Deactivate and then Activate the dataset again. Refreshing the counts may be desirable if many new documents have been added since the counts were last conducted, which could change the frequency of the Synonyms in the dataset.

The Deactivate button retains the Synonyms dataset in the system but makes them unavailable to users. The Remove button deletes the dataset entirely from the database.

If the counting process was interrupted (for example, by the server being patched and restarted) the system should be able to recover from where it left off and complete the counts, but it must be restarted manually be pressing the the Resume button which will be displayed when an interrupted process is detected. The counting process can also be stopped manually by pressing the Stop button. When counts have been stopped before completion, the Synonyms will be available but without counts.

There are several things displayed in the table that may be useful. Some of these are described below:

To understand fields, their values, and filters on their values, a brief discussion of Solr, indexes, and the structure of documents is useful.

An index in Solr is a collection of documents which have been indexed, meaning you can find the document given the values they are indexed under. A more familiar example of an index in general would be the index of a textbook, which indexes the pages of the textbook by words, phrases, or concepts. Thus, you look up a concept like "mesoscale convective vortex" in the index of a meteorology textbook, and it tells you the pages that talk about that concept. Solr is similar, but instead of indexing the pages of a single book, we index documents of the institution-wide health record. Thus, the index tells us the document(s) in which a word or phrase appears.

Solr documents differ from pages of a book in one important way: Solr documents are not just a block of text, but a block of text with metadata. Metadata associated with the document can be anything you wish to load with the document, but typically includes things like the department that produced the document, the clinician that wrote the note, etc. Each piece of metadata is placed in a different "field" alongside the main text of the document. In fact, the main text of the document is contained in just another field, so the representation of a document is uniform, and could include many large chunks of free text (but EMERSE does not do so).

Look at the below screenshot of Solr, which shows a document in JSON format:

Here, we can see a number of fields written in bold text, such as DEPT or CLINICIAN, and their values Urology and Harper, Bradley. These are what we would call metadata fields, and RPT_TEXT can be seen to have a html-formatted value which runs off the page — this is the main text of the document.

Documents in the documents index represent medical records, notes, reports, etc, whereas documents in the patient index represent patients themselves, with no main text field. See the below screenshot for the raw JSON-formatted patient document.

The values of a field are all the values that appear for that field across all documents in the index. For instance, the values of the DEPT field are the departments. In the system pictured here, these values are human-readable, however you may have department codes loaded into Solr.

As seen above, there are a number of different kinds of values that can be stored in Solr. The managed-schema file in Solr defines the fields, and their types. These fields look like:

There are three types, and each field can either be multi-valued or not. Multi-valued fields appear as an array of the underlying values in Solr. A document can be found by searching for any value in the array, and you can’t search for a document that contain a specific array of values. Consequently, the order of values in the array is not important.

Clicking on a patient or document field allows you to customize the following things:

Normally, when a field is shown in EMERSE, the value of that field is shown. However, EMERSE can group field values together, showing the group label for every value in the group, instead of the value itself. This is configured on the values tab of the field.

The mapping between what value is in what group is stored in the SQL database. The first 100 values of that mapping is shown in the table on this page. Each value has a count associated with it, which is the number of documents in Solr that have that value. Despite the name, groups don’t have multiple values assigned. A group is essentially a user-readable label, so you can assign a single value to a label (group) or multiple values can be mapped to a group. A single value can be assigned to more than one group. For example, if the value is "XRAY CHEST AND ABDOMEN", you might want to assign it to a group called "Chest" and also to a group called "Abdomen" to denote the different body locations.

The field that plays the SOURCE special role tells EMERSE which source a document is from. A source is generally a source system of documents, such as the main EHR system, like Epic, or other more specialized systems, like SoftPathDx for Pathology, or legacy EHR systems. EMERSE allows documents from different sources to use and show Solr fields differently. For instance, the DEPT field might store department information if the document is from the main EHR source, but for the pathology source, it may store the lab instead. Thus, you would want to label the field "Department" for the main EHR, but "Lab" for pathology. To do this, EMERSE allows you to set a source for a field. Doing so will move the field to a separate group just for that source on the left-hand side. Once a field is associated with a source, it is only used on documents from that source. If a field is associated with a specific source, it cannot play a special role in EMERSE.

Sources are described as the values of the field that plays the SOURCE special role. If you go to the values tab for that field, you will get a different interface:

Here are a few differences between this sources page and other field value pages:

Once sources have been defined, any document field can be made source-specific by setting the source on its attributes tab. Each source has its own group of fields in the left-hand sidebar, underneath the "Document Fields" group which are for document fields not associated with a specific source. Setting the field’s source will move it into the source’s group, where you can now order it with respect to the other source-specific fields. Source-specific fields always appear after cross-source fields whenever both are shown (just like they are in the sidebar).

Source specific fields are only shown when showing a document from that specific source. Similarly, when loading field values from Solr, a source-specific field only looks at documents from that particular source, instead of all documents loaded into the index.

A single field in Solr can be the backing Solr field of many fields in EMERSE. That is, you can create multiple fields in the fields tab, and select the same Solr Field for all of them. This allows you to take the same field, and group its values two different ways.

Suppose you have a single Solr field called IMAGE_DESC which stores a kind of code of the imaging done (for example "XRAY ABDOMEN"), which practically speaking contains two pieces of information: what was imaged (ABDOMEN), and what the imaging modality was (XRAY). Even though there is only one field in Solr, we can split this into two fields in EMERSE by creating an imaged body part field and a imaging modality field, both backed by the same IMAGE_DESC field in Solr. We then group the values different. For the imaged body party field, the groups are body parts, putting any imaging description that mentions the body part under the group for that body part. We do the same for the imaging modality, but where the groups are the modalities.

Another way to use the ability to map the same field multiple times is to provide different levels of granularity of some hierarchy. You may have the field mapped once without grouping, and then once with grouping.

However, since a value can appear in many groups, it is possible to have all levels in a hierarchy in a single mapping by having both granular groups and coarse groups defined side-by-side. Similarly, you can define groups for imaged body parts and for imaging modality for a single EMERSE field, rather than splitting that into two fields.

The choice is of what to do is mainly a preference, and primarily what the users would expect, but remember that a document must match all filters to match the search (except filters for a source the document isn’t from), and to match a filter, the document must have only one of the values required by the user. Thus, selecting multiple values for a single filter is different than selecting a single value in a multiple filters (even when they are mapped to the same underlying Solr field).

In other words, selecting the Ankle and CT groups in a filter on an imaging description field has the semantics of matching documents marked as either ankle or CT, whereas selecting Ankle in a filter on the imaged body part field, and selecting CT in a filter on the imaging modality field has the semantics of matching documents marked as both ankle and CT.

To understand many of the statistics, it’s best to explain the internal scheduling mechanism used to run heatmap queries.

When a user views a page of patients in the Overview table, this triggers the backend to add a batch of overview-table rows to a data structure we call the fair batch heap. All of the rows of a page of the table go into the same batch.

There are then a configurable number of threads that each pull a row from the highest-priority batch in the fair batch heap, and run queries to fill out that row. If there are N columns in the table (ie, N sources), then there are 2N queries done to produce simple counts of matching documents, or N + NC queries done for the mosaic view, where C is the number of colors in the term bundle.

After filling out the row, the thread sends the row to the browser, goes back to the fair batch heap, and re-proirities the batch based on the amount of time it took to fill out the row. It then starts again, pulling a row from the highest-priority batch.

The prioritization of batches is determined by a priority number maintained on the batch. Batches start with zero priority. If a row from a batch takes M milliseconds to fill out, and there are N other batches in the fair batch heap, then the batch that contained the processed row is docked MN priority, and every other batch is given M more priority. If a batch completes but has non-zero priority, it’s remaining priority is distributed evenly between the remaining batches in the fair batch heap. This ensures there is no "drift" from zero of the avarage priority of a batch in the heap. (In the code, this is actually phrased with "penality" which is just negative priority.)