EMERSE Installation Guide

To install EMERSE, someone with sys admin skills is desirable. This person should be able to install and configure software, edit database tables, etc. No actual programmer experience is needed for a default installation and configuration of EMERSE. Knowledge of servers, Java, Linux, and Apache software will be useful skills for this installation.

The initial setup described in this guide can probably be done in a day or less. More time will be required for additional customizations and planning related to localized needs, document extraction and indexing, regulatory approval, etc.

A Linux/Unix based server is suggested for installing the application server and for hosting the indexing services. No specific type of storage is required, but in general the server should be connected to the highest speed storage available. EMERSE performs a lot of disk reads to retrieve the documents, so read performance is important. We have conducted some experiments using SSD storage and have found it to provide nearly a two-fold increase in system performance for EMERSE, most noticeably in the area where all documents are retrieved and highlighted for a pre-defined set of patients.

Storage capacity is dependent on number of documents to be indexed. While no longer our latest statistics, at one point at Michigan Medicine 2.2TB was used to host approximately 166 million documents in both TXT and HTML format on the production server. An additional 4 TB was available for index optimization, as optimization requires about 2x the size of the index. If your documents are heavily formatted (e.g., HTML instead of TXT), storage requirements may be higher.

EMERSE doesn’t place great demand on the Oracle database, so a relatively small server can be used with 10-50 GB of storage allocated for user tablespaces. While no longer up-to-date, at one point at Michigan Medicine we were using 8 GB of database space for production EMERSE with 600+ users and 5 years of data (audit logs, user settings, etc).

The first step in installing EMERSE is to download and install a Java Development Kit (JDK) on the server. Java 17 or higher is required because it was compiled under that version.

Operating Systems often include a Java runtime, and is available on the PATH variable. We will use the java available from the linux distribution’s package manager, and install it like so:

The latter command should print out the java version installed, and confirms that Java is on the path.

EMERSE is packaged as a Java based WAR file that will be deployed to the Tomcat Servlet engine. Tomcat depends on a Java JDK. Tomcat version 9 is required. The Tomcat download page lists a number of components available for download. Only the "Core" software is required. See Required Software section.

We’ll navigate to the apache website and click on the "tar.gz" link under the "Core" bullet point. This should download apache from a mirror near you. Then unzip the in the /app directory:

Let’s start tomcat to confirm it works:

Open a browser and navigate to http://localhost:8080/. A Tomcat welcome screen should appear.

To stop tomcat, you just need to do:

(Though you don’t need to do that now.)

EMERSE leverages Apache Solr for searching documents. See Required Software section for the specific version to use. The Solr version should have the same major version as specified, and a minimum of the minor version. Otherwise, the sample indexes provided may not be usable, and EMERSE may not function.

Navigate your browser to https://solr.apache.org/downloads.html and download the latest .tgz BINARY release of Solr. We’ll expand it under /app and rename it, just like tomcat:

Let’s start solr to confirm it’s installed correctly:

and then navigate your browser to http://localhost:8983/solr/. You should the solr admin tool dashboard.

A database is needed for EMERSE. For the purposes of getting started, we’ll use the MariaDB database, since it’s often installable using linux package manages.

Database servers are often configured to be automatically started up as a "service" when the computer is booted. Different linux systems do this different ways, but the most common, especially for RedHat systems, is systemd. For Fedora, this looks like:

If you don’t want it started at boot, don’t "enable" the service. It can be started while "disabled."

To confirm it’s running, run the mariadb command, which attempts to connect to the database:

You should either get a command prompt for the database, or a permission error, such as "Access denied for user…​" You should not get an error saying it cannot get to the server, or cannot find a file named mysql.sock or a similar name.

If you are using Microsoft SQL Server as the backend database, we recommend two additional steps to ensure adequate performance.

From here on out, you’ll start installing software developed here at Michigan Medicine. Our software, along with SQL install scripts, and related files are all stored in our private EMERSE project page on GitHub. To access this repository and hence the rest of the installation materials, please contact the EMERSE team and provide us with your GitHub ID.

All installation files are "assets" on release pages of the EMERSE project. Generally, we don’t duplicate all the files needed to install EMERSE on each release, because many files don’t change, and many files that don’t change are extremely large (such as the example index files). For this reason, we only attach the changed files for each release. Thus, if you need a file named, say, emerse.properties you should search from the most recent release to the least recent release for the file with that name, or a very similar name. For instance, if you see emerse.properties on releases 6.1.1, 6.1, and on 6.0, you should use the one from 6.1.1. Similarly, if you are looking for emerse-solr-6.0.jar and you see it on release 6.0 but also see a emerse-solr.jar on release 6.1, you should use the 6.1 version. (We try not to have versions in our file names so that the file name is an exact match across versions.)

While not necessary, it can be helpful to have the system startup every time the server is started or rebooted, rather than entering the startup commands for each component every time. To do this, create a shell script in /app that will contain the commands to launch the two components:

Make the script executable:

This script assumes the database starts up on reboot already. If you chose not to do that when you install the database and what to do that now, it should be a command similar to

but it depends on your linux distribution, or how you installed your database. Alternately, you can add the database start and stop commands to this script.

Now add that startup script to a crontab. To create the file, type:

If you have followed the directions outlined in this document and installed Redhat Linux, it may open up the file using the vi editor. If so, type i for insert, then type one line as follows:

Then, to save the file, type esc [escape] and then :wq and enter to close and save the file. To verify that the crontab is there, type

It should show the line above (beginning with @reboot). At this point, restarting the server should allow all of the required components to startup automatically.

EMERSE has three tables describing Solr fields and field values:

Solr’s schema design and these tables are independent in the sense that changing one doesn’t affect the other, but they need to be consistent in order for EMERSE to work correctly. Despite EMERSE having the concept of multiple sources, all documents go into the same index in Solr, and so there is only one set of fields available to all documents, but not all documents need to use them, though they have to use fields consistently at a basic type level. For instance, a field PROVIDER can’t store text for one source and then a number for another source. (Of course, it could store the number as text, but that precludes range queries, for instance.)

Once we have declared the fields we will use in Solr to EMERSE, we should setup Solr to actually use those fields. You can delete the data directories inside each core:

Then edit the /app/indexes/documents/conf/managed-schema to remove all fields, but keep all the data types, then add our fields:

Generally, all fields should have docValues="true" unless it is one of the text fields. Fields are stored and indexed by default, but the field mapped to by the RPT_TEXT_NOIC emr intent (TEXT_CS in our case) should not be stored since its value should be a copy of the field mapped to by the RPT_TEXT emr intent (TEXT in our case) so we already have it stored, and don’t need a second copy of it. The two text fields also need additional attributes set as shown for EMERSE to be able to highlight and search.

Once we have rewritten our index schema, we can start solr again and start indexing.

You will need to develop an ETL process to load documents into EMERSE from your document sources. The EMERSE team has an application called "EMERSE Indexing" that indexes documents formed from queries to databases into Solr which we will use to show how this could be done. You can contact the EMERSE team to get the application. Looking at the code should give you an example of how you can do this yourself, or if your data source are all in databases, you may be able to use the application yourself.

EMERSE Indexing requires three things:

At UM, we use a instance Tomcat just for indexing, but for this guide, we’ll use the same one as EMERSE. Put the indexing.war in /app/tomcat/webapps/indexing.war and adjust add some lines to /app/tomcat/bin/setenv.sh to tell it where its configuration files are:

Now let’s define the properties:

The field solr.fields should list out all our solr fields. The properties starting like ds.<name> give connection details for a database called <name>. We’ll use two database, one called "dw" which would be your data warehouse, and "emerse-indexing" which is a required database that stores the tables EMERSE Indexing itself uses. We’ll create a database in MariaDB for this.

Create a database and run the install sql script for the indexing project, much like for EMERSE:

Finally, the indexing application requires a durable message broker called ActiveMQ "classic" edition. We can download and install it much like tomcat, from apache’s website. Expand the zip in /app/, renaming it to activemq, then starting it:

We can use the default configuration for activemq. You should see the admin interface at http://localhost:8161/admin/. The username and password are both admin.

Next, we will write the document-sources.xml. A quick overview of the project is needed, but I won’t cover everything here. You can see the documentation for the project on GitHub for more details.

The indexing application has a notion of a "source", and is defined by a <source> element in the XML. It has a name attribute, used within the application to load documents from the source, plus a source attribute which is assigned to the SOURCE field of the documents produced from it. (That is, it’s the EMERSE source.) Thus, multiple indexing sources can go into the same source bucket in EMERSE.

There are two kinds of queries in the indexing application. A document metadata query, which produces the metadata for documents, one document per row produced by the query, and a text query, which produces the text of the document. The text query is run once for each document, and the first column of all rows of produced by the query are appended together to produce the text of the document.

There are three variations on the metadata query, which differ only which documents they pick out, but always return the same set of columns. They are:

Each of these queries are defined under a common element <documentQueries> under the <source>. However, since the queries return the same data, but only for different subsets of all documents, they differ only slightly in the where clause. For that reason, instead of writing out the whole query under each element, they are defined as a substitution of "fragment references" on a single <baseDocumentQuery> (under the <source> element). The fragment references are written like ##fragment-name##. The replacement for the fragment references are defined by a like-named element under the variation’s element under <documentQueries>. Some example XML is helpful here:

Note that if multiple fragment references are present in the base query, then you’d define each as additional elements under the variation’s element (as a sibling to <selection-criteria> in the example above). Also note that the different variations have different query parameters, written in the form of :parameterName. These are fixed for each of the variations, and are as shown in the example.

Finally, in the text query, you reference the column aliases of the metadata query as query parameters. Thus, you can return some columns only needed in the text query (to make it run faster by doing fewer joins). The column aliases of the metadata query do double-duty by naming the field that column should be placed in. If the column alias isn’t in the solr.fields property in emerse-indexing.properties it won’t be added to the solr document.

We will define two indexing sources, matching our EMERSE sources. We will only work with the range metadata query, since it’s the most straightforward.

Notice in particular, that in the reports source, we produced the PROC_ID column in the metadata query, but it isn’t listed in the solr.fields property. We did this so we can use it in the text query, where we include the name of the procedure at the top of the document’s text, without having to do any joins to find it. Also notice that we construct the text of the document as a series of unions of queries, part of them are fixed text that can be used as section titles. The first column of the rows are joined together to make the text. Other columns are ignored, and only used to order the rows correctly.

Now you should be able to start tomcat and navigate to http://localhost:8080/indexing/. There you will see a very simple interface, where you can tell it to do one of the three variations on the metadata query. Run the range query for a date range that makes sense, and the application should index into Solr.