EMERSE Configuration and Optimization Guide

While it is not strictly required to have a web server such as apache’s httpd, or nginx, in front of Tomcat, it is typical. The only special configuration EMERSE currently needs is that the socket timeout should be set to around 10 minutes in order for synonym upload to work on very large synonym sets (such as those we distribute).

Solr has its own configuration files in two places. There is the installation directory, which is where you unzip the distribution, often referred to as solr-8.8.2/ in these guides. Then there is the configruation of the indexes, also called "cores" in Solr. The indexes reside in the Solr "home" directory, $SOLR_HOME which is either set by you in solr-8.8.2/bin/solr.in.sh or specified with -s when running solr start. If left unset, $SOLR_HOME is solr-8.8.2/server/solr/.

Each core has its own configuration files: - $SOLR_HOME/core_dir/core.properties - $SOLR_HOME/core_dir/conf/solrconfig.xml - $SOLR_HOME/core_dir/conf/managed-schema

The core.properties file defines the name of the core. It usually contains a line like name=documents or name=patient. This name doesn’t have to match the name of the directory it is in, but it’s less confusing if it does. This name determines the URL used to query the core, and how it appears in Solr’s admin app. EMERSE must be told the name of the core, not the name of the directory, since it uses the web API to talk to the core.

The solrconfig.xml file defines the version of Lucene used to talk to the index. Changing this will require a re-index since that may change the format of the index on disk.

In addition, solrconfig.xml contains the configuration of request handlers, such as the search, spell check and query validation handlers. These and other settings can be changed without re-indexing.

EMERSE assumes a number of endpoints are defined in this file, and those themselves rely on EMERSE’s Solr plugin. To install the plugin, place it in $SOLR_HOME/lib/. (You will have to create the directory.) The definitions of the endpoints that must be present in the solrconfig.xml files are kept in "configsets". These are basically templates for a Solr core. They have the same structure, but miss some files, such as the core.properties file.

The managed-schema file describes the format of documents for the core. It says what fields exist and how they should be indexed and stored. This will likely need to be modified to match your local document metadata structures, and will need to match what the appropriate database tables. For more information see the Data Guide.

EMERSE loads configuration data through Spring’s "environment", which is an abstraction over a number of ways to configure processes generally, Java programs specifically, and Java web applications more specifically. It presents configuration much like process-level "environment variables" which map textual names to textual values. The name/value pair generally is referred to as a "property" in Java. Spring’s environment searches all manners of configuration in some order, the first property found wins. The order of search is roughly as follows:

The emerse.properties file is searched for in the following order:

Generally, all EMERSE configuration is placed in the emerse.properties file, though configuration may be split across any of the six places Spring’s environment searches.

In practice, we tend to put the emerse.properties file under the home directory of the user which tomcat runs as. However, if this is not desirable, or if multiple versions of EMERSE are running on the same server, you must specify the location of the properties file using another property, emerse.properties.filepath specified in one of the first five ways. We tend to do this using either Java system properties, or JNDI.

Java system properties are passed as arguments to the JVM, in the form -Dproperty.name=value. Since the Tomcat scripts are what actually start the JVM, you need to configure them to pass this argument, and the easiest way to do that is to set and export the CATALINA_OPTS environment variable before starting tomcat:

If you merely want to put the properties file somewhere other than $HOME/emerse.properties you can create a file bin/setenv.sh inside the tomcat installation and put the export statement there; catalina.sh automatically runs such a file internally if it exists. (See the comments inside catalina.sh or catalina.bat for more information.)

The downside of this is that each tomcat instance can host at most one deployment of EMERSE. To have multiple deployments of EMERSE in the same tomcat instance, you can set up each deployment with a different JNDI environment, which can give EMERSE a different location for the emerse.properties file.

The JNDI environment for a war file is configured by a file inside Tomcat’s configuration directory: conf/Catalina/localhost/emerse.xml. (The directories other than conf will need to be created.) The filename emerse.xml should match the name of the war-file in the webapps directory. Inside, add the XML:

You can add other properties here in the same way as well. They will take precedence over the values specified in any emerse.properties file loaded. So, if two deployments have the same configuration, except for which Solr instance or database they connect to, you could just set the solr.serviceURL or ds.url properties here.

The EMERSE version number is distributed as part of the WAR file that gets deployed to the server. This is not really a configurable option, but it is being mentioned here for the sake of completeness. It can be found in the WAR manifest, under META-INF/MANIFEST.MF. (WAR files are just zip files with the wrong extension. This is true of jar files too.) Try unzip -c emerse.war META-INF/MANIFEST.MF.

EMERSE uses Spring profiles to enable optional services within the application. Active profiles are those listed in the value of the spring.profiles.active property in Spring’s environment. Profile names should be comma-separated.

Currently, there are two interesting profiles: ldap and no-scheduler.

ldap turns of the LDAP authentication mechanism which must be further configured with other properties. Additional information can be found in the LDAP section below.

no-scheduler turns off all scheduled jobs typically configured with the task.*.cron properties, described below. Activating this profile is used for testing only. (The scheduler is on by default, which is why activating the no-scheduler profile turns it off.)

See the Loading Configuration section for more details about how to put spring.profiles.active into Spring’s environment.

The following are all the properties that configure EMERSE. These are looked up through Spring’s environment, as described in Loading Configuration. Generally, we set them in the emerse.properties file.

To reduce the frequency of garbage collection and memory recollection use -Xmx and -Xms switches to control how JVM handles its heap memory. We recommend setting up tomcat to use between 1 and 2 gig of memory. You can add these arguments in the setenv.sh script in the bin directory of Tomcat. If it doesn’t exist, just create it. See Tomcat’s catalina.sh script for more details on this script.

Over time we have found that many document changes occur as they get updated or deleted (a deletion might be required if, for example, a document was found to be created under the wrong patient). It is possible to clear out these deleted/inactivated documents and potentially improve the performance of Solr by Optimizing the documents. This can be invoked manually using the Optimize button in the Solr Administration User Interface. Optimizing also reduces the index segment sizes which can also improve system performance. During the optimization process the original index is left in place while the new, optimized index is being created. This means that you will need empty storage about 2-3 times the original index’s size for optimization to proceed. Additionally, we have found that it can take about 10+ hours to conduct an optimization and it also uses substantial computational resources, meaning that system performance might suffer for users. Thus, it might be best to run this on weekends during times of low use. At Michigan Medicine we optimize infrequently and copy the indexes to a different server with more space and then copy the indexes back after optimization is complete. We also need to ensure that no new documents are added to the original index during this time.

Solr has a number of caches. Two important ones which are not configured by default are the filter cache and the document cache. These caches are index-specific, and are specified in the solrconfig.xml guide:

The maxRamMB attribute tells Solr how large the cache should be allowed to grow to before evicting entries. The size attribute tells Solr how many entries are allowed before evicting becomes necessary. Only one should be specified, but on earlier versions of Solr 8, the maxRamMB can’t be specified on the documentCache. (You’ll get an error if you do.)

We’d recommend configuring these caches both in the documents index and the patient-slave index, since both are queried by EMERSE. The documents index should have larger caches since it is used more heavily.

Not all queries that are sent to Solr are cached. We currently can cache the following queries (in the standard lucene syntax): 1. SOURCE:XXX 2. MRN:XXX 3. the collection of "filters" specified in EMERSE as a single query

To get a sense of the size of the queries, at UM, we have 200 million documents, and the first kind of query takes up 20-30 MB of space for each source in the system. (We have about five sources.) The second kind tends to take 2-30 KB (about a thousand times less), and the third kind takes between those two, depending on how selective the filters as a whole are. (The more selective, the smaller.)

Solr (on Linux at least) memory maps the index files, meaning it’s virtual memory will be about as large as the index file size. This means the virtual memory size of the application can be vastly larger than physical memory. (For instance, our Solr has at least 1 TB of virtual memory.)

These memory-mapped files are not a part of the Java heap space and so don’t contribute toward the -Xmx flag. In addition, the OS manages what of those files actually in physical memory and what isn’t. (Depending on the tool you use to look at memory on the box, the memory used for memory-mapped files may appear used or not; the OS is free to use that memory for another purpose, so it is, in a sense, free.) Solr internally caches the results of certain queries or parts of queries, so that if they are used frequently, the search doesn’t need to be done again. These do reside in the Java heap space.

So, you must strike a balance by having enough Java heap memory for query caching, and enough otherwise free memory on the box so that the OS has plenty of space to cache Solr’s memory-mapped files. At University of Michigan, we currently allocated 3GB to Solr’s heap, and have at least 20 GB of free memory on the box for the OS to cache files.

Allocating more heap space for Solr doesn’t mean you won’t have to tweak some of Solr’s cache settings, though there is some cache-sizing based on the max heap size. We haven’t done a ton of testing with caching, so we’ll say no more on this for now.

Solr’s memory can be configured with flags to the solr start command, or set as a default when starting solr by adding it to the solr.in.sh configuration.

If you are concerned with the performance of the garbage collector or free memory, you can see the frequency and duration of garbage collection in Solr’s GC log, contained in SOLR_INSTAL_DIR/server/logs/solr_gc.log.

EMERSE has two indexes used to keep track of patients: patient and its replica patient-slave.

The patient index is created by copying the patients from the database table, PATIENT over to the corresponding Solr index, patient. This is done automatically by the system once per day as a scheduled event. The schedule of the jobs can be found in the properties file. The default time set for the EMERSE distribution is 7:30 AM. This was done with the assumption that the patients in the Database table would be updated once every night during non-peak hours. If you are fine with that time, no changes need to be made.

After the PATIENT table is copied over, the replica is told to replicate from the master. To do this, the index must be configured with the master’s URL, which is done inside the solrconfig.xml file of the patient-slave core. There should be a handler defined like so:

Simply adjust the masterURL parameter as needed. Uncomment the username and password elements and just them if you set up basic auth on the Solr instance. If you have an SSL certificate (which you should if you use basic auth), you must use a domain name that is issued to the certificate.

The Overview screen in EMERSE is computationally expensive to show. Currently, it takes generally two searches for each cell of the table, more for the mosaic view.

To complete this work quickly and fairly among multiple concurrent users, EMERSE internally has a priority queue of batches of rows from this table. When a user goes to a page of the Overview table, EMERSE adds the rows of that page as a batch in the priority queue. Worker threads complete rows from the most-neglected batch in the queue. As rows are completed for the most-neglected batch, other batches will eventually become the most-neglected, and then rows will start being completed that batch. In this way, roughly equal time is spent on each batch.

There is one setting you can tweak on this search priority-queue.

Solr also provides a REST API that can be accessed with tools such as curl. By default this is not locked down and should be secured with basic authentication if the Solr ports are not firewalled to external communication.

Solr can be set up to use SSL/TLS, and require authentication with basic auth. Both of these features are supported by Solr Cloud, but EMERSE does not yet support Solr Cloud. However, the Jetty servlet engine embedded by stand alone Solr can be modified to require authentication and use SSL.

Passwords specified in the project.properties files can themselves be encrypted using Jasypt.

The Add/Remove users tab can be used to manage users of the EMERSE application. When you add new users, note that there are an expanded set of roles that can be applied to a user. For general users, you want to select/check “User with full privs” option and leave the others unchecked. The password field is required but will be ignored if security is set up to use LDAP. Although there is now a role for “limited access” type of user, we aren’t doing much with it yet locally.

Roles and Privileges for EMERSE users can be customized. Details about how this is done can be found in the Administrator Guide.

The Synonyms tab allows the admin user to update synonyms in the EMERSE application by uploading them from a TSV file. Multiple datasets can be added and managed through the interface in the Admin app. More details are discussed in the Administrator Guide.

The admin application has an option to "synchronize" various data between the database and Solr. While this happens automatically overnight it can be useful to force this more frequently, especially during initial system setup and testing. Details can be found in the Administrator Guide.