EMERSE Configuration and Optimization Guide

EMERSE loads configuration data mainly via property file configuration provided by the Spring Framework. Multiple paths can be defined for locating a file named project.properties, and Spring will use the first one it finds.

Normally this file is located inside the EMERSE war file. Once deployed, the file is located at TOMCAT_HOME/webapps/EMERSE/WEB-INF/classes. Using this location is usually not desirable in production environments, as deploying new versions of the code will cause this file to be replaced. Alternatively, the file can be placed in the Tomcat users’ home directory (that tomcat runs as) in a file called emerse.properties.

The EMERSE version number is distributed as part of the WAR file that gets deployed to the server. This is not really a configurable option, but it is being mentioned here for the sake of completeness. The number is set in the Project Object Model (POM) files that can be found in the META-INF directory within the Tomcat webapps folder.

EMERSE uses Spring profiles to enable optional services within the application. These profiles are "activated" by supplying a parameter to the EMERSE JVM at startup. Currently this is only useful to activate LDAP related spring security features. If the profile "ldap" is enabled, EMERSE will use enable LDAP for user authentication. Activation of profiles is added by adding a JVM system property at startup. In the example below, the tomcat startup file is modified to enable LDAP service via the system property. You should not have to change or swap XML files to enable LDAP. Toggling between the XML files is enabled at runtime via a Spring profile being activated.

Additional information can be found in the LDAP section below.

The following settings can be modified in the project.properties file, or emerse.properties as discussed previously:

To reduce the frequency of garbage collection and memory recollection use -Xmx and -Xms switches to control how JVM handles its heap memory. We recommend setting up tomcat to use between 1 and 2 gig of memory. One way to do this is to add the following snippet to the tomcat startup file - startup.sh.

Over time we have found that many document changes occur as they get updated or deleted (a deletion might be required if, for example, a document was found to be created under the wrong patient). It is possible to clear out these deleted/inactivated documents and potentially improve the performance of Solr by Optimizing the documents. This can be invoked manually using the Optimize button in the Solr Administration User Interface. Optimizing also reduces the index segment sizes which can also improve system performance. During the optimization process the original index is left in place while the new, optimized index is being created. This means that you will need empty storage about 2-3 times the original index’s size for optimization to proceed. Additionally, we have found that it can take about 10+ hours to conduct an optimization and it also uses substantial computational resources, meaning that system performance might suffer for users. Thus, it might be best to run this on weekends during times of low use. At Michigan Medicine we optimize infrequently and copy the indexes to a different server with more space and then copy the indexes back after optimization is complete. We also need to ensure that no new documents are added to the original index during this time.

Solr’s memory can be configured by a simple flag at startup. When working with millions of documents, we had some issues when Solr was using its default settings of 512 megabytes. Currently the EMERSE production instance at Michigan Medicine is configured to use 3 gigabytes. In the example below, 3 gigabytes are being allocated.

EMERSE has two indexes used to keep track of patients: patient and patient-slave (the patient-slave index is the one actually used by EMERSE while it is running).

The patient index is created by copying the patients from the database table, Patient over to the corresponding Solr index, patient. This is done automatically by the system once per day as a scheduled event. The schedule of the jobs can be found in the properties file. The default time set for the EMERSE distribution is 7:30 AM. This was done with the assumption that the patients in the Database table would be updated once every night during non-peak hours. If you are fine with that time, no changes need to be made.

The patient-slave Solr index gets replicated from the master patient Solr index, and it is possible to change the frequency of this replication. We currently have it set to do this every minute, and we do not see any reason that it should need to be changed. However, if you do wish to change it, this can be found in the solrconfig.xml file within the patient-slave core. The parameter to change is pollInterval. For example:

In a deployment of EMERSE where the main application and Solr are running on the same server, each process should be given adequate memory. At some point, however, allocating memory to these processes may actually reduce performance. Most modern operating systems will cache files in all available memory. As memory is allocated to these processes, less memory will be available to the operating system to cache files.

EMERSE enqueues user searches as they use the application, and a pool of workers concurrently pulls from that queue and executes searches. This parallel execution of searches allows fast searches to complete while long ones continue, and helps prevent a single user’s long-running query from slowing down the entire system for all users. Workers from the pool take a batch of requests from the queue at once, and process that batch sequentially. However, if the requests they are running are slow, they may take a smaller batch size, but never smaller than the minimum size. There are a few parameters in the project.properties file that control this construct, although we would not expect that these should need to be changed unless specific performance issues need to be tweaked.

The dynamic adjustment of batch sizes is based on the following rules, which are not configurable at this time. Each thread processes a search for a given user. A search is done for each patient (not patient/source). When the last search for the user has taken…​
…​< 1.5 seconds, max batch size (7) is used for next search
…​>=1.5 seconds and < 5 seconds, max batch size / 2 (round down, so 3 in this case) is used for next search
…​>=5 seconds, initilal batch size (1) is used for next search

Solr also provides a REST API that can be accessed with tools such as curl. By default this is not locked down and should be secured with basic authentication if the Solr ports are not firewalled to external communication.

Solr can be set up to use SSL/TLS, and require authentication with basic auth. Both of these features are supported by Solr Cloud, but EMERSE does not yet support Solr Cloud. However, the Jetty servlet engine embedded by stand alone Solr can be modified to require authentication and use SSL.

ActiveMQ broker can be setup to use SSL/TLS and also require authentication. ActiveMQ webapp can also be configured to use SSL. EMERSE allows setting user/password via configuration properties mentioned above [activemq-properties]

Passwords specified in the project.properties files can themselves be encrypted using Jasypt.

EMERSE provides a function for exporting password-protected Excel files containing patient lists and and associated comments/tags. These files are generated on demand by the user and stored on the EMERSE server inside the exploded EMERSE war file, with a unique download link provided to the user. Because there is no straightforward way to know when a file has been successfully downloaded, the Excel file persists on the server. We currently have a shell script on the server that executes every 30 minutes and deletes files older than 60 minutes.

The Add/Remove users tab can be used to manage users of the EMERSE application. When you add new users, note that there are an expanded set of roles that can be applied to a user. For general users, you want to select/check “User with full privs” option and leave the others unchecked. The password field is required but will be ignored if security is set up to use LDAP. Although there is now a role for “limited access” type of user, we aren’t doing much with it yet locally.

Roles and Privileges for EMERSE users can be customized. Details about how this is done can be found in the Administrator Guide.

The Synonyms tab allows the admin user to update synonyms in the EMERSE application by uploading them from a CSV file.