High priority- Log4J security announcement hannahb  2021-12-10 13:55
Status: Closed
 

Yesterday, researchers announced a critical vulnerability (CVE-2021-44228) in Log4J, a popular Java logging library that LabKey Server versions 20.11 and later use.

All LabKey Server administrators must take action to protect their installations immediately, especially installations that are open on the public Internet. We are making these changes for all hosted customers today which will involve a short server downtime. You will receive notice of this downtime shortly.

Right now, administrators must add a system property (-Dlog4j2.formatMsgNoLookups=true) so that Java will disable the part of Log4J with the vulnerability. The exact mechanism for setting this property depends on how Tomcat is installed and launched.

For Linux installations, locate the JAVA_OPTS or CATALINA_OPTS environment variables for your Tomcat installation. These settings are typically located under the /etc/init.d/tomcat startup file, the /etc/systemd/system/tomcat.service startup file, the /etc/default/tomcat file, or the CATALINA_HOME/bin/setenv.sh script. Once the JAVA_OPTS or CATALINA_OPTS line is located, add -Dlog4j2.formatMsgNoLookups=true to the end of the line and restart Tomcat.

For Windows installations running Tomcat as a service, typically admins should use the “tomcat9w.exe” utility to adjust the settings. The following doc page shows how to adjust these kinds of settings in a variety of environments.

https://www.labkey.org/Documentation/wiki-page.view?name=configWebappMemory

If you are setting the system property, you can confirm that it was picked up after restarting Tomcat. Go to the Admin Console->System Properties and confirm that the “log4j2.formatMsgNoLookups” property is shown with the value “true”.

https://www.labkey.org/Documentation/wiki-page.view?name=siteManagement#diagnostics

Early next week we will release a patched version of LabKey Server that incorporates Log4J 2.15.0 that eliminates the need for the system property going forward. We will post this version to support portals as soon as it is available.

Please reach out via your support portal with questions and requests for help in making these changes.

For more details on the Log4J vulnerability itself, see https://github.com/advisories/GHSA-jfh8-c2jp-5v3q

 
 
Will H. responded:  2021-12-14 19:53

I saw that LabKey v21.11.2 was released on Dec 14, 2021. I downloaded it in hopes that it contained Log4J v2.16.0. It contains v2.15.0 which is no longer considered safe. I'm posting this so others know v21.11.2 isn't the fix we are waiting for.

-Will

 
hannahb responded:  2021-12-15 07:55

Following up on the critical Log4J vulnerability we communicated about on Friday December 10, we have released LabKey Server 21.11.2 and 21.7.10 that include Log4J version 2.15.0. We have posted updated installers on the Community Edition download page and on customer support portals.

Any installation running LabKey Server 20.11.x or later should immediately upgrade to 21.11.2 or 21.7.10 if an administrator has not already added the -Dlog4j2.formatMsgNoLookups=true system property. For servers that have already added the system property, deploying 21.11.2 or 21.7.10 is optional but recommended. Either approach mitigates the severe vulnerability in Log4J (CVE-2021-44228).

Unfortunately, the Log4J developers today announced a related, moderate vulnerability (CVE-2021-45046). Based on the information available, we have good reason to believe that LabKey Server deployments are NOT impacted. However, we are erring on the side of caution and will be creating new LabKey Server hotfixes that incorporate Log4J 2.16.0. Installations can be updated to this followup release on a non-emergency basis.

Do not wait for this followup hotfix to protect your servers, you should take action NOW by installing 21.7.10 or 21.11.2 or using the -D system property.

We are sorry for the rapid stream of updates and recommendations, but we want to ensure that all LabKey Server installations are secure and that we keep everyone informed.

 
hannahb responded:  2021-12-15 11:00

Thanks, Will. As hopefully you saw, we are working on another hotfix to address the most recent moderate vulnerability. More information to come when we know more about the release date.

 
jeckels responded:  2021-12-15 11:42

I wanted to follow up on Will's post. I want to make sure that no one is waiting for another build before protecting their server, either via the -D system property or by deploying LabKey Server 21.11.2 or 21.7.10. It is imperative to get that level of protection in place.

We are aware of the followup changes in Log4J 2.16.0 that address a related but significantly less severe problem. Based on the information released, we don't believe that LabKey Server is vulnerable but are opting on the side of caution and taking the Log4J team's recommendation of updating to 2.16.0.

I know everyone would prefer to only need to take one set of mitigation steps, but do not wait to protect yourselves from the first critical vulnerability.

Thanks,
Josh

 
jeckels responded:  2021-12-16 10:26

We have finished testing and have released LabKey Server versions 21.11.3 and 21.7.11 which incorporate Log4J 2.16.0. We are in the process of pushing out new builds to customers' support portals and the Community Edition download page. These versions address both CVE-2021-44228 and CVE-2021-45046, and with the new builds you no longer need the -D system property as a mitigation (though it will cause no problems if you leave it in place).

We have separately been asked about older LabKey Server versions that use Log4J 1.2.17 which is vulnerable to CVE-2019-17571. Our assessment is that LabKey Server is not impacted, because it doesn't make use of the specific feature within Log4J 1.x that is vulnerable, a SocketServer. Thus, no mitigations are required, but we encourage all deployments to migrate to 21.11.3, the current supported, maintained version of LabKey Server.

Thanks,
Josh

 
Will H. responded:  2021-12-16 11:37

Thank you to everyone at LabKey for getting these updates out quickly!

 
jeckels responded:  2021-12-19 09:49

Log4J has announced yet another variant on these same issues, CVE-2021-45105. Fortunately, it only affects applications that use a non-default configuration of Log4J with a Pattern Layout that uses a Context Lookup.

LabKey Server does not use this configuration, so like the previous vulnerability, CVE-2021-45046, this newest announcement does not impact our users. Regardless, we will adopt the latest Log4J release, 2.17.0, for everyone's peace of mind, but it does not need to be rolled out as an emergency patch.

Thanks,
Josh

 
adam responded:  2021-12-30 11:53
Status: Active

Yesterday, Log4J announced another potential vulnerability, CVE-2021-44832. In this case, an attacker would need write permissions to an application's log4j2.xml configuration file, which should never be the case in a LabKey Server deployment. Despite not being vulnerable, we have updated all current release branches to the latest Log4J release, 2.17.1, for additional assurance. The next maintenance releases (e.g., 21.11.5) will include this version but there's no need for emergency patching.

Thanks,
Adam