Workaround instructions to address CVE-2021-44228 and CVE-2021-45046 in vCenter Server and vCenter Cloud Gateway (87081)

- CVE-2021-44228 & CVE-2021-45046 – VMSA-2021-0028
Highlighted sections indicate the most recent updates. See the Change log at the end of this article for all changes and subscribe to the article for updates.
- VCHA needs to be removed before executing the steps in this article. It can be reconfigured afterward.
- Environments with external PSCs need to have the steps taken on both vCenter and PSC appliances.
- Restoring from a File-Based Backup will put the environment into a vulnerable state again. Use the vc_log4j_mitigator.py script after restoring to correct this
- Upgrading the vCenter Appliance to an unmitigated version will put the environment into a vulnerable state again. Use the vc_log4j_mitigator.py script after upgrading to correct this
IMPORTANT: vc_log4j_mitigator.py will now mitigate CVE-2021-44228 and CVE-2021-45046 on vCenter Server end-to-end without extra steps. This script replaces the need to run remove_log4j_class.py and vmsa-2021-0028-kb87081.py independently. However, it is not necessary to run if you’ve already used those in your environment.
Please use the below 3 scenarios to verify if you’re previous steps were sufficient:
Completed remediation scenarios:
1. Used vc_log4j_mitigator.py
2. Used vmsa-2021-0028-kb87081.py script from KB 87088 and remove_log4j_class.py from this KB.
3. Used the manual workaround steps in this KB and remove_log4j_class.py.
If you are unsure if the previous steps were sufficient, run the vc_log4j_mitigator.py script with the “–dryrun” option to verify if the environment still has vulnerable files. If any are found, run it without the dryrun flag to correct them.
Automated Workaround (Recommended)
To apply the workaround for CVE-2021-44228 and CVE-2021-45046 with the unified script:
1. Download the script attached to this KB (vc_log4j_mitigator.py)
2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)
3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work
4. Execute the script copied in step 1:
python vc_log4j_mitigator.py
This will stop all vCenter services, updates all necessary files with the formatMsgNoLookups flag, removes the JndiLookup.class from all jar/war files on the appliance, and finally starts all vCenter services. The files that the script modifies will be reported as the script runs.
5. To verify that no more vulnerable files exist, execute the script again with the dry run flag:
python vc_log4j_mitigator.py -r
The list of vulnerable files should be zero
Note: Certain 3rd party vSphere Client plugins may be detected as vulnerable even after remediation is done. This is because the jar files are recreated each time the client is started. These must be addressed by either patching the plugin to an unaffected version, marking it incompatible, or removing the plugin via the vCenter MOB. Disabling the plugin from the vSphere Client will not work.
This has been found with older versions of the Pure Storage HTML client plugin 4.3.1 and below . The file continually marked as vulnerable will be: /usr/lib/vmware-vsphere-ui/server/work/Catalina/localhost/ROOT/eclipse/configuration/org.eclipse.osgi/321/0/.cp/log4j-core-2.11.2.jar.
Optional Arguments
-h, –help show this help message and exit
-d dirnames [dirnames …], –directories dirnames [dirnames …]
space separated list of directories to check
recursively for CVE-2021-44228 vulnerable java archive
files.
-a, –accept-services-restart
acccept the restart of the services without having
manual prompt confirmation for the same
-r, –dryrun Run the script and log vulnerable files without
mitigating them. The vCenter services are not
restarted with this option.
-b BACKUP_DIR, –backup-dir BACKUP_DIR
Specify a backup directory to store original files.
-l LOG_DIR, –log-dir LOG_DIR
Specify a directory to store log files.
Manual Workaround
To manually apply the workaround for CVE-2021-44228 and CVE-2021-45046 to vCenter Server Appliance 7.x and 6.x, skip to the relevant sections below and follow the full instructions there:
- Click here for vCenter Server Appliance 6.7.x workaround
- Click here for vCenter Server Appliance 6.5.x workaround
- Click here for vCenter Server Appliance 6.0.x workaround
vCenter Server Appliance 7.0.x Workaround
vMON Service
1. Backup the existing java-wrapper-vmon file:
- vCenter 7.0 Update 3, 3a, 3b
- vCenter 7.0 Update 2, 2a, 2b, 2c, 2d
exec $java_start_bin $jvm_dynargs $security_dynargs $original_args
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args
NOTE: The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below:
- vCenter 7.0 GA, 7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d
- vCenter 7.0 Update 1, U1a, U1c, U1d
exec $java_start_bin $jvm_dynargs “$@”
Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
service-control –start –all
Update Manager Service
Analytics Service
This is now remediated with remove_log4j_class.py at the end of this section.
Run the remove_log4j_class.py script
1. Download the script attached to this KB (remove_log4j_class.py)
2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)
3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work
4. Execute the script copied in step 1:
python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.
Verify the changes
Check if the processes includes -Dlog4j2.formatMsgNoLookups=true.
java -jar start.jar –list-config
System Properties:
——————
log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)
4. Verify the script successfully removed JndiLookup.class from all java files with the following command:
python remove_log4j_class.py -r
The list of vulnerable files should now be empty.
2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
===== Summary =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.
vCenter Server Appliance 6.7.x Workaround
vMON Service
1. Backup the existing java-wrapper-vmon file:
2. Update the java-wrapper-vmon file with a text editor such as vi:
3. At the very bottom of the file, replace the very last line with 2 new lines:
Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”
4. Restart vCenter Services:
service-control –start –all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
Secure Token Service
1. Back up and edit the the vmware-stsd file:
vi /etc/rc.d/init.d/vmware-stsd
{
perform_pre_startup_actions
JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
Identity Management Service
1. Back up and edit the the vmware-sts-idmd file:
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
service-control –start vmware-sts-idmd
Analytics Service and CM Service
Run the remove_log4j_class.py script
1. Download the script attached to this KB (remove_log4j_class.py)
2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)
3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work
4. Execute the script copied in step 1:
python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.
Verify the changes
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
2. Verify the Analytics Service changes:
Note: This should return 0 lines.
4. Verify the script successfully removed JndiLookup.class from all java files with the following command:
python remove_log4j_class.py -r
The list of vulnerable files should now be empty.
2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
===== Summary =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.
vCenter Server Appliance 6.5.x Workaround
vMON Service
Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”
service-control –start –all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
Secure Token Service
1. Back up and edit the the vmware-stsd file:
{
perform_pre_startup_actions
local retval
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
service-control –start vmware-stsd
Identity Management Service
1. Back up and edit the the vmware-sts-idmd file:
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
service-control –start vmware-sts-idmd
PSC Client Service
1. Back up and edit the vmware-psc-client file
vi /etc/rc.d/init.d/vmware-psc-client
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Djava.io.tmpdir=”$CATALINA_BASE/temp” \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
service-control –start vmware-psc-client
CM Service
This is now remediated with remove_log4j_class.py at the end of this section.
Run the remove_log4j_class.py script
1. Download the script attached to this KB (remove_log4j_class.py)
2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)
3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work
4. Execute the script copied in step 1:
python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.
Verify the changes
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
Note: This should return 0 lines.
3. Verify the script successfully removed JndiLookup.class from all java files with the following command:
python remove_log4j_class.py -r
The list of vulnerable files should now be empty.
2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
===== Summary =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.
vCenter Server Appliance 6.0 U3j Workaround
2: Back up the log4j-core-2.11.2.jar file
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak
3. Run the zip command to disable the class
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
3. Stop the service and then start it through service-control:
service-control –start vmware-perfcharts
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-core-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-api-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-core-2.1.jar
- /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-api-2.1.jar
VCHA needs to be removed before executing the steps in this KB.
Change log:
- December 12th 2021 – 11:20 PST: Updated Knowledge Base article advising workarounds for 6.5/6.7 partially address the vulnerability.
- December 12th 2021 – 16:00 PST: Added steps for stsd, idmd and psc-client services to the workaround sections for 6.5 and 6.7.
- December 12th 2021 – 17:00 PST: Added workaround for VCSA 6.0U3j.
- December 12th 2021 – 18:30 PST: Added additional information for older versions of VCSA 6.0.
- December 12th 2021 – 20:30 PST: Updated versions with vulnerable jar files in VCSA 6.0.
- December 13th 2021 – 01:57 PST: Updated the steps in the vMON service commands for permission validation
- December 13th 2021 – 16:30 PST: Simplified and re-arranged beginning of workaround section (no new content). Added clarification to 6.7U3p analytics verification steps. Updated cm service restart commands. Added highlighting to changelog.
- December 13th 2021 – 20:05 PST: Removed DBCC Utility Workaround step from this article. VMware has tested this and does not have any impact or risk hence the workaround section was removed.
- December 15th 2021 – 11:00 PST: Added a notice at the top concerning the recent updates on CVE-2021-44228 from Apache.
- December 15th 2021 – 17:30 PST: Formatting changes for readability (no new content)
- December 16th 2021 – 14:15 PST: Added new remove_log4j_class.py script which replaces several steps in this article and addresses new findings in CVE-2021-44228 and CVE-2021-45046. Added guidance in each section for its use.
- December 17th 2021 – 08:00 PST: Added clarification in the workaround steps that both remediation scripts are still necessary for completely un-remediated environments. Added step to vCenter Server 6.0 U3j instructions to manually remove the JndiLookup.class class
- December 17th 2021 – 16:30 PST: Updated remove_log4j_class.py to output a change summary and better handle “File is not a zip file” issues. Added section with new cmd line options for remove_log4j_class.py. Added “–dryrun” verification step to all manual sections.
- December 20th 2021 – 18:00 PST: Added vc_log4j_mitigator.py which unifies the previous steps and scripts into a single workaround. Added note about upgrades and file-based backup/restore operations to Impacts section.
- December 21st 2021 – 12:45 PST: Added note about 3rd party plugins reverting to a vulnerable state after the vsphere client is started. Added clarification that CVE-45046 is also addressed with all the changes since the “December 16th 2021 – 14:15 PST” update.
- December 22nd 2021 – 12:00 PST: Added extra information about known older versions of the Pure Storage plugin which need extra steps to rememdiate