Workaround instructions to address CVE-2021-44228 and CVE-2021-45046 in vCenter Server and vCenter Cloud Gateway (87081)

23.12.2021
48
Workaround instructions to address CVE-2021-44228 and CVE-2021-45046 in vCenter Server and vCenter Cloud Gateway (87081)
REKLAM ALANI

Highlighted sections indicate the most recent updates. See the Change log at the end of this article for all changes and subscribe to the article for updates.

Impact / Risks
  • VCHA needs to be removed before executing the steps in this article. It can be reconfigured afterward.
  • Environments with external PSCs need to have the steps taken on both vCenter and PSC appliances.
  • Restoring from a File-Based Backup will put the environment into a vulnerable state again. Use the vc_log4j_mitigator.py script after restoring to correct this
  • Upgrading the vCenter Appliance to an unmitigated version will put the environment into a vulnerable state again. Use the vc_log4j_mitigator.py script after upgrading to correct this
Resolution
The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 and CVE-2021-45046 when available.

IMPORTANT: vc_log4j_mitigator.py will now mitigate CVE-2021-44228 and CVE-2021-45046 on vCenter Server end-to-end without extra steps. This script replaces the need to run remove_log4j_class.py and vmsa-2021-0028-kb87081.py independently. However, it is not necessary to run if you’ve already used those in your environment.

Please use the below 3 scenarios to verify if you’re previous steps were sufficient:

ARA REKLAM ALANI

Completed remediation scenarios:
1. Used vc_log4j_mitigator.py
2. Used vmsa-2021-0028-kb87081.py script from KB 87088 and remove_log4j_class.py from this KB.
3. Used the manual workaround steps in this KB and remove_log4j_class.py.

If you are unsure if the previous steps were sufficient, run the vc_log4j_mitigator.py script with the “–dryrun” option to verify if the environment still has vulnerable files. If any are found, run it without the dryrun flag to correct them.

Workaround

Automated Workaround (Recommended)

To apply the workaround for CVE-2021-44228 and CVE-2021-45046 with the unified script:

1. Download the script attached to this KB (vc_log4j_mitigator.py)

2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)

3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work

4. Execute the script copied in step 1:

python vc_log4j_mitigator.py

This will stop all vCenter services, updates all necessary files with the formatMsgNoLookups flag, removes the JndiLookup.class from all jar/war files on the appliance, and finally starts all vCenter services. The files that the script modifies will be reported as the script runs.

5. To verify that no more vulnerable files exist, execute the script again with the dry run flag:

python vc_log4j_mitigator.py -r

The list of vulnerable files should be zero

Note: Certain 3rd party vSphere Client plugins may be detected as vulnerable even after remediation is done. This is because the jar files are recreated each time the client is started. These must be addressed by either patching the plugin to an unaffected version, marking it incompatible, or removing the plugin via the vCenter MOB. Disabling the plugin from the vSphere Client will not work.

This has been found with older versions of the Pure Storage HTML client plugin 4.3.1 and below . The file continually marked as vulnerable will be: /usr/lib/vmware-vsphere-ui/server/work/Catalina/localhost/ROOT/eclipse/configuration/org.eclipse.osgi/321/0/.cp/log4j-core-2.11.2.jar.

Optional Arguments
-h, –help            show this help message and exit
-d dirnames [dirnames …], –directories dirnames [dirnames …]
space separated list of directories to check
recursively for CVE-2021-44228 vulnerable java archive
files.
-a, –accept-services-restart
acccept the restart of the services without having
manual prompt confirmation for the same
-r, –dryrun          Run the script and log vulnerable files without
mitigating them. The vCenter services are not
restarted with this option.
-b BACKUP_DIR, –backup-dir BACKUP_DIR
Specify a backup directory to store original files.
-l LOG_DIR, –log-dir LOG_DIR
Specify a directory to store log files.

 

Manual Workaround

VMware highly recommends using the vc_log4j_mitigator.py script instead of these manual steps to avoid errors in this process.

To manually apply the workaround for CVE-2021-44228 and CVE-2021-45046 to vCenter Server Appliance 7.x and 6.x, skip to the relevant sections below and follow the full instructions there:

Note: For vCenter Cloud Gateway, only the steps for the vMon Service, Analytics Service, and remove_log4j_class.py script are necessary.

vCenter Server Appliance 7.0.x Workaround

vMON Service

1. Backup the existing java-wrapper-vmon file:

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
2. Update the java-wrapper-vmon file with a text editor such as vi:
vi /usr/lib/vmware-vmon/java-wrapper-vmon
3. At the very bottom of the file, replace the very last line with 2 new lines.
NOTE:  Execute this step number ‘3’ based on the vCenter version running in your environment. The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below:
  • vCenter 7.0 Update 3, 3a, 3b
  • vCenter 7.0 Update 2, 2a, 2b, 2c, 2d
Original

exec $java_start_bin $jvm_dynargs $security_dynargs $original_args

Updated

log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args

NOTE: The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below:

  • vCenter 7.0 GA, 7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d
  • vCenter 7.0 Update 1, U1a, U1c, U1d
Original

exec $java_start_bin $jvm_dynargs “$@”

Updated

log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”

4. Ensure the file permissions are set correctly with the below commands:
chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
5. Restart vCenter Services
service-control –stop –all
service-control –start –all

Update Manager Service

1. Back up the existing start.ini file.
cp -rfp /usr/lib/vmware-updatemgr/bin/jetty/start.ini /usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak
2. Update the start.ini file.
vi /usr/lib/vmware-updatemgr/bin/jetty/start.ini
3. Add the following line to the end of the file:
-Dlog4j2.formatMsgNoLookups=true
4. Restart the Update Manager Service with this command:
service-control –restart vmware-updatemgr

Analytics Service

This is now remediated with remove_log4j_class.py at the end of this section.

Run the remove_log4j_class.py script

1. Download the script attached to this KB (remove_log4j_class.py)

2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)

3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work

4. Execute the script copied in step 1:

python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.
1. Verify if the vMon services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups

Check if the processes includes -Dlog4j2.formatMsgNoLookups=true.

2. Verify the Update Manager changes are shown under “System Properties” in the output of the following two commands:
cd /usr/lib/vmware-updatemgr/bin/jetty/
java -jar start.jar –list-config

System Properties:
——————
log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini
)

3. Verify the Analytics Service changes:
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
Note: This should return 0 lines.

4. Verify the script successfully removed JndiLookup.class from all java files with the following command:

python remove_log4j_class.py -r

The list of vulnerable files should now be empty.

2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
=====     Summary     =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.

vCenter Server Appliance 6.7.x Workaround

vMON Service

1. Backup the existing java-wrapper-vmon file:

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak

2. Update the java-wrapper-vmon file with a text editor such as vi:

vi /usr/lib/vmware-vmon/java-wrapper-vmon

3. At the very bottom of the file, replace the very last line with 2 new lines:

Original
exec $java_start_bin $jvm_dynargs “$@”

Updated

log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”

4. Restart vCenter Services:

service-control –stop –all
service-control –start –all

Note: If the services do not start, ensure the file permissions are set correctly with these commands:

  • chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
  • chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

 

Secure Token Service

Note: These steps are only applicable to embedded vCenter and external PSC nodes

1. Back up and edit the the vmware-stsd file:

cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak
vi /etc/rc.d/init.d/vmware-stsd
2. Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
start_service()
{
perform_pre_startup_actions
  local retval
JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events  \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
3. Restart the vmware-stsd service:
service-control –stop vmware-stsd
service-control –start vmware-stsd

Identity Management Service

Note: These steps are only applicable to embedded vCenter and external PSC nodes

1. Back up and edit the the vmware-sts-idmd file:

cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak
vi /etc/rc.d/init.d/vmware-sts-idmd
2. Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
3. Restart the vmware-sts-idmd service:
service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd

Analytics Service and CM Service

These are now remediated with remove_log4j_class.py at the end of this section.

Run the remove_log4j_class.py script

1. Download the script attached to this KB (remove_log4j_class.py)

2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)

3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work

4. Execute the script copied in step 1:

python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.
1. Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

Note: This will also verify the workaround for the Analytics service in 6.7 U3p

2. Verify the Analytics Service changes:

grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
Note: This should return 0 lines. This does not apply to vCenter 6.7 U3p.
3. Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

Note: This should return 0 lines.

4. Verify the script successfully removed JndiLookup.class from all java files with the following command:

python remove_log4j_class.py -r

The list of vulnerable files should now be empty.

2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
=====     Summary     =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.

vCenter Server Appliance 6.5.x Workaround 

vMON Service

1. Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
2. Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
3. At the very bottom of the file, replace the very last line with 2 new lines.
Original
exec $java_start_bin $jvm_dynargs “$@”

Updated

log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”

4. Restart vCenter Services:
service-control –stop –all
service-control –start –all

Note: If the services do not start, ensure the file permissions are set correctly with these commands:

  • chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
  • chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

 

Secure Token Service

Note: These steps are only applicable to embedded vCenter and external PSC nodes

1. Back up and edit the the vmware-stsd file:

cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak
vi /etc/rc.d/init.d/vmware-stsd
2. Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
start_service()
{
perform_pre_startup_actions

local retval
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events  \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start

3. Restart the vmware-stsd service
service-control –stop vmware-stsd
service-control –start vmware-stsd

Identity Management Service

Note: These steps are only applicable to embedded vCenter and external PSC nodes

1. Back up and edit the the vmware-sts-idmd file:

cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak
vi /etc/rc.d/init.d/vmware-sts-idmd
2. Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
3. Restart the vmware-sts-idmd service:
service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd

PSC Client Service

Note: These steps are only applicable to embedded vCenter and external PSC nodes

1. Back up and edit the vmware-psc-client file

cp -rfp /etc/rc.d/init.d/vmware-psc-client /root/vmware-psc-client.bak
vi /etc/rc.d/init.d/vmware-psc-client
2. Insert a new line near line 300, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Djava.io.tmpdir=”$CATALINA_BASE/temp” \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start

3. Restart the vmware-psc-client service:
service-control –stop vmware-psc-client
service-control –start vmware-psc-client

CM Service

This is now remediated with remove_log4j_class.py at the end of this section.

Run the remove_log4j_class.py script

1. Download the script attached to this KB (remove_log4j_class.py)

2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)

3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It’s necessary to enable the bash shell before WinSCP will work

4. Execute the script copied in step 1:

python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as “VULNERABLE FILE” as the script runs.

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.
1. Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

2. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

Note: This should return 0 lines.

3. Verify the script successfully removed JndiLookup.class from all java files with the following command:

python remove_log4j_class.py -r

The list of vulnerable files should now be empty.

2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
=====     Summary     =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.

vCenter Server Appliance 6.0 U3j Workaround

vCenter Server Appliance 6.0 U3j is no longer in general support but has also been identified as vulnerable to CVE-2021-44228 due to the Performance Charts service. The remove_log4j_class.py script is not verified in vCenter 6.0. Please use the manual steps below to remove the class. Mitigation steps have been identified as follows:
1. Back up and edit /usr/lib/vmware-perfcharts/wrapper/conf/wrapper.conf on the appliance and add a new line just below “wrapper.java.additional.13=-Dlog4j.configurationFile=file:/etc/vmware-perfcharts/log4j2.xml” (line 72) with the following content:
wrapper.java.additional.14=-Dlog4j2.formatMsgNoLookups=true

2: Back up the log4j-core-2.11.2.jar file

cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak

3. Run the zip command to disable the class

zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

3. Stop the service and then start it through service-control:

service-control –stop vmware-perfcharts
service-control –start vmware-perfcharts
Note: vCenter Server Appliance versions 6.0GA – 6.0U3i are not vulnerable. However, versions 6.0 U3a/b/c/d/e/f were found to contain the following unused vulnerable jar files. No impact on the product has been observed after removing these jar files.
  • /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-core-2.1.jar
  • /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-api-2.1.jar
  • /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-core-2.1.jar
  • /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-api-2.1.jar
Related Information
To revert the workarounds, replace the modified files with the backups created in each step.
VCHA needs to be removed before executing the steps in this KB.

Change log:

  • December 12th 2021 – 11:20 PST: Updated Knowledge Base article advising workarounds for 6.5/6.7 partially address the vulnerability.
  • December 12th 2021 – 16:00 PST: Added steps for stsd, idmd and psc-client services to the workaround sections for 6.5 and 6.7.
  • December 12th 2021 – 17:00 PST: Added workaround for VCSA 6.0U3j.
  • December 12th 2021 – 18:30 PST: Added additional information for older versions of VCSA 6.0.
  • December 12th 2021 – 20:30 PST: Updated versions with vulnerable jar files in VCSA 6.0.
  • December 13th 2021 – 01:57 PST: Updated the steps in the vMON service commands for permission validation
  • December 13th 2021 – 16:30 PST: Simplified and re-arranged beginning of workaround section (no new content). Added clarification to 6.7U3p analytics verification steps. Updated cm service restart commands. Added highlighting to changelog.
  • December 13th 2021 – 20:05 PST: Removed DBCC Utility Workaround step from this article. VMware has tested this and does not have any impact or risk hence the workaround section was removed.
  • December 15th 2021 – 11:00 PST: Added a notice at the top concerning the recent updates on CVE-2021-44228 from Apache.
  • December 15th 2021 – 17:30 PST: Formatting changes for readability (no new content)
  • December 16th 2021 – 14:15 PST: Added new remove_log4j_class.py script which replaces several steps in this article and addresses new findings in CVE-2021-44228 and CVE-2021-45046. Added guidance in each section for its use.
  • December 17th 2021 – 08:00 PST: Added clarification in the workaround steps that both remediation scripts are still necessary for completely un-remediated environments. Added step to vCenter Server 6.0 U3j instructions to manually remove the JndiLookup.class class
  • December 17th 2021 – 16:30 PST: Updated remove_log4j_class.py to output a change summary and better handle “File is not a zip file” issues. Added section with new cmd line options for remove_log4j_class.py. Added “–dryrun” verification step to all manual sections.
  • December 20th 2021 – 18:00 PST: Added vc_log4j_mitigator.py which unifies the previous steps and scripts into a single workaround. Added note about upgrades and file-based backup/restore operations to Impacts section.
  • December 21st 2021 – 12:45 PST: Added note about 3rd party plugins reverting to a vulnerable state after the vsphere client is started. Added clarification that CVE-45046 is also addressed with all the changes since the “December 16th 2021 – 14:15 PST” update.
  • December 22nd 2021 – 12:00 PST: Added extra information about known older versions of the Pure Storage plugin which need extra steps to rememdiate
REKLAM ALANI
ETİKETLER: ,
YORUMLAR

Henüz yorum yapılmamış. İlk yorumu yukarıdaki form aracılığıyla siz yapabilirsiniz.