20 september 2017

We’ve spent a bunch of time investigating Cisco ASA devices and their firmware while looking into exploiting CVE-2016-1287, CVE-2016-6366, and other bugs. Part of this research has involved data mining numerous Cisco ASA firmware files to generate new exploit targets.

We took the time to write some tools to more effectively analyse or debug certain aspects of the exploits and automate a lot of the tasks we found repetitive. Many of these tools help speed up investigations of problems/bugs, debugging vulnerabilities and exploit states, and the general poking around of Cisco devices for research.

We decided to share some of our research and the associated tools. We are hoping this information and code will encourage others to investigate and research Cisco ASA devices.

To relay this information, we will be releasing a series of blogs describing a variety of topics, with this being the introductory article.

The goal of this introduction is to present some basic information about the Cisco ASA and to acknowledge the past research in this area that we found invaluable.

We plan to release blogs discussing the following topics:

Introduction to blog series and Cisco ASA
asafw: A set of scripts for automating Cisco ASA firmware analysis and datamining
asadbg: A set of scripts for automating initialization and runtime debugging of Cisco ASA devices/emulators
libdlmalloc: A gdb plugin for analysing the dlmalloc-2.8.x heap, as well as looking at how Cisco uses this heap
libptmalloc: A gdb plugin for analysing the ptmalloc2 heap, as well as how Cisco uses this heap
libmempool: A gdb plugin for analysing mempool-related chunk and mstate structures and how Cisco wraps heap functionality
Checkheaps: A look at how Cisco’s Checkheaps watchdog works on an ASA and how to trivially bypass it given a suitable write primitive
CVE-2016-1287 via IKEv1: A look into how exploitation of the IKE fragmentation vulnerability differs on IKEv1 vs IKEv2

In the future, we may choose to release additional information beyond these topics as well. You can access the main repository to pull all NCC Group Cisco ASA-related tool projects at https://github.com/nccgroup/asatools.

What is the Cisco ASA?

Several ASA firewall models [1]

Past Cisco IOS ASA research

Before going over the basics of the Cisco ASA, we’ll start by acknowledging some previous Cisco-related research that is helpful when learning about Cisco device exploitation, as well as setting up basic debug environments for a Cisco ASA5505 device.

FX and Michael Lynn detailed some techniques to exploit Cisco Internetwork Operating System (IOS) devices in Router Exploitation [2] and The Holy Grail: Cisco IOS Shellcode And Exploitation Techniques [3]. Alec Stuart-Muirk detailed several techniques [4] to get started with Cisco ASA, as well as vulnerabilities, in WebVPN.

David Barksdale, Jordan Gruskovnjak, and Alex Wheeler from Exodus Intel detailed a critical vulnerability in IKE and released a blog post [5] as well as some slides [6] and a working exploit [7]. Xorcat looked at using and improving the EXTRABACON [8] (CVE-2016-6366) exploit and discussed the Cisco ASA.

Silent Signal released some research [9] about analysing and improving the Equation Group’s EXTRABACON exploit. William Webb wrote [10] an article about looking into the Cisco ASA as a follow up to Exodus Intel’s research and encouraged people to look into ASA internals.

It is worth noting that the main information we used for starting to debug and analyse the Cisco ASA devices was the work done by both Alec Stuart in his BreakingBricks presentation/demos [11], and the researchers at Exodus Intelligence [12]. We would like to specifically call out their papers and presentations and thank them for publicly sharing their work.

It should be noted that some of the information provided in our write-ups will occasionally overlap with some of the details provided in other ASA-related work and this is simply to make it easier to follow along with what we do. We do still highly recommend reading through all previous research.

Test devices

The majority of our research was carried out on two ASA5505 devices for 32-bit (x86) and emulated devices for 64-bit (x86_64) using the Graphical Network Simulator-3 (GNS3) [13]. We did some testing on an ASA 5512-X (64-bit), however this was more limited. We extensively used GNS3 for 64-bit testing, as it is easier and faster to debug than a physical 64-bit device.

As you can probably deduce from above, as with both Exodus Intelligence and Alec Stuart-Muirk, we recommend picking up an ASA5505 device for 32-bit research. They can be found for a low cost online. If you do opt to purchase a 64-bit device, the ASA5506-X appears to be the lowest tier device that is also reasonably priced online.

ASA architecture

Command Line Interface (CLI)

In general, all configuration and administration is done using the Cisco’s Command Line Interface (CLI), which will be familiar to those who have worked with other Cisco devices. We won’t go into too much detail about CLI commands here because they are very similar to the commands available on Cisco IOS. However, they are different enough to confuse people already familiar with Cisco IOS. Refer to the Cisco documentation [14] for a complete command listing.

A list of useful commands we used regularly includes:

Command	Use
`terminal pager 0`	Disable slow scrolling
`enable`	Enter privileged mode. This is required to execute most of the interesting commands
`copy flash:/new_config running-config`	Load a config file from flash (non-persistent)
`copy running-config flash:/saved_config`	Save the running config
`copy running-config startup-config`	Save running config to default config (persistent)
`show running-config`	Show the current config
`config terminal`	Enter config mode. This is to configure most interesting settings
`configure factory-default`	Reset to default settings
`no call-home reporting anonymous`	Disable anonymous reporting
`format flash`	Format an inserted flash card
`dir flash:/`	List files on the flash
`verify /md5 my_file`	Verify md5sum of file
`crashinfo save disable`	Disable crash dumps

lina

When investigating the ASA, it is generally useful to know a little bit about Cisco IOS internals. In Felix Linder’s Router Exploitation paper [15], it was noted that IOS uses a single monolithic binary (ELF) running directly on the main CPU. Historically, IOS would typically be found on PPC32, MIPS32 or MIPS64 CPU architectures. He notes that they use their own run-to-completion scheduler for internal processes.

Interestingly, unlike traditional IOS, the ASA firmware is a Linux operating system running a single ELF called lina. Allegedly, though we are not personally familiar with it, the older PIX devices ran on a custom OS called PIX OS and it may be that they had a similar binary containing most IOS-like functionality running on them.

The lina program manages many internal processes to handle different features and some versions appear to use something along the lines of a run-to-completion scheduler to manage these internal processes. Note that the internal processes are handled by a scheduler implemented in lina and are different from Linux processes. In other words, it is up to lina to choose how important they are and manage scheduling. It is important from an exploitation point of view when dealing with feng shui and concurrency.

Although the ASA doesn’t officially run on Cisco IOS, we suspect that lina is essentially the old Cisco IOS ELF binary ported enough to run inside userland on Linux. This was suggested by Exodus Intel in their CanSecWest slides.

The processes above will not be observed from a traditional ps -ef from a Linux debug shell. Instead, you will just see lina running. Furthermore, the processes shown above don’t directly correlate to individual process threads running inside lina that you could observe in gdb. Instead, lina apparently implements its own internal logic for process creation and management and includes an internal scheduler to manage said processes. We won’t be going into great detail about the internal process management or scheduling, however.

Serial console

Cisco ASA devices generally provide a dedicated serial port that can be used to interact with the system console. In the examples we show throughout this series where we interact with the bootloader or console, we are simply using the serial console provided by our devices unless otherwise noted. You usually need two cables to access the serial port. We opted for the ones below but others should work too.

RJ45 Male to D-Sub RS232 DB9 Female cable [16] (a.k.a. Cisco console cable) (~£15)
FTDI USB to Serial RS232 adapter [17] (~£20)

Boot chain

When the ASA boots, the initial bootloader (called ROMMON) looks for a firmware to boot (e.g. asa924-k8.bin). This firmware is on the flash that will typically be located on a CompactFlash (CF) card [18] or in an embedded USB (eUSB) [19] device, which is a flash chip with a USB interface. The specific version of firmware to load is generally dictated by a default or saved ASA configuration on the device. This firmware contains the Linux kernel and the initial root filesystem. When the kernel loads, a script is responsible for starting lina_monitor. Finally, lina_monitor starts lina.

ASA boot chain

The ROMMON bootloader can be interrupted by hitting ESC or BREAK when the below text is displayed:

Launch BIOS Extension to setup ROMMON

Cisco Systems ROMMON Version (1.0(12)13) #0: Thu Aug 28 15:55:27 PDT 2008

Platform ASA5505

Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.
Boot in 10 seconds

There are several commands available, shown in the listing below. You can download an image from a TFTP server using tftpdnld. Notice the boot command that allows you to manually boot a firmware image:

Use ? for help.
rommon #0> ?

   Variables: Use "sync" to store in NVRAM
ADDRESS=       local IP address
CONFIG=        config file path/name
GATEWAY=       gateway IP address
IMAGE=         image file path/name
LINKTIMEOUT=    Link UP timeout (seconds)
PKTTIMEOUT=     packet timeout (seconds)
PORT=          ethernet interface port
RETRY=          Packet Retry Count (Ping/TFTP)
SERVER=        server IP address
VLAN=           enable/disable DOT1Q tagging on the selected port

   Commands: 
?                    valid command list
address        local IP address
boot           boot an image, valid args are:
       - "image file spec" and/or 
       - "cfg="
...
server         server IP address
set                  display all variable settings
show            display cmd-specific information
sync                 save variable settings in NVRAM
tftpdnld             TFTP download
timeout         packet timeout (seconds)
trace                toggle packet tracing
unset       unset a variable name

After executing the boot command, and waiting a few seconds (or minutes, depending on your device’s CPU speed), we get the Cisco CLI. At this stage lina is already started, as it is what actually provides the interface. Below, you can see that we can specify not only a specific asa*.bin file to load, but also a specific config file (with cfg=) both of which will be looked up from flash.

rommon #0> boot asa924-k8.bin cfg=config-924
Launching BootLoader...
Boot configuration file contains 1 entry.


Loading asa924-k8.bin....... Booting...
Platform ASA5505
[...]
Type help or '?' for a list of available commands.
ciscoasa>

The ability to interrupt the boot process and load any firmware version and config allows for some fairly powerful automation when it comes to testing exploits against numerous builds. As much of the past research showed, it allows someone to modify scripts at runtime to enable gdb (assuming they don’t want to do the changes offline for whatever reason). We will detail in a future blog post how we leveraged this to easily debug multiple ASA versions.

As an aside, it’s worth noting that in the previous research on Cisco IOS devices, FX leveraged ROMMON to execute a ROP payload as there are far fewer versions of ROMMON than IOS firmware.

ASA model types file formats

From a high level perspective, two different ASA models are provided by Cisco: 32-bit and 64-bit (the latter known as the X-series). Each device supports a variety of different firmware versions with different features depending on the purchased license. Because this is license-based, the same firmware is generally used for the different hardware (as long as they are the same architecture). A firmware such as asa924-k8.bin is used on 32-bit devices (5500 models [20]). asa924-smp-k8.bin is the equivalent for 64-bit (5500x models [21]). Note that, generally, SMP stands for symmetric multiprocessing [22], but most firmware with -smp in the name are actually limited to use for 64-bit hardware, which is a useful way to discriminate between firmware types prior to data extraction.

SPA firmware files, such as asa971-8-lfbff-k8.SPA, are used for newer ASA with FirePOWER [23]. Finally, firmware such as asav941-200.qcow2 are used for GNS3 [24].

Note that the k8 is apparently used for models with DES only while k9 indicates 3DES/AES support. Depending on what you’re working on, using k8 vs k9 shouldn’t affect your work in a meaningful way.

Heaps found on the ASA

Both 32-bit and 64-bit versions of Cisco ASA include a slightly customised version of dlmalloc-2.8.3 [25]. On 32-bit, this heap is used for almost all general purpose and DMA-related allocations. On 64-bit, general purpose allocations are instead done through glibc, and thus use glibc’s fork of ptmalloc2 [26]. These heaps are used to manage larger memory structures referred to by Cisco as mempools.

The Exodus Intelligence blog and slides briefly mention that the heap used is dlmalloc, however no explicit version is indicated. The dlmalloc behavior differs heavily between 2.6.x, 2.7.x, and 2.8.x, so it is important to make the distinction to better understand exploitation scenarios.

Both dlmalloc-2.8.x and ptmalloc2 allocations are wrapped by mempool shim functions that modify the allocation sizes and inject sub-structures into the resulting chunks for book keeping and debugging purposes.

We will go into more detail about the heaps observed, as well as the mempool logic, in future articles.

GNS3

Cisco officially supports emulating newer ASA firmware using a tool called GNS3. You can use this tool to emulate all sorts of different Cisco devices and create virtual networks. Instead of loading an asa*.bin firmware, you would load an asav*.qcow2 (Qemu Copy-on-Write [qcow]) which is an image qemu understands and which contains the asa*.bin and the equivalent of a flash partition.

ASA on GNS3

It is technically possible to load non-qcow ASA images into GNS3, although it’s not officially supported and your mileage may vary. For instance, we had difficulty creating 32-bit GNS3 images that could both work on the network (in order to exploit remotely) and that could have gdb attached. In the end, for 32-bit research we opted to stick with physical devices, whereas we were able to do most of our 64-bit investigations using GNS3.

Convenience tips

If you are going to be playing with Cisco ASA devices for research, there are a few general tips that we found will make your life easier, especially if you use some of the tools we will be releasing:

Configure a static IP address for the device so you always know where to talk to. This will make using the scripts a lot easier.
Consider using a USB-based Ethernet adapter. This simplifies networking and makes sure you don’t have some rootable firmware lying around directly on your network. This will allow you to more easily isolate traffic of interest in a tool like Wireshark or similar.
Use a dedicated switched power strip. The Cisco ASA devices we used did not have a physical reboot button or power button. In order to power cycle, you must unplug and replug the power cable. This quickly becomes tedious, so do yourself a favor early and have a switch handy.
Invest in a bigger CF card. The default one on an ASA5505 is 128MB but it does not allow storing lots of different firmware (each being ~30MB). This 8GB CF card [27] did the job for us for £9.
Buy a CF card reader (£9) [28]. You can always copy them over SSH but it is a lot faster to use a reader if you have several firmware files to copy.
While replacing a CF card that was malfunctioning, we noticed that the ASA 5505 (and likely other models as well) appears to have an 8GB size limit for CF cards. Inserting a 16GB CF card will simply cause the BIOS to fail POST. This size limit doesn’t appear to be well documented and we didn’t investigate further. Interestingly this drops you to an extended BIOS debug (EBDEBUG) shell (which doesn’t seem to be well documented).
There is a maximum length for the name of a firmware .bin that will be loaded. Be aware that if you use a long naming scheme and firmware isn’t loading as you’d expect, this could be the culprit.
We tried to replace the default RAM of 512MB with 1GB but haven’t noticed much debugging speed improvement.

Testing out configurations

If you want to do exploit development on the Cisco ASA then a good starting point is trying to test out either the public Exodus Intelligence exploit for CVE-2016-1287 or Equation Group’s or Metasploit’s EXTRABACON exploit for CVE-2016-6366. Both of these can target version 9.2.4 and we therefore recommend using this version as your starting point (as we did).

Taking the time to setup and understand how to configure the device for exploitation is a good opportunity to familiarize yourself with the CLI interface, as well as give some hints as to how a real-world device might be configured.

That said, we provide an example configuration in Appendix A to help people get started quickly if they want to get up and running with their testing on either IKE or SNMP.

Historical vulnerabilities

In addition to CVE-2016-1297 and CVE-2016-6366 (that are probably the better known vulnerabilities now) there are numerous other Cisco ASA vulnerabilities of interest. Many of these were mentioned in the Breaking Bricks presentation. Of particular note is that you can use things like the WebVPN vulnerabilities CVE-2014-3398 and CVE-2016-1295 to remotely leak the version via the external interface. These can aid in exploiting a bug like CVE-2016-1297 where you can’t leak the exact version over IKE directly.

Conclusion

This post has introduced our blog series and provided a very brief overview of the Cisco ASA device line. If you want to do some research on Cisco ASA, test out the Exodus Intelligence exploit or NSA’s EXTRABACON-based exploits, or some of the debugging tools we will release in the future, you can hopefully use the information provided above as a quick introduction and starting point.

We would appreciate any feedback or corrections. If you would like to contact us we can be reached by email or twitter: aaron(dot)adams(at)nccgroup(dot)trust / @fidgetingbits and cedric(dot)halbronn(at)nccgroup(dot)trust / @saidelike.

We would like to thank Vlad Ionescu, Katy Winterborn, Nick Dunn and Elliot Morrow for their review of the blogs in this series.

Appendix A: Cisco ASA configuration

The following is an example configuration from an ASA 5505 that has IKEv1, IKEv2, SSH, and SNMP enabled. It assumes the IP address of an external host is 192.168.210.78, and configures the Cisco ASA device to be 192.168.210.77

: Saved
: Hardware: ASA5505, 1024 MB RAM, CPU Geode 500 MHz
!
ASA Version 9.2(4) 
!
hostname ciscoasa-924-ikev1-2-ssh
enable password 8Ry2YjIyt7RRXU24 encrypted
names
!
interface Ethernet0/0
 switchport access vlan 200
!
interface Ethernet0/1
 switchport access vlan 100
!
interface Ethernet0/2
 shutdown
!
interface Ethernet0/3
 shutdown
!
interface Ethernet0/4
 shutdown
!
interface Ethernet0/5
 shutdown
!
interface Ethernet0/6
 shutdown
!
interface Ethernet0/7
 shutdown
!
interface Vlan100
 nameif inside
 security-level 100
 ip address 192.168.211.77 255.255.255.0 
!
interface Vlan200
 nameif outside
 security-level 0
 ip address 192.168.210.77 255.255.255.0 
!
boot config disk0:/config-ip-only
ftp mode passive
object network obj_any
 subnet 0.0.0.0 0.0.0.0
access-list RA_ACCESS_LIST_V2 extended permit ip host 192.168.210.78 host 192.168.210.77 
access-list RA_ACCESS_LIST_V1 extended permit ip host 192.168.210.78 host 192.168.210.77 
pager lines 24
logging asdm informational
mtu inside 1500
mtu outside 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
aaa authentication ssh console LOCAL 
http server enable
snmp-server host outside 192.168.210.78 community cisco
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart warmstart
crypto ipsec ikev1 transform-set TEST_TRANSFORM_V1 esp-des esp-sha-hmac 
crypto ipsec ikev2 ipsec-proposal TEST_PROPOSAL_V2
 protocol esp encryption des
 protocol esp integrity md5
crypto ipsec security-association pmtu-aging infinite
crypto map CRYPTO_M_V1 1 match address RA_ACCESS_LIST_V1
crypto map CRYPTO_M_V1 1 set peer 192.168.210.78 
crypto map CRYPTO_M_V1 1 set ikev1 transform-set TEST_TRANSFORM_V1
crypto map CRYPTO_M_V2 1 match address RA_ACCESS_LIST_V2
crypto map CRYPTO_M_V2 1 set peer 192.178.1.78 
crypto map CRYPTO_M_V2 1 set ikev2 ipsec-proposal TEST_PROPOSAL_V2
crypto map CRYPTO_M_V2 interface outside
crypto ca trustpool policy
crypto ikev2 policy 10
 encryption des
 integrity sha
 group 2
 prf sha
 lifetime seconds 86400
crypto ikev2 enable outside
crypto ikev1 enable outside
crypto ikev1 policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
telnet timeout 5
ssh scopy enable
no ssh stricthostkeycheck
ssh 0.0.0.0 0.0.0.0 inside
ssh 0.0.0.0 0.0.0.0 outside
ssh timeout 5
ssh version 2
ssh key-exchange group dh-group1-sha1
console timeout 0

dhcpd auto_config inside
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
username user password XEaJpJFaYvDqZKxJ encrypted privilege 15
tunnel-group 192.168.210.78 type ipsec-l2l
tunnel-group 192.168.210.78 ipsec-attributes
 ikev1 pre-shared-key mysharedkey1
tunnel-group TEST_GROUP_V2 type remote-access
tunnel-group TEST_GROUP_V2 ipsec-attributes
 peer-id-validate nocheck
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
 message-length maximum client auto
 message-length maximum 512
policy-map global_policy
 class inspection_default
 inspect dns preset_dns_map 
 inspect ftp 
 inspect h323 h225 
 inspect h323 ras 
 inspect rsh 
 inspect rtsp 
 inspect esmtp 
 inspect sqlnet 
 inspect skinny 
 inspect sunrpc 
 inspect xdmcp 
 inspect sip 
 inspect netbios 
 inspect tftp 
 inspect ip-options 
!
service-policy global_policy global
prompt hostname context 
no call-home reporting anonymous
Cryptochecksum:43af56f7fb6035189d9a8e30d5cb6cb1
: end