Intel® Software Guard Extensions (SGX): A Researcher’s Primer

05 January 2015

By Ollie Whitehouse

Research Research

tl;dr

Intel SGX is a trusted execution environment which provides a reverse sandbox. It’s not yet available but those who have had access to the technology have shown some powerful applications in cloud use cases that on the face of it dramatically enhance security without the performance constraints of homomorphic encryption.

However, there is enough small print to warrant both validation and defensive assessment activities when the technology becomes more generally available.

Introduction

There is a new set of features coming to Intel CPUs that have massive potential for cloud security and other applications such as DRM. However, as with all things that can be used for good there is also the potential for misuse. These features come in the guise of Software Guard Extensions (SGX).

In this post we’ve collated what we know about the technology, what others have said about it and how it is being applied in real-world applications.

What is SGX?

To quote Intel:

Intel® Software Guard Extensions (Intel® SGX) is a name for Intel Architecture extensions designed to increase the security of software through an “inverse sandbox” mechanism. In this approach, rather than attempting to identify and isolate all the malware on the platform, legitimate software can be sealed inside an enclave and protected from attack by the malware, irrespective of the privilege level of the latter.

So in short this means we can create a secure enclave (or a Trusted Execution Environment – TEE – if you wish) at the CPU level which is protected from the OS upon which it is running.

Architecturally Intel SGX is a little different from ARM TrustZone (TZ). With TZ we often think of a CPU which is in two halves i.e. the insecure world and the secure world. Communication with the secure world occurs from the insecure world via the SMC (Secure Monitor Call) instruction. In Intel SGX model we have one CPU which can have many secure enclaves (islands):

Source:  Intel® Software Guard Extensions Programming Reference Rev 2 (#329298-002) – Page 13

Intel SGX is like the Protected Processes Microsoft introduced in Windows Vista but with the added benefit that it is hardware enforced so that even the underlying OS kernel can’t tamper or snoop.

Intel® and Third Party High-Level and Low-Level Background Material

Intel provides a background as to the design goals of SGX in a series of three (currently) blog posts:

Three other useful resources from Intel on how SGX actually works come in the guise of:

From the Microsoft Haven research (see later) we know that revision 2 of the SGX specifications resolved issues with:

  • Exception handling: page faults and general protection faults were not reported in the enclave.
  • Permitted instructions: CPUID, RDTSC and IRET where not permitted previously. Of these RDTSC and IRET now are.
  • Thread-local storage: there were problems due to segment register usage which has been fixed.

Finally a good third-party presentation on Intel SGX is from the Technische Universität Darmstadt, Germany in a lecture on embedded system security titled Trusted Execution Environments Intel SGX. It provides a good summary of the functionality and how key operations occur to save you wading through the programming manual.

Previous Security Analysis

While no CPUs with SGX are available nor any emulators/simulators (publically at least – see later in this post) others have passed initial comment on the potential security ramifications of SGX in both good and evil contexts over the past 18 months.

Application of SGX in the Real World

So how far has the application of SGX in the real world come? As already mentioned given the lack of CPU support the ability to experiment with the technology has been limited to the few. Two examples of groups who have been afforded access are Microsoft Research and the United States Air Force Academy.

The application of the SGX by these two groups is quite radically different. Microsoft have focused on server side applications whilst the Air Force Academy seem to be focusing on client side use cases.

Secure Enclaves-Enabled Technologies: Browser Extension

According to a 2014 National Security Innovation Competition Proceedings Report on Secure Enclaves-Enabled Technologies from the United States Air Force Academy, there is at least one company who has access to the technology and has funding:

Secure Enclaves-Enabled Technologies is a digital security firm to be launched in the coming year. It is born from a unique relationship between Intel Labs and the Department of Homeland Security seeking to develop a revolutionary solution to cyber security problems

Then they reveal their intended application of SGX:

SE Enabled Technologies seeks to exploit the capabilities of SGX through the creation of software solutions. Currently, we are seeking to compliment Intel’s hardware solution through the use of a browser extension application. Using the browser extension, we can offer a wide array of security solutions from secure storage and transmission of documents to secure video streaming.

As of April 12, 2014 according to the Colorado Springs Undergraduate Research Forum proceedings things had moved on:

In today’s increasingly digital world, more and more sensitive information is stored electronically, and more and more often this information comes under attack. With the continual evolution of offensive attack techniques, the need for more impressive defensive counter-measures is becoming apparent.

As the requisite to fill this capability gap grows, so does the opportunity for businesses. Replacing the need for a countermeasure, recent developments in micro-processer technology have created a veritable impenetrable fortress to be placed inside modern day computer systems.

The answer lies in Secure Enclaves – Enabled Technology, a software company that utilizes Intel Labs’ revolutionary Software Guard Extensions technology. Instead of relying on encryption and software, this technology is hardware-based, and is so secure that an NSA Red Team could not crack it.

This technology has tremendous application to both government and private organizations concerned about security. For completing a proof of concept case with the Department of Veterans’ Affairs, Secure Enclaves – Enabled Technologies will receive $500,000 that it will channel into the development of a commercially available security software package.

Further details around Secure Enclaves-Enabled Technologies can be found in the 2014 National Security Innovation Competition proceedings (page 57 onwards) including these points of interest:

SGX will be a standard component of Intel’s chipsets beginning in 2015. However, new software must be developed or current software must be adapted in order to have the ability to utilize the new set of instructions provided by the chipset. Without this critical software development, the cyber security solution afforded by SGX will lie dormant.

Microsoft: VC3

VC3 was published in October 2014 in the paper titled VC3: Trustworthy Data Analytics in the Cloud, for which the abstract states:

We present VC3, the first practical framework that allows users to run distributed MapReduce computations in the cloud while keeping their code and data secret, and ensuring the correctness and completeness of their results.

VC3 runs on unmodified Hadoop, but crucially keeps Hadoop, the operating system and the hypervisor out of the TCB; thus, confidentiality and integrity are preserved even if these large components are compromised. VC3 relies on SGX processors to isolate memory regions on individual computers, and to deploy new protocols that secure distributed MapReduce computations. VC3 optionally enforces region self-integrity invariants for all MapReduce code running within isolated regions, to prevent attacks due to unsafe memory reads and writes

An interesting aspect of the Microsoft VC3 work is the adversary model they considered:

We consider a powerful adversary who may control the entire software stack in a cloud provider’s infrastructure, including hypervisor and OS. The adversary may also record, replay, and modify network packets.

The adversary may also read or modify data after it left the processor using probing, DMA, or similar techniques. Our adversary may in particular access any number of other jobs running on the cloud, thereby accounting for coalitions of users and data center nodes. Further, we assume that the adversary is unable to physically open and manipulate at least those SGX-enabled processor packages that reside in the cloud provider’s data centers.

Microsoft: Haven

Haven was presented at USENIX in October 2014 in a paper titled Shielding applications from an untrusted cloud with Haven (Slides etc. are available from USENIX.), for which the abstract states:

Our prototype, Haven, is the first system to achieve shielded execution of unmodified legacy applications, including SQL Server and Apache, on a commodity OS (Windows) and commodity hardware.

Haven leverages the hardware protection of Intel SGX to defend against privileged code and physical attacks such as memory probes, but also addresses the dual challenges of executing unmodified legacy binaries and protecting them from a malicious host. This work motivated recent changes in the SGX specification.

Further Applications

Two other papers that are related to how SGX can be applied are (both from Intel):

Finally there was some speculation that Ubuntu’s LXD announced early November 2014 might use SGX:

Ubuntu do state that: We’re working with silicon companies to ensure hardware-assisted security and isolation for these containers, just like virtual machines today.

Emulator and CPU Support Status

By now you will no doubt be salivating at the prospect of SGX and want to proto-type:

  1. New first of its kind malware which uses SGX in preparation for BlackHat USA 2015

OR

  1. A Soft-HSM

OR

  1. DRM extensions which use SGX

So what is the status of emulator and CPU support for SGX?

Emulators

Alas no emulators are available to public, although one does exist inside of Intel and shared with select partners.

In June 2014 Intel said in the comments section of the Intel® SGX for Dummies (Intel® SGX Design Objectives) blog post:

Intel is not ready to announce plans for availability of SGX emulation platform yet, but this forum will be updated when we are ready.

So how did Microsoft develop their VC3 and Haven proto-types? In the Microsoft VC3 paper from February 2014 Microsoft said:

We successfully tested our implementation in an SGX emulator provided by Intel

More interestingly they then go on to say:

However, since that emulator is not performance accurate, we have implemented our own software emulator for SGX. Our goal was to use SGX as specified in [31] as a concrete basis for our VC3 implementation and to obtain realistic estimates for how SGX would impact the performance of VC3. Our software emulator does not attempt to provide security guarantees.

The emulator is implemented as a Windows driver. It hooks the KiDebugRoutine function pointer in the Windows kernel that is invoked on every exception received by the kernel. Execution of an SGX opcode from [31] will generate an illegal instruction exception on existing processors, upon which the kernel will invoke our emulator via a call to KiDebugRoutine.

The emulator contains handler functions for all SGX instructions used by VC3, including EENTER, EEXIT, EGETKEY, EREPORT, ECREATE, EADD, EEXTEND, and EINIT. We use the same mechanism to handle accesses to model specific registers (MSR) and control registers as specified in [31]. We also modified the SwapContext function in the Windows kernel to ensure that the full register context is loaded correctly during enclave execution.

In Microsoft Haven research in October 2014 they said:

We developed Haven on an instruction-accurate SGX emulator provided by Intel, but evaluate it using our own model of SGX performance. Haven goes beyond the original design intent of SGX, so while the hardware was mostly sufficient, it did have three fundamental limitations for which we proposed fixes (§5.4). These are incorporated in a revised version of the SGX specification, published concurrently by Intel.

Note: The revised version they refer to is Rev 2 of the Intel® Software Guard Extensions Programming Reference.

Aside from this there was a project at Georgia Institute of Technology to add Intel SGX Emulation using QEMU which they appear to have achieved between their plan presentation on October 20, 2014 and their summary presentation on December 1, 2014. However a quick search of the QEMU commits finds no reference to their commits.

CPU Support

Currently there is no CPU on the market which supports the SGX or SGX2 instruction set.

Future Security Research

So first off from reading the programming reference some initial questions would be:

  • Do the ‘SGX Enclave Control Structures’ (SECS) have their integrity ensured? The SECSs contain meta-data used by the CPU to protect the enclave and will thus be a prime target.
  • Do the ‘Thread Control Structures’ (TCS) have their integrity ensured? The TCSs contain meta-data used by the hardware to save and restore thread specific information when entering / exiting the enclave.
  • Does the ‘Enclave Page Cache Map’ (EPCM) have its integrity ensured? The EPCM is used to keep track of which bits of memory are in the ‘Enclave Page Cache’ (EPC) and is ‘managed by the processor’.

The reason for the above questions are in part driven by the obvious but also by the fact that Intel mention that they stop certain concurrent operations to protect the integrity of these structures.

Also it is important to differentiate between integrity provided by the Memory Encryption Engine (MEE – more on this later as it provides external memory modification protection) and that afforded by the microcode operating on them. In the presentation Dynamic Root of Trust and Trusted Execution we see that MEE does provide integrity protection from external modification:

Some other questions which spring to mind also include:

  • What are the algorithm and key generation mechanism for the ‘Enclave Page Cache’ (EPC)? The EPC is the RAM used by the secure enclaves. Where it is part of the RAM they are protected by an encryption engine.
  • The whole debug problem. If you compromise or own the underlying OS before attestation of the enclave has occurred then there is likely the obvious bootstrap problem and where the Trusted Computing Base (TCB) is designed to help. This is also where remote attestation (as discussed by Joanna Rutkowska) will become critical to make sure that the aggressor has pre-owned the environment allowing them to catch the provisioning process within the supposed secure enclave.

Source:  Intel® Software Guard Extensions Programming Reference Rev 2 (#329298-002) – Page 177

Aside from the topics mentioned it is clear that the value of any microcode vulnerability or errata that allow subversion. For example the diagram below shows Processor Reserved Memory (PRM) which is used to house some of the structures mentioned in the question section.

Source:  Intel® Software Guard Extensions Programming Reference Rev 2 (#329298-002) – Page 40

How is integrity of the memory ensured in the EPC implementation? Looking at this related patent Parallelized counter tree walk for low overhead memory replay protection

As the lower-level counters (including L2, LI and L0 counters and the version nodes 260) are off the processor die and therefore are susceptible to attacks, each counter and each version node are encoded with an embedded Message Authentication Code (MAC) (shown as the blocks with hatched lines) to ensure their integrity.

In one embodiment, each embedded MAC is computed over the line in which they are embedded, using a corresponding counter from the next higher level as input. In the example of Figure 2, the embedded MAC for the version block 250 associated with L03 (shown in Figure 2 as the middle version block) is computed using the values of Vo – Vp and its corresponding L0 counter (LO3).

The value of this embedded MAC is stored striped in the line of the version blocks 250 (shown as striped boxes in Figure 2). The embedded MAC for each line of L0, LI and L2 is computed similarly. L3 counters do not need embedded MACs because the contents of L3 counters are within the trusted boundary 205.

Finally, from a defensive standpoint the impact on memory forensics and similar techniques is likely going to be substantial. Understanding the finer details will become critical.

Conclusions

Intel SGX is an interesting technology for numerous parties both good and bad outside of DRM. However, it is also clear that there is enough small print that the implementations across all families of CPU will warrant investigation when they become generally available.

From a vulnerability/attack research perspective vulnerabilities in enclave protected code (including brokers) as well as CPU microcode will become incredibly valuable as will any attestation aspects. There will likely be renewed focus on understanding the exploitability of issues noted in Intel CPU Errata to potentially subvert or other influence control of an enclave.

From a defensive standpoint cloud and sensitive compartmentalised client side operations become feasible without reliance on the security of underlying hypervisors or the performance / usability trade-offs of homomorphic encryption.

Finally imagine a world where LSASS on Microsoft Windows runs in an SGX enclave so even certain attacks implemented by Mimikatz are no longer possible. 

Published date:  05 January 2015

Written by:  Ollie Whitehouse

Ollie Whitehouse

Ollie Whitehouse

Formally Group CTO for NCC Group 2017 - 2022