Secure Calls

“Secure calls” are the current name for Cheriette inter-compartment calls. They are not in principle a new idea, every CHERI OS has a notion of a “compartment” and a CHERI capability-secured way of context switching between them. Cross-compartment calls are a CHERI-enforced RPC mechanism..

Cheriette borrows the caller-callee stack sharing from CompartOS & CHERIoT, and the general idea from CheriOS.

In Cheriette, secure calls are especially fast, because Cheriette uses a single global address space, and all process context is held in CHERI capabilities, explicitly none in the MMU. That means that switching between compartments using Secure Calls is as fast as setting up the call, saving registers to the stack and calling cinvoke, and zeroing the stack on return.

From an engineering perspective, secure calls in Cheriette have interfaces that mimick function calls. The fact that secure calls cross trust-domain boundaries means that the caller has to save and zero all non-argument registers before crossing the trust domain boundary and the callee has to zero out the stack it used before returning.

Instead of zeroing out the “shared” stack before returning to the caller, the handler stack could also

be allocated on-demand by an intermediary in-between the call. This design requires a trusted intermediary, and begs the question of where the intermediary gets its stack from. Depending on the design, the intermediary becomes a point of serialization for all secure calls in the system.
be pre-allocated and associated with the “channel” used for the secure call. This design requires no runtime allocation or sealing, but limits the number of concurrent calls for all tasks that share a channel to the number of pre-allocated stacks (In the current Cheriette, all tasks share the same “channel”, because a channel so far only had to be sealed pointers to the handler) .

What these alternatives have in common is that the handler stack is kept secret either by being sealed and “locked” to the channel or by being managed by an intermediary. The intermediary may have to zero out stacks for re-use on occasion, but that could happen asynchronously.

For now, Cheriette re-uses a restricted subset of the caller stack as the handler stack. This design is simplest, and requires no additional allocation decisions. Philosophically, this design decision follows the principle that the caller must provide the resources for the handler to run, not the other way around. In this way, the caller and handler merge into a single, continuous stream of execution that can be arbitrarly nested or even recursive.

Theory of Operation

High-level:

Task begins secure call
Create continuation cap that handler can return to
Create continuation data cap that handler passes back
Seal continuation cap
Seal complete stack cap
Store sealed complete stack cap in continuation data
Create restricted handler stack cap
Set stack pointer to restricted stack cap
cinvoke(sealed pc, sealed data)
Extract call arguments from register values
Call handler
Store return value in continuation data cap
Zero out restricted stack capability
Return through continuation cap
Recover complete stack from sealed cap
Return to task

Linking secure calls

Compartments are built independently of each other and get included into the OS as ELF files. The kernel loads the ELF binaries into memory and makes them runnable. During this process the kernel links all compartments together by resolving global symbol relocations. For this process to work, each compartment has to provide a set of symbols for each secure call it exposes:

scall_receive: A function that is the entry point when the compartment is entered through a secure call. This function is responsible for extracting the passed arguments, calling the correct handler and returning execution to the caller This only has to be defined once per compartment.
service_scall_handler_$SERVICE-NAME_$SECURE-CALL-NAME: A function that defines the handler for a defined secure call. This should be the “implementation” of the secure call.
service_scall_secure_package_argument_$SERVICE-NAME_$SECURE-CALL-NAME: An argument that is passed to a secure call handler. This argument is hidden from the caller. Not used at the moment.
service_scall_secure_package_$SERVICE-NAME_$SECURE-CALL-NAME: An object that contains a pointer to the handler and the secure argument.
scall_package_$SERIVCE-NAME_$SECURE-CALL-NAME: This object contains a pointer to scall_receive and contains the secure package. It contains everything that is needed for an outside compartment to call that specific secure call (scall_receive, scall_handler, scall_secure_argument). This is the object that other services will define as an external reference, which will then get linked by the kernel when booting. When performing a scall, the caller will provide a pointer to this package.

Secure calls across exception levels

Securing the stack

The stack capability is a capability like any other, and as such can be stored and loaded from memory at will, unless precautions are taken. This is an issue, because secure calls use a restricted subset of the caller stack as the callee stack.

An attacker could run two tasks in the caller: caller, and reader. caller stores its stack cap in a shared location where the reader task can accesss it. The caller then calls into the victim compartment. The victim callee stores secret data and caps on the stack provided by the caller. If the caller task running in the victim compartment is interrupted and the reader is scheduled, the reader can now read out private data of the victim compartment by accessing the callee stack.

┌──────┐         ┌──────┐                         ┌──────┐                ┌──────────────┐
│Caller│         │Reader│                         │Callee│                │Callee secrets│
└──┬───┘         └──┬───┘                         └──┬───┘                └──────┬───────┘
   │                │                                │                           │
   │Shares stack cap│                                │                           │
   │───────────────>│                                │                           │
   │                │                                │                           │
   │       Secure call with reduced stack cap        │                           │
   │────────────────────────────────────────────────>│                           │
   │                │                                │                           │
   │                │                                │Store private data on stack│
   │                │                                │──────────────────────────>│
   │                │                                │                           │
   │                │Read callee stack from stack cap│                           │
   │                │───────────────────────────────>│                           │
┌──┴───┐         ┌──┴───┐                         ┌──┴───┐                ┌──────┴───────┐
│Caller│         │Reader│                         │Callee│                │Callee secrets│
└──────┘         └──────┘                         └──────┘                └──────────────┘

Luckily, CHERI provides architectural means to work around this issue: Capabilities are either “colored” local or global. Global capabilities can be stored via any capabiliy that has the store permission, but local capabilities can only be stored by capabilities that have the explicit store local permission.

All stacks must now be made local capabilities. This ensures that the stack cannot be shared by storing the stack capability to memory.

How does the `caller` store and recover a `local` stack? CHERIoT: Register file has special cap with store local permission. May require trusted intermediary.

What about sharing through registers? One task may never “fork” and create two tasks with the same stack, or pass its stack cap as a cap argument to a new task to be created. A task may pass its stack along to another compartment, but then that task is locked into executing in the other compartment.

Clearing the Stack upon Return

Zeroing the stack before returning is a major overhead. CHERIoT adds changes to the hardware to track the stack high water mark and then only zeroes the stack that the callee actually used.

Absent hardware changes, it may be possible to have stacks designated as special areas for memory management, reserve physical memory space for stack usage, and remap the physical memory to the stack space.

Then, upon return, the callee stack could be switched out for zero-stack pages, and the dirty stack memory cleared asynchronously.

CheriOS

CheriOS keeps a dedicated stack per compartment that a thread could enter and switches to that stack when the thread enters the compartment. This requires creating and maintaining stacks per thread.

See: cherios/system/dylink/src/platform/riscv/stubs.S

CompartOS

CompartOS uses a per-thread stack, which is donated to the called compartment and reset on return.

See: https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-976.pdf

CherIoT

Uses restricted caller stack in callee.

See: cheriot-rtos/core/switcher/tstack.h