Inclavare, pronounced as [ˈinklɑveə], is the Latin etymology of the word enclave, which means to isolate the user's sensitive workload from the untrusted and uncontrollable infrastructure in order to meet the protection requirement for the data in use.

Inclavare Containers is an innovation of container runtime with the novel approach for launching protected containers in hardware-assisted Trusted Execution Environment (TEE) technology, aka Enclave, which can prevent the untrusted entity, such as Cloud Service Provider (CSP), from accessing the sensitive and confidential assets in use.

Inclavare Containers has the following salient features:

• Confidential computing oriented. Inclavare Containers provides a general design for the protection of tenant’s workload.
  • Create the hardware-enforced isolation between tenant’s workload and privileged software controlled by CSP.
  • Remove CSP from the Trusted Computing Base (TCB) of tenant in untrusted cloud.
  • Construct the general attestation infrastructure to convince users to trust the workloads running inside TEE based on hardware assisted enclave technology.
• OCI-compliant. The component rune is fully compliant with OCI Runtime specification.
• Cloud platform agnostic. It can be deployed in any public cloud Kubernetes platform.

Please refer to Terminology for more technical expressions used in Inclavare Containers.

Inclavare Containers is a sandbox project of the Cloud Native Computing Foundation (CNCF). If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF.

Inclavare Containers is helping to keep tenants' confidential data secure so they feel confident that their data is not being exposed to CSP or their own insiders, and they can easily move their trusted applications to the cloud.

Inclavare Containers follows the classic container runtime design. It takes the adaption to containerd as first class, and uses dedicated shim-rune to interface with OCI Runtime rune. In the downstrem, init-runelet employs a novel approach of launching enclave runtime and trusted application in hardware-enforced enclave.

The major components of Inclavare Containers are:

• rune
  rune is a CLI tool for spawning and running enclaves in containers according to the OCI specification. rune is already written into OCI Runtime implementation list.

• shim-rune
  shim-rune resides in between containerd and rune, conducting enclave signing and management beyond the normal shim basis. In particular shim-rune and rune can compose a basic enclave containerization stack for confidential computing, providing low barrier to the use of confidential computing and the same experience as ordinary container. Please refer to this doc for the details.

• enclave runtime
  The backend of rune is a component called enclave runtime, which is responsible for loading and running trusted and protected applications inside enclaves. The interface between rune and enclave runtime is Enclave Runtime PAL API, which allows invoking enclave runtime through well-defined functions. The softwares for confidential computing may benefit from this interface to interact with cloud-native ecosystem.

  One typical class of enclave runtime implementations is based on Library OSes. Currently, the recommended enclave runtime interacting with rune is Occlum, a memory-safe, multi-process Library OS for Intel SGX. And another typical class of enclave runtime is WebAssembly Micro Runtime (WAMR) with Intel SGX, a standalone WebAssembly (WASM) runtime with a small footprint, including a VM core, an application framework and a dynamic management for WASM applications.

  In addition, you can write your own enclave runtime with any programming language and SDK (e.g, Intel SGX SDK) you prefer as long as it implements Enclave Runtime PAL API.

Inclavare Containers implements Enclave Attestation Architecture (EAA), a universal and cross-platform remote attestation infrastructure. EAA can prove that sensitive workloads are running on a genuine and trusted hardware TEE based on confidential computing technology. The formal design of EAA will be published for RFC.

The major components of EAA are:

• Rats-TLS Rats-TLS enhances the standard TLS to support the trusted communications between heterogeneous hardware TEEs based on confidential computing technology, which is evolved from the ra-tls (deprecated). Even a non-hardware TEE platforms using Rats-TLS can communicate with a hardware TEE, e.g, SGX Enclave, through the attested and secured channel to transmit the sensitive information. In other words, the boundary of TCB is extended from execution environment to network transmission with Rats-TLS. In addition, Rats-TLS has an extensible model to support various hardware TEE. Refer to this design doc for more details.

• Confidential Container
  Confidential container in the form of the enclave runtime Occlum responds to the request from Inclavared, and then sends back the attestation evidence of confidential container to Inclavared. Confidential container plays the role of the attester.

• Inclavared
  Inclavared is responsible for forwarding the traffic between the confidential container and Shelter. The communication process is protected by the attested Enclave-TLS channel.

• Shelter
  Shelter, as the role of the verifier deployed in the off-cloud, records the launch measurements of enclave runtime, and afterward establishes the attested Enclave-TLS channel to communicate with Inclavared. Eventually, it retrieves the evidence about enclave runtimes for verification.

• sgx-tools
  sgx-tools is a CLI tool, used to interact Intel SGX AESM service to retrieve various materials such as launch token, quoting enclave's target information, enclave quote and remote attestation report from IAS. Refer to this tutorial for the details about its usage.

• epm
  epm is a service that is used to manage the cache pools to optimize the startup time of enclave. Refer to this tutorial for the details about its usage.

Please refer to Inclavare Containers Roadmap for the details. This document outlines the development roadmap for the Inclavare Containers project.

It's recommended to use Inclavare Containers development docker image to build Inclavare Containers from scratch.

Note that the environment of launching Inclavare Containers development docker image must be capable of hardware TEE and install the corresponding software stack, e.g, Intel SGX and Intel SGX SDK & PSW for Linux.

The exact command to run the docker image requires to be specified explicitly according to the type of SGX device driver.

• For legacy out-of-tree driver:

docker run -it -v /var/run/aesmd:/var/run/aesmd \
  -device /dev/isgx \
  inclavarecontainers/dev:$version-$os

• For DCAP and in-tree driver:

docker run -it -v /var/run/aesmd:/var/run/aesmd \
  -v /dev/sgx_enclave:/dev/sgx/enclave -v /dev/sgx_provision:/dev/sgx/provision \
  inclavarecontainers/dev:$version-$os

where:

• $version denotes the version of Inclavare Containers in use.
• $os denotes the OS type of development docker image, which may be ubuntu18.04 or alinux2.

Please be aware of running the commands listed below in the development container launched by Inclavare Containers development docker image.

1. Download the latest source code of Inclavare Containers

mkdir -p "$WORKSPACE"
cd "$WORKSPACE"
git clone https://github.com/alibaba/inclavare-containers

2. Build Inclavare Containers

cd inclavare-containers
# build rune, shim-rune, epm, sgx-tools, enclave-tls, shelter and inclavared
make

After build Inclavare Containers on your system, you can use the following command to install Inclavare Containers on your system.

sudo make install

{rune,shim-rune,epm,sgx-tools,shelter,inclavared} will be installed to /usr/local/bin/{rune,containerd-shim-rune-v2,epm,sgx-tools,shelter,inclavared} on your system. Enclave-TLS SDK will be installed to /opt/enclave-tls. {enclave-tls-server,enclave-tls-client} will be installed to /usr/share/enclave-tls/samples.

If you don't want to build and install Inclavare Containers from latest source code. We also provide RPM/DEB repository to help you install Inclavare Containers quickly. Please see the steps about how to configure repository firstly. Then you can run the following command to install Inclavare Containers on your system.

• On Ubuntu 18.04 server

sudo apt-get install rune shim-rune epm sgx-tools enclave-tls shelter inclavared

Inclavare Containers can be integrated with dockerd, containerd, and pouchd.

The former targets using docker to deploy Inclavare Containers. Specifically, you need to install the preferred enclave runtime when building container images, and then launch the enclave runtime through rune and enclave runtime specific PAL.

The latter targets using K8s to deploy Inclavare Containers. In this scenario, shim-rune and rune can compose an enclave containerization stack, so enclave runtime is not required and installed when building container images, providing with the same experience as ordinary containers.

Add the assocated configurations for rune in dockerd config file, e.g, /etc/docker/daemon.json, on your system.

{
        "runtimes": {
                "rune": {
                        "path": "/usr/local/bin/rune",
                        "runtimeArgs": []
                }
        }
}

then restart dockerd on your system.

You can check whether rune is correctly enabled or not with:

docker info | grep rune

Note that the systemd is not installed by default, so please manually start up dockerd:

dockerd -b docker0 --storage-driver=vfs &

Inclavare Containers is added to the adopters list of containerd. Besides, shim-rune supports containerd shim v2 API. So you can add the assocated configurations for shim-rune in the containerd config file, e.g, /etc/containerd/config.toml, on your system.

        [plugins.cri.containerd]
          ...
          [plugins.cri.containerd.runtimes.rune]
            runtime_type = "io.containerd.rune.v2"

then restart containerd on your system.

Add the assocated configurations in pouchd config file, e.g, /etc/pouch/config.json, on your system.

	"add-runtime": {
		"rune": {
            		"path": "/usr/local/bin/rune",
            		"runtimeArgs": null,
            		"type": "io.containerd.rune.v2"
        	},
		...
	}	

where:

• @path: specify the path of OCI Runtime, such as the pach of rune.
• @runtimeArgs: specify the arguments of the pouchd runtime, such as --platform, --network.
• @type: specify the shim template from the following candidates:
  • io.containerd.rune.v2: correspond to shim-rune
  • io.containerd.runtime.v1.linux: correspond to containerd-shim
  • io.containerd.runc.v1: correspond to containerd-shim-runc-v1

then restart pouchd on your system.

You can check whether rune is correctly enabled or not with:

pouch info | grep rune

Inclavare Containers can be deployed with Occlum LibOS and WebAssembly Micro Runtime (WAMR).

Please refer to this guide to run Occlum with rune and docker.

Please refer to this guide to deploy an enclave container in a Kubernetes cluster. Currently, Hello-world application image and web application images based on OpenJDK 11, Dragonwell, and Golang are provided. These images don't contain enclave runtime. They are only used for the deployment with containerd.

Please refer to this guide to run inclavare-containers with pouchd.

Please refer to this guide to run WAMR with rune.

WebAssembly Micro Runtime (WAMR) application image is provided. WAMR image contains enclave runtime, because it doesn't adapt to shim-rune and use off-cloud signing. It is only used for the deployment with dockerd.