Kloudless Blog

Kloudless Unified APIs enable you to code once and integrate many

Background

Here at Kloudless, we provide a Docker Container for Kloudless Enterprise that makes it easy to manage a Kloudless Enterprise cluster using industry standard tools like Docker Swarm or Kubernetes.

For some background, Kloudless provides a unified API that enables engineering teams to quickly integrate several software apps with a single implementation. This means our primary users are other developers and engineering teams.

However, downloading the container from our Kloudless Enterprise web portal is inconvenient. Users previously had to download the archived image and manually load it into their Docker daemon to use it. There also wasn’t a way to check which images were available without visiting the portal through a browser. To improve the experience, we decided to provide a private Docker Registry that would allow our users to not only pull images, but also query tags and take advantage of other useful features that the Docker Registry provides.

Private Docker Registry Architecture

To reduce our operational load, we use the Elastic Container Registry (ECR) that AWS provides as a managed Docker Registry. This allows us to work with Docker images without having to worry about maintaining the registry service or the underlying storage.

The primary concern is authenticating end-user access to this registry. ECR relies on short-lived auth tokens that are valid for 12 hours. This is problematic since we either have to provision per-user IAM accounts for each user accessing our registry, or repeatedly provide an auth token our app generates from our IAM credentials. Neither of those cases are very desirable, so we thought of an alternative.

Fortunately, our own application’s API provides tokens that authorize our users to access and manage different aspects of our platform. One way to leverage this is to have nginx accept API requests to our Docker Registry from clients that authenticate using our API’s tokens instead, and then replace the Kloudless tokens with the Docker ECR auth token. The infrastructure is roughly as shown below:

, Custom Authentication to a private Docker Registry with Nginx, Lua, and AWS ECR

ECR Authentication

It is straightforward to manage the proxy’s access to ECR. Since we are running the server in EC2, we can create an IAM role to read the relevant repository, describe repositories, and provision authorization tokens for ECR:

The server needs to store and refresh ECR authorization tokens to allow nginx to perform requests to ECR. A simple cron job that executes every 8 hours handles this process:

Proxying the requests

Since our customers only require read access, we can directly proxy the Docker Registry API requests and replace the authentication—after validating the token of course. We take advantage of the ngx-lua module to handle this within nginx itself. The OpenResty framework includes this library by default, but it is possible to install it separately as well. The following configuration snippet demonstrates how to safely proxy the Docker Registry API requests:

nginx doesn’t read from the file containing the ECR token on each request to ensure the requests are handled efficiently. Instead, nginx loads the tokens when the file changes. The following lua file referenced above by require("aws") handles this:

Custom Kloudless Authentication

The nginx configuration displayed earlier uses HTTP Basic Authentication to ensure compatibility with Docker command line tools. The developer’s email is the username, while their account’s API token is the password. In the access_by_lua block, nginx decodes the Basic Auth header, reads the Kloudless token, and uses that to perform a request to the Kloudless Meta API to list Licenses. This validates the token and also provides information on which Docker releases the developer has access to. The TODO comment in the earlier nginx config contains the following snippet to authorize the requests using nginx sub-requests:

Load-balancing

Kloudless’ setup uses an ELB in front of the proxy server for high availability and easier SSL/TLS termination. This allows Kloudless to easily scale up the proxies using auto-scaling groups and handle any individual instance’s failure. It also allows Kloudless to provide a much more user-friendly host name over HTTPS such as docker.kloudless.com rather than the long ECR domain name.

Conclusion

That’s it! The configuration above is modular enough that you can substitute any authentication or authorization method you want, including a different web service or a generic database. This private, read-only registry greatly simplifies how our customers get started with Kloudless Docker containers and allows them to better utilize industry standard tools with our appliance.

Categories: