We blogged about the alpha launch of Shipped back in June, but today I want to talk about the service it relies on: Mantl.
Mantl's goal is to provide a fully functional, instrumented, and portable container based PaaS for your business at the push of a button. It's an opinionated set of services, configurations and tools that provide a standard interface for you to operate your container-based applications and infrastructure. Typically companies write a lot of unnecessarily repeated, similar-looking glue code to deploy, inspect and manage their services and the infrastructure they run on. This code is difficult to effectively version control and is seldom the same between teams. Mantl is an attempt to turn these glued together pieces into an easy to use complete solution for managing your services.
The user experience that addresses this glue is called "Shipped" and it provides an integrated Build, Deploy, and Run experience. The Build experience is integrated to the developers existing tools, frameworks, and development languages (so there are no changes to how you develop your application). Deploy has become very complex over the last several years as cloud services and providers require you to refactor or rewrite your application for their environment. We do not believe this is the right approach.
Mantl's design is to enable you to deploy your application across any internal, private, public, or hybrid combinations of environments and technologies. You get the benefit of not being locked in and the reduced complexity of having a consistent deployment process. Finally, to support any deployment, the Run experience is very key. The goal of the Run experience is to be able to support your application across deployment locations in a CI/CD & Application Intent Framework focused on business objectives and not technologies.
You may find Mantl useful if:
You can adopt as much or as little of it as you'd like, but before we get into those decisions, let's look at what it provides out of the box.
What's does it do?
Mantl is a collection of tools from a lot of great service providers, the big ones are below.
Provisioning
Terraform
Terraform, by HashiCorp, provides an abstraction layer that allows you to provision compute resources on a large number of providers. It bootstraps the raw infrastructure that will run Mesosand all the corresponding services on top of it.
Scheduling
Mesos
Mesos manages resources in your cluster and handles scheduling and isolating your tasks. It allows you to bin-pack your servers, efficiently running multiple tasks on each node. It's the key to making all of this work. Mesos relies on Zookeeper for high availability, so you'll also get that for free.
Kubernetes
Kubernetes solves similar problems as Mesos and support is planned in the near future.
Service Discovery
Consul
Consul, another Hashicorp product, handles service discovery. It makes it easy to coordinate cluster membership and also does some other handy things like arbitrary key value storage for configuration management, etc. A lot of the coordination in Mantl uses Consul to make the magic happen.
Logging & Search
Mantl uses the "ELK" stack to handle logging and indexing.
Elastic Search
This handles indexing logs and metrics from each host and each container. It's how you'll monitor the performance of your services.
Logstash
This little daemon runs on each node and basically pipes logs into elastic search for later consumption/analysis.
Kibana
Kibana is basically a visualization layer for elastic search. If you annotate your logs nicely, Kibana will make pretty graphs for you (yay!).
Monitoring
CollectD
This is another daemon that runs on each node and collects information so you can monitor your cluster in a standard way.
Container Management
Docker
Mesos has built-in support for Docker, a very popular portable container implementation. It's typical to build your application into a deployable docker container.
Marathon
Marathon manages long running tasks, like your applications. It's the typical deploy target for consumers of Mantl. If your servers or containers die, marathon takes care of rescheduling them so your services are always available.
Application Routing
HAProxy
Marathon allows you to scale the number of containers running your application, but once you've done so, you have a new problem: routing traffic between instances of your application. Mantl manages this for HTTP(S) traffic by keeping the HAProxy configuration up to date with Consul entries for each service running in Marathon.
Glue
Mesos-Consul / Marathon-Consul
These are the data-bridges that keep Consul up to date. Whenever Mesos or marathon runs a new task, these lets Consul know about it so the DNS entries are always current.
HAProxy-Consul
This is a Docker container purpose built as a load balancer for your application deployments. The proxy configuration is kept up to date by Consul-Template to load balance between containers running the same application. For now, this only works for stateless HTTP(s) services, but eventually it may support other protocols.
Security
Calico
This handles virtual network isolation, which is useful for multi-tenant environments or maintaining strict isolation for security and compliance reasons.
Vault
Most systems have secrets, Mantl provides Vault out of the box to set you up to handle your secrets and personally identifiable information in a safe way.
How does Shipped use Mantl?
Shipped is purpose built to make deploying docker applications super easy, so Mantl was built alongside it to make hosting these applications easy.
Shipped actually runs itself alongside other tenant applications on Mesos workers. Shipped allows developers to build their application into a docker image. Once you have a containerized app, you can deploy with the push of a button. Behind the scenes, this goes something like this:
Mantl Architecture
Shipped wraps all this up for you in a tidy management interface:
How do I get started?
Cisco has prepared pretty solid getting started docs in the Mantl repository. You can provision a cluster on OpenStack, Google Compute Engine, Amazon Web Services, DigitalOcean, VMware vSphere or SoftLayer without too much trouble. After that, getting to know the tools and how they all fit together is still left as a bit of an exercise to the user. I've done my best to link to the components above, and the docs on the repo are constantly improving. The repo has been starred 1500 times, so get involved! You can also try it out from Shipped
Let me know your thoughts and give it a try!
Looking for help managing your highly available micro-services?
Reach out to Must Win!
-
Mike Ihbe is a founding partner of The Must Win All Star Web & Mobile Consultancy.
Mike is an expert in a dizzying array of technologies and has loads of experience managing fast-moving dev teams, designing systems, and scaling large applications. When not on the job, he can be found cooking delicious meals on ski slopes in exotic locales.