• A gRPC interface to the key-value store using the ‑‑grpc option
• Prometheus-compatible metrics collection endpoint using the ‑‑prometheus option
• CLI adjustements
• Performances improvements

What's next?

As you can probably guess from our change log, we're actively working on the key-value layer of our storage platform right now.

We're especially focused on improving and stressing our in-house key-value store. At the same time we're cleaning a lot of code so it can be open sourced in the following weeks.

You may wonder: why another key-value store? What will make it different from the etcd, Consul, ZooKeeper and other key-value stores is the use of a decentralized model (i.e peer-to-peer) where every node is equipotent, making it easily scalable, built with resilience and security in mind.

If you want to know more about it, have a look at the video or the slides from our DockerCon presentation a few weeks ago.

As always, don't hesitate to get in touch with us if you have any feedback or would like to chat about our releases. We're on reddit, Twitter, Slack or IRC.

Presentation

First and foremost, we wanted to share the slides and video of the presentation for all the ones who couldn't attend to enjoy (hopefully). Through this talk, I explain why we've designed Infinit's clustering technology this way, how it differentiate from other ones and what benefits it brings to the table:

Slides: https://www.slideshare.net/Docker/infinits-next-generation-keyvalue-store-julien-quintard-and-quentin-hocquet-docker

Use Cases

During the four or so days at DockerCon, I've met with numerous people, from partners, customers and users. Such exchanges helped me get feedback and better understand everyone's expectations when it comes to storage. What came out of those discussions is that most people want storage to be taken care of, being the number one challenge to the containerization of their applications.

However, it also appeared that most people don't really know what type of data they intend to store, hence what they should use. This is especially true for enterprises that have thousands of legacy applications.

When it comes to cloud native developers, the number one application in need of a resilient and scalable storage backend seems to clearly be the database. Over the years at Infinit, we've always taken extra care explaining the specifics of a database when it comes to storage and as such what should be used.

What makes a database specific compared to other distributed applications is that the data stored is only accessed by a single node: a database frontend listens on a port, handles incoming requests and synchronizes its in-memory state on disk. Should the database be scaled out by spawning another frontend on another node, this new node would also synchronize its in-memory state on disk, but that disk would be completely isolated from the first node's.

Many other distributed applications need the state to be shared between multiple nodes. This is not the case for a database for which state is node specific.

Database-specific redudancy

This brings me to how such a state should be managed. In theory, one should always rely on the database's software to scale and handle redundancy. As such, if your database, such as MongoDB, provides a replication feature, we advise you to rely on it because the database application has more context to replicate data and optimize placement, load balancing etc. rather than rely on the storage backend.

We however understand that configuring and tuning a database is not everyone's hobby. Assuming throughput is not a problem and that all you want is for your database's state to be persisted, then you could use a resilient storage backend. Now the type of storage you decide to use will directly impact your database's performance.

Persistent storage

Let us now come back to my previous and bold statement "it appears that most people don't really know what type of data they intend to store". The Infinit storage platform has been designed to provide multiple policy-customizable storage logics on top of a scalable and resilient storage clustering mechanism. By using the Infinit storage platform, you can decide to use one or more storage logic, most likely between an object storage, a block device or a (distributed) file system.

In the context of a database, most people I've talked to were drawn to the idea of deploying a distributed file system to back their database, because of the compliance to the POSIX standards. A distributed file system is able to support concurrent accesses from multiple nodes and handle conflicts. As such, since such multi-node concurrent accesses are not possible when it comes to a database (whose state is specific to a single node), using a distributed file system would be too expensive and you would end up paying the price for high-end functionalities that are unnecessary for a database.

A block device on the other end is, in theory (depending on the file system it is formatted in), only accessed from a single node. It therefore perfectly matches the requirements of a database application. In addition and because of this limitation, a block device will be more efficient that a distributed file system, doing away with locking, conflicts, cache invalidations etc.

As a summary, databases are likely to be the first and most deployed application requiring persistent storage. But because people have different needs, I believe that operators will want to have fine-grained control over their deployments and should as such achieve persistent state through the database's redundancy capability itself. As for developers, they don't want to waste time configuring and tuning a database to replicate itself. Those users could in this case rely on a block device storage logic, accessible through an interface such as NBD, iSCSI or else, so as to achieve persistent storage without much effort.

We are it has created a wave of interest and led to a few suggestions and merge requests. We are especially proud that Elle was featured on the 2nd of March, 2017 cppcast (podcasts I highly recommend if you are interested in C++).

However, because of the history behind Elle, a 7-year-old library, which used to be closed-source, designed and used by a small team, Elle was more an holdall for our core libraries. The source code was barely documented and 4 namespaces were cohabiting.

We chose to revamp Elle, by cleaning everything, unifying coding style, documenting, writing examples, and more, to give Elle a new face. And it's live!

A quick reminder of what Elle is

Elle is a “framework“, written in C++14, that contains:

• reactor: Asynchronous framework based on coroutines
• das: Compile-time introspection via symbol-based metaprogramming
• cryptography: Object-oriented API around OpenSSL
• protocol: Versioned and multiplexed communication protocol for remote procedure calls (RPCs)
• And more...

In addition, Elle is cross-platform. It works on common environments and architectures, including ARM (Elle has been tested on Android, iPhone and Raspberry Pi). To date, Elle was deployed on close to a million devices, thanks to Infinit's previous product infinit.io.

The new face of Elle

As an open library, Elle now follows the same structure Boost uses:

• A unique namespace elle
• Most classes are now documented (abstract, example and public methods documentation)
• Libraries names are now prefixed by elle_, e.g. libelle_<module>.so

Therefore, if you already use Elle, you will need to adjust your code because:

• reactor:: became elle::reactor::, infinit::cryptography:: became elle::cryptography::, infinit::protocol:: became elle::protocol::, etc.
• libreactor.so became libelle_reactor.so, libelle.so became libelle_core.so, etc.
• etc.

Now that things are getting serious, Elle even has its own logo. Read more...

Interested in trying Elle?

As mentioned earlier, Elle is already accessible on GitHub (here) and includes a few examples to demonstrate what features it provides.

Elle doesn't have, strictly speaking, a standard release life cycle. For the sake of simplicity, we will weekly upload the newest 64-bits version for the LTS Ubuntu here along with a Docker image (infinitd/elle) containing Elle, built and installed, examples and a development environment (a compiler, buildsystems, etc.) so you can give Elle a quick try.

For more details about building or testing Elle, you can consult Elle's wiki on GitHub.

With all this change, we decided we needed to refresh our image a bit to better match our new direction towards storage.

Logotype

People usually love our logo; it's super recognizable and has been the pillar of our image culture for years so the Infinit symbol isn't going anywhere.

Side note: our Infinit symbol and name actually come from the Tibetan endless knot.

Our handwritten logotype also conveyed a nice playful character which was well adapted for our file transfer applications. With our shift to the storage world, we wanted a more serious connotation to better fit this new market and our enterprise focus.

However, it had to keep some of the cheerfulness of our previous logo since we still want to make creating storage infrastructure easy for any developer.

We went with the Linotte font, designed by Joël Carrouché which perfectly met all of our requirements. It's more readable at small size, has a solid geometric construction while maintaining a few imperfections conveying exactly the feeling we had in mind.

Projects

Over the last few months we've been working hard to open source our technology, component by component (you can follow our progress on our projects page).

We started with our build system written in Python 3, Drake, and we recently added our coroutine-based asynchronous C++ development framework, Elle.

With our infinite symbol as the model, we reused the same mechanics playing with the loop widths to get these two logos.

The Drake logo represents the mystical drake animal as you probably guessed.

The Elle logo... sit tight. In French, Elle is pronounced like the letter "L", and is our feminine gender third-person singular subject pronoun. Well, it means "she". We had a look at what the L letter represented and found that the symbol for the L hieroglyph is a lion, which is also an astrological sign represented by this symbol. Playing with the curve widths, we quickly agreed on this logo which can either look like the astrological sign, or a woman's hair — because it means she, get it? :)

Our key-value store will follow next, but we're still in some intense epic discussions about what the name will be.

Even though the C++ standard has been moving pretty fast over the last few years, with a new version every 3 years, the C++ ecosystem lacks some important features. Boost C++ libraries has been providing reference implementations for these missing features, helping programmers get a few steps ahead.

In 2009, Infinit's founder Julien mycure Quintard started developing Elle as the core library for his PhD project: the prototype of the Infinit storage platform. After years and years of development, Elle has grown and is now a meta-library composed of a set of sub-libraries that we are proud to finally share with the community.

Elle is currently (mid February 2017) at stage 1 (access); you can clone the source code, read it, build it and tweak it. However, until it reachs stage 2 (develop), the documentation and hierarchy might seem confusing. Hopefully, within two months, Elle will reach stage 2 and be well documented.

Yet another library?

Elle is a software development framework, in the sense that it changes the way we approach asynchronism (spoiler: it uses coroutines).

Some frameworks like Qt with its signals/slots, rely on event-driven programming, which tends to produce "spaghetti code": it's hard to follow the execution without knowledge of a wide range of classes. Like every other paradigm, event-driven programming and coroutines have their pros and cons. However, as you can see, a few projects have emerged to introduce support for coroutines in Qt (qt-couroutine or QFiber). I'm not blaming Qt people. Qt is an extremely mature, well designed framework and its popularity speaks for itself. Event-driven programming fits their needs perfectly.

Elle is much more than just an asynchronous framework; it comes with tons of built-in features. Being the core library of the Infinit storage platform, Elle provides all the tools you need to create projects which require cryptography, networking and/or consensus. With Elle, one could very easily develop, say, an HTTP server in C++ capable of handling tens of thousands of clients without breaking a sweat.

Welcome to Elle

Elle is based on Boost, OpenSSL, zlib, curl, and a few other libraries which forms a unified C++ environment, including:

• A complete asynchronous framework based on coroutines in C++
• Compile-time introspection via symbol-based metaprogramming
• Object-oriented cryptography
• Versioned and multiplexed communication protocol for remote procedure calls (RPCs)
• And more...

And, good news everyone, it's now open-source and available on our GitHub account.

Reactor: an asynchronous framework for C++

For those who are not familiar with coroutines, natively supported by dozens of languages (including Go, Haskell, Python, D, etc.), they allow for non-preemptive multitasking (unlike threads) offering concurrency with a reduced need for locking and fit extremely well in I/O bound programming. They also allow for an illusion of sequential code execution in an asynchronous environment.

Basically, a coroutine can choose to yield, in which case its context (state of the program) is saved and the routine goes to sleep. Then, another coroutine is woken, has its context restored, runs until it yields or completes and so on until all the coroutines have completed their tasks. Unlike thread programming, the current coroutine cannot be preempted.

Coroutines are one of the current hot topics for C++, with a proposal for a potential addition to the standard. Boost.Coroutine already provides "templates for generalized subroutines". Elle's reactor sub-library, in addition to providing an elegant and portable implementation of coroutine, is the complete asynchronous framework mentioned earlier, that includes networking sockets (TCP, UDP, µTP), support for HTTP(s), finite state machines (FSM), synchronization mechanisms, etc.

Here is an example of a coroutine in charge of echoing back a message. As you will see, the code is written as a natural sequence of instructions to execute, same for exception handling: no hooks, no promises and no callbacks.

void  
echo_server()  
{
  // ...
  // For each connection, store the coroutine it was assigned to.
  client_threads.emplace_back(
    std::make_shared<reactor::Thread>(
      sched, "serve",
      // Echo a message back to it sender.
      [socket]
      {
        // Until the connection is closed.
        try
        {
          while (true)
          {
            // Yield until a the message is read until '\n'.
            auto line = socket->read_until("\n");
            // Yield until the message is written.
            socket->write(line);
          }
        }
        catch (reactor::network::ConnectionClosed const&)
        {}
      }));
}

It is as simple to perform HTTP requests.

// Parse a stream, expecting JSON.
auto process_json = [] (elle::IOStream const& body)  
                    {
                      // Do something clever.
                    };
// Create the HTTP request.
auto r = reactor::http::Request{  
  "http://maps.googleapis.com/maps/api/geocode/json",
  reactor::http::method::GET,
  "application/json"};
// Set the query.
r.query_string({{"address", "white house"},  
                {"sensor", "false"}});
// Finalize the operation.
r.finalize();  
if (r.status() == reactor::http::StatusCode::OK)  
  // Parse the output as a stream.
  process_json(r);
// ...

Das: compile-time introspection

Even if C++ offers powerful metaprogramming capabilities, Das uses a symbol-based paradigm to push it further. Named symbols provide powerful compile-time introspection, allowing for more flexibility and decision making. It also provides convenience features like:

• Printing and serialization

Writing printers and basic serialization implementation is a time consuming and redundant task. Das provides a non-intrusive default serialization implementation and printer for C++ objects.

• Named arguments

With minimal intrusion, Das offers the possibility to have named arguments in C++ for clearer and less error prone function calls.

• CLI

Often, command line tools have a 1-1 mapping to the underlying functions actually invoked and force manual type-conversion from the CLI argument to the type of the actual function argument. Named symbols, by design, perfectly solve the problem, offering an almost zero cost CLI binding.

Here is an example demonstrating how, in a non-intrusive way, you can provide serialization and printing for a random object.

// The symbols used for introspection.
DAS_SYMBOL(artist);  
DAS_SYMBOL(title);

struct Record  
{
  std::string artist;
  std::string title;
};
// The Model, which declares the symbols associated to a class or struct.
using Model = das::Model<Record,  
                         elle::meta::List<Symbol_title,
                                          Symbol_artist>>;
// Set Model as the default model for Record.
DAS_MODEL_DEFAULT(Record, Model);  
// Make all class::Model printable.
using das::operator <<;

// You know have access Das' features:
// - Printing:
// > Record(artist = "Darude", title = "Sandstorm")
std::cout << Record{"Darude", "Sandstorm"} << std::endl;  
// - Serialization
// > {"artist": "Darude", "title": "Sandstorm"}
elle::serialization::json::SerializerOut serializer(std::cout, false);  
das::serialize(Record{"Darude", "Sandstorm"}, serializer);  

Cryptography: object-oriented wrapper for OpenSSL

Cryptography is a tough and complex area but thanks to OpenSSL, we don't have to re-implement cryptographic algorithms. Elle's cryptography sub-library is a convenience wrapper over OpenSSL, offering modern C++ interfaces, object-oriented and stream-oriented APIs, automatic resource management, transparent module initialization and deinitialization, etc.: The power of OpenSSL with the elegance of modern C++.

#include <cryptography/rsa/KeyPair.hh>
#include <cryptography/SecretKey.hh>

// This is a bad implementation of cryptography::envelope.
// It stands here as an example.
std::tuple<elle::Buffer, elle::Buffer>  
hybrid_encrypt(cryptography::rsa::PublicKey const& peer_key)  
{
  // The payload you want to send.
  auto data = elle::Buffer{"super important data to encrypt"};
  // Generate a random password of 1024 characters.
  auto symmetric_key = cryptography::SecretKey(
    cryptography::generate<elle::Buffer>(1024));
  // Encipher the payload with the secret key.
  auto encrypted_payload = symmetric_key.encipher(
    data,
    cryptography::Cipher::blowfish, // default: aes256
    cryptography::Mode::cfb, // default: cbc
    cryptography::OneWay::sha512 // default: sha256
  );
  // Encrypt the key itself.
  auto encrypted_key = peer_key.encrypt(symmetric_key);

  // Return both encrypted payload and key, so the peer can decrypt the key
  // using his private key and use that key to decipher the original payload.
  return {encrypted_payload, encrypted_key};
}

Protocol: unified communication channels

Because the Infinit storage platform relies on sending data between nodes, setting up a sustainable channel of communication is crucial. Unification and versioning of the communication protocol itself are mandatory for robustness. Elle's protocol sub-library is designed to transparently take care of those two requirements on top of every stream.

It also allows for multiplexing communications channels in a single stream in order to support our remote procedure call (RPC) system.

And more...

Elle contains additions to the standard library, such as printers for standard types, new algorithms, but also classes and functions related to:

• Logging (hierarchical logging system)
• Serialization (JSON and binary)
• Distributed consensus (Paxos)
• Exception handling (nesting, serializable exceptions, etc.)
• Lazy evaluation wrappers, generators, etc.
• UUID
• And many more...

Usage, support and contribution

Elle is officially maintained by Infinit (which is now part of Docker). Elle will evolve according to our own needs, therefore, APIs may change over time. Also, binary compatibility won't be preserved between versions for now.

You can legitimately ask why you should use Elle then. Elle is a library which provides, in our opinion, great tools to write beautiful, easy to maintain and efficient code in C++. At present, we can't ensure backward compatibility, mostly because it requires a lot of effort, but open-sourcing it will push us towards this. Also, its stability and efficiency has been proven by the two projects Infinit has developed, the Infinit file transfer applications (deployed on close to a million devices, both desktop and mobile) and our current project, the Infinit storage platform.

If you have any questions or issues, we are available:

• by email: open+elle@infinit.sh
• on Slack: channel #elle
• on GitHub: https://github.com/infinit/elle

If you want to contribute, you are more than welcome. However, until the project reaches its final stage, contributions may take longer than expected to be integrated.

Hyper-scale

Hyper-scale is the ability of an architecture to scale appropriately as increased demand is added to the system. This typically involves the ability to seamlessly provision and add compute, memory, networking, and storage resources to a given node or set of nodes that make up a larger computing environment — from Wikipedia.

In short: every component can scale separately.

This hyper-scale model has been popularized by the giants Google, Facebook, Amazon and their large distributed sites. But a new model is emerging these last years, first with the virtual desktop infrastructure (VDI) and now with the attractiveness of containers.

Hyper-convergence

Hyper-convergence is a type of infrastructure system with a software-centric architecture that tightly integrates compute, storage, networking and virtualization resources and other technologies from scratch — from TechTarget.

In short: everything scales together. Let's visualize it with a schema.

As you can imagine, these two models come from different use cases and there is not one solution prevailing over the other. This is how Software Defined Storage (SDS) solutions like Infinit come into play.

Infinit default hyper-scalable architecture

By default, Infinit allows you to build your own custom storage infrastructure without taking consideration of your application, basically providing you the storage part on the hyper-scalable architecture schema above.

One big advantage of this system is that if you want to add and remove storage nodes on the fly without interfering with your application, you can. Because of its resilience and decentralized architecture, Infinit is able to take care of rebalancing data between all the nodes, or to heal itself if one node were to crash.

This architecture is perfect for operators wanting complete isolation and control over their storage component, independently of their computing resources.

Building hyper-convergent architectures with Infinit

But it's easy to imagine different use cases where developers want an easy way of scaling storage alongside their applications.

Dealing with two different clusters — one for storage, one for applications — is particularly cumbersome for developers that don't want to tweak every detail of their architecture.

That's why hyper-convergence architecture are interesting for them, making storage scale alongside their application cluster. In the context of containers, such an application cluster would be managed by an orchestrator, such as Docker Swarm, Kubernetes or Mesos.

Well, Infinit also provides a simple way to add hyper-converged storage to your Docker Swarm, meaning that as a compute node is added to a Swarm, Infinit aggregates its storage capacity to the virtual pool.

This global persistent storage can then be used by any container running on the Swarm to provide a place to store common files or share state. As nodes join or leave the Swarm, Infinit automatically scales the storage and rebalances or replicates blocks to ensure high availability and redundancy of the data stored.

Infinit provides a simple way to add hyper-converged storage to your Docker Swarm.

We made a prototype during a Docker hackathon in 2016, have a look!

Choosing what's best for you

There are no right answers as to which architecture is better.

If you're a developer looking for a simple solution that lets you focus on your code rather than tweaking your storage layer, go for an hyper-convergent architecture. If you're an operator, deploying applications in production, in need of an elastic and resilient storage layer, go for an hyper-scalable one.

But choosing a software defined storage product like Infinit will allow you to adapt to every situation. Whether you need an empowered hyper-converged Docker Swarm with persistent storage or an independent hyper-scalable storage infrastructure for your applications, Infinit won't lock you up to an unique solution.

Akim started working with us on our storage platform a couple of months ago. He has now joined Docker to officially take part in our 7-people team!

Let's introduce Akim with a small biography.

PhD years

Everything started when Akim was a PhD student.

He was sitting next to the printer, where he could see all the different ways that people wasted paper: single-sided printing, large fonts for plain text, print PostScript source instead of rendered PostScript, etc.

So he decided to help out by writing scripts named after the printers. These scripts made it easy to print double-sided, to print several "virtual minipages" on single physical pages, etc.

At some point, he also used a small C program to convert plain text files into decent PostScript: a2ps.

Over the years he added several features to a2ps which eventually integrated all the features from the original shell-scripts, including pretty-printing for more programming languages, reordering pages to print duplex booklets, and many, many other features. This new a2ps became very successful (eventually it was dubbed GNU a2ps), so he started to get reports about portability issues.

That’s how he ended up using Autoconf, and then Automake. To provide extensible pretty-printing features, he implemented a description language using Bison and Flex. And he used regular expressions heavily. Eventually, he contributed a lot to Bison, Autoconf, Automake and many other open source projects. He even became the official maintainer of Bison and Autoconf for a while.

Teaching at EPITA

Right after his PhD defense, he was hired at EPITA, a French private university for computer science, by Thierry Géraud, founder of LRDE, EPITA's R&D lab. He spent almost eighteen years there, teaching courses such as Formal Languages, Compiler Construction, C++, Formal Logic, Object-Oriented Design, etc.

The legend has it that he was the teacher of Julien & Quentin (former CEO & CTO of Infinit respectively) for a few years.

He was also involved in several projects:

• Vaucanson, a C++ platform dedicated to automata and rational expressions;
• Transformers, a source-to-source workbench for C++ programs;
• Tiger, a compiler written in C++ based on Andrew Appel's brilliant "Modern Compiler Implementation" books which is used at EPITA to teach long-term software development.

Robotics at Gostai

For six years, starting in 2006, he was a part-time employee of Gostai, a French start-up working on robotics. With Quentin Hocquet, Matthieu Nottale, and Antony Méchin (who was still an intern back then). They worked on the Urbi platform, in particular the urbiscript programming language.

Then, when he went back full time to EPITA, he spent most of his research time on the Vcsn project (GitLab link), a reboot of the Vaucanson project. Its core is a C++14 templated library, on top of which sits a "regular" and easy to use C++ library, thanks to type erasure and run-time code generation. Python and IPython bindings for it provide a very pleasant interactive graphical user experience. Don't tell anyone but he's quite proud of what Vcsn has become :)

Time for a new adventure

After 6494 days spent at EPITA with its great team and passionate students, he's now part of the Infinit team at Docker, working on the distributed storage platform. Here, he will indulge himself in some of his passions: modern C++, frisbee, and coffee.

Akim will be working on the platform backend alongside Matthieu and Quentin. You can find him on our Slack channel at @akim.

Welcome Akim!

Drake is a language agnostic build-system, officially adopted and supported by Infinit since July 2012. At Infinit, we use Drake to build our products (C++ and Objective-C), our servers (Python), our websites (HTML, CSS and JS), to run tests and to package applications, etc.

We moved from CMake to Drake for several reasons, the main one being that Drake configuration files (known as a drakefile) are plain Python3 files; it's easy to read and benefits from the rich Python3 ecosystem, making most tasks trival.

Drake is a bottom-up designed build system. It relies on 3 simple parts: nodes (files: sources or targets), builders (that creates nodes) and dependencies (conditions for builders); builders transforms a list of source nodes into a list of target nodes when all its dependencies have been fulfilled.

The drakefile (comparable to both a CMakefile and the resulting Makefile) is a Python3 file listing nodes and defining builders, in order to create a dependency graph. Once defined, the dependency graph is explored and Drake creates target nodes by running the associated builder. Greenlets allow for parallelizing builders and a cache (based on mtime and sources, targets and builder hashing) prevents unnecessary operations.

Drake and C++

Even though Drake is language agnostic, it includes a module for C++ designed to ease integration, including:

• Header exploration: Drake recursively explores includes and adds headers to the builder's sources; editing one, even a system header, will cause Drake to recompile.
• Configuration composition: C++ configurations describe include paths, library paths, flags, etc and can be summed up together.
• Tookits: Drake determines tools related to your compiler and detects source and target architectures, thereby, ease cross-compilation.

Here is a drakefile for a dummy hello_world.cc example:

import drake  
import drake.cxx

def configure(cxx_toolkit = None,  
              cxx_config = drake.cxx.Config(),
              path = '.'):

  # Create a default C++ toolkit if none is provided.
  # This will use the default system compiler.
  cxx_toolkit = drake.cxx.GccToolkit(cxx_toolkit)

  # List the sources required.
  sources = drake.nodes(
    'hello_world.cc',
  )

  # Declare a builder to create our 'hello_world' executable.
  hello_world = drake.cxx.Executable(
    # Path in the build directory where the executable will be
    # output to.
    path = drake.Path('%s/hello_world' % path),
    sources = sources, # Sources on which the executable depends.
    tk = cxx_toolkit,  # C++ toolkit to use.
    cfg = cxx_config,  # C++ compiler configuration to use.
  )

  # Create a 'build' rule.
  # Invoking this rule using //build will build all targets
  # added to the rule (and their dependencies).
  build = drake.Rule('build')

  # Add the 'hello_world' executable to the rule's targets.
  build << hello_world

Assuming the following hierarchy for the current working directory:

$ tree . --noreport
.
├── drakefile
└── hello_world.cc

Here is an example showing how easy Drake makes changing configuration. Assuming both the PYTHONPATH and PATH are correctly set:

$ drake . --cxx_toolkit="g++" --path=linux //build
<...>  
Compile hello_world.o  
Link linux/hello_world  
<...>  
$ ./linux/hello_world
Hello, world!  

$ drake . --cxx_toolkit="x86_64-w64-mingw32-g++" --path=windows //build
<...>  
Compile hello_world.o  
Link windows/hello_world.exe  
<...>  
$ wine64 windows/hello_world.exe
Hello, world!  

Easy!

Simple as a brick building game

Drake has been designed to be extensible! Because it relies on simple bricks, writing new modules is a simple task.

So, feel free to extend it: write modules for your favorite language, for documents, pictures, sound or video manipulation (why not!).

For example, here are the basics to add Go support to Drake. This can be used with a simple drakefile to build Docker machine:

import drake.go  
import drake.git

def configure(goconfig = drake.go.Configuration(),  
              gotoolkit = drake.go.Toolkit()):

  config = drake.go.Configuration(goconfig)
  toolkit = drake.go.Toolkit(gotoolkit)
  build = drake.Rule('build')
  machine = drake.go.Source('cmd/machine.go')
  git = drake.git.Git()

  config.add_ldflags(
    ['-X',
     '%s.GitCommit=%s' % (toolkit.list('version'),
                          git.rev_parse('HEAD', short = True)),
     '-w',
     '-s'
   ])
  build << drake.go.Executable(source = machine,
                               toolkit = toolkit,
                               config = config)

Everyone in our team has been using open-source software since we started programming. We believe in the power of open-source and feel that it is only natural to give back to the community and be as transparent as possible regarding our technology. We hope that by open-sourcing our code, more developers will join us and contribute to improve the core technology and build awesome tools on top of it.

We committed a few months ago to make our code available to the community. This has taken longer than we wanted but we're getting closer.

As a small team, resource allocation is always problematic but we finally made open-sourcing one of our priorities. Hence, expect some activity on our GitHub account in the weeks to come.

What are we open-sourcing?

For the past 5 years, we have developed two products, a file transfer application and a storage platform. Both rely on the same set of libraries and use our open-source build system: Drake.

We'd like to make Elle, our suite of core libraries, available to the community. We believe this powerful set of libraries will be useful to developers as it provides clean abstractions for common tasks and make use of interesting technologies (including coroutines). In addition, Elle's reliability, stability and performance have already been tested on close to a million of devices.

Once this has been completed, we would like to keep the momentum and open-source components of our storage platform. The first of these will be our horizontally-scalable, resilient and policy-based key-value store benefiting from cryptographic access control.

This will be followed by various bricks built on top of it, including tools to configure and monitor an infrastructure but, more importantly, developer APIs for creating various types of end user interfaces, allowing one to create file systems, object stores or block devices. We will include our implementations for a POSIX-compliant file system using FUSE, an Amazon S3 compatible object store and a Network Block Device.

How do we proceed?

We want the open-source experience that developers (and contributors) have to be flawless. We know the frustration of a project that doesn't compile, doesn't behave as documented or has no documentation at all. To ensure the best experience possible, we will follow a strict methodology in our open-sourcing.

Going forward, we want open-source to be a cornerstone of our development process. We hope to have Elle available publicly in the coming weeks. In the meantime, you can always keep track of our progress.

To anyone following the container space, this may not come as a surprise because persistent storage remains the number one challenge when it comes to container technologies.

Still, for Infinit, this is a huge milestone, closing the loop that has taken the team through quite an incredible journey.

Docker has changed the way developers work. By providing an ecosystem of tools based on container technologies, developers and operators have been able to gain in agility and security. At the same time, Docker has succeeded at reconciling operators and developers (through what has been dubbed the DevOps movement) by making them speak the same language.

This whole revolution is still in its early stages even though both ends of the spectrum, startups and enterprises, have jumped on the train. Indeed, at this point, containers have been used primarily to deploy stateless applications because these can be stopped and relaunched easily, allowing for easy scalability.

Unfortunately, this is no longer enough as developers want more. Namely, they want the ability to scale stateful applications, such as databases, as easily as their stateless counterparts.

With stateful applications being the next natural step in this evolution, Infinit comes into play by providing the Docker community with a storage platform that is scalable, resilient and easy to deploy.

There are numerous ways for Infinit to improve the Docker ecosystem, from a default volume plugin for stateful applications, to the distributed storage of images for the Docker Trusted Registry or to providing a hyper-converged storage environment in a Docker Swarm (something that we demonstrated in the latest Docker Hackathon).

There is a long road ahead as Infinit is still in its infancy. Our plan for the coming months is to improve stability, resilience and scalability so that we can quickly release a production-ready version 1.0 of the software.

#Docker Acquires Distributed Storage Startup @infinit to Provide Support for Stateful Containerized Applications

We are also preparing to open source the different components of our software (core libraries, decentralized key-value store and storage platform), something that a lot of people have been (rightfully) requesting. Unfortunately until now, we lacked the resources to do it properly and we did not want to just throw the sources at the community without proper support.

In addition to this, we are hard at work developing more storage interfaces (iSCSI, NBD, AWS S3 etc.) in order to cover more use cases: databases, logs, website media files, concurrent accesses, etc.

Even though Docker has already made its mark in history, we believe that what the community sees today is really just the tip of the iceberg.

Docker has the potential to reshape infrastructure software at large. This is why, even though Infinit was about to close a significant round of financing, the team has preferred to join Docker in order to have a bigger and more direct impact.

Stay tuned, this is just a new beginning! Don't hesitate to subscribe (below) to this blog to follow the development of the Infinit storage platform.

You can read Docker's announcement on their blog.

⌘ ❤︎ 🐳

With the 0.7.0 release we've added any S3 compatible service to this list. This means that if you have an S3 compatible block store (such as Minio) you can now use this to store Infinit blocks.

To use this feature, you simply need to add the credentials for the service using infinit-credentials, then create your S3 type storage with an --endpoint option as follows:

$> infinit-credentials --add --aws --name minio
$> infinit-storage --create --name minio \
    --s3 \
    --account minio \
    --region us-east-1 \
    --bucket my-bucket \
    --endpoint http://192.168.1.53:9000

Excluding the --endpoint option will make the storage default to using an Amazon S3 endpoint.

If you'd like to give it a try, we updated our S3 documentation. If you'd like us to add another provider, you can upvote or add it here.

During his presentation, Quentin talked about how the Infinit storage platform is designed and showed our new tool infinit-cluster which simplifies setting up common storage deployments. We're still working on the tool so it's not yet included in Infinit but expect to see it in a future release.

What follows is the demo that Quentin gave at the Docker Distributed Systems Summit. The demo creates three storage nodes (a.k.a servers) to store the blocks of data (with a default replication factor of three). We can then create a volume backed by this storage that can be accessed by other containers for example.

The first step is to initialize the Infinit cluster:

$> docker run --name node1 --rm \
    infinit:0.7.0 \
    infinit-cluster --init

Once this has been done, other nodes can join the cluster by connecting and authenticating to the first node. Those of you familiar with Docker Swarm will see that infinit-cluster's syntax is very similar to Docker Swarm's.

$> docker run --name node2 \
    --link node1 \
    infinit:0.7.0 \
    infinit-cluster --join node1

$> docker run --name node3 \
    --link node1 --link node2 \
    infinit:0.7.0 \
    infinit-cluster --join node1

Once the Infinit cluster is running, we have a place to store our data. We can now launch a new container in which we will create a volume and mount it.

$> docker run --name client --privileged \
    --link node1 --link node2 --link node3 \
    infinit:0.7.0 \
    bash
$> infinit-volume --create docker-summit
$> infinit-volume --run docker-summit --mountpoint /mnt/infinit &

Note that we need the --privileged mode so that we can mount the Infinit volume that we created.

Files and folders can be created in the /mnt/infinit/ mountpoint and they will persist beyond the life of the newly launched container. They will also be accessible by other containers.

We can access the volume we created from other containers using the Infinit Docker volume plugin. This is part of the Infinit daemon which is also containerized and can be started using the following command:

$> docker run --name plugin --privileged \
    --volume /run:/run:shared \
    --link node1 --link node2 --link node3 \
    infinit:0.7.0 \
    sh -c 'infinit-cluster --connect node1 && infinit-daemon --run'

Once the Infinit daemon is running, the docker-summit volume is exposed as a Docker volume that you can use in any container.

$> docker run --name my-service \
    --volume infinit/docker-summit:/mnt/infinit \
    alpine \
    touch /mnt/infinit/something

Quentin's slides from his presentation can be found here:
http://www.slideshare.net/infinit-one/persistent-storage-tailored-for-containers-dockersummit

Check out the video here:
https://www.youtube.com/watch?v=k0m-XwCoxo0

Most evil slide of the #DockerSummit was definitely the @infinit total knowledge pentagram pic.twitter.com/NQGXR9WOuJ

— *aanand, **kwaanand (@AanandPrasad) October 8, 2016

• Infinit for Windows in alpha
• IPV6 support
• New Koordinate overlay
• S3 compatible storage support
• Bandwidth and CPU usage optimization
• Full changelog
  
  

Infinit for Windows

We finally have something to share with you, fellow Windows users. You can now find our Windows binaries on our website, and our Get Started guide is updated with some nice dir commands.

Download Upgrade

Our GUI client, Infinit Drive, is now also available on Windows.

As always, don't hesitate to get in touch with us if you have any feedback or would like to chat about our releases. We're on reddit, Twitter, Slack or IRC.

Docker's lightweight virtualization, portable container format and orchestration tools are driving a revolution in software engineering and infrastructure. Lighter weight virtualization allows complex services to be broken into single applications per container which simplifies and clarifies system design.

A portable container format and suite of orchestration tools mean that deploying services is easier, more robust and more deterministic than ever.

This ease of deployment and scaling is particularly the case for stateless applications. Scaling out is as simple as launching more application containers on available hosts and load balancing across them. Stateful applications are more complex. In the simplest form, application state can be stored on the container's host but this requires that an application is run on a specific host.

In addition, if this host were to be lost the application's state data would be lost too. Different applications have vastly different storage needs: a database may require raw block storage while a website may require an object store for images and yet another application a full POSIX compliant filesystem.

Clearly a networked storage solution which provides different types of storage is required. A modern solution should also scale well – as close to linearly as possible – and not have any single points of failure. These goals can only truly be met by a decentralized system.

At Infinit we are building such a storage platform and with the 0.6.2 release we have introduced a Docker volume plugin. Once you have setup the Infinit daemon on your hosts, your containers will be able to mount Infinit volumes irrespective of the host that they're running on.

To get started with our Docker plugin, read the documentation.

Get Started

Container technologies and the micro-services philosophy allow for more agility when it comes to developing and deploying applications. In addition to such revolutionary technologies are software-defined infrastructure solutions, from networking to storage, which have been in used in development environments for years but are now about to disrupt enterprise IT.

These two trends have been obvious to developers and operators for quite some time already, in particular following the exceptional growth of Docker. But it happens that enterprises are also embracing these radical changes as Keith Fulton, CTO of ADP, made clear in his talk at DockerCon'16.

Keith Fulton talk during DockerCon'16 (from 1:06:00)

Enterprises feel that it is about time, having learned the hard way that their storage infrastructure is stuck in another age. Expensive appliances, non-scalable protocols, monolithic setups and non-programmable components are just too far off to keep pace with ever-changing business requirements.

As a result, many enterprises are no longer afraid to consider drastically modernizing their infrastructure. This process starts by replacing their hardware storage setup with a naturally distributed storage platform. By decoupling the hardware from the software, one can expect to reduce costs and greatly gain in flexibility.

This trend is already made clear both through market research analyses and the lost of market shares from all the major hardware vendors: EMC, NetApp, HP, Dell etc.

Software-defined storage is making its way into production environments.

Whether you like it or not, software-defined storage is making its way into production environments because of compelling advantages over existing hardware-based solutions:

1. Scalability: Software-defined solutions can scale up to several petabytes (even to exabytes if well-architectured) whereas appliance-based solutions are inherently limited in terms of storage capacity. In addition, software-based solutions can very quickly scale out to thousands of computer nodes, in particular if benefiting from a decentralized architecture (i.e no metadata server): Infinit, Cassandra, Tahoe-LAFS etc.

2. Cost: While enterprises have been locked in with existing vendors for decades, software-defined solutions allow them to finally decide which hardware to use: re-use legacy appliances, buy custom or commodity hardware. Enterprises can expect a 70+% reduction in total cost of ownership. In addition, because different servers can compose the software-defined infrastructure, it is much more cost effective than traditional solutions to keep pace and adapt to the exponentially growing amount of data to store by integrating modern hardware.

3. Security: While the firmware of hardware-based solutions is rarely updated, leading to critical security holes that are never fixed, software-based solutions can be updated on a regular basis to benefit from security patches and performance optimizations.

4. Flexibility: Software-based solutions can, by definition, run anywhere (or they shouldn't call themselves software-defined): bare metal, virtual machines or containers. Given this flexibility, such solutions are ideal for more complex environments: multi-site, hybrid cloud etc. In addition, some software-defined solutions (Infinit and Hedvig primarily) go further by allowing multiple storage infrastructure to evolve in parallel, each with its own set characteristics: topology, composition etc. This capability allows enterprises to choose which data goes where: to the cloud for rarely accessed data for instance, or on premise for benefiting from a faster access or to keep business critical information under control.

5. Programmatic: Even more important to modern flows is the possibility to control the behavior of a storage infrastructure through a set of properties: redundancy, security, data placement, versioning and more. This allows developers and enterprises to set up different infrastructure with different optimizations that will suit the needs of the overlying application/user.

For the past decade, several software-defined solutions have paved the way to what is now accepted as a viable solution to production challenges, most notably RedHat's Ceph and GlusterFS but also Nexenta.

Even though such storage solutions are not necessarily playing very well with the cloud, it is worth noting that software-defined storage, by definition, can be complementary by allowing for hybrid cloud deployments.

Also, thanks to the advent of containers, enterprises could also create their own cloud by relying on cloud-based bare metal for instance, and further reduce their costs: Packets, Scalway, etc.

A new and more modern wave of software-defined storage products like Infinit, Hedvig, Portworx and more) is however making its way to the market, in particular because well adapted to more dynamic environments such as large-scale containerized applications.

Given the diversity of workflows and setups, it is clear that this is not a winner-takes-all market, as the history has shown with numerous big players. However, the advent of container technologies and software-defined technologies are reshuffling the cards...