Elle, our C++ core library, is now open source

TL;DR. We are proud to finally open source Elle, Infinit's C++ core library providing a C++ asynchronous framework, based on coroutines, that comes with a set of modules to easily perform networking, serialization, RPCs, cryptography, distributed consensus and more.

Even though the C++ standard has been moving pretty fast over the last few years, with a new version every 3 years, the C++ ecosystem lacks some important features. Boost C++ libraries has been providing reference implementations for these missing features, helping programmers get a few steps ahead.

In 2009, Infinit's founder Julien mycure Quintard started developing Elle as the core library for his PhD project: the prototype of the Infinit storage platform. After years and years of development, Elle has grown and is now a meta-library composed of a set of sub-libraries that we are proud to finally share with the community.

Elle is currently (mid February 2017) at stage 1 (access); you can clone the source code, read it, build it and tweak it. However, until it reachs stage 2 (develop), the documentation and hierarchy might seem confusing. Hopefully, within two months, Elle will reach stage 2 and be well documented.

Yet another library?

Elle is a software development framework, in the sense that it changes the way we approach asynchronism (spoiler: it uses coroutines).

Some frameworks like Qt with its signals/slots, rely on event-driven programming, which tends to produce "spaghetti code": it's hard to follow the execution without knowledge of a wide range of classes. Like every other paradigm, event-driven programming and coroutines have their pros and cons. However, as you can see, a few projects have emerged to introduce support for coroutines in Qt (qt-couroutine or QFiber). I'm not blaming Qt people. Qt is an extremely mature, well designed framework and its popularity speaks for itself. Event-driven programming fits their needs perfectly.

Elle is much more than just an asynchronous framework; it comes with tons of built-in features. Being the core library of the Infinit storage platform, Elle provides all the tools you need to create projects which require cryptography, networking and/or consensus. With Elle, one could very easily develop, say, an HTTP server in C++ capable of handling tens of thousands of clients without breaking a sweat.

Welcome to Elle

Elle is based on Boost, OpenSSL, zlib, curl, and a few other libraries which forms a unified C++ environment, including:

  • A complete asynchronous framework based on coroutines in C++
  • Compile-time introspection via symbol-based metaprogramming
  • Object-oriented cryptography
  • Versioned and multiplexed communication protocol for remote procedure calls (RPCs)
  • And more...

And, good news everyone, it's now open-source and available on our GitHub account.

Reactor: an asynchronous framework for C++

For those who are not familiar with coroutines, natively supported by dozens of languages (including Go, Haskell, Python, D, etc.), they allow for non-preemptive multitasking (unlike threads) offering concurrency with a reduced need for locking and fit extremely well in I/O bound programming. They also allow for an illusion of sequential code execution in an asynchronous environment.

Basically, a coroutine can choose to yield, in which case its context (state of the program) is saved and the routine goes to sleep. Then, another coroutine is woken, has its context restored, runs until it yields or completes and so on until all the coroutines have completed their tasks. Unlike thread programming, the current coroutine cannot be preempted.

Coroutines are one of the current hot topics for C++, with a proposal for a potential addition to the standard. Boost.Coroutine already provides "templates for generalized subroutines". Elle's reactor sub-library, in addition to providing an elegant and portable implementation of coroutine, is the complete asynchronous framework mentioned earlier, that includes networking sockets (TCP, UDP, µTP), support for HTTP(s), finite state machines (FSM), synchronization mechanisms, etc.

Here is an example of a coroutine in charge of echoing back a message. As you will see, the code is written as a natural sequence of instructions to execute, same for exception handling: no hooks, no promises and no callbacks.

void  
echo_server()  
{
  // ...
  // For each connection, store the coroutine it was assigned to.
  client_threads.emplace_back(
    std::make_shared<reactor::Thread>(
      sched, "serve",
      // Echo a message back to it sender.
      [socket]
      {
        // Until the connection is closed.
        try
        {
          while (true)
          {
            // Yield until a the message is read until '\n'.
            auto line = socket->read_until("\n");
            // Yield until the message is written.
            socket->write(line);
          }
        }
        catch (reactor::network::ConnectionClosed const&)
        {}
      }));
}

It is as simple to perform HTTP requests.

// Parse a stream, expecting JSON.
auto process_json = [] (elle::IOStream const& body)  
                    {
                      // Do something clever.
                    };
// Create the HTTP request.
auto r = reactor::http::Request{  
  "http://maps.googleapis.com/maps/api/geocode/json",
  reactor::http::method::GET,
  "application/json"};
// Set the query.
r.query_string({{"address", "white house"},  
                {"sensor", "false"}});
// Finalize the operation.
r.finalize();  
if (r.status() == reactor::http::StatusCode::OK)  
  // Parse the output as a stream.
  process_json(r);
// ...

Das: compile-time introspection

Even if C++ offers powerful metaprogramming capabilities, Das uses a symbol-based paradigm to push it further. Named symbols provide powerful compile-time introspection, allowing for more flexibility and decision making. It also provides convenience features like:

  • Printing and serialization

Writing printers and basic serialization implementation is a time consuming and redundant task. Das provides a non-intrusive default serialization implementation and printer for C++ objects.

  • Named arguments

With minimal intrusion, Das offers the possibility to have named arguments in C++ for clearer and less error prone function calls.

  • CLI

Often, command line tools have a 1-1 mapping to the underlying functions actually invoked and force manual type-conversion from the CLI argument to the type of the actual function argument. Named symbols, by design, perfectly solve the problem, offering an almost zero cost CLI binding.

Here is an example demonstrating how, in a non-intrusive way, you can provide serialization and printing for a random object.

// The symbols used for introspection.
DAS_SYMBOL(artist);  
DAS_SYMBOL(title);

struct Record  
{
  std::string artist;
  std::string title;
};
// The Model, which declares the symbols associated to a class or struct.
using Model = das::Model<Record,  
                         elle::meta::List<Symbol_title,
                                          Symbol_artist>>;
// Set Model as the default model for Record.
DAS_MODEL_DEFAULT(Record, Model);  
// Make all class::Model printable.
using das::operator <<;

// You know have access Das' features:
// - Printing:
// > Record(artist = "Darude", title = "Sandstorm")
std::cout << Record{"Darude", "Sandstorm"} << std::endl;  
// - Serialization
// > {"artist": "Darude", "title": "Sandstorm"}
elle::serialization::json::SerializerOut serializer(std::cout, false);  
das::serialize(Record{"Darude", "Sandstorm"}, serializer);  

Cryptography: object-oriented wrapper for OpenSSL

Cryptography is a tough and complex area but thanks to OpenSSL, we don't have to re-implement cryptographic algorithms. Elle's cryptography sub-library is a convenience wrapper over OpenSSL, offering modern C++ interfaces, object-oriented and stream-oriented APIs, automatic resource management, transparent module initialization and deinitialization, etc.: The power of OpenSSL with the elegance of modern C++.

#include <cryptography/rsa/KeyPair.hh>
#include <cryptography/SecretKey.hh>

// This is a bad implementation of cryptography::envelope.
// It stands here as an example.
std::tuple<elle::Buffer, elle::Buffer>  
hybrid_encrypt(cryptography::rsa::PublicKey const& peer_key)  
{
  // The payload you want to send.
  auto data = elle::Buffer{"super important data to encrypt"};
  // Generate a random password of 1024 characters.
  auto symmetric_key = cryptography::SecretKey(
    cryptography::generate<elle::Buffer>(1024));
  // Encipher the payload with the secret key.
  auto encrypted_payload = symmetric_key.encipher(
    data,
    cryptography::Cipher::blowfish, // default: aes256
    cryptography::Mode::cfb, // default: cbc
    cryptography::OneWay::sha512 // default: sha256
  );
  // Encrypt the key itself.
  auto encrypted_key = peer_key.encrypt(symmetric_key);

  // Return both encrypted payload and key, so the peer can decrypt the key
  // using his private key and use that key to decipher the original payload.
  return {encrypted_payload, encrypted_key};
}

Protocol: unified communication channels

Because the Infinit storage platform relies on sending data between nodes, setting up a sustainable channel of communication is crucial. Unification and versioning of the communication protocol itself are mandatory for robustness. Elle's protocol sub-library is designed to transparently take care of those two requirements on top of every stream.

It also allows for multiplexing communications channels in a single stream in order to support our remote procedure call (RPC) system.

And more...

Elle contains additions to the standard library, such as printers for standard types, new algorithms, but also classes and functions related to:

Usage, support and contribution

Elle is officially maintained by Infinit (which is now part of Docker). Elle will evolve according to our own needs, therefore, APIs may change over time. Also, binary compatibility won't be preserved between versions for now.

You can legitimately ask why you should use Elle then. Elle is a library which provides, in our opinion, great tools to write beautiful, easy to maintain and efficient code in C++. At present, we can't ensure backward compatibility, mostly because it requires a lot of effort, but open-sourcing it will push us towards this. Also, its stability and efficiency has been proven by the two projects Infinit has developed, the Infinit file transfer applications (deployed on close to a million devices, both desktop and mobile) and our current project, the Infinit storage platform.

If you have any questions or issues, we are available:

If you want to contribute, you are more than welcome. However, until the project reaches its final stage, contributions may take longer than expected to be integrated.


Get started with Infinit in less than 10 minutes! Test it!

Antony Méchin

Software engineer at Docker (from the former Infinit team)

Paris

Subscribe to Infinit's Blog

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!