Building a Data Annotation Platform

Machine Learning and Big Data in modern application development

The fields of Machine Learning and Big Data are becoming vital in modern application development. We can see artificial intelligence being used in systems ranging from e-commerce to health industry applications.

With these advancements in technology, data has become extremely valuable and there are many free services that make a lot of money solely through collecting and selling data of their users, which is often later used to train various neural networks.

Why does it matter?

Data on its own is rarely enough - machine learning often requires some input from the user, especially in the training phase. For example, if we want to train a neural network to recognize pictures of cats, we have to provide a large number of pictures and manually label the ones that actually contain cats.

This process is called "data annotation" and can be very time consuming with large datasets. This was the main motivation for the Data Annotation Platform.

What is Data Annotation Platform?

An app to prototype Machine Learning applications much quicker
Data Annotation Platform (or DAP for short) - an application that delegates annotating to the users, who in turn can earn a little bit of money for their work. This can tremendously help data scientists and allow them to prototype Machine Learning applications much quicker.

We've built a simple Proof of Concept version of the DAP first. In this state it was a React Native mobile application with a Python (Django) backend. It contained all the fundamental functionalities, from user management to the actual data annotation, but lacked the payment system.

Recently, we've decided to come back to this project and integrate it with the emerging cryptocurrency from Facebook - Libra.

What is Libra?

Libra is the upcoming cryptocurrency created by Facebook and developed by the Libra Association. There are two main differences between Libra and other cryptocurrencies:

• Lack of decentralization - Libra is not going to be entirely decentralized. The majority of the blockchain will be running on the Libra Association's servers. This might seem like a downside, as decentralization was supposed to be a core value for cryptocurrencies. But in practice it might significantly increase security and accessibility of the network (e.g. speed of transactions).
• Libra's value - Libra is supposed to be extremely stable, thanks to it being backed by a number of the world's most stable currencies. Even if the value of one of them goes down, the rest should keep the price of Libra at the same level. This is a huge deal because other cryptocurrencies have a history of rapid losses of value, which can scare investors and users alike.

Why Libra?

Libra - a currency for almost 2 billion people

The main motivation of Libra is to provide a currency for almost 2 billion people that currently live without bank accounts. These people are exploited by financial services companies that charge enormous fees for money transfers.

In contrast, Libra transfers are completely free of charge and the currency can be bought in a grocery store. This creates an enormous market for applications built on the Libra blockchain.

Why did we choose Libra for Data Annotation Platform?

At Netguru, we usually prefer working with mature, tested technologies, especially for commercial projects. But in our R&D department we make certain bets on the technologies that we believe will be a big part of application development in the future.

Libra has been on our radar since it's announcement and we were eager to try it. DAP seemed like a perfect candidate to use it - it's an internal project and we could safely test Libra's stability without impacting users.

Another reason to use Libra was to simply get more experience working with cryptocurrencies. The demand for such services will grow and we want to be ready to deliver to our clients' needs.

A minimum viable product (MVP)

How it turned out

The MVP version of the platform is ready. We've decided to implement the Libra integration in a separate micro-backend using Ruby on Rails. This helped us deliver the functionality very quickly, but also gave us a couple challenges.

First of all, as of now, there are still no Ruby libraries for interacting with Libra's blockchain. We've tried to write our own, but it turned out to be too big of a project. Looking for quick wins, we've found a JavaScript library that could do what we required.

So we decided to write a small Node.js program that acted as an interface between our RoR backend and the Libra library, allowing us to easily create wallets, make transactions, and check account balances.

If we were to write a similar app in the future, we'd probably use Node.js from the start to unify the tech stack (unless Ruby gets better support for Libra in the meantime).

See for yourself how it works!