Augmented Reality-Based Mobile App for iOS Users

One of our latest R&D projects aims to show people that no matter where we are, art is all around us. It’s just a matter of perception.

We’ve developed an Augmented Reality-based mobile app that instantly shifts your surroundings based on a given painter’s style.

What was the challenge?

At Netguru our vision is to integrate our research capability into solving real-life problems. In addition to our commercial activities, we spend time creating educational and non-profit projects that can benefit the community.

By creating Kunster we wanted to bring art closer to people and engage both gen Z digital natives and older generations.

Now you don’t have to leave your house to enjoy the art and see the styles of the greatest world painters. Our app will help you to experience it without having to go to the museum, if your access is limited.

We decided to pick the most characteristic paintings and famous artists, including:

• Van Gogh and his Starry Night
• Leonardo da Vinci, the author of Mona Lisa
• Edvard Munch, famous because of his Scream
• Pablo Picasso and his The Weeping Woman
• Witkacy and his Fight (Tree Felling) - in partnership with Muzeum Sztuki in Łódź
  
  

Enjoy art wherever you are

See for yourself how it works!

How did we do it?

Our iOS application is based on Machine Learning and Augmented Reality. We used ARKit, CoreML, and TensorFlow Lite to achieve our goals.

PyTorch allowed us to quickly develop a pipeline to experiment with style transfer - training the network, stylizing videos, incorporating stabilization, and providing the necessary evaluation metrics to improve the model. coremltools was the framework we used to integrate our style transfer models into the iPhone app, converting the model into the appropriate format and running video stylization on a mobile device.

How does it work?

The app uses AR to overlay a portal over the user’s physical surroundings. Step through it, and you can view the world transformed to look like it was painted by the artist you selected.. Users can record videos and share them with friends and/or social media. They can also learn something more about the painter, take a quiz about his/her work, or find museums where the artworks are being displayed.