Building a Virtual Try-on Manicure App for COSMO Group’s NEONAIL

Its cosmetics reach distribution points all over Poland, but also abroad, including Germany, Spain, Italy, the Czech Republic, Austria, England, South Africa, Greece, and more. NEONAIL, a company from the COSMO Group portfolio, leads with unique and innovative nail styling and care products, including 400 shades of UV gel polish and numerous accessories.

Source: COSMO Group

An app to pick the perfect polish

Netguru worked with NEONAIL to design and develop Color Match NEONAIL – a virtual try-on app for manicures available on both the App Store and Google Play.

The app allows users to test a wide range of nail colors, shapes, and decorations in real-time, bookmark their favorites, and purchase them via NEONAIL’s connected ecommerce platform.

The lifelike color transfer experience was made possible using augmented reality, computer vision algorithms, and machine learning models. By being able to realistically test products on their own nails, the app helps users feel more confident about their purchasing decisions.

Source: COSMO Group

Boosting online sales with a virtual try-on solution

NEONAIL noticed its customers were more inclined to purchase nail products in brick-and-mortar stores where they could confirm if a color or design matched their expectations.

By building a virtual try-on app for manicures, the client wanted to provide its customers with a realistic way to try out nail products from the comfort of their homes to help them decide if they want to purchase them.

One of the main goals was to increase sales by enabling users to test out NEONAIL products online and purchase them from the ecommerce store.

Source: COSMO Group

Client’s expectations

• Develop a virtual try-on app for manicures
• Provide a natural and lifelike 3D effect of colored nails in real-time
• Provide a flawless user experience with an intuitive and easy-to-use app that’s connected to the ecommerce platform, enabling users to seamlessly purchase products after testing them
• Develop an app that’s responsive on multiple platforms and devices (not just the latest iPhones), available for iOS and Android

Netguru’s role and services provided

NEONAIL came to us with the idea of building a virtual try-on app. Our role was to bring this idea to life by consulting and developing the app from scratch.

We provided services across the whole software development lifecycle including project management, user research, product design, development, and quality assurance. More specifically, this included:

• Consulting and proposing solutions and technologies
• Creating the project roadmap
• Building the algorithms and machine learning models to create an authentic nail salon experience in real-time using augmented reality
• Building the proof of concept (PoC)
• Developing the app for both iOS and Android
• Connecting the app to the ecommerce shopping cart to enable users to seamlessly purchase products
• Providing project management and quality assurance services to ensure the cooperation ran smoothly and was completed to the highest quality possible from start to finish

Netguru assembled a team of experts to complete the project including machine learning specialists, developers, product designers, UX researchers, quality assurance engineers, and project managers.

Overcoming technical complexity

The main obstacles we faced upon commencing the project revolved around technological complexity. In short: how can we create algorithms and train machine learning models to provide an accurate, close-to-natural experience?

To achieve this, there were many factors to consider: how do we ensure the app runs smoothly across multiple devices and operating systems? How do we ensure a great experience even when the video coming from the camera is not high quality? How can we stabilize nail polish patterns and make sure they display in the right direction? And how do we provide a realistic color transfer taking into consideration nail/skin boundaries, different fingernail shapes, and existing nail polish?

Netguru’s step-by-step approach

To achieve the realistic and responsive user experience NEONAIL was hoping for, our machine learning experts ran through a number of steps:

1. Building the PoC based on images scraped from the internet to validate the idea of transferring complicated nail patterns.
2. Gathering data requirements to train the machine learning models and for the nail patterns used in the application e.g. images without glare and reflections.
3. Brainstorming potential future directions.
4. Data gathering

• Gathering data for training the machine learning model, including movies from different mobile devices with various qualities, backgrounds, and hands.
• High-quality images of nail polish samples in proper lighting conditions with no reflections. We later switched to a model created by the client’s designer.
• Maintaining data versioning and data updates during the model development.

5. Labeling and quality assurance. Describing the labeling process, organizing labeling workforce and creating a quality review process for annotations.
6. Machine learning model training for fingernail segmentation and detection of nail direction based on movies from several devices with differing noise, backgrounds, lighting conditions, and quality.
7. Optimizing the detection criteria so that nails are detected with a proper balance between sensitivity and specificity.
8. Visualization of the experiments and results for step by step debugging purposes, to show how much nail is covered with polish, the results of experiments enhancing realism, machine learning model effectiveness.
9. Using Common C++ engine for iOS, Android, and Python to maintain the same image processing pipeline across devices and significantly reduce development time.
10. Improving the realism of the effect by smoothing the nail-skin boundary and stabilizing nail detection.
11. Increasing FPS and decreasing memory requirements through pruning and quantization of the machine learning models as well as optimizing the computer vision code in C++.

An agile, flexible, and creative collaboration

We opted to use the Agile methodology and the Scrum framework for our collaboration with NEONAIL.Throughout the product development process Netguru experts did user tests thanks to which we could gather users observations, feedback and suggestions for improvements.

Thanks to regular checkins presenting the progress of the iOS& Android app development as well as demos of the machine learning models in non-technical terms, the client was able to constantly review the state of our work. This ensured smooth communication, adaptability, and transparency throughout the project.

By presenting results of experiments to the client and welcoming feedback and opinions, we were able to reach decisions on the best next steps together. We also provided machine learning model training infrastructure and documentation for management.

Results of the collaboration

The NEONAIL app was created thanks to the great cooperation with the client and cross team collaboration.

• The Color Match NEONAIL app was delivered in line with all requirements to support NEONAIL in achieving its business goals

• Users can try on various nail colors, shapes, and patterns in real-time while enjoying the realistic effect

• Netguru’s intricate engineering work facilitated the true-to-life, high-quality color transfer experience. Colors and patterns are flawlessly copied to the user’s nails while achieving a realistic 3D effect.
• The app is available on different platforms and devices, for both iOS and Android

• The client received a self-organized team of Netguru experts ready to tackle the project's strategic and technological challenges and deliver solutions to bring the idea to life.