Providing Confidence in Fiverr's AI Performance via Mona's Monitoring Solution

AI has gained a lot of popularity and continues to get widely adopted across virtually all industries as more use cases are established. Businesses deploy AI solutions for fast business growth and to stay competitive within their market. That’s exactly what Fiverr did. Now, Fiverr is an industry leader as a forward-thinking technology company.

Founded in Tel Aviv-Yafo, Israel, over ten years ago, Fiverr is a platform that connects freelance services with businesses worldwide. This has resulted in a significant amount of data that Fiverr obtains, processes, and stores. As Fiverr accumulates this data, they apply advanced analytics, AI and machine learning as a basis to create new features and products.

As the Director of Data Science at Fiverr, Yuval Ben Zion recognizes that AI and machine learning will continue to be a factor in accelerating business growth. Having been in the data science field for over 8 years, Yuval has seen many challenges that companies face when deploying models into production and has formed strategies on how to overcome them. Now at Fiverr, he oversees all areas related to algorithms, machine learning, and artificial intelligence. During his time with the company, Yuval has built a team of data scientists that develop and deploy machine learning models, adding new features and creating new products for Fiverr’s business. They understand the impact that AI can achieve for the overall business and focused their efforts to get the algorithms right.

Fiverr utilizes AI in many functions within the organization and continues to evolve its capabilities. They leverage AI to solve a myriad of business problems including spam detection, recommendations, personalizations, promoted ads, NLP and vision tasks. Just a few years ago, teams were not as aware of the impact that machine learning can bring to a business. Although businesses already have algorithms set up, machine learning utilizes that information to make products better. For example, search results are displayed using an algorithm that matches with the keyword but with artificial intelligence, it adds another layer to search by being able to rank the results in the predicted order of the intended query. With recommendations, algorithms will display similar items that have previously been viewed, but AI will allow those recommendations to be tailored to the interest of the user. There are some aspects of Fiverr’s business that relies on artificial intelligence solutions to operate, as it is a tool that has proven success to support optimizations and increase business growth.

Being forward thinking, Fiverr understood that although AI-driven systems can bring tremendous value, they can also hurt the business if not properly optimized. Yuval and his team expect issues around data integrity and model accuracy, among other things, to arise over time across their models and pipelines. It is no secret that scaling AI is about continuous evolution. Therefore, they sought a solution to automatically track their AI systems and proactively surface insights regarding issues, or opportunities to optimize so, Yuval and his team started to research the field to evaluate the tools and platforms on the market.

As Fiverr’s data science group started to investigate the different AI monitoring solutions, their main objective was finding an all-in-one monitoring platform to support all of their different AI use cases. With that in mind, the team defined the most important required features for their qualification process and decided to evaluate the different alternatives. Once they researched the field, they met with a few companies and decided to run Proofs of Concept (POCs) with two companies, one of which was Mona.

As Fiverr’s business has multiple AI-enabled use cases, they decided to test each monitoring platform with two unrelated use cases, ultimately testing the flexibility of each solution. Fiverr decided that the first use case would be related to search results ranking and the second would be the customer lifetime value prediction. These two use cases were chosen intentionally due to their major differences, in which the results would provide Fiverr with the confidence in the ability to monitor the rest of their AI use cases. On one hand, the search results ranking occurs in real time and is oriented towards Fiverr’s customers. On the other hand, the customer lifetime value predictions are received in batch processes and are not product-oriented, used internally by Fiverr's marketing team. Although they are different in dimensions, Fiverr needed a monitoring solution to work equally effectively with both of these use cases.

Upon assessment, Mona won by accommodating both use cases with its flexible configuration. Fiverr was impressed with Mona’s extensibility and the ability to integrate with their existing tech stack. Such things as being able to create flexible monitoring schemas for any use-case represented the intricacy of the Mona platform and it was exactly what Fiverr was looking for.

As Fiverr was in the evaluation process, they needed a solution that could also accommodate their massive data sets which Mona is able to do, and do well. Mona has devised methods to smartly handle huge data sets in both batches and in real-time. Fiverr realized Mona’s platform was the most mature and capable solution for big-data.

Fiverr also liked the fact it was easy to define specific metrics to track. For instance, Fiverr needed to add custom fields for the separation of their logic for train, test, and inference, as well as model version tracking and different production environments. Mona’s platform allowed them to easily implement and track these things and make sure only relevant data is used in the different monitoring schemes. This ensured that the proper metrics were being tracked and that they could quickly detect any issues that arose, avoiding risks associated with their business KPIs. For all these reasons, Mona was chosen as the monitoring solution that Fiverr wanted to move forward with.

In 12 months, Fiverr has expanded its monitoring capabilities, leveraging the Mona platform, to 15 model use cases, and there are dozens of additional use cases planned further down the road. As a full e-commerce marketplace, Fiverr handles many different use cases from search engine to fraud detection. Having deployed Mona, Fiverr’s team is able to detect issues before they impact business KPIs, leveraging a single monitoring platform for all of their use cases. This is the value that Fiverr needed all along. Mona is the only platform that provides the capabilities of defining your own custom metrics and custom dimensions within your data for any given use case.

Mona’s platform is able to perform an exhaustive search within the data to find segments with anomalous behaviors such as data drifts, biases and outliers. Once Mona detects and alerts an issue within the model, Fiverr’s team is able to instantly troubleshoot and examine how the model was configured. Being able to be automatically alerted and the ability to address problems quickly enables their team to optimize their models to ensure the best performance while also finding data issues and bugs. Working closely with Mona’s customer success team, Yuval continues to be pleased with the professionalism and the experience of the team. They are readily available to assist with everything from implementation to adding new features, specific to Fiverr’s business.

In the long term, there are three main values from monitoring models. The first is about catching underperformance issues and sending alerts to relevant team members. The second one is extending research capabilities using real production data. Finally, the third is having the operations run smoothly with automated and granular testing. Businesses are always looking to deploy best practices for their teams and do fewer things manually. Even if AI-related issues aren’t the main thing hurting your business KPIs, monitoring your models allows you to have the most confidence in your processes and have visibility into how things are going. It helps you benchmark where you were at pre-model and how things are going after your model has been deployed.

Artificial intelligence has proven to accelerate growth for businesses such as Fiverr, but in order to achieve similar results, businesses need to stay ahead of the curve. It is always important to think about any risks that can occur to your business and ways to avoid them. As new use cases start to appear, more models are added in order to support fast business growth and it is necessary to know if the predictive models are performing as intended. Being able to have complete visibility into AI systems becomes necessary to optimize on processes and guarantees that the system is running at optimum performance. Fiverr is one of the leading organizations that has successfully implemented digital transformations, especially in the area of AI.