At Schibsted Tech Experiments we strive to be on top of emerging technology trends and facilitate Schibsted experiments with the most relevant tech trends. One of these big trends is image recognition (a.k.a. computer vision). We believe this technology represents great potential for products and services across Schibsted in terms of increased value for our users and improved user experience.
“How?”, you might think. Below we present four use cases that we believe are valuable to our users.
What is image recognition?
Up until recently, computers could at best see our world. Advances in artificial intelligence have made it possible for machines to both see and understand our world. Images are data and this data is processed through algorithms that are trained to recognize patterns in that data. Different models can return different results depending on the desired use case.
Why use image recognition in our marketplaces?
Marketplaces are all about matchmaking (a.k.a. liquidity); that is bringing buyers and sellers together and facilitating a transaction between them. While we are good at this, we can improve!
Image recognition can help us in several ways, e.g. improve the quality of our data and the user experience, amongst other things. We have selected some of the most promising use cases and presented these below.
Use cases for image recognition
While companies like Google, Amazon, eBay and (in particular) Pinterest have been working on their own solutions for image recognition, and all of them have lately scaled their investments into the technology, Schibsted has not been completely at rest. Some features are already live and more are in the pipeline. Stay tuned!
Use case 1: Visual search
In production (BETA) for mobile browsers.
A beta version of visual search is already in use on Finn. It can be tested in mobile browsers (on iPhone only Safari is supported)
With visual search, you take a photo with your mobile phone – and the results show products that match what you photographed.
Our culture is already dominated by visual stimuli. Half of the human brain is (directly or indirectly) devoted to the processing of visual information. It seems only natural that search and discovery start with an image.
Visual search enables quicker search and more accurate results, catering to better user experience in the search and discovery phases. This drives better matchmaking and is the fuel of success for our marketplaces and one of Schibsted’s strategic pillars.
TRY FOR YOURSELF! (Click the link on a smartphone and open in browser)
Use case 2: Recommend visually similar ads
This is in production for all categories on FINN Torget.
Image recognition can capture remarkably more facets of an image than what a human is able to articulate in a text search or what collaborative filtering (a common recommendation method) is able to account for. Thus recommendations based on visual similarity can be more relevant, which makes it easier for a buyer to find exactly what she is looking for.
Another useful distinction between recommendations based on collaborative filtering versus visual similarity is that while the former tends to favor recently published ads (because of users’ behavior), the latter is often indifferent to the age of the ads (as similarity trumps age). This means recommendations based on visual similarity can give new life to older ads and hopefully provide them with a new owner.
Use case 3: Metadata enrichment
We have a prototype ready for how to add extra metadata to a product based on the images. This prototype won the jury prize at FINN Hackdays in May 2019.
From FINN Insights we know that some sellers «forget» to include significant information about their object in the ad. Meanwhile, some buyers find it hard to find exactly what they are looking for even though it might exist in the marketplace.
We cut out the humans from the equation and jump straight to automation. Image recognition allows us to populate an ad with additional metadata which makes it more «searchable» and consequently easier to find.
That way we boost liquidity – and both seller and buyer are happy!
What’s more, it could even help valuate an object and autogenerate relevant alt text for images which is great for accessibility and SEO.
Use case 4: Cropped visual search
This concept has been inspired by Pinterest and subject to further exploration at Finn.
Much like Google, our platforms started out as a place to search for and find stuff. We have learned that users might need help and we have put significant resources into discovery features like personalized feeds and similar results.
Over the past few years though, we have seen a rise of inspirational platforms like social media and Pinterest. Instagram and Pinterest are incredibly good at monetizing the search for inspiration. So why are we not doing the same?
We know that a large number of marketplace users browse the real-estate vertical for inspiration or just to dream. Why not help them find the things they like on our generalist marketplace or at Prisjakt?
Will continue the work
This is just the beginning of computer vision technology in Schibsted. Although these cases concern the marketplace side of Schibsted, the possibilities with image recognition are practically endless. We continue to work with some new ideas and improvements on existing products.