How accurate is trade mark search powered by computer vision technology?
When we first began the Artificial Intelligence project with Lanternfish in 2017, we brainstormed many possible types of AI applications in the intellectual property field, such as patent and trade mark search, translation, prediction of case results, drafting patents, designing logos, etc.
These are all theoretically within the capability of AI technology, but after further analysis and evaluation, from both a technical and business perspective, we finally decided to focus on three main areas: image trade mark search, patent translation and patent search.
At Kangxin and Lanternfish, each of these has a specific project team which includes AI engineers in the respective fields, such as AI in machine vision for the image trade mark search project; and natural language processing for patent translation and patent search projects, with the teams supplemented by IP professionals such as trade mark attorneys, and patent attorneys.
In this blog I would like to share with you particular insights from the image trade mark search project.
1. What is computer vision (CV), and how can it be used in image trade mark search?
I tried to search for an accurate while easy-to-understand definition of this technical term, and found the following version from a website of an analytic company called SAS, which I like the most:“Computer vision is a field of artificial intelligence that trains computers to interpret and understand the visual world. Using digital images from cameras and videos and deep learning models, machines can accurately identify and classify objects — and then react to what they ‘see’.”
In the field of CV, there are some specific machine learning techniques, such as object detection in images, which are very relevant to image trade mark search.
2. What are the differences compared to the traditional Vienna code search?
In our traditional search experience using Vienna code, there are several unavoidable steps to be conducted by professional trade mark attorneys:
- Recognise what the objects are in the target trade mark image. Since many trade marks are designed to be abstract, philosophical, and flexible to support various interpretations by the brand owners, it is often a challenge to accurately describe exactly what the objects are in the logos, even for the most experienced attorneys, and there are very likely to be various possible and reasonable descriptions
- Check the Vienna Classification and find all the codes that correspond to the descriptions, this is similar to using a dictionary. There are over 1,600 codes, such as 4.2.20 “Other beings partly human and partly animal”
- Input all the selected Vienna codes, and search. You then may receive thousands of results in a random sequence, and must try to find the relevant ones, which may take hours
- Even with all these efforts, there is still a significant risk of missing some similar marks if they are not accurately coded in the official database, or if the objects in the mark do not belong to the same code, despite having a similar visual appearance.
AI-powered image search only has two steps:
- Upload the target trade mark picture, and search
- The search results will show the similar marks with the ranking based on degree of similarity.
3. What are the differences between the AI engine for image trade mark search and Google image search?
Google image search is trained by big data from all kinds of images, including many kinds of pictures which are not trade mark-like. Trade mark images are substantially different from general pictures, and the amount of trade mark images only accounts for a fraction of the whole of big data; furthermore, the criteria of similarity between trade marks is a legal issue, which is different from the general expectation of using Google search to find a restaurant.
The quality and relevance of the big data used to train the engine will determine the final performance of the search, which can be evaluated by the following KPIs:
- Do the search results miss any similar marks?
- Is the ranking reasonable, is there any less similar mark ranked higher than the more similar mark?
- Speed of search
4. Challenges of AI in image trade mark search
What if the Trade Mark Offices continue to search the traditional way, by using Vienna codes, and cite some prior marks which are not considered similar by the AI search engine?
In some cases, AI is wrong, and the missed cited marks should be considered similar. AI deep learning is different from traditional rule-based software, because you do not teach the engine the rules, the engine learns by itself through data training. Thus if you find a certain search result from AI to be odd or not satisfactory, you cannot simply change the rule to correct it.
Changing results generated by AI is complicated, you may need to feed the engine with more data of a certain type, to check whether the original data has certain defects, or to add more layers to the neural network... Therefore, many people say AI is like a black box. The good news is that the ‘box’ can be continuously improved with deep learning, and its learning speed is faster than human beings.
In other cases, AI is probably right, i.e. the missed marks should not have been cited to reject the later mark, which is also why some official rejections could be overcome in the review of refusal by arguing the dissimilarity.
Consistency of examination criteria among all examiners has always been a challenge, as the determination of trade mark similarity is rather subjective when conducted manually.
It seems that some Trade Mark Offices, such CNIPA and EUIPO, are also exploring the use of AI technology in conducting image trade mark search and examination. Once the application of AI image search becomes more widely used, the consistency between various algorithms may be easier to reach than the consistency between human judgements.
Whilst writing this blog, I noticed a news item where Joseph Redmon, the creator of the YOLO algorithm, declared that he will stop conducting computer vision research due to military applications and privacy concerns.
No doubt any new technology would face the risk of being misused, and the evolution of laws and related enforcement systems is always slower than the development of technology. It is hoped that with the wisdom and effort of both IP practitioners and AI engineers, AI technology can help make the life of IP attorneys and in-house counsels easier and happier, without any side effects.
Next steps
- Consider joining FICPI’s Study & Work Committee on Software, High-Tech, and Computer Related Issues (CET 6)
How FICPI makes IP attorneys more effective
FICPI’s members are worldwide and from a host of specialisms in the intellectual property arena, ensuring that insights from the community can be shared with fellow members, whether to develop professional excellence, improve IP asset management, or keep up with the latest in technology enhancements.