Getting started with the Meta AI Video Similarity Challenge

In this post, we walk through the steps for getting started with the Meta AI Video Similarity Challenge.

Michael Schlauch
DrivenData

Why a video similarity competition

The ability to identify and track harmful material on social media platforms is crucial to the experience of users on these platforms. Content tracing systems allow new content to be matched with its the original source, and can also be used to identify violent content, such as the Christchurch shooting, or to address other challenges such as identifying misinformation and enforcing copyright protections.

Manual content moderation can be difficult to scale to meet the large volume of content on platforms like Instagram and Facebook, where tens of thousands of hours of video are uploaded each day. Users may be able to create copies of harmful content faster than they can be reviewed. Copy detection systems allow scaling of content moderation, by applying moderator decisions to new copies of that content.

Video copy detection poses unique challenges, due to the amount of information in each video. Violating content may be a small section within a video, and may require localized matching methods to detect.

This competition allows you to test your skills in building this key component of content moderation systems, and in so doing contribute to making social media more trustworthy and safe for the many who use it.

A new video dataset

The Meta team has created a new dataset for this challenge comprised of thousands of videos, some of which are manipulated versions of other videos in the dataset.

Here's an example of one such video pair. The video on the left is a reference video. The video on the right is a query video that contains a clip that has been derived from the reference video -- in this case, a portion of the query video (from 0:00.3 to 0:05.7) contains a portion of the reference video (from 0:24.0 to 0:29.43) that has been surrounded in all four corners by cartoon animals and had a static effect applied:

reference video (credit: walkinguphills) query video derived from reference video


Your task in this competition -- whether you are participating in the matching track, descriptor track, or both -- will be to develop methods for determining whether a given query video is in fact derived from one of the videos in a large reference set.

Suggested steps for getting started

The Video Similarity Challenge is somewhat more complex than a typical challenge. It has two tracks and two phases. There is a code execution component. Some special rules apply in developing your solution. But we try to make it as easy as possible for you to participate.

Here are the steps we'd suggest for getting familiar with the challenge, so that you're soon on your way to making submissions and hopefully climbing up the leaderboard.

1. πŸƒπŸ½Click the β€œCompete” button on the sidebar to enroll in the competition (matching, descriptor).

2. πŸ“– Read the Problem Descriptions for each track (matching, descriptor) to get familiar with the competition data, problem framing, and evaluation metric. You may want to start thinking about which track to focus on, at least initially, although you are encouraged to participate in both tracks if possible. Solutions you develop for one track can often be adapted to work with the other track later on.

3. πŸ“– Read the Code Submission Format pages (matching, descriptor) to understand what you'll need to submit. For now, the main thing to note is that this will eventually include a script to run inference for your solution, which we execute on our compute cluster to ensure solutions meet requirements for being computationally efficient. The script is not required in order to make a submission and be scored, but it will be required in order to be prize-eligible. You may choose to focus on the other parts of your solution initially, and turn to the code execution component once your solution is more developed.

4. πŸ”½ Download the data. Metadata and ground truth files can be directly downloaded from the competition website (matching, descriptor). The downloading-video-archives.txt file on these pages also contains instructions for downloading the video archives, which are hosted on S3 as individual video files. The entire Phase 1 dataset is about 150 GB. If you just want to download a small subset of the data for now, refer to the section of the instructions for downloading a "nanoset".

5. πŸ” Research some possible approaches. Now might be a good time to get familiar with how others have approached this or similar problems. Here are a couple suggestions directly related to this challenge, but we would encourage you to look well beyond these examples:

Unlike a typical supervised machine learning competition, developing a solution for this challenge is not as simple as just training a model on the provided labeled training data. We expect that successful competitors will need to be creative in assembling training data, potentially bringing in external datasets and using self-supervised approaches.

On that note, you will probably want to check out the data augmentations library Augly, which was used in creating the competition dataset. Augly could be a useful tool for generating your own labeled video pairs. See the documentation on working with video data here.

6. βš–οΈ Understand the rules. You'll want to do both of the following:

  • Review the Official Rules (matching, descriptor). These govern your overall participation in the challenge and your commitments if you are selected as a winner. Much of this is fairly standard for participating in DrivenData competitions.
  • Review the Rules on Data Use (matching, descriptor). These outline some of the technical requirements for prize-eligible solutions.

7. πŸ“… Review important dates for this competition.

  • March 24 2023: Phase 1 ends. You are required to make a valid submission including a code execution script prior to this date if you would like to be eligible for prizes in Phase 2. If working on a team, your team cannot change after this date.
  • April 2-7 2023: Phase 2 will take place over the course of one week from April 2-9. Make sure that you'll be available to download the new Phase 2 data, which will consist of roughly 15 GB of new query videos, and run your model on the new data in this relatively short timespan.

8. πŸŽ‰ Have fun!

Good luck with this competition! We hope you enjoy it. Feel free to hit up the discussion forum if you have any questions or things to discuss with other participants. We're looking forward to seeing what you can do!

Stay updated

Join our newsletter or follow us for the latest on our social impact projects, data science competitions and open source work.

There was a problem. Please try again.
Subscribe successful!
Protected by reCAPTCHA. The Google Privacy Policy and Terms of Service apply.

Latest posts

All posts

winners

Meet the winners of Phase 2 of the PREPARE Challenge

Learn about how winners detected cognitive decline using speech recordings and social determinants of health survey data

resources

Open-source packages for using speech data in ML

Overview of key open-source packages for extracting features from voice data to support ML applications

tutorial

Getting started with LLMs: a benchmark for the 'What's Up, Docs?' challenge

An introduction to using large language models via the benchmark to a document summarization challenge.

winners

Meet the Winners of the Goodnight Moon, Hello Early Literacy Screening Challenge

Learn about the results and winning methods from the early literacy screening challenge.

resources

Where to find a data job for a good cause

Finding data jobs for good causes can be difficult. Learn strategies, job lists, and tips to find organizations with open positions working on causes you care about.

winners

Meet the Winners of the Youth Mental Health Narratives Challenge

Learn about the winning solutions from the Youth Mental Health Challenge Automated Abstraction and Novel Variables Tracks

winners

Meet the winners of the Forecast and Final Prize Stages of the Water Supply Forecast Rodeo

Learn about the winners and winning solutions from the final stages of the Water Supply Forecast Rodeo.

insights

10 takeaways from 10 years of data science for social good

This year DrivenData celebrates our 10th birthday! We've spent the past decade working to use data science and AI for social good. Here are some lessons we've learned along the way.

tutorial

Goodnight Moon, Hello Early Literacy Screening Benchmark

In this guest post from the MIT Gabrieli Lab, we'll show you how to get started with the literacy screening challenge!

tutorial

Youth Mental Health: Automated Abstraction Benchmark

Learn how to process text narratives using open-source LLMs for the Youth Mental Health: Automated Abstraction challenge

winners

Meet the winners of Phase 1 of the PREPARE Challenge

Learn about the top datasets sourced for Phase 1 of the PREPARE Challenge.

resources

Teaching with DrivenData Competitions

Inspiration and resources for teaching students data science, machine learning, and AI skills with DrivenData competitions.

winners

Meet the winners of the Pose Bowl challenge

Learn about the top solutions submitted for the Pose Bowl: Spacecraft Detection and Pose Estimation Challenge.

winners

Meet the winners of the Water Supply Forecast Rodeo Hindcast Stage

Learn about the winning models for forecasting seasonal water supply from the first stage of the Water Supply Forecast Rodeo.

tools

Cookiecutter Data Science V2

Announcing the V2 release of Cookiecutter Data Science, the most widely adopted data science project template.

resources

How to make data science projects more open and inclusive

Key practices from the field of open science for making data science work more transparent, inclusive, and equitable.

winners

Meet the winners of the Kelp Wanted challenge

Dive into the solutions from the super segmenters who best detected kelp in Landsat imagery!

winners

Meet the winners of the SNOMED CT Entity Linking Challenge

Meet the winners with the best systems for detecting clinical terms in medical notes.

winners

Meet the winners of the Pale Blue Dot challenge

Learn about the top visuals created for the Pale Blue Dot: Visualization Challenge and the solvers behind them.

tutorial

NASA Pose Bowl - Benchmark

An introduction to the NASA Pose Bowl competition, with a benchmark solution for the object detection track

Work with us to build a better world

Learn more about how our team is bringing the transformative power of data science and AI to organizations tackling the world's biggest challenges.