by
Michael Schlauch
Welcome to the benchmark notebook for the Pose Bowl: Object Detection challenge!
If you are just getting started, first checkout the competition homepage and problem description.
Pose Bowl: Object Detection¶
In this challenge, you will help to develop new methods for conducting spacecraft inspections by identifying the position and orientation of a target spacecraft in an image.
The goal of this $12,000 challenge is to help NASA address key operational hurdles in spacecraft inspection, including the ability to detect a variety of different types of target spacecraft, as well as being able to run this code efficiently on relatively low-cost hardware. For the latter reason, this challenge is a code execution challenge, and your submission will be run on our code execution platform in an environment that simulates a small, off-the-shelf computer board used on R5 NASA spacecraft.
If you're not familiar with code execution challenges, don't worry! This benchmark notebook/blog post will help you get comfortable with the setup and on your way to producing your own challenge submissions.
We'll cover two main areas in this post:
Section 1. Getting started: An introduction to the challenge data, including image examples and some explanation about the type of variation to expect across the dataset.
Section 2. Demo submission: A demonstration of how to run the benchmark example and produce a valid code submission.
Section 1: Getting started¶
Get this notebook¶
First, clone the runtime repository with:
git clone https://github.com/drivendataorg/spacecraft-pose-object-detection-runtime.git
The repository includes a copy of this notebook in /notebooks/data-exploration-and-benchmark.ipynb
. We'd encourage you to run this notebook yourself as part of this exercise.
You will also need to set up your environment. For the purposes of this notebook, you will just need to the following libraries, which you can install with the following pip
command.
pip install jupyterlab pandas pyarrow opencv-python matplotlib ultralytics
Download some data¶
Let's first download some challenge imagery from the Data Download page. The imagery dataset comes in the form of .tar
files, each of which is about 2.5 GB. For this benchmark example, we'll just download the 0.tar
file.
Downloading imagery may take some time, so feel free to do something else for a little while. (The code submission format page and this runtime repository's README are relevant background reading for what we're about to do next.)
Once the tar file has been downloaded, you can extract it to the data_dev
directory (the command below should work on a Unix-based system). We suggest saving the images to the data_dev
directory (or something similarly named) rather than the data
directory because the data
directory plays a special role in simulating the test data that is available when your submission runs in the code execution environment.
tar -xzvf 0.tar -C data_dev/
You'll also want to download the submission_format.csv
, train_labels.csv
and train_metadata.csv
from the Data Download page and save these in the same data_dev
directory.
Once everything is downloaded, your data_dev
directory should look like this.
spacecraft-pose-object-detection-runtime/
└── data_dev
├── images
│ ├── 0001954c9f4a58f7ac05358b3cda8d20.png
│ ├── 00054819240f9d46378288b215dbcd3a.png
│ ├── 000dbf763348037b46558bbcb6a032ac.png
│ ...
│
├── submission_format.csv
├── train_labels.csv
└── train_metadata.csv
Explore the data¶
Now it's time to get acquainted with the challenge data. Below we define locations for some of the important files.
from pathlib import Path
import numpy as np
import pandas as pd
PROJ_DIRECTORY = Path.cwd().parent
DATA_DIRECTORY = PROJ_DIRECTORY / "data"
DEV_DIRECTORY = PROJ_DIRECTORY / "data_dev"
IMAGES_DIRECTORY = DEV_DIRECTORY / "images"
Let's take a look at two of the metadata files.
train_meta = pd.read_csv(DEV_DIRECTORY / "train_metadata.csv", index_col="image_id")
train_labels = pd.read_csv(DEV_DIRECTORY / "train_labels.csv", index_col="image_id")
The train_metadata.csv
contains information about the type of spacecraft and background used to generate the image. You should consider ways to use this information effectively in stratifying your train and test splits. The test data that you will be evaluated on includes spacecraft types that are not included in the training set, so you will want to consider strategies for ensuring your model generalizes well. You may also want to consider ways of generating additional training data, e.g. to generate a more diverse variety of background imagery.
The train_metadata.csv
contains information about all images in the training set. But since we haven't downloaded all the images yet, let's filter the dataset down to just the images we've saved locally.
# we didn't download the full training set, so add a column indicating which images are saved locally
train_meta["exists"] = train_meta.index.to_series().map(lambda x: (IMAGES_DIRECTORY / f"{x}.png").exists())
# filter our metadata down to only the images we have locally
train_meta = train_meta[train_meta.exists]
train_meta.head()
The train_labels.csv
contains the bounding box information for the target spacecraft in each image.
train_labels.head()
Let's look at a few example images to get a feel for what's in this dataset.
import cv2
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
def get_bbox(image_id, labels):
"""Get bbox coordinates as list from dataframe for given image id."""
return labels.loc[image_id].loc[["xmin", "ymin", "xmax", "ymax"]].values.tolist()
def display_image(image_id, images_dir=IMAGES_DIRECTORY, show_bbox=False, labels=train_labels):
"""Display image given image ID. Annotate with bounding box if `show_bbox` is True."""
img = cv2.imread(str(images_dir / f"{image_id}.png"))
fig, ax = plt.subplots()
# cv2 reads images as BGR order; we should flip them to RGB for matplotlib
# ref: https://stackoverflow.com/questions/54959387/rgb-image-display-in-matplotlib-plt-imshow-returns-a-blue-image
ax.imshow(np.flip(img, axis=-1))
if show_bbox:
xmin, ymin, xmax, ymax = get_bbox(image_id, labels)
patch = Rectangle((xmin, ymin), xmax - xmin, ymax - ymin, fill=False, edgecolor='white', linewidth=1)
ax.add_patch(patch)
Some images are relatively straightforward with an easily identifiable spacecraft. Below is one example. Note that the spacecraft is not necessarily fully in view. Also note that the background in this case consists entirely of our planet, with no view of outer space, while in other images it may be entirely space, or more commonly a mix of planet and space.
display_image("07dd7b5a0b7b224abc0a7fea8e78de76", show_bbox=True)
There are many examples where the spacecraft will be more difficult to detect.
One challenging type of image involves target spacecraft that appear very small, due to their distance from the chaser spacecraft. Here is one example of that situation.
display_image("0d4b4eda4bf7c0251399047d71cc4188", show_bbox=True)
Lens flare and other visual artifacts from the refraction of light on the camera lens may also complicate your detection algorithm, as in the example below.
display_image("01fcd95cdcf8bb84ec4dfa7b87bf2abc", show_bbox=True)