blog

NASA Pose Bowl - Benchmark


by Michael Schlauch

spacecraft with bounding box

Welcome to the benchmark notebook for the Pose Bowl: Object Detection challenge!

If you are just getting started, first checkout the competition homepage and problem description.

Pose Bowl: Object Detection

In this challenge, you will help to develop new methods for conducting spacecraft inspections by identifying the position and orientation of a target spacecraft in an image.

The goal of this $12,000 challenge is to help NASA address key operational hurdles in spacecraft inspection, including the ability to detect a variety of different types of target spacecraft, as well as being able to run this code efficiently on relatively low-cost hardware. For the latter reason, this challenge is a code execution challenge, and your submission will be run on our code execution platform in an environment that simulates a small, off-the-shelf computer board used on R5 NASA spacecraft.

If you're not familiar with code execution challenges, don't worry! This benchmark notebook/blog post will help you get comfortable with the setup and on your way to producing your own challenge submissions.

We'll cover two main areas in this post:

  • Section 1. Getting started: An introduction to the challenge data, including image examples and some explanation about the type of variation to expect across the dataset.

  • Section 2. Demo submission: A demonstration of how to run the benchmark example and produce a valid code submission.

Section 1: Getting started

Get this notebook

First, clone the runtime repository with:

git clone https://github.com/drivendataorg/spacecraft-pose-object-detection-runtime.git

The repository includes a copy of this notebook in /notebooks/data-exploration-and-benchmark.ipynb. We'd encourage you to run this notebook yourself as part of this exercise.

You will also need to set up your environment. For the purposes of this notebook, you will just need to the following libraries, which you can install with the following pip command.

pip install jupyterlab pandas pyarrow opencv-python matplotlib ultralytics

Download some data

Let's first download some challenge imagery from the Data Download page. The imagery dataset comes in the form of .tar files, each of which is about 2.5 GB. For this benchmark example, we'll just download the 0.tar file.

Downloading imagery may take some time, so feel free to do something else for a little while. (The code submission format page and this runtime repository's README are relevant background reading for what we're about to do next.)

Once the tar file has been downloaded, you can extract it to the data_dev directory (the command below should work on a Unix-based system). We suggest saving the images to the data_dev directory (or something similarly named) rather than the data directory because the data directory plays a special role in simulating the test data that is available when your submission runs in the code execution environment.

tar -xzvf 0.tar -C data_dev/

You'll also want to download the submission_format.csv, train_labels.csv and train_metadata.csv from the Data Download page and save these in the same data_dev directory.

Once everything is downloaded, your data_dev directory should look like this.

spacecraft-pose-object-detection-runtime/
└── data_dev
    ├── images
    │   ├── 0001954c9f4a58f7ac05358b3cda8d20.png
    │   ├── 00054819240f9d46378288b215dbcd3a.png
    │   ├── 000dbf763348037b46558bbcb6a032ac.png
    │   ...
    │
    ├── submission_format.csv
    ├── train_labels.csv
    └── train_metadata.csv

Explore the data

Now it's time to get acquainted with the challenge data. Below we define locations for some of the important files.

In [1]:
from pathlib import Path
import numpy as np
import pandas as pd

PROJ_DIRECTORY = Path.cwd().parent
DATA_DIRECTORY = PROJ_DIRECTORY / "data"
DEV_DIRECTORY = PROJ_DIRECTORY / "data_dev"
IMAGES_DIRECTORY = DEV_DIRECTORY / "images"

Let's take a look at two of the metadata files.

In [2]:
train_meta = pd.read_csv(DEV_DIRECTORY / "train_metadata.csv", index_col="image_id")
train_labels = pd.read_csv(DEV_DIRECTORY / "train_labels.csv", index_col="image_id")

The train_metadata.csv contains information about the type of spacecraft and background used to generate the image. You should consider ways to use this information effectively in stratifying your train and test splits. The test data that you will be evaluated on includes spacecraft types that are not included in the training set, so you will want to consider strategies for ensuring your model generalizes well. You may also want to consider ways of generating additional training data, e.g. to generate a more diverse variety of background imagery.

The train_metadata.csv contains information about all images in the training set. But since we haven't downloaded all the images yet, let's filter the dataset down to just the images we've saved locally.

In [3]:
# we didn't download the full training set, so add a column indicating which images are saved locally
train_meta["exists"] = train_meta.index.to_series().map(lambda x: (IMAGES_DIRECTORY / f"{x}.png").exists())
# filter our metadata down to only the images we have locally
train_meta = train_meta[train_meta.exists]

train_meta.head()
Out[3]:
spacecraft_id background_id exists
image_id
0001954c9f4a58f7ac05358b3cda8d20 24 247 True
00054819240f9d46378288b215dbcd3a 14 10 True
000dbf763348037b46558bbcb6a032ac 19 17 True
000e79208bebd8e84ce6c22fd8612a0d 14 15 True
000f13aff94499d03e3997afc55b0aa0 28 15 True

The train_labels.csv contains the bounding box information for the target spacecraft in each image.

In [4]:
train_labels.head()
Out[4]:
xmin ymin xmax ymax
image_id
0001954c9f4a58f7ac05358b3cda8d20 0 277 345 709
00054819240f9d46378288b215dbcd3a 753 602 932 725
000dbf763348037b46558bbcb6a032ac 160 434 203 481
000e79208bebd8e84ce6c22fd8612a0d 70 534 211 586
000f13aff94499d03e3997afc55b0aa0 103 0 312 193

Let's look at a few example images to get a feel for what's in this dataset.

In [5]:
import cv2
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle

def get_bbox(image_id, labels):
    """Get bbox coordinates as list from dataframe for given image id."""
    return labels.loc[image_id].loc[["xmin", "ymin", "xmax", "ymax"]].values.tolist()

def display_image(image_id, images_dir=IMAGES_DIRECTORY, show_bbox=False, labels=train_labels):
    """Display image given image ID. Annotate with bounding box if `show_bbox` is True."""
    img = cv2.imread(str(images_dir / f"{image_id}.png"))
    fig, ax = plt.subplots()
    # cv2 reads images as BGR order; we should flip them to RGB for matplotlib
    # ref: https://stackoverflow.com/questions/54959387/rgb-image-display-in-matplotlib-plt-imshow-returns-a-blue-image
    ax.imshow(np.flip(img, axis=-1))

    if show_bbox:
        xmin, ymin, xmax, ymax = get_bbox(image_id, labels)
        patch = Rectangle((xmin, ymin), xmax - xmin, ymax - ymin, fill=False, edgecolor='white', linewidth=1)
        ax.add_patch(patch)

Some images are relatively straightforward with an easily identifiable spacecraft. Below is one example. Note that the spacecraft is not necessarily fully in view. Also note that the background in this case consists entirely of our planet, with no view of outer space, while in other images it may be entirely space, or more commonly a mix of planet and space.

In [6]:
display_image("07dd7b5a0b7b224abc0a7fea8e78de76", show_bbox=True)

There are many examples where the spacecraft will be more difficult to detect.

One challenging type of image involves target spacecraft that appear very small, due to their distance from the chaser spacecraft. Here is one example of that situation.

In [7]:
display_image("0d4b4eda4bf7c0251399047d71cc4188", show_bbox=True)

Lens flare and other visual artifacts from the refraction of light on the camera lens may also complicate your detection algorithm, as in the example below.

In [8]:
display_image("01fcd95cdcf8bb84ec4dfa7b87bf2abc", show_bbox=True)