\documentclass[a4paper]{article}

\usepackage{url,graphicx}

\graphicspath{{Figures/}}

\begin{document}

\title{The PASCAL Visual Object Classes Challenge 2009 (VOC2009) Development Kit}
\author{Mark Everingham \and John Winn}
\maketitle

\tableofcontents
\newpage

\section{Challenge}

The goal of this challenge is to recognize objects from a number
of visual object classes in realistic scenes (i.e.\ not
pre-segmented objects). There are twenty object classes:
\begin{itemize}
    \item person
    \item bird, cat, cow, dog, horse, sheep
    \item aeroplane, bicycle, boat, bus, car, motorbike, train
    \item bottle, chair, dining table, potted plant, sofa,
    tv/monitor
\end{itemize}
There are three main tasks:
\begin{itemize}
    \item Classification: For each of the classes predict the presence/absence of
at least one object of that class in a test image.
    \item Detection: For each of the classes predict the bounding boxes of each
object of that class in a test image (if any).
    \item Segmentation: For each pixel in a test image, predict the class of the
    object containing that pixel or `background' if the pixel
    does not belong to one of the twenty specified classes.
\end{itemize}
In addition, there is a single ``taster'' task:
\begin{itemize}
    \item Person Layout: For each `person' object in a test image
    (indicated by a bounding box of the person), predict the presence/absence
    of parts (head/hands/feet), and the bounding boxes of those
    parts.
\end{itemize}

\section{Data}

The VOC2009 database contains a total of 14,743 annotated images.
The data is released in two phases: (i)~training and validation
data with annotation is released with this development kit;
(ii)~test data \emph{without} annotation is released at a later
date.

\subsection{Classification/Detection Image Sets}

For the classification and detection tasks there are four sets of
images provided:
\begin{enumerate}
    \item[\texttt{train}: ] Training data
    \item[\texttt{val}: ] Validation data (suggested). The
    validation data may be used as additional training data (see
    below).
    \item[\texttt{trainval}: ] The union of \texttt{train} and
    \texttt{val}.
    \item[\texttt{test}: ] Test data. The test set is not provided
    in the development kit. It will be released in good time
    before the deadline for submission of results.
\end{enumerate}

Table~\ref{tbl:stats} summarizes the number of objects and images
(containing at least one object of a given class) for each class
and image set. The data has been split into 50\% for
training/validation and 50\% for testing. The distributions of
images and objects by class are approximately equal across the
training/validation and test sets. To increase the amount of data,
the dataset includes images from the 2008 dataset, indicated by
the `2008' prefix. The assignment of images to training/test sets
follows the 2008 assignments i.e.\ the 2008 training/test sets are
a subset of the corresponding 2009 sets. Note that no annotation
for the 2008 test set has been released.

\begin{table}
    \begin{center}
        \caption{Statistics of the main image sets. Object statistics
        list only the `non-difficult' objects used in the evaluation.}
        \label{tbl:stats}
        \input{tbl_main.tex}
    \end{center}
\end{table}

\subsection{Segmentation Image Sets}

For the segmentation task, corresponding image sets are provided
as in the classification/detection tasks.  To increase the amount
of data, the training and validation image sets include images
from the 2007/2008 segmentation tasters, indicated by the
`2007/2008' prefix. The test set contains only 2008/2009 images
(i.e.\ those for which no annotation has been released), and is a
subset of the test set for the main tasks for which pixel-wise
segmentations have been prepared. Table~\ref{tbl:segstats}
summarizes the number of objects and images (containing at least
one object of a given class) for each class and image set, for the
combined 2007--9 data. In addition to the segmented images for
training and validation, participants are free to use the
un-segmented training/validation images supplied for the main
classification/detection tasks, and any annotation provided for
the main challenge e.g.\ bounding boxes.

\begin{table}
    \begin{center}
        \caption{Statistics of the segmentation image sets.}
        \label{tbl:segstats}
        \input{tbl_seg.tex}
    \end{center}
\end{table}

\subsection{Person Layout Taster Image Sets}

For the person layout taster task, corresponding image sets are
provided as in the classification/detection tasks. A person is
indicated by a bounding box, and each person has been annotated
with part layout (head, hands, feet). As in the segmentation task,
the training and validation image sets include images from the
2007/2008 person layout tasters, indicated by the `2007/2008'
prefix. The test set contains only 2008/2009 images (i.e.\ those
for which no annotation has been released), and is disjoint from
the test set for the main tasks. Table~\ref{tbl:laystats}
summarizes the number of `person' objects annotated with layout
for each image set.

\begin{table}
    \begin{center}
        \caption{Statistics of the person layout taster image sets. Object
        statistics list only the `person' objects for which layout
        information (parts) is present.}
        \label{tbl:laystats}
        \input{tbl_layout.tex}
    \end{center}
\end{table}

\subsection{Ground Truth Annotation}
\label{sec:gt}

Objects of the twenty classes listed above are annotated in the
ground truth. For each object, the following annotation is
present:
\begin{itemize}
    \item \textbf{class}: the object class e.g.\ `car' or `bicycle'
    \item \textbf{bounding box}: an axis-aligned rectangle specifying the
    extent of the object visible in the image.
    \item \textbf{view}: `frontal', `rear', `left' or `right'. The
    views are subjectively marked to indicate the view of the
    `bulk' of the object. Some objects have no view specified.
    \item \textbf{`truncated'}: an object marked as `truncated' indicates that
    the bounding box specified for the object does not correspond
    to the full extent of the object e.g.\ an image of a person
    from the waist up, or a view of a car extending outside the
    image.
    \item \textbf{`occluded'}: an object marked as `occluded'
    indicates that a significant portion of the object within the
    bounding box is occluded by another object.
    \item \textbf{`difficult'}: an object marked as `difficult'
    indicates that the object is considered difficult to recognize,
    for example an object which is clearly visible but unidentifiable
    without substantial use of context. Objects marked as difficult
    are currently \emph{ignored} in the evaluation of the challenge.
\end{itemize}
In preparing the ground truth, annotators were given a detailed
list of guidelines on how to complete the annotation. These are
available on the main challenge web-site~\cite{voc}.

\subsection{Segmentation Ground Truth}
\begin{figure}[h!]
    \begin{center}
        \begin{tabular}{ccc}
            \includegraphics[height=5cm]{inpseg.eps} &
            \includegraphics[height=5cm]{clsseg.eps} &
            \includegraphics[height=5cm]{objseg.eps} \\
            a. & b. & c.
        \end{tabular}
        \caption{Example of segmentation ground truth. \textbf{a.}
        Training image \textbf{b.} Class segmentation showing background,
        car, horse and person labels.  The cream-colored `void' label is
        also used in border regions and to mask difficult objects.
        \textbf{c.} Object segmentation where individual object instances
        are separately labelled.}
        \label{fig:segfig}
    \end{center}
\end{figure}

For the segmentation image sets, each image has two corresponding
types of ground truth segmentation provided:
\begin{itemize}
    \item class segmentation: each pixel
is labelled with the ground truth class or background.
    \item object segmentation: each pixel is labelled with an object
number (from which the class can be obtained) or background.
\end{itemize}
Figure~\ref{fig:segfig} gives an example of these two types of
segmentation for one of the training set images. The ground truth
segmentations are provided to a high degree of accuracy, but are not
pixel accurate, as this would have greatly extended the time
required to gather these segmentations. Instead, they were labelled
so that a bordering region with a width of five pixels may contain
either object or background. Bordering regions are marked with a
`void' label (index 255), indicating that the contained pixels can
be any class including background. The void label is also used to
mask out ambiguous, difficult or heavily occluded objects and also
to label regions of the image containing objects too small to be
marked, such as crowds of people.  All void pixels are ignored when
computing segmentation accuracies and should be treated as
unlabelled pixels during training.

In addition to the ground truth segmentations given, participants
are free to use any of the ground truth annotation for the
classification/detection tasks e.g.\ bounding boxes.

\subsection{Person Layout Taster Ground Truth}

For the person layout taster task, `person' objects are
additionally annotated with three `parts':
\begin{itemize}
    \item head -- zero or one per person
    \item hand -- zero, one, or two per person
    \item foot -- zero, one, or two per person
\end{itemize}
For each annotated person, the presence or absence of each part is
listed, and for each part present, the bounding box is specified.
The test images for the person layout taster are disjoint from the
main image sets. There are no `difficult' objects.

\section{Classification Task}

\subsection{Task}

For each of the twenty object classes predict the presence/absence
of at least one object of that class in a test image. The output
from your system should be a real-valued confidence of the
object's presence so that a precision/recall curve can be drawn.
Participants may choose to tackle all, or any subset of object
classes, for example ``cars only'' or ``motorbikes and cars''.

\subsection{Competitions}

Two competitions are defined according to the choice of training
data: (i)~taken from the VOC
\texttt{trainval} data provided, or (ii)~from any source excluding
the VOC
\texttt{test} data provided:
\begin{center}
    \begin{tabular}{|c|c|c|c|}
    \hline
    No. & Task & Training data & Test data \\
    \hline
    1 & Classification & \texttt{trainval} & \texttt{test} \\
    2 & Classification & \textbf{any but} VOC \texttt{test} & \texttt{test} \\
    \hline
    \end{tabular}
\end{center}

In competition 1, any annotation provided in the VOC
\texttt{train} and
\texttt{val} sets may be used for training, for example bounding
boxes or particular views e.g.\ `frontal' or `left'. Participants
are \emph{not} permitted to perform additional manual annotation
of either training or test data.

In competition 2, any source of training data may be used
\emph{except} the provided \texttt{test} images. Researchers who have pre-built
systems trained on other data are particularly encouraged to
participate. The test data includes images from ``flickr''
(\texttt{www.flickr.com}); this source of images may
\emph{not} be used for training. Participants who have acquired
images from flickr for training must submit them to the organizers
to check for overlap with the test set.

\subsection{Submission of Results}

A separate text file of results should be generated for each
competition (1 or 2) and each class e.g.\ `car'. Each line should
contain a single identifier and the confidence output by the
classifier, separated by a space, for example:
\begin{verbatim}
comp1_cls_test_car.txt:
    ...
    2009_000001 0.056313
    2009_000002 0.127031
    2009_000009 0.287153
    ...
\end{verbatim}
Greater confidence values signify greater confidence that the
image contains an object of the class of interest. The example
classifier implementation (section~\ref{sec:examplecls}) includes
code for generating a results file in the required format.

\subsection{Evaluation}
\label{sec:clseval}

The classification task will be judged by the precision/recall
curve. The principal quantitative measure used will be the average
precision (AP). Example code for computing the precision/recall
and AP measure is provided in the development kit.

Images which contain only objects marked as `difficult'
(section~\ref{sec:gt}) are currently \emph{ignored} by the
evaluation. The final evaluation may include separate results
including such ``difficult'' images, depending on the submitted
results.

Participants are expected to submit a \emph{single} set of results
per method employed. Participants who have investigated several
algorithms may submit one result per method. Changes in algorithm
parameters do \emph{not} constitute a different method -- all
parameter tuning must be conducted using the training and
validation data alone.

\section{Detection Task}

\subsection{Task}

For each of the twenty classes predict the bounding boxes of each
object of that class in a test image (if any). Each bounding box
should be output with an associated real-valued confidence of the
detection so that a precision/recall curve can be drawn.
Participants may choose to tackle all, or any subset of object
classes, for example ``cars only'' or ``motorbikes and cars''.

\subsection{Competitions}

Two competitions are defined according to the choice of training
data: (i)~taken from the VOC
\texttt{trainval} data provided, or (ii)~from any source excluding
the VOC
\texttt{test} data provided:
\begin{center}
    \begin{tabular}{|c|c|c|c|}
    \hline
    No. & Task & Training data & Test data \\
    \hline
    3 & Detection & \texttt{trainval} & \texttt{test} \\
    4 & Detection & \textbf{any but} VOC \texttt{test} & \texttt{test} \\
    \hline
    \end{tabular}
\end{center}

In competition 3, any annotation provided in the VOC
\texttt{train} and
\texttt{val} sets may be used for training, for example bounding
boxes or particular views e.g.\ `frontal' or `left'. Participants
are \emph{not} permitted to perform additional manual annotation
of either training or test data.

In competition 4, any source of training data may be used
\emph{except} the provided \texttt{test} images. Researchers who have pre-built
systems trained on other data are particularly encouraged to
participate. The test data includes images from ``flickr''
(\texttt{www.flickr.com}); this source of images may
\emph{not} be used for training. Participants who have acquired
images from flickr for training must submit them to the organizers
to check for overlap with the test set.

\subsection{Submission of Results}

A separate text file of results should be generated for each
competition (3 or 4) and each class e.g.\ `car'. Each line should
be a detection output by the detector in the following format:
\begin{verbatim}
    <image identifier> <confidence> <left> <top> <right> <bottom>
\end{verbatim}
where \texttt{(left,top)-(right,bottom)} defines the bounding box
of the detected object. The top-left pixel in the image has
coordinates $(1,1)$. Greater confidence values signify greater
confidence that the detection is correct. An example file excerpt
is shown below. Note that for the image \texttt{2009\_000032},
multiple objects are detected:
\begin{verbatim}
comp3_det_test_car.txt:
    ...
    2009_000026 0.949297 172.000000 233.000000 191.000000 248.000000
    2009_000032 0.013737 1.000000 147.000000 114.000000 242.000000
    2009_000032 0.013737 1.000000 134.000000 94.000000 168.000000
    2009_000035 0.063948 455.000000 229.000000 491.000000 243.000000
    ...
\end{verbatim}
The example detector implementation (section~\ref{sec:exampledet})
includes code for generating a results file in the required
format.

\subsection{Evaluation}

The detection task will be judged by the precision/recall curve.
The principal quantitative measure used will be the average
precision (AP). Example code for computing the precision/recall
and AP measure is provided in the development kit.

Detections are considered true or false positives based on the
area of overlap with ground truth bounding boxes. To be considered
a correct detection, the area of overlap $a_o$ between the
predicted bounding box $B_p$ and ground truth bounding box
$B_{gt}$ must exceed $50\%$ by the formula:
\begin{equation}
    a_o = \frac{area(B_p \cap B_{gt})}{area(B_p \cup B_{gt})}
\end{equation}
Example code for computing this overlap measure is provided in the
development kit. Multiple detections of the \emph{same} object in
an image are considered \emph{false} detections e.g.\ 5 detections
of a single object is counted as 1 correct detection and 4 false
detections -- it is the responsibility of the participant's system
to filter multiple detections from its output.

Objects marked as `difficult' (section~\ref{sec:gt}) are currently
\emph{ignored} by the evaluation. The final evaluation may include
separate results including such ``difficult'' images, depending on
the submitted results.

Participants are expected to submit a \emph{single} set of results
per method employed. Participants who have investigated several
algorithms may submit one result per method. Changes in algorithm
parameters do \emph{not} constitute a different method -- all
parameter tuning must be conducted using the training and
validation data alone.

\section{Segmentation Task}

\subsection{Task}

For each test image pixel, predict the class of the object
containing that pixel or 'background' if the pixel does not belong
to one of the twenty specified classes. The output from your
system should be an indexed image with each pixel index indicating
the number of the inferred class (1-20) or zero, indicating
background.

\subsection{Competitions}

A single competition is defined:
\begin{center}
    \begin{tabular}{|c|c|c|c|}
    \hline
    No. & Task & Training data & Test data \\
    \hline
    5 & Segmentation & \texttt{trainval} & \texttt{test} \\
    \hline
    \end{tabular}
\end{center}

Any annotation provided in the VOC
\texttt{train} and
\texttt{val} sets may be used for training, for example segmentation, bounding
boxes or particular views e.g.\ `frontal' or `left'. Both the
images with and without segmentation provided may be used if
desired, although the training data used must be reported as part
of the submission (see below). Participants are \emph{not}
permitted to perform additional manual annotation of either
training or test data.

\subsection{Submission of Results}

Submission of results should be as collections of PNG format
indexed image files, one per test image, with pixel indices from 0
to 20. The PNG color map should be the same as the color map used
in the provided training and validation annotation (MATLAB users
can use
\texttt{VOClabelcolormap} -- see section~\ref{sec:voclabelcolormap}).
The example segmenter implementation
(section~\ref{sec:exampleseg}) includes code for generating
results in the required format.  Participants may choose to include segmentations for only a subset of
the 20 classes in which case they will be evaluated on only the included classes.

Along with the submitted image files, participants \textbf{must} also
state whether their method used segmentation training data only or
both segmentation and bounding box training data.  This information will be used
when analysing and presenting the competition results.

\subsection{Evaluation}

The segmentation task will be judged by average segmentation
accuracy across the twenty classes and the background class. The
segmentation accuracy for a class will be assessed using the
intersection/union metric, defined as the number of correctly
labelled pixels of that class, divided by the number of pixels
labelled with that class in either the ground truth labelling or
the inferred labelling.  Equivalently, the accuracy is given by
the equation,
\begin{eqnarray}\label{eq:unionIntersection}
    \mathrm{segmentation~accuracy} = \frac{\mathrm{true~positives}}{\mathrm{true~positives} + \mathrm{false~positives} +
    \mathrm{false~negatives}} \nonumber
\end{eqnarray}


Code is provided to compute segmentation accuracies for each class,
and the overall average accuracy (see section~\ref{sec:VOCconfusion}).

Participants are expected to submit a \emph{single} set of results
per method employed. Participants who have investigated several
algorithms may submit one result per method. Changes in algorithm
parameters do \emph{not} constitute a different method -- all
parameter tuning must be conducted using the training and
validation data alone.

\section{Person Layout Taster}

\subsection{Task}

For each `person' object in a test image (their bounding box is
provided) predict the presence/absence of parts (head/hands/feet),
and the bounding boxes of those parts. Each prediction for a
person layout should be output with an associated real-valued
confidence of the layout so that a precision/recall curve can be
drawn. Multiple estimates of layout may be output for each person,
but estimates other than the first correct are treated as
false positives as in the detection task.

As noted, the bounding box of the person is provided. To be
considered a correct estimate of the layout, two conditions must
be satisfied: i)~correct prediction of parts present/absent;
ii)~correct prediction of bounding boxes for all parts.

\subsection{Competitions}

Two competitions are defined according to the choice of training
data: (i)~taken from the VOC
\texttt{trainval} data provided, or (ii)~from any source excluding
the VOC
\texttt{test} data provided:
\begin{center}
    \begin{tabular}{|c|c|c|c|}
    \hline
    No. & Task & Training data & Test data \\
    \hline
    6 & Layout & \texttt{trainval} & \texttt{test} \\
    7 & Layout & \textbf{any but} VOC \texttt{test} & \texttt{test} \\
    \hline
    \end{tabular}
\end{center}

In competition 6, any annotation provided in the VOC
\texttt{train} and
\texttt{val} sets may be used for training, for example bounding
boxes or particular views e.g.\ `frontal' or `left'. Participants
are \emph{not} permitted to perform additional manual annotation
of either training or test data.

In competition 7, any source of training data may be used
\emph{except} the provided \texttt{test} images. Researchers who have pre-built
systems trained on other data are particularly encouraged to
participate. The test data includes images from ``flickr''
(\texttt{www.flickr.com}); this source of images may
\emph{not} be used for training. Participants who have acquired
images from flickr for training must submit them to the organizers
to check for overlap with the test set.

\subsection{Submission of Results}

To support the hierarchical (person+parts) nature of this task, an
XML format has been adopted for submission of results. A separate
XML file of results should be generated for each competition (6 or
7). The overall format should follow:
\begin{verbatim}
<results>
    <layout>
        ... layout estimate 1 ...
    </layout>
    <layout>
        ... layout estimate 2 ...
    </layout>
</results>
\end{verbatim}
Each detection is represented by a \texttt{<layout>} element. The
order of detections is not important. An example detection is
shown here:
\begin{verbatim}
    <layout>
        <image>2009_000183</image>
        <object>1</object>
        <confidence>-1189</confidence>
        <part>
            <class>head</class>
            <bndbox>
                <xmin>191</xmin>
                <ymin>25</ymin>
                <xmax>323</xmax>
                <ymax>209</ymax>
            </bndbox>
        </part>
        <part>
            <class>hand</class>
            <bndbox>
                <xmin>393</xmin>
                <ymin>206</ymin>
                <xmax>488</xmax>
                <ymax>300</ymax>
            </bndbox>
        </part>
        <part>
            <class>hand</class>
            <bndbox>
                <xmin>1</xmin>
                <ymin>148</ymin>
                <xmax>132</xmax>
                <ymax>329</ymax>
            </bndbox>
        </part>
    </layout>
\end{verbatim}
The \texttt{<image>} element specifies the image identifier. The
\texttt{<object>} specifies the index of the object to which the layout relates
(the first object in the image has index 1) and should match that
provided in the image set files (section~\ref{sec:perlayimgset}).
The
\texttt{<confidence>} element specifies the confidence of the
layout estimate, used to generate a precision/recall curve as in
the detection task.

Each \texttt{<part>} element specifies the detection of a
particular part of the person i.e.\ head/hand/foot. If the part is
predicted to be absent/invisible, the corresponding element should
be omitted. For each part, the \texttt{<class>} element specifies
the type of part: \texttt{head}, \texttt{hand} or \texttt{foot}.
The \texttt{<bndbox>} element specifies the predicted bounding box
for that part; bounding boxes are specified in image co-ordinates
and need not be contained in the provided person bounding box.

To ease creation of the required XML results file for MATLAB
users, a function is included in the development kit to convert
MATLAB structures to XML. See the \texttt{VOCwritexml} function
(section~\ref{sec:vocwritexml}). The example person layout
implementation (section~\ref{sec:examplelay}) includes code for
generating a results file in the required format.

\subsection{Evaluation}

The person layout task will be judged by the precision/recall
curve. The principal quantitative measure used will be the average
precision (AP). Example code for computing the precision/recall
and AP measure is provided in the development kit.

To be considered a true positive, each layout estimate must
satisfy two criteria:
\begin{itemize}
    \item set and number of predicted parts
    matches ground truth exactly e.g. \{head, hand, hand\} or
    \{head, hand, foot\}
    \item predicted bounding box of each part overlaps ground
    truth by at least 50\%
\end{itemize}
The overlap between bounding boxes is computed as in the detection
task. Note that in the case of multiple parts of the same type
e.g.\ two hands, it is \emph{not} necessary to predict which part
is which.

\section{Development Kit}
The development kit is packaged in a single gzipped tar file
containing MATLAB code and (this) documentation. The images,
annotation, and lists specifying training/validation sets for the
challenge are provided in a separate archive which can be obtained
via the VOC web pages~\cite{voc}.

\subsection{Installation and Configuration}

The simplest installation is achieved by placing the development
kit and challenge databases in a single location. After untarring
the development kit, download the challenge image database and
untar into the same directory, resulting in the following
directory structure:
\begin{verbatim}
    VOCdevkit/                           % development kit
    VOCdevkit/VOCcode/                   % VOC utility code
    VOCdevkit/results/VOC2009/           % your results on VOC2009
    VOCdevkit/local/                     % example code temp dirs
    VOCdevkit/VOC2009/ImageSets          % image sets
    VOCdevkit/VOC2009/Annotations        % annotation files
    VOCdevkit/VOC2009/JPEGImages         % images
    VOCdevkit/VOC2009/SegmentationObject % segmentations by object
    VOCdevkit/VOC2009/SegmentationClass  % segmentations by class
\end{verbatim}

If you set the current directory in MATLAB to the
\texttt{VOCdevkit} directory you should be able to run the
example functions:
\begin{itemize}
    \item \texttt{example\_classifier}
    \item \texttt{example\_detector}
    \item \texttt{example\_segmenter}
    \item \texttt{example\_layout}
\end{itemize}

If desired, you can store the code, images/annotation, and results
in separate directories, for example you might want to store the
image data in a common group location. To specify the locations of
the image/annotation, results, and working directories, edit the
\texttt{VOCinit.m} file, e.g.
\begin{verbatim}
% change this path to point to your copy of the PASCAL VOC data
VOCopts.datadir='/homes/group/VOCdata/';

% change this path to a writable directory for your results
VOCopts.resdir='/homes/me/VOCresults/';

% change this path to a writable local directory for the example code
VOCopts.localdir='/tmp/';
\end{verbatim}

Note that in developing your own code you need to include the
\texttt{VOCdevkit/VOCcode} directory in your MATLAB path, e.g.
\begin{verbatim}
>> addpath /homes/me/code/VOCdevkit/VOCcode
\end{verbatim}

\subsection{Example Code}

Example implementations are provided for all tasks. The aim of
these (minimal) implementations is solely to demonstrate use of
the code in the development kit.

\subsubsection{Example Classifier Implementation}
\label{sec:examplecls}

The file \texttt{example\_classifier.m} contains a complete
implementation of the classification task. For each VOC object
class a simple classifier is trained on the
\texttt{train} set; the classifier is then applied to the
\texttt{val} set and the output saved to a results file in the format
required by the challenge; a precision/recall curve is plotted and
the `average precision' (AP) measure displayed.

\subsubsection{Example Detector Implementation}
\label{sec:exampledet}

The file \texttt{example\_detector.m} contains a complete
implementation of the detection task. For each VOC object class a
simple (and not very successful!) detector is trained on the
\texttt{train} set; the detector is then applied to the \texttt{val} set
and the output saved to a results file in the format required by
the challenge; a precision/recall curve is plotted and the
`average precision' (AP) measure displayed.

\subsubsection{Example Segmenter Implementation}
\label{sec:exampleseg}

An example segmenter is provided which converts detection results
into segmentation results, using
\texttt{create\_segmentations\_from\_detections} (described below). For
example:
\begin{verbatim}
>> example_detector;
>> example_segmenter;
\end{verbatim}
This runs the example detector, converts the detections into
segmentations and displays a table of per-class segmentation
accuracies, along with an overall average accuracy.

\subsubsection{Example Layout Implementation}
\label{sec:examplelay}

The file \texttt{example\_layout.m} contains a complete
implementation of the person layout task. For each
specified person
a simple (and not very successful!) layout
predictor is trained on the \texttt{train} set; the layout predictor is
then applied to the
\texttt{val} set and the output saved to a results file in the
format required by the challenge; a precision/recall curve is
plotted and the `average precision' (AP) measure displayed.

\subsection{Non-MATLAB Users}

For non-MATLAB users, the file formats used for the VOC2009 data
should be straightforward to use in other environments. Image sets
(see below) are vanilla text files. Annotation files are XML
format and should be readable by any standard XML parser. Images
are stored in JPEG format, and segmentation ground truth in PNG
format.

\section{Using the Development Kit}

The development kit provides functions for loading annotation
data. Example code for computing precision/recall curves and
segmentation accuracy, and for viewing annotation is also
provided.

\subsection{Image Sets}

\subsubsection{Classification/Detection Task Image Sets}

The \texttt{VOC2009/ImageSets/Main/} directory contains text files
specifying lists of images for the main classification/detection
tasks.

The files \texttt{train.txt}, \texttt{val.txt},
\texttt{trainval.txt} and \texttt{test.txt} list the image identifiers for the
corresponding image sets (training, validation,
training+validation and testing). Each line of the file contains a
single image identifier. The following MATLAB code reads the image
list into a cell array of strings:
\begin{verbatim}
imgset='train';
ids=textread(sprintf(VOCopts.imgsetpath,imgset),'%s');
\end{verbatim}
For a given image identifier \texttt{ids\{i\}}, the corresponding
image and annotation file paths can be produced thus:
\begin{verbatim}
imgpath=sprintf(VOCopts.imgpath,ids{i});
annopath=sprintf(VOCopts.annopath,ids{i});
\end{verbatim}
Note that the image sets used are the same for all classes. For
each competition, participants are expected to provide output for
all images in the \texttt{test} set.

\subsubsection{Classification Task Image Sets}

To simplify matters for participants tackling only the
\emph{classification} task, class-specific image sets with
per-image ground truth are also provided. The file
\texttt{VOC2009/ImageSets/Main/<class>\_<imgset>.txt} contains image identifiers and
ground truth for a particular class and image set, for example the
file \texttt{car\_train.txt} applies to the `car' class and
\texttt{train} image set.

Each line of the file contains a single image identifier and
ground truth label, separated by a space, for example:
\begin{verbatim}
    ...
    2009_000040 -1
    2009_000042 -1
    2009_000052  1
    ...
\end{verbatim}

The following MATLAB code reads the image list into a cell array
of strings and the ground truth label into a corresponding vector:
\begin{verbatim}
imgset='train';
cls='car';
[ids,gt]=textread(sprintf(VOCopts.clsimgsetpath, ...
                    cls,imgset),'%s %d');
\end{verbatim}

There are \emph{three} ground truth labels:
\begin{itemize}
    \item[\texttt{-1:}] Negative: The image contains no objects of the class
    of interest. A classifier should give a `negative' output.
    \item[\texttt{ 1:}] Positive: The image contains at least one
    object of the class of interest. A classifier should give a `positive' output.
    \item[\texttt{ 0:}] ``Difficult'': The image contains only
    objects of the class of interest marked as `difficult'.
\end{itemize}

The ``difficult'' label indicates that all objects of the class of
interest have been annotated as ``difficult'', for example an
object which is clearly visible but difficult to recognize without
substantial use of context. Currently the evaluation ignores such
images, contributing nothing to the precision/recall curve or AP
measure. The final evaluation may include separate results
including such ``difficult'' images, depending on the submitted
results. Participants are free to omit these images from training
or include as either positive or negative examples.

\subsubsection{Segmentation Image Sets}

The \texttt{VOC2009/ImageSets/Segmentation/} directory contains
text files specifying lists of images for the segmentation task.

The files \texttt{train.txt}, \texttt{val.txt},
\texttt{trainval.txt} and \texttt{test.txt} list the image identifiers for the
corresponding image sets (training, validation,
training+validation and testing). Each line of the file contains a
single image identifier. The following MATLAB code reads the image
list into a cell array of strings:
\begin{verbatim}
imgset='train';
ids=textread(sprintf(VOCopts.seg.imgsetpath,imgset),'%s');
\end{verbatim}
For a given image identifier \texttt{ids\{i\}}, file paths for the
corresponding image, annotation, segmentation by object instance
and segmentation by class can be produced thus:
\begin{verbatim}
imgpath=sprintf(VOCopts.imgpath,ids{i});
annopath=sprintf(VOCopts.annopath,ids{i});
clssegpath=sprintf(VOCopts.seg.clsimgpath,ids{i});
objsegpath=sprintf(VOCopts.seg.instimgpath,ids{i});
\end{verbatim}
Participants are expected to provide output for all images in the
\texttt{test} set.

\subsubsection{Person Layout Taster Image Sets}
\label{sec:perlayimgset}

The \texttt{VOC2009/ImageSets/Layout/} directory contains text
files specifying lists of image for the person layout taster task.

The files \texttt{train.txt}, \texttt{val.txt},
\texttt{trainval.txt} and \texttt{test.txt} list the image identifiers for the
corresponding image sets (training, validation,
training+validation and testing). Each line of the file contains a
single image identifier, and a single object index. Together these
specify a `person' object for which layout is provided or to be
estimated, for example:

\begin{verbatim}
    ...
    2009_000595  1
    2009_000595  2
    2009_000606  1
    ...
\end{verbatim}

The following MATLAB code reads the image list into a cell array
of strings and the object indices into a corresponding vector:

\begin{verbatim}
imgset='train';
[imgids,objids]=textread(sprintf(VOCopts.layout.imgsetpath, ...
                            VOCopts.trainset),'%s %d');
\end{verbatim}

The annotation for the object (bounding box only in the test data)
can then be obtained using the image identifier and object index:

\begin{verbatim}
    rec=PASreadrecord(sprintf(VOCopts.annopath,imgids{i}));
    obj=rec.objects(objids{i});
\end{verbatim}

\subsection{Development Kit Functions}

\subsubsection{\texttt{VOCinit}}

The \texttt{VOCinit} script initializes a single structure
\texttt{VOCopts} which contains options for the PASCAL functions
including directories containing the VOC data and options for the
evaluation functions (not to be modified).

The field \texttt{classes} lists the object classes for the
challenge in a cell array:
\begin{verbatim}
VOCopts.classes={'aeroplane','bicycle','bird','boat',...
                 'bottle','bus','car','cat',...
                 'chair','cow','diningtable','dog',...
                 'horse','motorbike','person','pottedplant',...
                 'sheep','sofa','train','tvmonitor'};
\end{verbatim}

The field \texttt{trainset} specifies the image set used by the
example evaluation functions for training:
\begin{verbatim}
VOCopts.trainset='train'; % use train for development
\end{verbatim}

Note that participants are free to use both training and
validation data in any manner they choose for the final challenge.

The field \texttt{testset} specifies the image set used by the
example evaluation functions for testing:
\begin{verbatim}
VOCopts.testset='val'; % use validation data for development
\end{verbatim}

Other fields provide, for convenience, paths for the image and
annotation data and results files. The use of these paths is
illustrated in the example implementations.

\subsubsection{\texttt{PASreadrecord(filename)}}

The \texttt{PASreadrecord} function reads the annotation data for
a particular image from the annotation file specified by
\texttt{filename}, for example:
\begin{verbatim}
>> rec=PASreadrecord(sprintf(VOCopts.annopath,'2009_000067'))

rec =

       folder: 'VOC2009'
     filename: '2009_000067.jpg'
       source: [1x1 struct]
         size: [1x1 struct]
    segmented: 0
      imgname: 'VOC2009/JPEGImages/2009_000067.jpg'
      imgsize: [500 334 3]
     database: 'The VOC2009 Database'
      objects: [1x6 struct]
\end{verbatim}

The \texttt{imgname} field specifies the path (relative to the
main VOC data path) of the corresponding image. The
\texttt{imgsize} field specifies the image dimensions as
\texttt{(width,height,depth)}. The \texttt{database} field
specifies the data source (VOC2009). The \texttt{segmented} field
specifies if a segmentation is available for this image. The
\texttt{folder} and \texttt{filename} fields provide an alternative
specification of the image path, and \texttt{size} an alternative
specification of the image size:
\begin{verbatim}
>> rec.size

ans =

     width: 500
    height: 334
     depth: 3
\end{verbatim}
The \texttt{source} field contains additional information about
the source of the image e.g.\ web-site and owner. This information
is obscured until completion of the challenge.

Objects annotated in the image are stored in the struct array
\texttt{objects}, for example:
\begin{verbatim}
>> rec.objects(2)

ans =

        class: 'person'
         view: 'Right'
    truncated: 0
     occluded: 0
    difficult: 0
        label: 'PASpersonRight'
     orglabel: 'PASpersonRight'
         bbox: [225 140 270 308]
       bndbox: [1x1 struct]
      polygon: []
         mask: []
     hasparts: 1
         part: [1x4 struct]
\end{verbatim}
The \texttt{class} field contains the object class. The
\texttt{view} field contains the view:
\texttt{Frontal}, \texttt{Rear}, \texttt{Left} (side view, facing
left of image), \texttt{Right} (side view, facing right of image),
or an empty string indicating another, or un-annotated view.

The \texttt{truncated} field being set to 1 indicates that the
object is ``truncated'' in the image. The definition of truncated
is that the bounding box of the object specified does not
correspond to the full extent of the object e.g.\ an image of a
person from the waist up, or a view of a car extending outside the
image. Participants are free to use or ignore this field as they
see fit.

The \texttt{occluded} field being set to 1 indicates that the
object is significantly occluded by another object. Participants
are free to use or ignore this field as they see fit.

The \texttt{difficult} field being set to 1 indicates that the
object has been annotated as ``difficult'', for example an object
which is clearly visible but difficult to recognize without
substantial use of context. Currently the evaluation ignores such
objects, contributing nothing to the precision/recall curve. The
final evaluation may include separate results including such
``difficult'' objects, depending on the submitted results.
Participants may include or exclude these objects from training as
they see fit.

The \texttt{bbox} field specifies the bounding box of the object
in the image, as \texttt{[left,top,right,bottom]}. The top-left
pixel in the image has coordinates $(1,1)$. The \texttt{bndbox}
field specifies the bounding box in an alternate form:
\begin{verbatim}
>> rec.objects(2).bndbox

ans =

    xmin: 225
    ymin: 140
    xmax: 270
    ymax: 308
\end{verbatim}

For backward compatibility, the \texttt{label} and
\texttt{orglabel} fields specify the PASCAL label for the
object, comprised of class, view and truncated/difficult flags.
The \texttt{polygon} and \texttt{mask} specify polygon/per-object
segmentations, and are not provided for the VOC2009 data.

The \texttt{hasparts} field specifies if the object has sub-object
``parts'' annotated. For the VOC2009 data, such annotation is
available for a subset of the `person' objects, used in the layout
taster task. Object parts are stored in the struct array
\texttt{part}, for example:
\begin{verbatim}
>> rec.objects(2).part(1)

ans =

        class: 'head'
         view: ''
    truncated: 0
     occluded: 0
    difficult: 0
        label: 'PAShead'
     orglabel: 'PAShead'
         bbox: [234 138 257 164]
       bndbox: [1x1 struct]
      polygon: []
         mask: []
     hasparts: 0
         part: []
\end{verbatim}
The format of object parts is identical to that for top-level
objects. For the `person' parts in the VOC2009 data, parts are not
annotated with view, or truncated/difficult flags. The bounding
box of a part is specified in image coordinates in the same way as
for top-level objects. Note that the object parts may legitimately
extend outside the bounding box of the parent object.

\subsubsection{\texttt{viewanno(imgset)}}

The \texttt{viewanno} function displays the annotation for images
in the image set specified by \texttt{imgset}. Some examples:
\begin{verbatim}
>> viewanno('Main/train');
>> viewanno('Main/car_val');
>> viewanno('Layout/train');
>> viewanno('Segmentation/val');
\end{verbatim}

\subsection{Classification Functions}

\subsubsection{\texttt{VOCevalcls(VOCopts,id,cls,draw)}}

The \texttt{VOCevalcls} function performs evaluation of the
classification task, computing a precision/recall curve and the
average precision (AP) measure. The arguments
\texttt{id} and
\texttt{cls} specify the results file to be loaded, for example:
\begin{verbatim}
>> [rec,prec,ap]=VOCevalcls(VOCopts,'comp1','car',true);
\end{verbatim}
See \texttt{example\_classifier} for further examples. If the
argument \texttt{draw} is true, the precision/recall curve is
drawn in a figure window. The function returns vectors of recall
and precision rates in \texttt{rec} and \texttt{prec}, and the
average precision measure in \texttt{ap}.

\subsection{Detection Functions}

\subsubsection{\texttt{VOCevaldet(VOCopts,id,cls,draw)}}

The \texttt{VOCevaldet} function performs evaluation of the
detection task, computing a precision/recall curve and the average
precision (AP) measure. The arguments
\texttt{id} and
\texttt{cls} specify the results file to be loaded, for example:
\begin{verbatim}
>> [rec,prec,ap]=VOCevaldet(VOCopts,'comp3','car',true);
\end{verbatim}
See \texttt{example\_detector} for further examples. If the
argument \texttt{draw} is true, the precision/recall curve is
drawn in a figure window. The function returns vectors of recall
and precision rates in \texttt{rec} and \texttt{prec}, and the
average precision measure in \texttt{ap}.

\subsubsection{\texttt{viewdet(id,cls,onlytp)}}

The \texttt{viewdet} function displays the detections stored in a
results file for the detection task. The arguments
\texttt{id} and \texttt{cls} specify the results file to be loaded, for example:
\begin{verbatim}
>> viewdet('comp3','car',true)
\end{verbatim}
If the \texttt{onlytp} argument is true, only the detections
considered true positives by the VOC evaluation measure are
displayed.

\subsection{Segmentation Functions}

\subsubsection{\texttt{create\_segmentations\_from\_detections(id,confidence)}}

This function creates segmentation results from detection results.

\texttt{create\_segmentations\_from\_detections(id)} creates segmentations
from the detection results with specified identifier e.g.\ comp3.
This is achieved by rendering the bounding box for each detection
in class order, so that later classes overwrite earlier classes
(e.g.\ a person bounding box will overwrite an overlapping an
aeroplane bounding box). All detections will be used, no matter
what their confidence level.

\texttt{create\_segmentations\_from\_detections(id,confidence)} does the same,
but only detections above the specified confidence will be used.

See \texttt{example\_segmenter} for an example.

\subsubsection{\texttt{VOCevalseg(VOCopts,id)}}
\label{sec:VOCconfusion}
The \texttt{VOCevalseg} function performs evaluation of the
segmentation task, computing a confusion matrix and segmentation
accuracies for the segmentation task. It returns per-class
percentage accuracies, the average overall percentage accuracy,
and a confusion matrix, for example:
\begin{verbatim}
>> [accuracies,avacc,conf,rawcounts] = VOCevalseg(VOCopts,'comp3')
\end{verbatim}
Accuracies are defined by the intersection/union measure. The optional fourth
output `rawcounts' returns an un-normalized confusion
matrix containing raw pixel counts. See \texttt{example\_segmenter} for
another example. This function will also display a table of overall
and per-class accuracies.

\subsubsection{\texttt{VOClabelcolormap}(N)}
\label{sec:voclabelcolormap}

The \texttt{VOClabelcolormap} function creates the color map which
has been used for all provided indexed images. You should use this
color map for writing your own indexed images, for consistency. The
size of the color map is given by N, which should generally be set
to 256 to include a color for the `void' label.

\subsection{Layout Functions}

\subsubsection{\texttt{VOCwritexml}(rec,path)}
\label{sec:vocwritexml}

The \texttt{VOCwritexml} function writes a MATLAB structure array
to a corresponding XML file. It is provided to support the
creation of XML results files for the person layout taster. An
example of usage can be found in \texttt{example\_layout}.

\subsubsection{\texttt{VOCevallayout(VOCopts,id,draw)}}

The \texttt{VOCevallayout} function performs evaluation of the
person layout task, computing a precision/recall curve and the
average precision (AP) measure. The arguments
\texttt{id} and
\texttt{cls} specify the results file to be loaded, for example:
\begin{verbatim}
>> [rec,prec,ap]=VOCevallayout(VOCopts,'comp6',true);
\end{verbatim}
See \texttt{example\_layout} for further examples. If the argument
\texttt{draw} is true, the precision/recall curve is drawn in a
figure window. The function returns vectors of recall and
precision rates in \texttt{rec} and \texttt{prec}, and the average
precision measure in \texttt{ap}.

\section*{Acknowledgements}

We gratefully acknowledge the following, who spent many long hours
providing annotation for the VOC2009 database: Jan Hendrik Becker,
Patrick Buehler, Kian Ming Adam Chai, Miha Drenik, Chris Engels,
Hedi Harzallah, Nicolas Heess, Sam Johnson, Markus Mathias,
Alastair Moore, Maria-Elena Nilsback, Patrick Ott, Florian
Schroff, Alexander Sorokin, Paul Sturgess, David Tingdahl. We also
like thank Andrea Vedaldi for additional assistance.

The preparation and running of this challenge is supported by the
EU-funded PASCAL2 Network of Excellence on Pattern Analysis,
Statistical Modelling and Computational Learning.

\bibliographystyle{plain}
\bibliography{local}

\end{document}
