visiongrader help


visiongrader is a scoring tool for detection algorithms. It is mainly used with face and pedestrians recognition, but nothing prevents you from using it with another type of data, provided you write a parser for the data.

Both the parser and the comparator can easily be changed, so it can adapt to quite a large range of situations. You can either score a single result or, if you got the confidence of the results, plot a ROC or a DET curve.


For the time being, visiongrader is a command-line program. You can run it bt typing either python or simply ./ if it has been correctly chmoded. The options depend on the kind of data you are scoring.

Common options

If you do not specify --roc or --det, it will just score a single file. See the next section to plot curves from confidences.

ROC/DET options

If you specify --roc or --det, then more options become available, in addition to the previous ones. If one of these options are set without any of --roc or --det set, it will either be ignored or raise an error.

Print saved/multiple curves

Saved curves are stored in a python pickle format, so you cannot read them easily. That's why there is the module. It allows you to plot one or several saved curves on the same graph. Basically, you use it typing

python [options] file1 [file2 [file3...]]

There are several options you can use :


Suppose you want to plot a DET curve from data generated by eblearn, on the INRIA pedestrian dataset. We assume that the eblearn data are named bbox.txt, and that the INRIA dataset is located at data/INIRA. We want to use the overlap50percent comparator. Then the command line would be :

python --input eblearn --input_parser eblearn --groundtruth data/INRIA --groundtruth_parser INRIA --comparator overlap50percent --det

Suppose now you want to save the curve in the file eblearn-inria.curve, and you now want a ROC curve. You also want only 100 points on the curve :

python --input eblearn --input_parser eblearn --groundtruth data/INRIA --groundtruth_parser INRIA --comparator overlap50percent --roc --saving-file eblearn-inria.curve --sampling 100

If you want to plot the curves eblearn-inria.curve, foo.curve and bar.curve on the same graph, with x between 0 and 100, and eblearn-inria.curve being highlighted, you type :

python --main_curve eblearn-inria.curve --xmin 0 --xmax 100 eblearn-inria.curve foo.curve bar.curve

Standard comparators

The standard comparators mainly use boxes. If the object has no box (like the groundtruth for the CMU face dataset), it generates a box from the data, which could lack precision.

Writing modules


By default, parsers are stored in the parsers directory. A parser will be named foo when used. Every .py file in the parser directory will be considerated as a parser, and an error will occur if it does not have the right sturcture.

A parser sould at least have one of the following functions, and can have both of them.


Comparators are similar to parsers in the way they are handled. They are in the comparators folder, and they are named by their file name without the .py .

The only recognized function in a comparator is compare_datasets(toscore, groundtruth), which takes two dataset.DataSet and returns a result.DataSetResult .

There are several common and useful functions for comparators that can be found in comparator_helpers .