visiongrader help

Intoduction

visiongrader is a scoring tool for detection algorithms. It is mainly used with face and pedestrians recognition, but nothing prevents you from using it with another type of data, provided you write a parser for the data.

Both the parser and the comparator can easily be changed, so it can adapt to quite a large range of situations. You can either score a single result or, if you got the confidence of the results, plot a ROC or a DET curve.

Usage

For the time being, visiongrader is a command-line program. You can run it bt typing either python main.py or simply ./main.py if it has been correctly chmoded. The options depend on the kind of data you are scoring.

Common options

If you do not specify --roc or --det, it will just score a single file. See the next section to plot curves from confidences.

• --help : displays the help.
• --input : the path to the input file. It is the file to score.
• --input_parser : the parser to use to read the input file.
• --groundtruth : the path to the groundtruth file.
• --groundtruth_parser : the parser to use to read the groundtruth file
• --parser_dir : the path to the directory where the parsers are located. If not specified, it is set to ./parsers .
• --comparator : the comparator to use. The comparator role is to tell if two boxes (or whatever object is used to describe the detections) match.
• --comparator_dir : the path to the directory there the parsers are located. If not specified, it is set to ./comparators .
• --crawl : instead of an input file, use an input directory. It will only have an effect if the input parser has a crawl support. The comportement can differ from a parser to another, but the idea is to crawl into the directory and concatenate all the result files.

ROC/DET options

If you specify --roc or --det, then more options become available, in addition to the previous ones. If one of these options are set without any of --roc or --det set, it will either be ignored or raise an error.

• --roc : make a ROC curve.
• --det : make a DET curve.
• --confidence_min : the mininal confidence to consider when making the curve.
• --confidence_max : the maxinal confidence to consider when making the curve.
• --sampling : the number of points used to make the curve.
• --show-no-curve : does not display the curve when computed.
• --saving-file : by default, the curve is not saved. If you specify file with this options, the curve is saved in that file.
• --xmin : the minimal x on the displayed curve.
• --xmax : the maximal x on the displayed curve.
• --ymin : the minimal y on the displayed curve.
• --ymax : the maximal y on the displayed curve.

Print saved/multiple curves

Saved curves are stored in a python pickle format, so you cannot read them easily. That's why there is the plotpickle.py module. It allows you to plot one or several saved curves on the same graph. Basically, you use it typing

python plotpickle.py [options] file1 [file2 [file3...]]

There are several options you can use :

• --help : displays the help.
• --main_curve : if there is a curve you want to highlight, specify the name of its file with this option. Don't forget you still have to put the name just like any curve, so the file name will appear twice.
• --xlegend : the legend on the x axis.
• --ylegend : the legend on the y axis.
• --xmin : the minimal x on the graph.
• --xmax : the maximal x on the graph.
• --ymin : the minimal y on the graph.
• --ymax : the maximal y on the graph.

Examples

Suppose you want to plot a DET curve from data generated by eblearn, on the INRIA pedestrian dataset. We assume that the eblearn data are named bbox.txt, and that the INRIA dataset is located at data/INIRA. We want to use the overlap50percent comparator. Then the command line would be :

python main.py --input eblearn --input_parser eblearn --groundtruth data/INRIA --groundtruth_parser INRIA --comparator overlap50percent --det

Suppose now you want to save the curve in the file eblearn-inria.curve, and you now want a ROC curve. You also want only 100 points on the curve :

python main.py --input eblearn --input_parser eblearn --groundtruth data/INRIA --groundtruth_parser INRIA --comparator overlap50percent --roc --saving-file eblearn-inria.curve --sampling 100

If you want to plot the curves eblearn-inria.curve, foo.curve and bar.curve on the same graph, with x between 0 and 100, and eblearn-inria.curve being highlighted, you type :

python plotpickle.py --main_curve eblearn-inria.curve --xmin 0 --xmax 100 eblearn-inria.curve foo.curve bar.curve

Standard comparators

The standard comparators mainly use boxes. If the object has no box (like the groundtruth for the CMU face dataset), it generates a box from the data, which could lack precision.

• overlap : it compares two boxes and returns true iff the boxes overlap, even if it is only one pixel.
• overlap50percent : it compares two boxes A and B, and returns true iff the boxes overlap, and
  area(inter(A, B))/area(union(A, B)) > 0.5 . Actually, you can change 0.5 to any value simply by editing the source file.
• garcia-delakis : this is a face comparator found in the article Convolutional Face Finder : A Neural Architecture for Fast and Robust Face Detection by C. Garcia and M. Delakis. It compares a box with a face, and returns true iff the eyes and the mouth are in the box, and if the box area does not exceed a times the size of a groundtruth box. a can be adjusted in the file.

Writing modules

Parsers

By default, parsers are stored in the parsers directory. A parser foo.py will be named foo when used. Every .py file in the parser directory will be considerated as a parser, and an error will occur if it does not have the right sturcture.

A parser sould at least have one of the following functions, and can have both of them.

• parse(filename, crawl = False) : if crawl is set to False, it should parse a single file, if it is set to True, it can either look for several files in the filename directory, or raise an error if the crawl mode is not supported. It should return a dataset.DataSet object.
• parse_multi(filename, crawl = False) : it has the same comportement than parse, except than it takes confidences into account, and returns a dataset.DataSetMulti object.

Comparators

Comparators are similar to parsers in the way they are handled. They are in the comparators folder, and they are named by their file name without the .py .

The only recognized function in a comparator is compare_datasets(toscore, groundtruth), which takes two dataset.DataSet and returns a result.DataSetResult .

There are several common and useful functions for comparators that can be found in comparator_helpers .