Blog for work on my Masters thesis - a survey of methods for evaluating media understanding, object detection, and pattern matching algorithms. Mostly, it is related to ViPER, the Video Performance Evaluation Resource. If you find a good reference, or would like to comment, e-mail viper at cfar.umd.edu.
Archives
Media Processing Evaluation Weblog
Thursday, October 20, 2005
Somethings been really, really annoying me
Okay, so I talked with Ilya this afternoon about getting ViPER-PE to perform per his requirements. He needs a list of all objects that are identified at all in a video. He doesn't much care that the objects are identified well. Originally I suggested a Framewise evaluation, but he doesn't care if an object is identified on each frame, just that it is identified sometime during its existance in the field of view. So, he should logically use an object evaluation with high thresholds and no statistical evaluation. But, he has identified a few bugs...
The first bug is that some items are making it through localization. I haven't noticed it, because they are caught during the statistical match. I've tracked this down to a bug in the 'clear' method of the 'FrameSpan' object, where it wasn't always clearing values. I've added a few more tests to the AttributeTest unit test set to exercise the framespan more.
The second bug arises in MULTIPLE matching, where some cromulent matches are thrown out.... MULTIPLE is almost what Ilya wants, although NONE is a better choice. I'll look in to fixing that; I thinkg what he really wants is an asymmetric multiple match, like multi-greedy, where each target descriptor is matched to 1..k candidates, and each candidate is matched to at most one target.