How do we measure Performance?
Overview about how we measure the performance of our released models
We calculate accuracy by comparing counts obtained by our traffic counting solution against a manually obtained ground truth (GT). Delivering correct and realistic accuracy measures is most important, and therefore we make a real effort obtaining our GT data.
We also make sure that scenes from any performance measurement never find their way into our training dataset, and thereby avoiding overtraining and unrealistic performance measurements which cannot be reached in real world usecases.
Crossing line & Virtual door
The following example describes counting accuracy calculation for crossing lines.
Example
Given the following results table:
Scene 1 has 2 errors (1 missed, 1 overcount)
Scene 2 has 1 error (1 missed)
In total, there are 3 errors and 16 Ground truth counts (5 + 3 + 3 + 5)
This gives us an accuracy of 16-3/16 = 81.25%
Origin/Destination
The following described counting accuracy calculation for origin/destination.
Scene 1 has 2 errors (1 missed, 1 overcount)
Scene 2 has 1 error (1 missed)
In total, there are 3 errors and 11 GT counts (5 + 3 + 3)
This gives us an accuracy of 11-3/11 = 72.72%
ANPR
For automated number plate recognition (ANPR), the accuracy logic is the same as for crossing lines, with two additional restrictions:
vehicle class is not taken into account
the number plate sent in the event is compared and has to fully match the ground truth
image-crop | ground-truth | model-result | correct |
85BZHP | 85BZHP | YES | |
BNW525 | BNW555 | NO | |
DY741WJ | DY741WJ | YES | |
GU278MB | GU278MB | YES | |
FW749XA | FW749XA | YES | |
ERZM551 | ERZM55 | NO |
For this example, we receive an accuracy of 4/6*100% = 66%
Devices used for Measuring
For our performance measures, we are using different types of hardware to guarantee a stable version of our software. When we receive different results in our Happy RTSP Performance lab, we are going to proudly announce the minimum percentage as our accuracy to be achieved.
In the table below, you can see the 4 different devices we are testing, as well as an example of results achieved. In this case, we would publish an accuracy of 90% as a target.
Device | Accuracy |
P101 | 91% |
Nvidia Jetson AGX | 91% |
Nvidia Jetson NX | 91% |
Nvidia GTX 1080 | 90% |
Last updated
