How do we measure Performance?
Overview about how we measure the performance of our released models
We calculate accuracy by comparing counts obtained by our traffic counting solution against a manually obtained ground truth (GT). Delivering correct and realistic accuracy measures is most important, and therefore we make a real effort obtaining our GT data.
We also make sure that scenes from any performance measurement never find their way into our training dataset, and thereby avoiding overtraining and unrealistic performance measurements which cannot be reached in real world usecases.

Crossing line & Virtual door

The following example describes counting accuracy calculation for crossing lines.
Confusion Matrix Measuring Accuracy for CL & VD

Example

Given the following results table:
  • Scene 1 has 2 errors (1 missed, 1 overcount)
  • Scene 2 has 1 error (1 missed)
  • In total, there are 3 errors and 16 Ground truth counts (5 + 3 + 3 + 5)
This gives us an accuracy of 16-3/16 = 81.25%

Origin/Destination

The following described counting accuracy calculation for origin/destination.
Confusion Matrix Measuring Accuracy for OD
Scene 1 has 2 errors (1 missed, 1 overcount)
Scene 2 has 1 error (1 missed)
In total, there are 3 errors and 11 GT counts (5 + 3 + 3)
This gives us an accuracy of 11-3/11 = 72.72%

ANPR

For automated number plate recognition (ANPR), the accuracy logic is the same as for crossing lines, with two additional restrictions:
  • vehicle class is not taken into account
  • the number plate sent in the event is compared and has to fully match the ground truth
image-crop
ground-truth
model-result
correct
85BZHP
85BZHP
YES
BNW525
BNW555
NO
DY741WJ
DY741WJ
YES
GU278MB
GU278MB
YES
FW749XA
FW749XA
YES
ERZM551
ERZM55
NO
For this example, we receive an accuracy of 4/6*100% = 66%

Devices used for Measuring

For our performance measures, we are using different types of hardware to guarantee a stable version of our software. When we receive different results in our Happy RTSP Performance lab, we are going to proudly announce the minimum percentage as our accuracy to be achieved.
In the table below, you can see the 4 different devices we are testing, as well as an example of results achieved. In this case, we would publish an accuracy of 90% as a target.
Device
Accuracy
P101
91%
Nvidia Jetson AGX
91%
Nvidia Jetson NX
91%
Nvidia GTX 1080
90%
Last modified 1d ago