How do we measure Performance?

Overview about how we measure the performance of our released models

We calculate accuracy by comparing counts obtained by our traffic counting solution against a manually obtained ground truth (GT). Delivering correct and realistic accuracy measures is most important, and therefore we make a real effort obtaining our GT data.

We also make sure that scenes from any performance measurement never find their way into our training dataset, and thereby avoiding overtraining and unrealistic performance measurements which cannot be reached in real world usecases.

Crossing line & Virtual door

The following example describes counting accuracy calculation for crossing lines.

Confusion Matrix Measuring Accuracy for CL & VD

Example

Given the following results table:

• Scene 1 has 2 errors (1 missed, 1 overcount)

• Scene 2 has 1 error (1 missed)

• In total, there are 3 errors and 16 Ground truth counts (5 + 3 + 3 + 5)

This gives us an accuracy of 16-3/16 = 81.25%

Origin/Destination

The following described counting accuracy calculation for origin/destination.

Confusion Matrix Measuring Accuracy for OD

Scene 1 has 2 errors (1 missed, 1 overcount)

Scene 2 has 1 error (1 missed)

In total, there are 3 errors and 11 GT counts (5 + 3 + 3)

This gives us an accuracy of 11-3/11 = 72.72%

ANPR

For automated number plate recognition (ANPR), the accuracy logic is the same as for crossing lines, with two additional restrictions:

• vehicle class is not taken into account

• the number plate sent in the event is compared and has to fully match the ground truth

image-crop	ground-truth	model-result	correct
	85BZHP	85BZHP	YES
	BNW525	BNW555	NO
	DY741WJ	DY741WJ	YES
	GU278MB	GU278MB	YES
	FW749XA	FW749XA	YES
	ERZM551	ERZM55	NO

For this example, we receive an accuracy of 4/6*100% = 66%

Devices used for Measuring

For our performance measures, we are using different types of hardware to guarantee a stable version of our software. When we receive different results in our Happy RTSP Performance lab, we are going to proudly announce the minimum percentage as our accuracy to be achieved.

In the table below, you can see the 4 different devices we are testing, as well as an example of results achieved. In this case, we would publish an accuracy of 90% as a target.

Device	Accuracy
P101	91%
Nvidia Jetson AGX	91%
Nvidia Jetson NX	91%
Nvidia GTX 1080	90%