Use this technical documentation to learn, apply and share about SWARM technology.

Who should read this?

This documentation is a technical guideline for partners and customers who are interested in integrating our software in a new or existing data analyzing and IoT environment.

Number plate area code (Early Availability Feature)

In addition to the number plate raw text and country code, we now support the number plate area code. Some number plates include an area code associated with a geographical area, such as "W" for Vienna (Austria) or "M" for Munich (Germany).

The following 13 countries are supported

• Austria

• Bulgaria

• Switzerland

• Czech Republic

• Germany

• Greece

• Croatia

• Ireland

• Norway

• Poland

• Romania

• Slovakia

• Slovenia

How does this work?

• For supported countries, we detect spaces between letters and parse the raw license plate text according to a predefined format.

• In the case of countries that are not supported (e.g. Italy), the generated event won't contain an area code.

How to configure?

All use cases based on ANPR are supported, no additional configuration is required.

• Barrierless Parking with ANPR

• Journey Time

How can I access the data?

The Data Analytics widgets "Journey Distributions" and "License Plates" allow segmenting by "License Plate Area"

If you are using a custom broker, the event schema has been adjusted to contain the area code.

Minor improvements and bug fixes

• Improved classification accuracy for the classes person and bicycle (Traffic & Parking Standard/Accuracy+)

• Fixed Data Analytics, some queries led to inconsistent data

• Fixed rule engine for Region of Interest occupancy, triggering a hardware output

• Fixed P100/OP100 slowdown introduced with 2023.3

• Reduced downtime during device updates

• Fixed debug video output pixelation of counting line overlays

• Fixed license notification for not enough SPS licenses

The SWARM Perception Platform consists of the following major components:

• SWARM Control Center (SCC): Manage and configure Perception Boxes and analyze your data with Data Analytics. For both, we provide APIs for integration.

• SWARM Perception Agent: The SWARM software running on our products P101/OP101/VPX. The video from an RTSP camera/USB camera is processed with the help of deep learning. Events are generated and sent via MQTT to either Data Analytics or a custom MQTT broker. Single or multi-camera processing is supported. The engine is configured solely through the SCC.

Data integration options

• Data Analytics API (high level)

  • Events are sent and stored in Azure Cloud environment managed by Swarm

  • Events are processed and stored by Swarm

  • The API enables easy integration with third party systems

• MQTT (low level)

  • Requires to operate an MQTT server

  • Requires the storing/processing of raw events

  • Enables on the edge processing for time-critical and/or offline use cases

Principles

The architecture is based on the following principles:

• Centralized data analytics, device configuration, and update management.

  • Hosted in the Microsoft Azure Cloud, maintained by Swarm.

• Decentralized camera stream processing at the edge

  • Generated data (Events) and management traffic are decoupled.

  • Support for heterogeneous network infrastructure

• Scale from one to thousands of SWARM Perception agents

Device Schedule (Running/Standby)

The device schedule allows you to define time slots when the device should be in operational mode (device state: running) or in power saving mode (device state: standby).

The power saving mode is mostly relevant for the battery powered system BMA Mobil since it extends the usage time for a single charge.

In our measurements, the BMA Mobil reduces the power consumption during standby by about 55%.

How to enable the schedule for a device?

1. Open a device in the control center and select the tab schedule

2. Switch the toggle to Enabled and select desired time slots when the device should be operational

3. Save Schedule

What will happen during standby?

The device reduces the power consumption by pausing the video signal processing.

Functions that are not available during standby:

• No events will be generated

• No camera image is visible

• No event triggers can be configured

Functions that are available during standby:

• The device will be online in the Control Center, including available device health, logs, and reboot functionality.

• Wake-up the device from standby (→Set the toggle to Disabled)

What is the expected battery lifetime?

In our measurements the BMA Mobil consumes 1,07A when active and 0,48A when in standby.

Examples

With an 100Ah battery degraded to 90Ah we get the following battery lifetimes:

• Always active: 90Ah / 1,07A ~3,5 days

• Always standby: 90Ah / 0,48A ~7,8 days

• 13h active, 59h standby (Example from screenshot, total 3days): 13h * 1,07A + 59h *0,48A = 42Ah out of 90Ah (~50%)

Improved latency for BMA/BMC

We have been able to improve latency for the BMA working together with the BMC to < 1s. Measured by vehicle is physically present in a zone to the point when the BMC switches a dry contact.

Other Improvements

• Fixed: On Curiosity startup the first Region of Interest event was always vacant (even if there were objects).

• Improved: Curiosity performance improvements by avoiding to re-classify already classified objects

• Improved: The Number Plate Column for DataAnalytics Parking widgets is optional

Device Update Size

2023.3 U1 → 2024.2: 155MB (BMA / P101 / Jetson Nano)

API

• There are no API breaking changes

Control Center

• Device Updates

  • Select individual devices for an update, e.g. this allows staged rollout site by site

  • Schedule the update, e.g. 5am to reduce the production impact to a minimum
• Device Camera View

  • Auto refresh the camera preview for a convenient calibration for up to 30seconds

  • The stream automatically expands when one stream is configured

  • Event triggers and focus zone is now visible in every camera preview

  • Show or hide the event triggers/focus area

  • The focus area is not hiding event triggers anymore
• The MQTT QoS level (0 - at most once, 1 - at least once, 2 -exactly once) for customer broker is configurable. Higher levels of QoS are more reliable, but involve higher latency and have higher bandwidth requirements.
• Side Menu
• API

  • Potentially breaking change: Authentication requests are throttled, limit: 100/hour. Note: The number of API calls for the Control Center is not throttled.

  • Potentially breaking change: Serial number format changed to string

Device

Models

• The detection accuracy for Traffic&Parking (Standard and Accuracy+) has been improved, in particular the classes: car with trailer, trucks (single unit, articulated, with trailer) and bicycle.

Tracker

We improve the data quality by enhancing the tracking system for:

• Non-linear movement (e.g. vehicles in roundabouts)

• Short occlusions (e.g. vehicles occluded by poles or signs)

• Track hijacking (one or more objects share the same track) when used in combination with the focus area

• Calibration visualisation

  • Tracks are coloured in blue when they are too small to classify.

  • Long tracks will not be cut-off

This is not a marketing video, it shows the engine overlay we ship with 2024.1. No hand tuning has been performed.

The Virtual Perception Box (VPX) is a software-only product that will be provided as docker containers (SPS license).

• You are fully responsible for the setup and maintenance of the hardware and software (OS, driver, networking, system applications, …)

• Production-ready checklist for devices with VPX (Swarm Analytics recommendation)

  • Integrate remote management (e.g. VPN/SSH)

  • Integrate update management (e.g. Ansible) to update system packages like IotEdge, Jetpack, Docker

  • Device monitoring

  • Check security (firewall, security patches applied, strong passwords/certificates)

P101 diagram

Once Step 6 is complete, return to the Getting Set-Up page to continue with Device Configuration.

The SWARM Perception Box is a managed black box; meaning you do not need to manage hardware/OS/driver/security patches. You also cannot access the system.

To get set up, follow these steps:

1. Mount and Go Online

Your first step includes:

• Mount the (O)P101/P401 to the desired location.

• Ensure your (O)P101/P401 is online in your SWARM Control Center.

2. Device Configuration

After the Perception Box is successfully connected with the SWARM Control Center, you must configure the used camera and the scenario you are fulfilling.

3. Data Analytics

​

Adaptive Traffic Control

Adaptive traffic control enables you to interface with hardware devices like traffic controllers using dry contacts. Use cases and benefits:

• 'Smart Prio' System: Prioritise certain traffic classes and ensure fluid traffic behavior in real-time (e.g. pedestrians, bicyclists, e-scooters, heavy traffic).

• Simplify infrastructure maintenance: Replace multiple induction loops with a single Swarm Perception Box. The installation does not require excavation work, and reduces the maintenance effort/costs.

Device Health - The healthiness of your device at a glance

The following metrics are supported:

• Device Uptime, Status, Restarts, Available Disk Space

• Device Temperature (support for P101/OP101/Jetson Nano)

• LTE Modem Traffic, Signal Strength and Reconnects (support for OP100/OP101)

• Camera status and Camera processing speed (FPS)

• Generated and Pending events

Model Improvements

Traffic & Parking (Standard and Accuracy+)

We improved the classification accuracy for both variants Standard and Accuracy+, especially for the classes: articulated truck, truck with trailer and car with trailer.

People Head

We fixed the class output, the event class output is now person (vs. previously head). Affected by this change are the devices P101/OP101/VPX. Not affected are the devices P100/OP100.

We improved the accuracy due to higher resolution processing.

People Full body

We improved the accuracy due to higher resolution processing.

Parking Fisheye

The model has been deprecated and will not be updated in the future. The model will continue to work as long as there are devices using it. Please consider switching to the Traffic & Parking model.

Device Metadata

Organize devices and generate events containing pre-defined device metadata. You can define up to five key and value pairs for a device. The keys and values can be freely defined by your choice, we support autocompletion for keys to avoid nasty typos.

Track Calibration History

Track calibration overlays the last 100 object tracks on a current camera frame. This enables you to position event triggers (e.g. counting lines) for optimal results. We extend the functionality with a history over the last 24 hours.

With track calibration history enabled you will be able to access the track calibration for every hour of the past 24 hours.

Device Update

You decide when your devices get an update. Please update soon in order to use the latest features (e.g. Adaptive Traffic Control, Track Calibration History) and to benefit from quality improvements (e.g. Model updates), bug fixes and security updates.

We will automatically update by 13.12.23.

Other improvements

• Control Center - User Management - Invite users and manage permission of existing users

• Control Center - MQTT client id can be defined for custom MQTT brokers

• Data Analytics - Speed-up of Origin-Destination widgets

• Data Analytics - Fix widget visualisation of the chord diagram (OD) and line charts

• Under some conditions an ROI in combination with a rule did not trigger events

P401 diagrams

Once Step 6 is complete, return to the Getting Set-Up page to continue with Device Configuration.

There are two major use cases for understanding the traffic situation across your city, urban streets as well as highways.

Find the Product Datasheet here.

Once your OP101DC is mounted and online, return to the Getting Set-Up page to continue with Device Configuration.

Install JetPack 5.1.2

Flash the system image JetPack 5.1.2 (L4T 35.4.1) onto your Jetson device. Follow the documentation from NVIDIA. Depending on your hardware capability, you have the option to use an SSD or internal storage.

Installation size

Jetpack 5.1.2 requires at least 32GB or more of storage. In order to free up more storage for additional software, either use a bigger storage device or follow the NVIDIA reference.

Install VPX Agent

With our installer script, installing the VPX agent is easy. Make sure to get the serial(s) from us in advance.

sudo apt update && sudo apt install curl -y
curl http://get-vpx.swarm-analytics.com/install-2.0.sh > install.sh  && chmod +x install.sh 
./install.sh

After the installation script is complete, the IoT Edge runtime will pull four docker containers as outlined below.

The device type needs to be set in SWARM Control Center. Currently, only our support team can do that. Please create a ticket, therefore.

Make sure that the container curiosity-arm64-jetpack5 is used.

swarm@VPX:~$ sudo docker ps -a
CONTAINER ID        IMAGE                                             COMMAND                   CREATED             STATUS              PORTS                                                                  NAMES
57eec104e917        swarm.azurecr.io/curiosity-arm64-jetpack5:7.1.0   "./curiosity"             2 weeks ago         Up 5 minutes                                                                               curiosity
82b106f9d3d7        swarm.azurecr.io/azure-module-arm64:1.11.1         "java -jar app.jar"       6 weeks ago         Up 10 days                                                                                 azure-module
27ffd61ab021        mcr.microsoft.com/azureiotedge-hub:1.4.10       "/bin/sh -c 'echo \"$…"   6 weeks ago         Up 10 days          0.0.0.0:443->443/tcp, 0.0.0.0:5671->5671/tcp, 0.0.0.0:8883->8883/tcp   edgeHub
5e96e96eb440        mcr.microsoft.com/azureiotedge-agent:1.4.10     "/bin/sh -c 'exec /a…"    6 weeks ago         Up 10 days                                                                                 edgeAgent

You will see in the SWARM Control Center an "Unnamed Device" with the corresponding registration ID:

You want to know the traffic situation of an urban street or highway? SWARM software is providing the solution to get the number of vehicles passing at the street split per vehicle type (Classes) and direction.

What data can be generated?

For this use case, SWARM software is providing you with any data needed for traffic counting - The counts of vehicles including classification can be covered. The counts are split between two directions (IN/OUT). Furthermore, several counts can be made in one video camera, e.g.: count each lane separately. On top, you have the opportunity to add a second counting line, calibrate the distance in between and estimate the speed of the vehicles passing both lines.

What needs to be considered for a successful analysis?

Environment specification

Hardware Specifications

The VPX agent needs the following hardware and software requirements to be met.

NVIDIA Jetson family

• Hardware

  • Supported NVIDIA Jetson devices

    • Nano 4 GB, TX2 NX, Xavier NX, AGX Xavier

    • Orin Nano 4GB/8GB, Orin NX 8GB/16GB, Orin AGX 32GB/64GB

  • Memory: at least 4 GB available

  • Storage: at least 6 GB free

• Software

  • JetPack 4.6 or Jetpack 5.1.2 (not all Jetson devices support newer Jetpack)

  • IotEdge 1.4

• Network requirements

NVIDIA GPUs

• Hardware

  • CPU: X86-64bit

  • Memory: At least 4 GB available

  • Storage: At least 15 GB free

  • NVIDIA Workstation GPUs

    • RTX series (e.g. RTX A2000)

    • Quadro RTX series (e.g. Quadro RTX 4000)

• Software

  • Ubuntu 20.04 LTS

  • NVIDIA Driver Version 470 (or newer)

  • Docker 19.0.1+

  • IotEdge 1.4

• Network requirements

You can find the benchmark results here: How many cameras can my Perception Box compute?

In case of uncertainty, please contact support

Install JetPack 4.6.0

Flash the system image JetPack 4.6.0 (L4T 32.6.1) onto your Jetson device. Follow the documentation from NVIDIA. Depending on your hardware capability, you have the option to use an SD card or internal storage.

Free up hard disk space on your Jetson device (optional)

sudo apt update && sudo apt install curl -y
curl http://get-vpx.swarm-analytics.com/clean_jetson.sh > clean_jetson.sh  && chmod +x clean_jetson.sh 
./clean_jetson.sh

Install VPX Agent

With our installer script, installing the VPX agent is easy. Make sure to get the serial(s) from us in advance.

sudo apt update && sudo apt install curl -y
curl http://get-vpx.swarm-analytics.com/install.sh > install.sh  && chmod +x install.sh 
./install.sh

After the installation script is complete, the IoT Edge runtime will pull four docker containers as outlined below.

Make sure that the container curiosity-arm64-tensorrtis used.

swarm@VPX:~$ sudo docker ps -a
CONTAINER ID        IMAGE                                             COMMAND                   CREATED             STATUS              PORTS                                                                  NAMES
57eec104e917        swarm.azurecr.io/curiosity-arm64-tensorrt:5.3.0   "./curiosity"             2 weeks ago         Up 5 minutes                                                                               curiosity
82b106f9d3d7        swarm.azurecr.io/azure-module-arm64:1.1.0         "java -jar app.jar"       6 weeks ago         Up 10 days                                                                                 azure-module
27ffd61ab021        mcr.microsoft.com/azureiotedge-hub:1.0.10.3       "/bin/sh -c 'echo \"$…"   6 weeks ago         Up 10 days          0.0.0.0:443->443/tcp, 0.0.0.0:5671->5671/tcp, 0.0.0.0:8883->8883/tcp   edgeHub
5e96e96eb440        mcr.microsoft.com/azureiotedge-agent:1.0.10.3     "/bin/sh -c 'exec /a…"    6 weeks ago         Up 10 days                                                                                 edgeAgent

You will see in the SWARM Control Center an "Unnamed Device" with the corresponding registration ID:

Install NVIDIA drivers and the Docker engine

Install the Docker Engine

Install the NVIDIA drivers and CUDA

Install the NVDIA container toolkit

After you followed the installation guidelines you must be able to get a similar output

docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.86       Driver Version: 470.86       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|

Install Azure IoTEdge

Install IotEdge 1.4 (only the package aziot-edge is required)

Configure Azure IotEdge

You will receive a ZIP file from Swarm with configuration files. (Replace $ID with the device ID you received from SWARM)

ID=<DEVICE-ID>
sudo mkdir /opt/swarm/
sudo mkdir /opt/swarm/config
sudo mkdir -p /etc/iotedge/
sudo cp $ID/VPX-$ID/config.yaml /etc/iotedge/config.yaml
sudo chown iotedge:iotedge /etc/iotedge/config.yaml
sudo cp $ID/VPX-$ID/config.toml /etc/aziot/config.toml

sudo mkdir -p /var/lib/iotedge/hsm/
sudo cp $ID/all/azure-production-certificates/* /var/lib/iotedge/hsm/
sudo chown aziotcs:aziotcs /var/lib/iotedge/hsm/iot-edge-device-SwarmEdgeDeviceCA-full-chain.cert.pem
sudo chown aziotcs:aziotcs /var/lib/iotedge/hsm/swarm-iot.root.ca.cert.pem
sudo chown aziotks:aziotks /var/lib/iotedge/hsm/iot-edge-device-SwarmEdgeDeviceCA.key.pem
sudo iotedge config apply -c '/etc/aziot/config.toml'

At this point check the IotEdge logs for any errors

iotedge system logs -- -f

You will now see your deployment in the SWARM Control Center as "Unnamed Device" with the registration ID:

At this stage, the IoT Edge runtime will pull the docker images and once finished the device can be configured in the Control Center.

docker ps -a
CONTAINER ID   IMAGE                                                    COMMAND                   CREATED        STATUS        PORTS                                                                                                                         NAMES
118c96889ec7   swarm.azurecr.io/curiosity-x64-tensorrt:6.6.33           "/usr/local/bin/nvid…"    6 days ago     Up 6 days     0.0.0.0:8090->8090/tcp, :::8090->8090/tcp                                                                                     curiosity
95d63beb336e   swarm.azurecr.io/azure-module-x64:1.7.1                  "java -jar app.jar"       6 days ago     Up 6 days                                                                                                                                   azure-module
3003fd8db258   mcr.microsoft.com/azureiotedge-hub:1.1.15                "/bin/sh -c 'echo \"$…"   3 months ago   Up 4 weeks    0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:5671->5671/tcp, :::5671->5671/tcp, 0.0.0.0:8883->8883/tcp, :::8883->8883/tcp   edgeHub
6f469d38d91d   mcr.microsoft.com/azureiotedge-metrics-collector:1.0.9   "/bin/sh -c 'echo \"$…"   3 months ago   Up 3 months                                                                                                                                 azure-monitor
88d10c824af0   mcr.microsoft.com/azureiotedge-agent:1.1.15              "/bin/sh -c 'exec /a…"    3 months ago   Up 3 months                                                                                                                                 edgeAgent

Next steps: Configure your use case.

With the SWARM Perception Platform, you are able to find a solution for each parking environment thanks to the offering of following use cases.

You have a parking space where you simply want to know your utilization by making an Entry/Exit count, SWARM provides a perfect solution for doing that quite easily. See yourself:

What data can be generated?

For this use case, SWARM software is providing you with any relevant data for your Entry/Exit parking space. The solution is gathering the number of vehicles in your parking space as well as the number of vehicles entering and exiting your parking space for customizable time frames.

The vehicles are classified in any classes the SWARM software can detect. Nevertheless, consider that the following configuration set-up is optimized to detect vehicles and not people and bicycles.

What needs to be considered for a successful analysis?

Environment requirements

Hardware Specifications

Do you want to know the average speed of your traffic in given areas? SWARM software is providing the solution to get the number of vehicles passing at the street split in different speed segments (10km/h) or even see the average speed of the given count over an aggregated time period.

What data can be generated?

What needs to be considered for a successful analysis?

Environment specification

Hardware Specifications

You want to know the flow of your intersections, SWARM Perception platform is providing the solution to get the number of vehicles starting in an origin zone and ending up in a destination zone per vehicle type (Classes).

What data can be generated?

For this use case, SWARM software is providing you with any data needed for Origin Destination Zones - The number of vehicles from one zone to another including classification according to BAST/TLS standards can be covered.

What needs to be considered for a successful analysis?

Environment specification

Hardware Specifications

You have a parking space where you want to know your utilization and parking times of your customers, then you can use the SWARM solution as following.

What data can be generated?

For this use case, SWARM software is providing you with any relevant data for your Entry/Exit parking space. The solution is gathering the number of vehicles in your parking space as well as the number of vehicles entering and exiting your parking space for customizable time frames.

The vehicles are classified in any classes the SWARM software can detect. Nevertheless, consider that the following configuration set-up is optimized to detect vehicles and not people and bicycles.

Thanks to the License plate recognition, the parking duration of your customers will be analyzed. On top of the License plate information, license plate origin country as well as license plate area codes are available as meta information. The country codes are according to ISO 3166 Alpha 2 standard. The country classification is working with excellent accuracy of 99%.

What needs to be considered for a successful analysis?

Find below some general settings for the installation of this use case. As the automatic number plate reading needs some more detailed information you will find additional and more detailed information on how to set it up in the following page:

Environment requirements

Hardware specification

How ANPR works

Automatic number plate recognition (ANPR) works in four steps. It needs to detect vehicles and license plates, read the plates, and trigger events.

1. Detect vehicle

Detect vehicles as cars, trucks, and buses and follow them in the video stream.

2. Detect license plate

For each detected vehicle, detect license plates and map them to the vehicles.

3. Read license plate

For each detected license plate, apply an optical character recognition (OCR) to read the plate.

4. Send event

If the vehicle crosses a counting line, send an event with the text from the detected license plate.

The main challenge for ANPR setups consists of clearly readable license plates. This means a sharp and well-illuminated image without occlusions or blurry objects is required to obtain correct results. The following guide shows how to set up our ANPR system and helps to avoid the most common issues.

Typical Setup

The system is designed for two typical setups which are described here.

Single lane from the side

For this setup, the camera is mounted at around 2m height, as closely as possible to the side of the lane to avoid a high horizontal angle. If possible, enforce vehicles to stay in lane for the ANPR section, as switching lanes can lead to inaccurate results.

Two lanes from above

When positioning the camera above the cars (e.g. entry/exit of a garage) a maximum of two lanes can be covered.

To work properly, vehicles should drive straight through the scene to have the license plate visible in the entire scene. The camera should be at a height of 3m and facing vehicles directly from the front or back to avoid a high horizontal angle to the license plate.

Camera Setup

Camera setup can sometimes be tricky and often requires some experimentation with the camera position and parameters to get optimal results.

The following sections describes common camera issues and how to avoid them.

Resolution

License plates need to be visible with 250 pixel-per-meter (PPM). For a standard European plate, this gives us a minimum height of 30px and a minimum width of 100px to get good recognition results.

For camera setups with object distances within the specification, a FullHD (1080p) resolution is sufficient. In some cases, it might help to choose a higher resolution (4MP or 2K) for a sharper image.

It is recommended to check the size of license plate crops manually during the setup phase.

Viewing Angle

License plates need to be visible from a direct viewing angle. While small angles (<20° horizontal, <30° vertical) and tilting <5° can be handled, larger angles cannot work at all. If view angles get bigger, the system is more likely to mix up characters or is not able to recognize characters close to the edges.

For camera positions from the side only a single lane is recommended, while with camera views from above, a maximum of two lanes works.

Lighting

Scene illumination has two major effects.

1. With good illumination, a lower shutter speed can be chosen and images get less blurred, especially for fast-moving vehicles.

2. Good lighting reduces the ISO value of the camera and images appear less grainy and sharper.

Some cameras offer additional illumination which can be useful. If the camera light is not sufficient, an external illumination of the scene is required.

Digital noise reduction (DNR) should be kept in a low range to further reduce graininess.

Shutter speed

A low shutter speed is important for moving objects to get a sharp image and avoid blurriness caused by motion.

While in general, faster is better, the selected shutter speed depends on the available light in the scene.

Depending on vehicle speed a shutter speed of 1/250 is a bare minimum for moving objects below 15 km/h. For faster vehicles, up to 40 km/h, a shutter speed of 1/500 is a good choice. For faster objects, an even lower shutter speed is required which only works with good illumination.

Encoding Quality

To stream the camera image, data is encoded. Different encodings can save data and reduce image quality. For the ANPR use case, high image quality is required. Select H.264 codec and a high bitrate of >6000kbps for FullHD (1080p) content and >8000 kbps for 4MP video material with 25 FPS.

Additional features such as BLC and WDR are not recommended, as postprocessing can reduce details. If they are necessary, the impact on the video quality should be checked.

A constant bitrate (CBR) usually leads to a better quality than variable bitrate (VBR).

Test setup in various conditions

When setting up the camera, it is recommended to take a few short test videos in different lighting conditions (morning, midday, evening, night) to check if license plates are clearly visible in all conditions.

Event Triggers

A suitable position of the event trigger (counting line) is essential for good ANPR results. If the line is positioned far in the back, the ANPR system has no time to detect and recognize the plate before an event is sent. If the line is in a position, where the license plate is only visible at a suboptimal angle, results will not be accurate.

For an optimal counting line position, a short debug video of the scene with 3-5 vehicles is required. In the analysis of the video, one should follow the vehicle through the optimal section with the best view on the plate (see Example 6). Just as the view on the plate gets worse (see Example 7), position the line right behind the center of the vehicle.

By this method, it’s guaranteed that the system can utilize the best video parts to detect and recognize the license plate and send the event just before suboptimal views worsen the result.

Accuracy

Our ANPR system is tested properly under various conditions. In our test setup, we have around five different scenes, and accuracies are calculated on the basis of > 800 European vehicles. Overall accuracy means the percentage of correctly identified vehicles plus license plates compared to all passing vehicles with readable license plates.

Under the specified conditions, the system reaches >95% overall accuracy in slow parking environments and >90% in environments with fast vehicles.

For a detailed analysis of potential errors, see limitations described below.

Limitations

A base limitation that cannot be solved is the general readability of license plates. Plates with occlusions, covered with dust or snow or incorrectly mounted plates cannot be read. Environmental limitations such as strong rain or snow which blocks the clear view on plates can also lead to inaccurate results.

General

There are a few hard limitations where the system cannot provide good results.

1. Illumination (day-only)

Currently, the system supports good illumination only. This limitation is usually for day-only, however it will also work for well-lighted night scenes if the license plates are clearly recognizable.

2. Single-line plates only

License plates with two lines (such as motorcycle plates) are not supported and recognition will not work.

3. EU license plates only

The recognition system is limited to standard EU license plates. It can work with license plates from other countries (and some older non-standard EU plates), but there are no accuracy guarantees.

Error Types

There are four potential errors that can occur within the ANPR system.

• No vehicle is detected (< 1% error rate)

• No plate is detected (< 0.1% error rate)

• Event is sent without a vehicle passing (< 0.1% error rate)

• Wrong plate text is recognized (< 5-10% error rate, depending on the scenario)

Typical Errors

The OCR system identifies character by character. In most error cases it’s a single character that is misclassified or a character that was missed. As in some countries, license plate characters look very similar (sometimes even exactly the same), most errors are caused by mixing up characters with lookalikes.

• B and 8 can be mixed-up

• D and 0 can be mixed-up

• 0 and O can be mixed-up

• I and 1 can be mixed-up

• 5 and S can be mixed-up

The best option to avoid these mixups is to get a clear front view on the plate. However, for systems that need to match in- and outgoing license plates, it might make sense to match them with a fuzzy search that takes mixups and duplicated characters into account. For example, the system could still match plate texts like S123A0 and SI23AO, when the second event exists on the same or following day.

Further improvements

If all setup recommendations are implemented and the camera configuration cannot be improved any further, there are some external improvements that can be made. Best results are achieved when combined.

1. Slow down vehicles in the ANPR section to have more time to detect and recognize the license plate.

2. Reduce distance from vehicle to camera, for example by limiting the entry to a single or narrow lane, reducing variation and angle to the camera.

3. If possible, use camera zoom to focus on the section with the best view on license plates. This can also help with a low resolution of plate crops.

4. If the Swarm system performance is an issue (low FPS), it can help to blackout any unnecessary image parts with vehicles (e.g. with a privacy zone). By this, the focus of the system is set on the ANPR section only.

You have a parking space where you simply want to know if your specific parking spaces are occupied or free, SWARM provides a perfect solution for doing that quite easily. See yourself:

What data can be generated?

For this use case, SWARM software is providing you with any relevant data for Single Space detection within your parking space. The solution is to provide you with the occupancy state of each of your configured parking lots.

The single space detection will give you information about the occupancy state of your parking lot (free or occupied) as well as the information about the object in your parking space, including the classification. Nevertheless, consider that the following configuration set-up is optimized to detect vehicles and not people and bicycles. On top the classification is depending on the camera installation, for a more top-down view the classification will be less accurate.

Camera placement

The main challenge in planning a camera installation is to avoid potential occlusions by other cars. We advise using the Axis lens calculator or generic lens calculator and testing your parking setup for the following conditions:

• put a car on one of the parking spaces

• put a large vehicle (high van, small truck - the largest vehicle that you expect in your parking) on all parking spaces next to your car

• if you still can see >70 % of the car, then this parking spot is valid.

General & easy recommendations for deciding where to place the camera:

• Parking spots have to be fully visible (inside the field of view of the camera). We do not guarantee full accuracy for cropped single parking spaces.

• Avoid objects (trees, poles, flags, walls, other vehicles) that occlude the parking spaces. Avoid camera positions, where cars (especially high cars like vans) occlude other cars.

• Occlusions by other parking cars, mainly happen if parking spaces are aligned in direction of camera-alignment lines.

Get a better overview for installations with more details on camera distance to objects and mounting height:

What needs to be considered for a successful analysis?

Environment requirements

Hardware Specifications

The following definitions will be used:

• Camera-angle: measured from a horizontal line.

• Camera-height: mounting height of the camera

• Distance camera to point in the image-center. This distance is already fixed with camera angle and camera mounting height. But for better estimation of your camera setup, we add it as an additional parameter.

In the following, we give standard examples, with the following assumptions:

• Car-width = 2 m

• Parking length/car-length = 5 m

• Height of the occluding vehicle = 2.5 m

Single parallel parking line

Two parallel parking lines

In order to detect journeys of vehicles, there is a need to detect the same vehicle at several predefined locations. This means there needs to be a dedicated identifier in order to tell if these are the same vehicles.

For vehicles, the unique identifier is obviously the license plate (LP). So, the LP will be taken as the unique identifier for matching vehicles across several locations. As LPs are considered personal data, a salt hashing function will be applied to pseudo-anonymize the personal data.

How does it work?

Based on the SWARM standard use case for traffic counting, the object (vehicle) will be detected and classified. If the journey time feature is enabled, the algorithm will run an LP detection and an LP reading for each detected vehicle. The raw string of the LP will then be pseudonymized with a so-called hashing mechanism, and the pseudonymized random text will be sent within the standard Counting Line (CL) event over the encrypted network.

In the upcoming section, more details of the single steps are described:

Detect vehicle

In each frame of the video stream, vehicles are detected and classified as cars, trucks, and buses. Alongside this, the vehicle is tracked across the frames of the video stream.

Detect license plate

For each classified vehicle, the license plate is detected and mapped to the object.

Read license plate

For each detected license plate, an optical character recognition (OCR) is applied to read the plate. The output of this part is a text which includes the raw string of the license plate.

Pseudonymize the license plate (Hashing)

In order to hash the LP, a salt shaker generates random salts in the backend (Cloud) and distributes the salts to the edge devices. A salt is random data that is used as an input to hash data, for example, passwords or in our case LPs. The salt will not be saved in the backend. The only point, where the salts are temporarily stored is in the edge device (Perception box).

In order to increase the safety of potential attacks, the salt has a validity window of 12 hours. After the validity window, a new randomly generated salt will be used. The graphic below illustrates an example of the hashing function used for LPs.

Salts 1-4 are generated by the salt shaker and distributed to each edge device. In order to always detect all journeys, each LP is hashed with two salts. Two salts are needed, as a journey could potentially have a longer travel time than the salt validity time. In the upcoming section, match event on possible journeys, it is shown why two salts per LP are needed.

Trigger and send event

If the vehicle crosses a counting line (CL), a CL event with the hashes (h) from the detected LP is sent via MQTT to the Cloud (Microsoft Azure) and saved in a structured database (DB).

Match event on possible journeys

On the cloud, the DB is regularly checked for possible matches within the hashes. As shown above, two hashes are created per detected vehicle. If one of the two hashes is the same for two different detections it will be saved as a journey with the journey time information, class, edge device names & GPS coordinates of the edge device.

In case the same hash is found in several locations, a multi-hop journey will be saved based on the sorting of the timestamps. (e.g.: Journey from location A to B to C)

Anonymize data

After 12h, which is the validity time of the salt used for pseudonymizing the license plate, the pseudonymized LP will be deleted. This action makes the pseudonymized data anonymized. In summary, it means, that after 12 hours past the detection of the vehicle and LP all data are anonymized.

This guide focuses on specific details to be considered for journey times and area-wide traffic flow on public roads, focusing on camera placement, camera settings, and event trigger configuration.

Camera placement

Perfect camera placement is critical in order to get a clear image and readable number plates. While some parameters such as distance from the camera to the number plate can be fine-tuned by zooming after installation, mounting height and angle between the camera and travel direction of vehicles can only be adjusted by physical and cost-intensive re-arrangement. The camera position has to be chosen in a way that passing vehicles are fully visible and can be captured throughout several frames of the video stream while making sure the number plates are large enough for the ANPR system to identify every single character.

We recommend mounting heights between 3 and 8 meters, therefore the suitable minimum capture distance ranges from 5 to 14 meters. Besides the vertical angle constraint, number plates should be visible with at least 250 pixels-per-meter (PPM), this constraint determines the minimum focal length (zoom) the camera has to be set to.

Why between 3 and 8 meters of camera mounting height?

The lower bound of 3 meters is determined by rather practical reasons and not technical limitations. Cameras mounted lower than 3 meters are often prone to vandalism. Also, headlights from passing vehicles can lead to reflections on the camera. The upper bound of 8 meters is determined by the resulting minimum capture distance of at least 14 meters for the needed camera resolution of 1920x1080p. License plates need to be visible with 250 pixel-per-meter (PPM).

As the Swarm Perception Box and cameras are mainly mounted on existing infrastructure such as traffic light poles, there are two general options to mount the cameras: side mounting or overhead mounting.

Overhead mounting

When positioning the camera above the vehicles, two lanes can be covered with one sensor.

Consider mounting height (1) and capture distance (2) which determine the vertical angle (3) between the camera and the travel direction of the vehicle. The distance between the center of the lane (4) and the camera determines the horizontal angle (5) between the camera and the travel direction of the vehicle.

Side mounting

When mounting the camera to the side of the road, two lanes can be covered, assuming the horizontal angle between the camera and the travel direction of the vehicles is not exceeding 20°.

Position the camera as close as possible to the side of the road to avoid a horizontal angle larger than 20°. Larger angles can lead to lower accuracy because parts of the number plate can become unreadable. While traveling directions (1) and (2) are the same for both vehicles, horizontal angle (3) is much larger than (4).

Night mode settings

While capturing sharp images during the day with good lighting conditions is relatively easy, low lightning and dark conditions make it a lot more difficult for cameras to deliver readable number plates from moving vehicles. The following section of this guide, therefore, provides an overview to fine-tune your camera to deliver readable number plates in such conditions.

However, the setting of the following parameters strongly depends on the specific camera mounting position and its environment. A light source such as a streetlamp or a vehicle passing on a different lane can send light to the camera sensors and influence the resulting image to a great extent. For this reason, this guide can only provide a general overview of relevant settings and their effect on image quality.

Day/Night Switch

We recommend that the Auto day/night switch mode from the cameras is used. As you can see in the examples below, it is crucial that the camera changes to night mode reliably.

Next to the traffic frequency at given locations, you are wondering about statistics on how long the vehicles take from one to another location and how the traffic flows across your city and municipality. With this solution, you can generate that data with a single sensor solution from SWARM.

What data can be generated?

For this use case, SWARM software is providing you with the most relevant traffic insights - The counts of vehicles including the classification of the SWARM main classes can be covered. On top, you have the opportunity to add a second counting line, calibrate the distance in between and estimate the speed of the vehicles passing both lines. By combining more sensors in different locations the journey time as well as the statistical traffic flow distribution will be generated.

The journey time and traffic flow distribution can be generated for vehicles only (car, bus and truck).

What needs to be considered for a successful analysis?

Environment specification

Accuracy

In this technical documentation, accuracy refers to the penetration rate of a single sensor, which is the percentage of correctly identified license plates divided by the total number of vehicles counted during a ground truth count.

The current penetration rate for this use case is 60%, taking into account different day/nighttimes, weather conditions, and traffic situations. When calculating journey time between two sensors, approximately 36% of journeys are used as the baseline, which is calculated by multiplying the penetration rate of both sensors.

The accuracy is sufficient to generate data that can be used to make valid conclusions about vehicle traffic patterns and journey times.

Hardware Specifications

Adaptive Traffic Control

Adaptive traffic control enables you to interface with hardware devices like traffic controllers using dry contacts. Use cases and benefits:

• 'Smart Prio' System: Prioritise certain traffic classes and ensure fluid traffic behavior in real-time (e.g. pedestrians, bicyclists, e-scooters, heavy traffic).

• Simplify infrastructure maintenance: Replace multiple induction loops with a single Swarm Perception Box. The installation does not require excavation work, and reduces the maintenance effort/costs.

What needs to be considered for a successful analysis?

Environment specification

Hardware Specifications