QR Code Recognition in Streaming Video

The customer has given us the following task: to implement an application for QR code recognition in streaming video. The main condition was a hard limit on time required to process one frame - no more than 20 milliseconds, and recognition accuracy of X%. Another condition, but not so hard, was to work on a Linux system, which is quite a common task for us.

As an input, we were provided with several video clips on which to test our application. However, due to the NDA, we cannot show these videos here. A little that we can tell you about the data is that it is a black and white video at 50 frames per second, where the QR code moves in the direction from right to left.

Experiments and implementation

To work with video, we immediately took OpenCV and decided to test the implementation of an empty processing loop in Python as the first option. After the first launch, we realized that the implementation of this idea would not bring the desired result, and we would spend too much time optimizing the code. It is worth noting that processing took an average of about 300 milliseconds, and this is with the missing processing code!

After this experiment, we immediately decided to use heavy artillery in the form of C++. Many teams think that using C++ for development is very labor intensive, but in practice, developing in C++ is not that difficult, although its standard is very bloated (our apologies to C++ adherents). Also note that if you use the latest features of the language, then writing software becomes much easier.

We divided the implementation of the program into three stages: the basic implementation, the code optimization stage, and the final version.

At the stage of the basic implementation, we wrote almost the same program code in Python, but at the same time we added QR code recognition based on the same OpenCV. On average, a frame was processed in 15 milliseconds, but there were large outliers up to 81 milliseconds, which did not fit into the given task framework.

The major complaints were about the QR code recognition code. Applying various approaches, we were faced with the fact that the QRCodeDetector class from the OpenCV library did not cope very well with recognition and did not behave very stable in terms of computing time.

Here we smoothly move on to the stage of code optimization. We decided to abandon QRCodeDetector and use the third-party quirc library. The decision turned out to be right, and we immediately got a good increase in processing speed - on average, it took from 10 to 11 milliseconds per frame, and the recognition accuracy became much higher.

Even although we achieved the result desired by the customer, and completed the main part of the task, we went a little further and improved the recognition accuracy even more. The simplest method by which we have slightly increased the accuracy of QR code recognition is the use of adaptive binarization using the mean metric.

The final version of the application has undergone a few more changes. Among the most significant are support for EAN codes recognition and adapting the application to work as a linux daemon.

Conclusion

We completed the project right on time. The client received an effective QR code recognition tool, which exceeds the initial accuracy requirements. Our software allowed the client to greatly optimize the production process and reduce costs.

With extensive experience in the implementation of non-standard and complex projects, the MST team is ready to develop even the most demanding software solutions to optimize business processes.