Разработка
Last time we told you about QR recognition in video streams. We've made a few more improvements since then. In this new text, we'll unveil more about this project and tell you what has change since the last time.
Logic update
First and foremost, let's look at the revamped functionality.
All the important parameters, both new and old, now live in the configuration file.
Among other critical introductions is frame cropping. From now on, not a whole frame is processed, but only a given part of it. You can set up the borders in the configuration file, because this parameter depends on the camera position.
Earlier, you had to equalize a histogram for each frame. But this was changed in this version. One of the key reasons for this change was that after cropping, the QR falls into a well-lit area, so there is just no point in equalizing a histogram. Moreover, it only worsens recognition accuracy.
But all these are slight changes. The biggest thing done is implementing a chain of recognition algorithms. The algorithms in the chain are sorted from the fastest to the slowest and shown in the figure.
Let's go in order and look at all the frame processing stages. At the very beginning, we pass the frame to the Quirc library, which is very fast and has a good recognition accuracy. If it can't recognize the QR code, it sends us the QR code location. After that, we start processing the small frame.
During small frame processing, we cut the QR code out of the frame and use zooming to slightly reduce the image size. Practice has shown that QR codes are worse recognized on FHD frames than on frames that are half as large. This is how we increase the number of recognized QR codes by a couple of percent. Of course, the scaling operation is quite specific and doesn't play out well every time. This is why it's not set up in the configuration file as well.
Affine transformations are then applied to the frame, along with small frame block algorithms: Quirc, zbar, WeChat QR.
At this stage, the QR code is usually recognized by one of the algorithms, and the recognition operation ends. Statistics shows that around 95% of frames are easily processed by the first two algorithms. But if there is no success here, we invoke the heavy guns, a full frame block.
The full frame stage mainly comes down to the WeChat QR library and two algorithms from OpenCV. WeChat QR is also located in the OpenCV library, but since the former is based on neural network methods, it's worth considering it a standalone library. At this stage, a frame rarely reaches the last algorithm, OpenCV decodeCurved. But even if it does, the QR code must be heavily distorted and processing it with the last method will take a while, around 250 ms.
Alongside these algorithms, the customer also wanted to special processing of a frame queue. One of the main settings is queue throttling. If there are more than 50 frames in the queue, we start queuing every third frame until handlers take all items from the queue. If the queue grows more than a certain threshold value (this value is also set in the configuration file), we stop adding new frames altogether.
Having implementing all this, we have significantly increased the QR recognition in video streams. But everything was not so rosy. Read along to learn why.
OpenCV is key
I think it's barely possible to have everything under control when you develop a sophisticated application in C++, especially one operating in multi-threaded mode. We couldn't do without segfaults here. But the weirdest thing is that they weren't on our application's side, but in the OpenCV library.
Here begins our detective story of finding bugs in OpenCV.
It all started with the analysis of the crash damp that led us to the divideIntoEvenSegments, function. More specifically, to this code fragment:
float temp_dist = distancePointToLine(*it, segment_start, segment_end);
if (temp_dist > max_dist_to_line)
{
max_dist_to_line = temp_dist;
}
Problem #1. The distancePointToLine function returns NaN.. Having entered this function we realized that division by zero was taking place, since an incorrect comparison of float with zero had been implemented there.
float QRDecode::distancePointToLine(Point2f a, Point2f b , Point2f c)
{
float A, B, C, result;
A = c.y - b.y;
B = c.x - b.x;
C = c.x * b.y - b.x * c.y;
float dist = sqrt(A*A + B*B);
if (dist == 0) return 0;
result = abs((A * a.x - B * a.y + C)) / dist;
return result;
}
By fixing this thing, we didn't resolve the main problem, so the app kept crashing. So we continued our investigation and found out that the createSpline function returns lines like this
[-nan, 177], [-nan, 178], [-nan, 179], [-nan, 180], [-nan, 181], [-nan, 182], [-nan, 183], [-nan, 184], [-nan, 185], [-nan, 186], [-nan, 187], [-nan, 188], [-nan, 189], [-nan, 190], [-nan, 191], [-nan, 192], [-nan, 193], [-nan, 194], [165, 177],
...
Diving deeper into the code, we identified the potential , problem area. When y_arr[i + 1] and y_arr[i] are very close (or equal), we get NaN due to further division by h[i].
for (int i = 0; i < n - 1; i++)
{
h[i] = static_cast(y_arr[i + 1] - y_arr[i]);
}
for (int i = 1; i < n - 1; i++)
{
alpha[i] = 3 / h[i] * (a[i + 1] - a[i]) - 3 / (h[i - 1]) * (a[i] - a[i - 1]);
}
We were happy at that moment, seeing our application no longer crash. But how naive we were.
While preparing the app container, we decided to check how the app operated inside the container. And that was the right decision. A segfault was waiting for us there. Here's the most interesting part. The application runs smoothly outside the container, but crashes inside.
We had to do some workarounds and get a crash dump inside the container. After we reviewed the dump, we found out that the problem lay in the straightenQRCodeInParts function.
#0 0x00007f8cbc799f45 in cv::QRDecode::straightenQRCodeInParts() () from /lib/x86_64-linux-gnu/libopencv_objdetect.so.406
We had to spend some time poking around in the code, we came to this:
for (int i = 0; i < NUM_SIDES; i++)
{
Point2f temp_point_start = segments_points[i].front();
Point2f temp_point_end = segments_points[i].back();
The code may seem good. It just gets the first and the last element. But the most interesting part starts when segments_points or segments_points[i] doesn't contain any elements.
Having added a check, we finally defeated that segfault!
If you're wondering what OpenCV version we used, it was 4.6.0. For the curious, here's the commit hash: b0dc474160e389b9c9045da5db49d03ae17c6a6b.
Containerization
Since we're using a patched version of OpenCV, a question arose of how to present the application to the customer. Without thinking too much, we decided to use docker.
That stage was way easier that the last two. There's nothing to describe, but I still want to say a few words.
We took the ubuntu:latest container as a basis, added the patched OpenCV libraries, and introduced all the other dependencies through apt-get. Of course, we put the application in the container, with the entrypoint registered.
We only need to create a service file for systemd, so that it's easy to start the container with the necessary flags (forwarding the configuration file to the container, log directory, network access), and the application is ready to go.
Отзыв клиента
В июле 2019 года компания ООО «Аэро-Трейд» искала подрядчика для выполнения работ по внедрению системы безналичной оплаты товаров самолетах авиакомпании «AZUR air». После тщательного изучения рынка, мы решили обратиться для реализации данного проекта в компанию ООО «МСТ Компани». Основной задачей было обеспечить каждый борт авиакомпании «AZUR air» терминалом, который сможет принимать платежи не только на земле, но и во время полёта.
В течении всего времени нашего сотрудничества, специалисты ООО «МСТ Компани» продемонстрировали отличные профессиональные навыки при подготовке проекта, и разработке документации. В результате мы получили гибкое и надёжное решение, которое удовлетворяет нашим требованиям.
По итогам работы с компанией ООО «МСТ Компани» хочется отметить соблюдение принципов делового партнерства, а также четкое соблюдение сроков работ и выполнение взятых на себя обязательств. ООО «Аэро-Трейд» выражает благодарность специалистам компании за проделанную работу в рамках внедрения системы безналичной оплаты на самолетах авиакомпании «AZUR air». И рекомендует компанию ООО «МСТ Компани» как надёжного партнёра в области платёжных решений.
