Recognize the pattern in images into list

[fbeleznay]

I am new to image processing, so this may not be an appropriate question here. If this is the case, I apologize.

Can this extension be used to create a tool similar to a QR code scanner (but without the guiding squares in the corners)?

More specifically, I would like a tool that can recognize the pattern in images like the one below.

I need a binary list of lists, for example

{

{0,0,0,0,1,1,0,1,1,0,1,1,0,0,0,0},

{0,0,0,0,1,1,0,1,1,0,1,1,0,0,0,0},

{0,0,1,1,1,1,1,1,1,1,1,1,1,1,0,0},

…

}

for the image above.

[Kevinkun]

If you already have this image on your phone, you don't even need an extension. you can load this image to canva, with the methods in canvas, you can achieve this.

and is the image width/height/columns/rows is fixed? that will be more easier.

[fbeleznay]

No, the image dimensions, and more importantly the square size is not fixed.

I can do this with a bit of manual adjustment (crop to get rid of the text from the bottom and manually set the square size), I was hoping that with a bit of image processing these can be automatic.

I do not know how I can automatically detect the text on the bottom using canvas.

I do not know how I can use canvas operations to detect the height of the rows.

[Kevinkun]

please send more images, like 10.

[fbeleznay]

Here are a few, more can be found on handweaving.net

[Kevinkun]

in your images, no. 1,2,3,7,8,9,10 with same grid size 11x11,
I guess no 4,5,6 orignal images with same grid size 11 x 11 also, but the image is resized/compressed when uploaded here?

[fbeleznay]

No, the images are not resized, they are simply downloaded from the handweaving.net website.

Indeed, most images on the website have grid size 11x11, but not all.

The file name contains the grid size, but I would not like to rely on it, because I am after a solution that only uses the information in the image itself (in case the file is renamed).

[ABG]

Those images look sharp.

Are they SVG?

SVG is based on text, which you can try to parse .

[fbeleznay]

These are png images, not svg.

[Kevinkun]

you need an extension to get color of one point/pixel of original image, not on AI2 canvas.

[TIMAI2]

@fbeleznay

I need a binary list of lists

I have created an example aia project using the canvas to do this using the #8776 design, I cropped off the caption at the bottom. This was made easier because #8776 was square with 32 x 32 "boxes", and only had two distinct colours. I used a canvas with a size of 320 x 320 pixels, meaning each box was 10 pixels, and placed the design image as the canvas background.

I then generated a list of box centre coordinates, used this to get the unique colours on the design, then iterated over the box centres again to set each box as a 0 or 1.

Blocks

canvasHandweave.aia (7.6 KB)

This example eliminates many of the variables that you can have with your designs, the aia can become much more complicated!

[fbeleznay]

Yes, this is the functionality I am after, but I would like to do this in an automated way.

I guess, this is what AI is for (after all the visual inspection of the patterns clearly shows the structure, an artificial intelligence should be able to reconstruct this human visual inspection). Also, qr code scanners do very similar things. I just need to experiment how to do it.

For example, for the crop, I was thinking about using the ocr tool of the openCV extension, but it would be good if it would return not only the text, but also some information on where it found the text.

After cropping, I can use the extension to convert the image to grayscale and then the threshold tool to get a binary black and white version of the pattern.

To get the grid size, I was thinking about several approaches.

One is to resize the image with different values and then resize it back without any filter (to get a sharp bitmap pattern). If the original image is close to the result, then the resize value is the grid size. Unfortunately, I do not see any options on what resize method the openCV extension use, so I do not know how to set that I do not need for example the cubic approximation. Also, I do not see any options for comparing two images. Maybe the erode and dilate functions instead of the resize can be helpful, I need to experiment.

The other option I had in mind is to find squares in the contour. I need to experiment with this, I do not know what is the result of GetAllContours and ContourFitMinRotatedRect or ContourFitMinCircle (to be honest, I do not know what it means to fit a minimum rectangle or circle, I would think that the largest circle or rectangle is more interesting).

A third approach is to look for vertical lines in the result of GetAllContours or the ToCanny edge detection (I am not sure how to do this, though, I need to experiment) and see how far these vertical lines are.

All in all, I need to experiment.

[TIMAI2]

Additional work:

Crop and Read: takes the same image: #8776, crops the design and the caption to images, then ocrs the caption to return the text. Uses the OCR.SPace extension by @Marco_Perrone. You will need to get an api key from here - free registration and use.

I also use my ImageConvertor extension. If you really think you need to grayscale the design, i can probably add that feature.

Blocks

cropAndRead.aia (44.5 KB)

If you want to avoid using extensions, you can use the canvas to do the cropping and web component with ocrspace.

I have now added a grayscale feature to ImageConvertor:

[TIMAI2]

Knowing you like to do things without extensions:

OCR Images and PDFs without extensions (ocr.space)

OCR Images and PDFs without extensions (ocr.space) An image file with text: [Screenshot_20260629-162506] A 3 page pdf: [Screenshot_20260629-162441] You can use @Marco Perrone's excellent OCR extension with ocr.space, but if you cannot or will not use extensions, here is a method to use ocr.space using only native components. Requirements: An online http / php server with storage space for file uploads. A php file to handle file uploads. Get a free ocr.space API key (Free OCR API). Rest…

I tested it with this image and it worked fine to return the text:

Handweaving.net #8776

[hotatp]

@TIMAI2

方框

Your reply gave me some inspiration. Without altering the image, I tried determining the box's dimensions by counting the number of consecutive pixels in a single row or column, and it actually worked out. Along the way, I noticed a one-pixel black border around the box. I decided to use this border as the boundary for counting and just return a fixed value, which saved me from worrying about the box's fill color. However, testing showed that this didn't yield a unique value, since the border doesn't always exist. Fortunately, it's pretty easy to figure out the correct value.

Once I got the box dimensions, the rest of the work became much easier with your tips. I then tried reconstructing the image using the extracted list. The reconstructed image made it really easy to spot the edges, which allowed me to effortlessly remove the unwanted parts.

The three buttons function as follows: 1. Get box dimensions, 2. Extract list, and 3. Reconstruct image.

There are a few test images to choose from in the list. While smaller images work fine, oversized images fail to display properly on the canvas, and the extracted list is also incorrect.

Canvas_To_List.aia (58.6 KB)

[ABG]

Could you post an English version of the blocks?
It looks interesting.

[hotatp]

Canvas_To_List_en.aia (10.6 KB)

[TIMAI2]

METRIC RAT AI2 - Canvas Image to Foreground