WO2025053827A1

WO2025053827A1 - A robotic system that can detect floor defects intended to be used for robotic systems and display them on the created map and a method for the operation of the said system

Info

Publication number: WO2025053827A1
Application number: PCT/TR2024/051039
Authority: WO
Inventors: Kemal Eren SERBETCİ
Original assignee: Ara Robotik Ve Yapay Zeka Teknolojileri AS
Current assignee: Ara Robotik Ve Yapay Zeka Teknolojileri AS
Priority date: 2023-09-06
Filing date: 2024-09-06
Publication date: 2025-03-13
Anticipated expiration: 2026-03-06

Abstract

The invention relates to a system and method that warns the user by detecting floor problems such as scanning the existing floor during normal walking/operation, predefined or newly encountered fractures, cracks, color differences, liquid ponding, uneven floor, separation, bending, protrusion and indentation, etc. by integrating on an existing autonomous mobile industrial robot platform and marking it with different visuals/signs on the digital map created by the mobile autonomous robot platform.

Description

A ROBOTIC SYSTEM THAT CAN DETECT FLOOR DEFECTS INTENDED TO BE USED FOR ROBOTIC SYSTEMS AND DISPLAY THEM ON THE CREATED MAP AND A METHOD FOR THE OPERATION OF THE SAID SYSTEM

Technical Field of the Invention

The invention relates to a system and method that warns the user by detecting floor problems such as scanning the existing floor during normal walking/operation, predefined or newly encountered fractures, cracks, color differences, liquid ponding, uneven floor, separation, bending, protrusion and indentation, etc. by integrating on an existing autonomous mobile industrial robot platform and marking it with different visuals/signs on the digital map created by the mobile autonomous robot platform. The main function aimed by the invention is to perform the function of autonomously scanning the existing floor of closed spaces and determining the above-mentioned floor problems before they cause any serious problems and warning the relevant unit. It is the method that allows the operation of this system and the system that includes electronic processors and equipment that will automatically move with the lidar and camera units on the autonomous device, scan and map the environment, determine a route accordingly, and the floor scan and examination subject to the invention are artificial intelligence supported cameras, electronic processors, and equipment to process the images from these cameras. This complete system will be able to identify the problems on the floor with the cameras on the obstacles/changes, process them on the map and record them.

State of the Art

Mobile autonomous robots are already used in open/closed spaces around the world. These robots have different functions and almost all of them work on maps that are preloaded or autonomously created by themselves.

When it comes to commercial/industrial area sizes, it is a fact that the size, diversity, and detection of any floor problems take a lot of time. Timely and accurate detection of problems such as fractures, cracks, color differences, liquid ponding, uneven floor, separation, bending, protrusion, and indentation in commercial/industrial floors is carried out with current workers, which is a waste of time and can sometimes lead to serious problems caused by human errors. Uneven floors pose a physical danger in the working areas and create discomfort for visitors in the indoor navigation areas.

They will be able to detect, process and report floor quality and problems with hardware/software support that can be added to robots that are already working for indoor floor maintenance, internal transportation or another function and can create maps autonomously.

When similar patent studies conducted so far are examined, although criteria such as cleaning type detection, pollution detection, obstacle, hole, crack, or gap detection have been highlighted according to the floor, there is no issue of visually processing them, showing them on a created map and warning the user.

Problems to be Solved by the Invention

The object of the invention is to create a hardware and software that warns the user by detecting floor problems such as scanning the existing floor during normal robotic walking/operation, predefined or newly encountered fractures, cracks, color differences, liquid ponding, uneven floor, separation, bending, protrusion and indentation, etc. that can be integrated on an existing autonomous mobile industrial robot platform and marking it with different visuals/signs on the digital map created by the mobile autonomous robot platform.

The main function aimed by the invention is to perform the function of autonomously scanning the existing floor of closed spaces and determining the floor problems before they cause any serious problems and warning the relevant unit. It is the method that allows the operation of this system and the system that includes electronic processors and equipment that will automatically move with the lidar and camera units on the autonomous device, scan and map the environment, determine a route accordingly, and the floor scan and examination subject to the invention are artificial intelligence supported cameras, electronic processors, and equipment to process the images from these cameras. This complete system will be able to identify the problems on the floor with the cameras on the obstacles/changes, process them on the map and record them.

The preferred embodiment of the invention describes the autonomous robotic floor recognition, detection, and processing equipment for ensuring the integrity, ease of use and safety of large floors, especially in commercial/industrial areas, and reducing floor-related occupational accidents.

In order for the invention to be applied on all kinds of floors, artificial intelligence supported cameras, processors with high graphics processing power, special cabling and connections, specially developed detection, discrimination algorithms and software are used.

The camera + processor + algorithm + software of the invention, detects the problems on the floor autonomously without the need for the user to take any additional action, and if it is necessary to avoid them in order to ensure the normal operation of the autonomous robot, it notifies the software that controls the driving mechanism of the robot, and at the same time processes all the floor problems it detects on the created map.

When the system subject to the invention encounters a floor problem that may prevent the robot from performing the task autonomously, it is also possible to calculate the location information and possible directions of the problem required to avoid them and report them to the software that controls the driving mechanism of the robot.

For example, if the autonomous robot is programmed to perform indoor cleaning, if there is a floor problem such as fracture, crack, height difference, color difference or liquid accumulation, etc. that is not in line with the previously entered system information, if the direct passing of the robot is a problem, a new temporary map information is calculated in line with the problem location information from the cameras and sensors and sent to the main navigation process system of the robot and an instant change/avoidance maneuver is ensured. In this way, both the physical integrity and the way the autonomous robot works are preserved, and the necessary information can be provided on the map by fully perceiving the problem on the floor. In addition to the existing robotic mechanism, the floor problems visual detection system can also be operated autonomously in such a way that it can perform different functions on its own thanks to the algorithm + software and camera equipment previously defined in the system without human help.

The main purpose of this camera and algorithm is to provide instant guidance against the problems that may arise from the existing autonomous robot, without disrupting and preventing its function, as well as to mark the problems that have occurred by constantly checking the integrity and suitability of the existing floor on a map and to digitally convey the warnings to the user. In this way, floor-based work accidents can be prevented, a safe environment can be provided for visitors, and since any robot that already roams autonomously on the floors is used, too much manpower and valuable work time will be prevented from being spent for floor control.

Description of the Figures

Figure 1. A schematic side view of the robotic mechanism

Description of References in Figures

1. Floor

2. Body

3. Camera

4. Cleaning element

5. Movement element

Description of the Invention

The invention relates to a system that allows the recognition and mapping of the defects of the floor (1) by using a variety of cameras (3) and artificial intelligence algorithms for robotic mechanisms aimed at advancing on a floor (1).

The preferred embodiment of the invention relates to a floor (1) defect recognition and mapping system designed especially for robotic floor cleaners. The system uses high- resolution cameras (3) and advanced artificial intelligence algorithms to detect and map a variety of floor (1) defects, such as cracks, bumps, and spots. The robotic floor cleaner can navigate efficiently, avoid obstacles, and perform effective cleaning operations by accurately identifying and mapping the floor (1) defects. This invention provides autonomous floor cleaning with improved performance and efficiency.

The robotic body (2) is equipped with at least one camera (3) positioned in front of the robot in the direction of movement and at least one camera (3) positioned behind the robot in the direction of movement.

In the event that the robotic mechanism of the invention is a robotic floor cleaner, the body (2) may also be equipped with at least one cleaning element (4) and at least one movement element (5).

For this purpose, a high-resolution camera (3) array is used within the preferred embodiment of the invention.

The system of the invention is also equipped with an operating data processing and analysis module related to said camera (3) arrays, a fault recognition module, a mapping and localization module, a navigation and cleaning optimization module, a real-time monitoring and updates module, a user interface module, and an integration module with a robotic floor cleaner.

The system of recognizing and mapping the floor (1) defects comprises an array of high- resolution cameras (3) designed to capture detailed images of the surface of the floor (1). The array of cameras (3) includes a plurality of high-resolution cameras (3) strategically positioned on the robotic floor cleaner (1) to ensure a comprehensive coverage of the floor area.

Data Processing and Analysis module:

The images captured from the array of cameras (3) are processed and analyzed using advanced artificial intelligence algorithms to identify and classify the various floor (1) defects. Algorithms use machine learning methods such as convolutional neural networks (CNNs) and deep learning to extract meaningful features from images and make accurate defect predictions.

Fault recognition module:

The system uses a multi-step defect recognition process to detect and classify floor defects. The stages include:

A. Image Preprocessing: Captured images undergo preprocessing methods, including noise reduction, image filtering, and color normalization, to improve image quality.

B. Feature Extraction: Relevant features such as texture, color variations, edges, and patterns are extracted from preprocessed images.

C. Classification: The extracted features are fed into artificial intelligence (Al) algorithms that are trained to accurately classify the floor defects. The classification results include categories such as cracks, bumps, spots, and other surface abnormalities.

Mapping and Localization module:

The system uses data from the camera array and the defect recognition process to create a comprehensive map of the floor, including the exact locations and types of defects detected. The mapping method combines captured images with simultaneous localization and mapping (SLAM) algorithms to ensure accurate positioning of the robot cleaner and efficient navigation around floor defects.

Navigation and Cleaning Optimization Module:

The generated floor defect map is used to optimize the cleaning path of the robotic floor cleaner. The navigation algorithm combines the defect map to plan the most efficient way to prevent detected defects. The robot cleaner can perform effective and efficient cleaning operations by avoiding obstacles and focusing on areas that require cleaning.

Real-Time Monitoring and Updates Module: The system provides real-time monitoring of the floor surface during the cleaning process. As the robotic floor cleaner moves, high resolution cameras continuously capture images and update the defect map, allowing dynamic adjustments in the cleaning path to account for newly detected defects.

User interface module:

The floor defect recognition and mapping system includes a user interface that allows users to view the generated defect map, monitor cleaning progress, and customize cleaning preferences. The user interface can be implemented through a viewing screen or a mobile application that provides a user-friendly experience.

Integration Module with Robotic Floor Cleaner:

The system for recognizing and mapping floor defects is designed to integrate with existing or new robotic floor cleaners.

The system can be seamlessly incorporated into the robot cleaner's control unit, enabling autonomous operation, and improving cleaning performance by efficiently detecting and mapping floor defects.

The invention also defines a method of operating a system that allows the recognition and mapping of the defects of the floor (1) by using a variety of cameras (3) and artificial intelligence algorithms for robotic mechanisms aimed at advancing on a floor (1).

This method, in its most basic form, comprises the following steps:

• collecting the image or video stream of the surface of the floor (1) for examination with the camera (3) arrays positioned on the robotic body (2),

• using the navigation map to mark the detected defects,

• capturing a frame from the input image or video stream,

• performing preprocessing steps such as resizing, normalization or color space conversion on the captured frame, • processing and sorting and evaluation of the preprocessed image/frame in the defect recognition model,

• identifying, for each detected defect, its location in the image/frame and matching the defect or finding location with the corresponding location on the navigation map,

• updating the relevant location on the navigation map to indicate the presence of a defect/finding,

• repeating the process within these frames if there are more frames or images to process,

• sending the navigation map to the user and updating the navigation map when the process is complete for the whole map or the marked section, highlighting the marked defect locations.

The map used during the implementation of the method described above may have been created by the robotic mechanism or may be the navigation map created during this process.

A pretrained artificial intelligence model can be used for the error/defect recognition described above. Accordingly, the navigation map is adjusted so that it has the same virtual dimensions as the floor surface. At this stage, the calibration and settings of the camera that will read and examine the floor are made. A threshold value for defect detection confidence is determined.

The preprocessed image/frame is processed in the defect recognition model and sorted and evaluated. Obtaining estimates for a variety of defects/findings, such as color change, cracks, height differences, discoloration, bumps, and ponding, and filtering estimates below the confidence threshold.

More specifically, the captured image goes through the following steps:

• pre-processing methods, including noise reduction, image filtering, and color normalization, to improve image quality,

• extracting relevant features such as texture, color variations, edges, and patterns from preprocessed images,

• feeding the extracted features to artificial intelligence algorithms trained to accurately classify ground defects and categorizing the classification results into cracks, bumps, stains, and other surface anomalies. For each detected defect, its position in the picture/frame is defined. The defect or finding position is matched to the corresponding position in the navigation map. The relevant location on the navigation map is updated to indicate the presence of a defect/finding.

A report summarizing the findings is generated when the entire map or marked section is completed. Relevant details such as error type, location, and severity can be added.

It sends the navigation map to the user by highlighting the marked defect locations. Defect/finding icons have been predetermined and added to the visual library. The system provides a visual representation of the floor surface with obvious defects. The created report is presented to the user visually or as a downloadable file.

This algorithm summarizes the basic steps for examining and recognizing floor defects, marking them on a navigation map, and reporting the results to the user. Specific application details, such as error recognition model selection or visualization methods, may vary depending on the application and available resources.

Floor defects/errors can be shown to the user on the map, or the map can be updated autonomously by the system and the robotic mechanism can be operated by considering the defects/errors.

Claims

1. A robotic system for robotic mechanisms aiming to move on a floor (1), which allows the recognition and mapping of defects of the floor (1) using various numbers of cameras (3) and artificial intelligence algorithms, characterized in that it comprises a robotic body (2) equipped with at least one camera (3) positioned in front of the robot in the direction of movement and at least one camera (3) positioned behind the robot in the direction of movement.

2. The robotic system according to claim 1, characterized in that in case the robotic mechanism is a robotic floor cleaner, the body (2) is further provided with at least one cleaning element (4) and at least one movement element (5).

3. The robotic system according to claim 1, characterized in that it comprises a high- resolution camera (3) array.

4. The robotic system according to claim 3, characterized in that it is equipped with a data processing and analysis module, a fault recognition module, a mapping and localization module, a navigation and cleaning optimization module, a real-time monitoring and updates module, a user interface module, and an integration module with a robotic floor cleaner, operating in conjunction with said camera (3) arrays.

5. A method of operating a robotic system for robotic mechanisms intended to move on the floor (1), allowing the recognition and mapping of defects in the floor (1) using various numbers of cameras (3) and artificial intelligence algorithms, characterized in that it comprises the following process steps,

• using the navigation map to mark the detected defects,

• capturing a frame from the input image or video stream,

6. The method for operating the robotic system according to claim 5, characterized in that the map used during its implementation is the navigation map, which is pregenerated by the robotic mechanism or generated during this process.

7. The method of operating the robotic system according to claim 5, characterized in that it comprises the following process steps:

• adjusting the navigation map to have the same virtual dimensions as the floor surface,

• performing calibration and settings of the camera that will read and examine the floor at this stage,

• determining a threshold value for defect detection confidence,

• processing and sorting and evaluation of the preprocessed image/frame in the defect recognition model,

• obtaining estimates for a variety of defects/findings, such as color change, cracks, height differences, discoloration, bumps, and ponding, and filtering out estimates below the confidence threshold,

• using a pretrained artificial intelligence model for error/defect recognition.

8. The method of operating the robotic system according to claim 5, characterized in that it comprises the following process steps: • pre-processing methods including noise reduction, image filtering, and color normalization, to improve image quality,

• feeding the extracted features to artificial intelligence algorithms trained to accurately classify ground defects and categorizing the classification results into cracks, bumps, stains, and other surface anomalies.

9. The method of operating the robotic system according to claim 5, characterized in that it comprises the process step of operating the robotic mechanism considering the defects/errors by autonomously updating the map as well as displaying the defects/errors to the user on the map.