CN116414112A - Control method of self-mobile device, self-mobile device and storage medium - Google Patents

Abstract

An embodiment of the present application provides a control method of a self-moving device, a self-moving device and a storage medium. In the embodiment of the present application, contour extraction is performed on an environmental image, and reflection detection is performed on an inner region of the extracted contour to be detected. Then automatically, accurately and quickly identify water surface areas based on vision technology. Further, based on the actual position information of the contour to be detected including the reflection in the geographical scene, the self-mobile device is controlled to avoid the water surface area under the geographical scene during the traveling process, thereby reducing the probability of the self-mobile device being damaged by wading. Furthermore, there is no need to arrange physical boundaries to help the self-mobile device identify the water surface area, and this method does not need to consume time and energy, and the cost is low.

Description

Self-moving device control method, self-moving device and storage medium

technical field

The present application relates to the technical field of artificial intelligence, and in particular to a control method of a self-moving device, a self-moving device and a storage medium.

Background technique

With the continuous development of computer technology, sensor technology and artificial intelligence technology, more and more outdoor robots that provide different services have emerged, such as smart lawn mowers, smart inspection robots, smart killing robots, smart Handling robots and intelligent outdoor cleaning robots. Compared with indoor robots, the working area environment of outdoor robots is more complicated, and there will be more dangerous areas such as swimming pools and ponds. This requires outdoor robots to avoid these dangerous areas during work and prevent abnormal interruption of tasks.

In the existing technology, magnetic strips, metal wires, magnetic nails, etc. are laid around the dangerous area in advance to form the physical boundary of the dangerous area. When the outdoor robot detects the physical boundary of the dangerous area, it confirms that there is a dangerous area ahead and executes avoiding the dangerous area. Actions. This method needs to lay physical boundary lines in advance, which increases work intensity and reduces work efficiency. And due to the complexity of the outdoor environment, water accumulation areas may appear in the working area at any time, but because the physical boundary is not arranged in advance, the outdoor robot cannot recognize the water accumulation area, resulting in damage to the outdoor robot.

Contents of the invention

Various aspects of the present application provide a control method of self-moving equipment, self-moving equipment and storage media, which do not need to lay physical boundary lines, and can avoid water surface areas during travel, thereby reducing self-mobile equipment wading probability of damage.

An embodiment of the present application provides a control method of self-mobile equipment, which has an image acquisition device, and the method includes: acquiring an environmental image collected by the image acquisition device, and the environmental image is an image in the direction of travel of the self-mobile equipment; Extract the contour of the image to obtain at least one contour to be detected; perform reflection detection on the inner area of the contour to be detected to determine whether there is a reflection in the contour to be detected; when there is a reflection in the contour to be detected, obtain the contour to be detected that has a reflection The actual location information in the scene can control the movement of the self-mobile device according to the actual location information, so as to avoid the water surface area in the geographical scene.

The embodiment of the present application also provides a self-moving device, including: a vehicle frame, a walking device, an image acquisition device, a memory, and a processor, wherein the walking device is arranged on the vehicle frame to drive the self-moving device to walk; the image acquisition device It is arranged on the vehicle frame; the memory is used to store computer programs; the processor is coupled to the memory to execute the above-mentioned control method of self-moving equipment.

An embodiment of the present application further provides a storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, the processor is caused to implement the above method for controlling the mobile device.

In the embodiment of the present application, the contour is extracted from the environment image, and the reflection detection is performed on the inner area of the extracted contour to be detected, and then the contour to be detected with reflection is automatically, accurately and quickly identified in the geographical area based on vision technology. The actual location information in the scene. Further, based on the actual position information of the contour to be detected including the reflection in the geographical scene, the self-mobile device is controlled to avoid the water surface area under the geographical scene during the traveling process, thereby reducing the probability of the self-mobile device being damaged by wading. Furthermore, there is no need to arrange physical boundaries to help the self-mobile device identify the water surface area, and this method does not need to consume time and energy, and the cost is low.

Description of drawings

The drawings described here are used to provide a further understanding of the application and constitute a part of the application. The schematic embodiments and descriptions of the application are used to explain the application and do not constitute an improper limitation to the application. In the attached picture:

FIG. 1 is a schematic diagram of an application scenario provided by an exemplary embodiment of the present application;

FIG. 2 is a schematic flow chart of a control method of a self-mobile device provided in an exemplary embodiment of the present application;

FIG. 3 is an exemplary schematic diagram of path planning;

FIG. 4 is a geometric relationship diagram between key contour points and moving distances provided by an exemplary embodiment of the present application;

FIG. 5 is a schematic diagram of a horizon in an exemplary environment image;

Fig. 6 is an exemplary pinhole imaging model;

Fig. 7 is an exemplary environment image;

Fig. 8 is a schematic structural diagram of a mobile device provided by an exemplary embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and corresponding drawings. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

An embodiment of the present application provides a control method of a self-moving device, a self-moving device, and a medium. In the embodiment of the present application, contour extraction is performed on an environment image, and reflection detection is performed on the inner area of the extracted contour to be detected, and then Automatically, accurately and quickly identify water surface areas based on vision technology. Further, based on the actual position information of the contour to be detected including the reflection in the geographical scene, the self-mobile device is controlled to avoid the water surface area under the geographical scene during the traveling process, thereby reducing the probability of the self-mobile device being damaged by wading. Furthermore, there is no need to arrange physical boundaries to help the self-mobile device identify the water surface area, and this method does not need to consume time and energy, and the cost is low.

The technical solutions provided by various embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

Fig. 1 is a schematic diagram of an application scenario provided by an exemplary embodiment of the present application. The application scenario shown in Figure 1 is that the smart lawnmower performs mowing work outdoors. Due to the complexity of the outdoor environment, in addition to objects such as stones, trees, and grass that need to be mowed, there are also dangerous areas such as ponds, swimming pools, puddles, and ditches in the outdoor environment. Therefore, the smart lawn mower needs to accurately identify whether the area ahead is a water-logged area during mowing operations, so as to reduce the probability of wading damage.

It is worth noting that the smart lawnmower shown in Figure 1 performing mowing operations outdoors is only a schematic application scenario, and the control method from the mobile device provided by the embodiment of the present application can be applied to indoor and outdoor scenarios , such as intelligent inspection robots performing inspection tasks outdoors or indoors, intelligent killing robots performing killing tasks outdoors or indoors, intelligent handling robots performing handling tasks outdoors or indoors, and intelligent outdoor cleaning robots performing cleaning tasks outdoors or indoors Of course, the embodiment of the present application does not limit the application scenario.

The technical solutions provided by various embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

Fig. 2 is a schematic flowchart of a method for controlling an autonomous mobile device provided by an exemplary embodiment of the present application. Referring to Figure 2, the method may include the following steps:

201. Acquire an environment image collected by an image acquisition device, where the environment image is an image in a traveling direction of the mobile device.

202. Perform contour extraction on the environment image to obtain at least one contour to be detected.

203. Perform reflection detection on the inner area of the contour to be detected to determine whether there is a reflection in the contour to be detected. If yes, perform step 204. If not, end. If there is no water surface area in the traveling direction of the mobile device, then the mobile device can continue traveling in the traveling direction.

204. Acquire the actual location information of the contour to be detected with reflection in the geographical scene, so as to control the self-mobile device to travel according to the actual location information, so as to avoid the water surface area in the geographical scene.

In the embodiment of the present application, the self-mobile device uses an image acquisition device to collect an environment image in its own traveling direction during the traveling process. Or the self-mobile device uses the image acquisition device to collect the environment image in the direction of its own travel when it is preparing to travel, for example, the self-mobile device uses the image acquisition device to collect the environment image in the direction of its travel at the starting point of travel. The image acquisition device may be a depth camera. Such as video cameras, cameras, etc. The geographic scene may be an actual usage scene, for example, the mobile scene from the mobile device as shown in FIG. 3 .

In the embodiment of the present application, the image acquisition device can be controlled to perform image acquisition in real time, or the image acquisition device can be controlled to perform image acquisition periodically, and the image acquisition period is set according to actual business requirements.

In the embodiment of the present application, the environment image is processed by a contour extraction algorithm to extract at least one contour in the environment image. For ease of understanding, the extracted contour is referred to as a contour to be detected. In addition, the contour extraction result of the environment image may also include but not limited to the following information: image position information, contour perimeter information, contour height information, and contour shape information of each contour to be detected in the environment image. Further, at least one contour is extracted from the environment image through a contour extraction algorithm. The at least one may be one or more. The plurality may be 2, 3, 4, etc. Taking the example shown in Figure 1, the intelligent lawn mower uses the image acquisition device to collect the environmental image of the front area in real time during the running process. By extracting the outline of the environmental image, the outline of the water surface area, the outline of the stone, the outline of the tree can be identified. The contour of and many other contours.

After contour extraction is performed on the environment image, reflection detection may also be performed on any extracted contour to be detected, so as to determine whether there is a reflection in the contour to be detected. For any contour to be detected in at least one contour to be detected, if it is detected that there is a reflection in the inner area of the contour to be detected, then the contour to be detected is the contour of the water surface area; if it is detected that there is no reflection in the inner area of the contour to be detected, then The contour to be detected is not the contour of the water surface area. Therefore, when there is no reflection in all the contours to be detected, it means that there is no water surface area in the traveling direction of the self-mobile device, and the mobile device can continue to travel in the traveling direction. Specifically, for example, after contour extraction is performed on the environment image, three contours to be detected are extracted. The interior of the three contours to be detected is detected one by one. When one of the three contours to be detected is detected or there is a reflection, it means that there is a contour of the water surface area among the three contours to be detected. In this way, it can be obtained The actual position information of the contour to be detected with reflection in the geographical scene, so as to control the self-mobile device to advance according to the actual position information, so as to avoid the water surface area under the geographical scene; when the 3 contours to be detected are detected When there are multiple reflections in the three contours to be detected, it means that there are multiple contours of the water surface area among the three contours to be detected. In this way, the actual position information of the multiple contours to be detected with reflections in the geographical scene can be obtained respectively, so as to be able to The location information controls the movement of the mobile device to avoid multiple water surface areas in the geographical scene; when it is detected that there is no reflection in the three contours to be detected, it means that there is no contour of the water surface area among the three contours to be detected , which can make the mobile device continue to travel in the direction of travel.

Further optionally, one implementation of step 203 is: extracting multiple first feature points within the contour to be detected; and performing calculations on the multiple first feature points on the environment image to determine the Whether the first feature point has a matching point that is symmetrical with respect to the horizon, and when the first feature point has a matching point that is symmetrical with respect to the horizon, then form a symmetrical point pair with the first feature point and the matching point ; Obtain the number of symmetrical point pairs, and determine that there is a reflection in the contour to be detected when the number of symmetrical point pairs satisfies a second preset value. Further, the plurality may be 2, 3, 4, 5, etc., which is not specified in this application. Of course, the more the number of first feature points, the more sampling points available for judging whether there is a reflection, and thus the higher the accuracy for judging whether there is a reflection. However, the larger the number of first feature points, the more calculation amount will be brought, so the number of first feature points can be selected according to actual needs.

In the embodiment of the present application, the matching point of the first feature point is a feature point symmetrical to the first feature point with respect to the horizon, and the first feature point may be considered as a reflection of the corresponding matching point. The second preset value is set according to actual business needs. If the number of symmetrical point pairs is greater than or equal to the second preset value, it is determined that the number of symmetrical point pairs meets the second preset value; if the number of symmetrical point pairs is less than the second preset value value, it is determined that the number of symmetrical point pairs does not meet the second preset value.

If it is counted that the number of symmetrical point pairs is greater than or equal to the second preset value, it is confirmed that the number of symmetrical point pairs is large, and there is a reflection in the contour to be detected, that is, the contour to be detected is a water surface contour. If it is counted that the number of symmetrical point pairs is less than the second preset value, it is confirmed that the number of symmetrical point pairs is small, and it is determined that there is no reflection in the contour to be detected, that is, the contour to be detected is not a water surface contour.

Further, referring to FIG. 7 , in one embodiment, feature point extraction is performed on the inner area of the contour to be detected in the environment image to obtain a plurality of first feature points. Traverse each first feature point in turn, and for each first feature point, look for a symmetrical feature point corresponding to the first symmetrical point with the horizon as the symmetric center in the environment image as a matching point, if found, make each first feature point A feature point and a symmetrical feature point form a symmetrical point pair. If not found, the first feature point does not have a feature point with the horizon as the center of symmetry. For example, as shown in FIG. 7 , three first feature points A, B and C are extracted in FIG. 7 . Further, the distances between the three feature points A, B and C and the horizon are calculated respectively. For example, the distances between the three feature points A, B and C and the horizon are L ₁ , L ₂ and L ₃ . Further, feature points at distances L ₁ , L ₂ , and L ₃ from the horizon can be found on the sides of the horizon far away from A, B, and C, respectively. As shown in Figure 7, in Figure 7, the feature points that are symmetrical to the three first feature points (A, B, and C) with the horizon as the center are found, respectively (A', B', and C') . The A', B' and C' are at distances L ₁ , L ₂ and L ₃ from the horizon, respectively. In this way, the following three feature point pairs are formed: (A, A'), (B, B') and (C, C'). For example, the second preset value is 2. Since the number (3) of feature point pairs is greater than the second preset value (2), it is determined that there is a reflection in the contour to be detected.

Further optionally, another implementation of step 203 is: step 2031: extract a first feature point in the contour to be detected as the current first feature point; step 2033: on the environment image, for the current first feature point Perform calculations to determine whether the current first feature point has a current matching point that is symmetrical to the horizon, and when the current feature point has a current matching point that is symmetrical to the horizon, then the current first feature point and the current matching point form a The current symmetrical point pair; step 2035: use the current symmetrical point pair as the checked symmetrical point pair, and record it in the checked symmetrical point pair set, and extract any first one that is different from the checked symmetrical point pair set in the contour to be detected. The feature point is used as the current first feature point, and steps 2033 to 2035 are repeated until the number of detected symmetrical point pairs satisfies the second preset value, and it is determined that there is a reflection in the contour to be detected.

Further, the extraction of the first feature point may adopt any of the following algorithms: ORB extraction algorithm, SIFT extraction algorithm, HARRIS extraction algorithm and the like.

After identifying the to-be-detected contour with reflection from at least one to-be-detected contour, acquire the actual position information of the to-be-detected contour with reflection in the geographical scene, and control the self-mobile device to travel according to the actual position information to avoid the geographical scene the subsurface area. Referring to Figure 3, when the self-mobile device is performing path planning, it plans a navigation path that avoids the water surface area based on the actual position information of the contour to be detected with reflection in the geographic scene, and avoids the geographical area during the travel along the navigation path. The water surface area under the scene, thereby reducing the probability of wading damage during travel.

In specific applications, the actual position information of the contour to be detected with reflection in the geographical scene can be marked as the actual position information of the water surface area in the environment map that assists in navigation processing from the mobile device, according to the actual position information of the marked water surface area The environmental map of the mobile device is controlled to avoid the water surface area and continue to travel. Among them, the environmental map may include but not limited to: Grid-based map, topological map, geometric feature map and hybrid map constructed based on SLAM (Simultaneous localization and mapping) technology .

In addition, before marking the actual position information of the water surface area in the environment map that assists in navigation processing from the mobile device, it can also be judged whether the coordinate system used by the environment map is the world coordinate system, and if so, the actual position of the water surface area can be directly The information is labeled into the environment map. If not, calculate the coordinate system transformation matrix between the world coordinate system and the coordinate system used by the environment map, and use the coordinate system transformation matrix to convert the actual position information of the water surface area to the coordinate system used by the environment map, and convert the water surface area The location information in the coordinate system used by the environment map is marked on the environment map to complete the task of adding information of the water surface area in the environment map. In some optional embodiments of the present application, based on the image position information of the contour to be detected with reflection in the environment image and the coordinate system transformation matrix between the image coordinate system corresponding to the environment image and the world coordinate system, the calculation of The actual location information of the contour to be detected in the geographic scene.

Further optionally, in order to obtain more accurate actual position information of the contour to be detected with reflection in the geographical scene, the specific implementation of step 204 is: obtain two target environment images collected by the image collection device, wherein, Each target environment image contains the contour to be detected with reflection; the second feature point corresponding to the contour to be detected with reflection in the two target environment images is obtained; the second feature point in the two target environment images is Match to obtain the second set of feature points that are successfully matched in the two target environment images; construct the three-dimensional space coordinates of the second feature point set in the geographic scene; according to the first geographic coordinate information of the image acquisition device when shooting the two target environment images , and the three-dimensional space coordinates of the second feature point in the geographic scene to obtain the geographic coordinate information of the contour to be detected with reflection, wherein the geographic coordinate information is the actual position information of the contour to be detected with reflection in the geographic scene.

In the embodiment of the present application, the second feature point may be a feature point obtained by processing any contour point on the contour to be detected with a reflection by using a feature point detection algorithm. Further optionally, the second feature point is a feature point corresponding to a vertex on the contour to be detected with reflection. Specifically, based on the shape information of the contour to be detected with reflection, the vertex formed by the intersection of two line segments can be selected from multiple contour points on the contour to be detected with reflection, and the vertex obtained by processing the vertex with a feature point detection algorithm Two feature points.

Further optionally, when constructing the three-dimensional space coordinates of the second feature point set in the geographic scene, the second feature point set can be used to calculate the rotation matrix and translation matrix between the two target environment images using epipolar constraints; according to the rotation The matrix and the translation matrix generate the three-dimensional space coordinates of the second feature point set in the geographic scene.

For ease of understanding, description will be made in conjunction with FIG. 4 . Take two environmental images of the same water surface area at different locations. The pixel points corresponding to the same object in the two environmental images satisfy the epipolar constraint relationship. As shown in Figure 4, assuming that the two target environment images are environment image 1 and environment image 2 respectively, for any contour point P of the water surface area under the geographic scene (that is, the real world), for example, the vertex of the water surface area contour, the The projection of the contour point P in the environment image 1 is the second feature point P1, and the projection of the contour point P in the environment image 2 is the second feature point P2. The position O1 is the image acquisition device when the image acquisition device is shooting the environment image 1 The position of the optical center of O2 is the position of the optical center of the image acquisition device when the image acquisition device captures the environmental image 2. e1 and e2 in Figure 4 are poles, assuming that the image position of the second feature point P1 is recorded as p1, and the image position of the second feature point P2 is recorded as p2, t is the translation matrix, R is the rotation matrix, and T is the rotation of the matrix Set, the essential matrix E=t∧R, E is a 3×3 essential matrix. According to the polar constraints, p1×t×R×p2=0 can be obtained, and the essential matrix E is solved by using the eight-point method. By decomposing the singular value of the essential matrix E, the rotation matrix R and translation matrix t between the two target environment images can be obtained. Based on any second feature point in the second feature point set, use the rotation matrix and translation matrix to perform coordinate transformation on the image position of the second feature point in the corresponding target environment image, and obtain the second feature point relative to the image The three-dimensional space coordinates of the acquisition device.

任一轮廓点P在一张目标环境图像的图像像素坐标记为(u₁,v₁)，任一轮廓点P在另一张目标环境图像的图像像素坐标记为(u₂,v₂)。将8个轮廓点P的两个图像像素坐标依次代入以下公式：/>

可以得到本质矩阵E＝(e1,e2,e3^T,。When using the eight-point method to solve the essential matrix E, it is necessary to obtain the image pixel coordinates of the eight contour points P in the two target environment images, assuming

The image pixel coordinates of any contour point P in a target environment image are marked as (u ₁ , v ₁ ), and the image pixel coordinates of any contour point P in another target environment image are marked as (u ₂ , v ₂ ) . Substitute the two image pixel coordinates of the eight contour points P into the following formula in turn: />

The essential matrix E=(e1,e2,e3 ^T , can be obtained.

The first geographic coordinate information of the image acquisition device when capturing two images of the target environment may specifically be: the three-dimensional space coordinates of the image acquisition device when capturing two images of the target environment. Further, when calculating the position information of the O1 position and the position information of the O2 position, firstly, the current position information of the self-mobile device can be obtained based on the positioning device of the self-mobile device, and according to the current position information of the self-mobile device and the image of the self-mobile device The relative positional relationship between the acquisition devices is calculated to obtain the current three-dimensional space coordinates of the image acquisition devices.

Combining with Figure 4, it can be seen that based on the position information of O1 position, O2 position information, the three-dimensional space coordinates of feature point P1, and the three-dimensional space coordinates of feature point P2, the contour point P can be calculated simply, accurately and quickly through geometric principles. The three-dimensional space coordinates of .

In the control method of the self-mobile device provided in the embodiment of the present application, firstly, the environment image in the traveling direction of the self-mobile device is obtained by the image acquisition device, and the contour of the environment image is extracted to obtain at least one contour to be detected; then, based on at least one According to the reflection detection result of the contour to be detected, the contour to be detected containing the reflection is determined from at least one contour to be detected; finally, based on the actual position information of the contour to be detected containing the reflection in the geographical scene, the self-mobile device is controlled to avoid Avoid the water surface area under the geographic scene. As a result, it is possible to automatically, accurately, and quickly help self-mobile devices identify water surface areas based on visual technology, and then based on the actual location information of water surface areas in geographical scenarios, self-mobile devices can be controlled to effectively avoid water surface areas during travel, reducing self-mobile devices. Probability of water damage to mobile devices. Furthermore, there is no need to arrange physical boundaries to help the self-mobile device identify the water surface area, and this method does not need to consume time and energy, and the cost is low.

In a specific application, whether it is an indoor environment or an outdoor environment, there may be a variety of objects or areas in the working environment. In this way, multiple contours to be detected can be identified by contour extraction on the environmental image. If reflection detection is performed on all contours to be detected, the recognition efficiency of the water surface area will be reduced.

Therefore, in an optional embodiment, the step extracts the contour of the environment image to obtain at least one contour to be detected, and the step of performing reflection detection on the inner area of the contour to be detected to determine whether there is a reflection in the contour to be detected, and The following steps may be performed: filtering at least one contour to be detected to delete contours to be detected above the horizon on the environment image in the at least one contour to be detected, and/or deleting contours to be detected whose circumference is smaller than a first preset value.

Generally speaking, if at least one contour to be detected is screened, the probability that the contour to be detected that is screened out is the contour of the water surface is relatively high, and the reflection detection of the contour to be detected that is screened out can improve the recognition efficiency of the contour of the water surface, and can also improve the water surface contour. Contour recognition accuracy.

In this embodiment of the present application, no limitation is imposed on a specific implementation manner of screening at least one contour to be detected. Here are several optional implementations:

Mode 1: Delete the contours to be detected whose circumference is smaller than the first preset value, that is, keep the contours to be detected whose circumference is greater than or equal to the first preset value. Wherein, the first preset value is a preset contour perimeter threshold, and the first preset value is set according to actual business requirements. Based on the comparison result between the circumference of each contour to be detected and the first preset value, contours to be detected with shorter circumferences may be filtered.

Mode 2: Delete the contours to be detected located above the horizon on the environment image, that is, retain the contours to be detected located below the horizon on the environment image.

In this manner, based on the horizon in the environment image and the relative positional relationship between the multiple contours to be detected, the contours to be detected below the horizon can be screened out.

Mode 3: Delete the contours to be detected whose perimeter is smaller than the first preset value, and delete the contours to be detected above the horizon on the environmental image from the remaining contours to be detected whose perimeter is greater than or equal to the first preset value. That is, the contours to be detected whose circumference is greater than or equal to the first preset value and which are located below the horizon in the environment image are retained.

The embodiment of the present application does not limit the manner of determining the horizon in the environment image. For example, the target detection algorithm can be used to identify buildings in the environment image, and detect the bounding box surrounding the building. Determined as the horizon in the environment image, as shown in Figure 5. For another example, in order to more accurately determine the horizon in the environment image, multiple horizons determined based on the bounding boxes corresponding to the multiple buildings may be weighted and summed to obtain the final horizon in the environment image.

Further optionally, in order to more accurately determine the horizon in the environment image, when determining the horizon in the environment image, it can be obtained according to the pose data of the mobile device when collecting the environment image and the optical center position of the image acquisition device Horizon in environment images. Wherein, the optical center position of the image acquisition device can be obtained from calibration parameters of the image acquisition device. The pose data of the self-mobile device when collecting the environment image may be provided by an inertial sensor (Inertial Measurement Unit, IMU), a three-axis attitude sensor, a six-axis attitude sensor, a nine-axis attitude sensor, etc. set on the self-mobile device. Due to the turbulent terrain, as shown in Figure 5, if the self-mobile device encounters stones during travel, it will tilt. Wire.

In the embodiment of the present application, the horizon in the environmental image is the intersection line between the absolute horizontal plane at the altitude where the optical center of the image acquisition device is located and the image plane where the environmental image is located, and the line segment passing through the optical center of the image acquisition device in Figure 6 is A line segment in an absolute horizontal plane. The calculation principle of the horizon in the environment image will be introduced below with reference to FIG. 6 . In Fig. 6, the O-x-y-z coordinate system is the coordinate system of the image acquisition device, the O'-x'-y' coordinate system is the pixel plane coordinate system, O is the optical center of the image acquisition device, and the focal length of the image acquisition device is f, the image The plane is also the physical imaging plane. The absolute horizontal plane at the altitude where the optical center of the image acquisition device is located is recorded as PQP'Q', the intersection line between the absolute horizontal plane of PQP'Q' and the image plane is P'Q', and P'Q' is the horizon in the environmental image .

Assuming that the optical center position of the image acquisition device is recorded as (X, Y, Z), the attitude data of the self-moving device (also the attitude data of the image acquisition device) is recorded as (roll, yaw, pitch), roll represents the roll angle, and yaw represents Yaw angle, pitch represents the pitch angle. Based on the optical center position (X, Y, Z) and attitude data (roll, yaw, pitch), the equation of the PQP'Q' horizontal plane can be calculated: Ax+By+Cz+D=0. Among them, A, B, C, D are constant coefficients. Taking two points P(X1, Y1, Z1) and Q(X2, Y2, Z2) on the horizontal plane of PQP'Q', according to the pinhole imaging principle, P(X1, Y1, Z1) can be pixel P'( u1, v1), and Q(X2, Y2, Z2) on the imaging plane pixel Q'(u2,v2), the connection between pixel P'(u1,v1) and pixel Q'(u2,v2) is the horizon in the environment image.

For ease of understanding, several scenario embodiments are introduced below to describe in detail the method for controlling the self-mobile device provided by the embodiment of the present application.

Scenario Example 1:

When the intelligent lawn mowing robot performs mowing work outdoors, it turns on the image acquisition device set at the front end of the intelligent lawn mowing robot to collect images, and extracts the contours of the collected environmental images, and separates the internal areas of the extracted multiple contours. Perform reflection detection. A contour is a water contour when there is a reflection in the contoured interior area. Add the actual position information of the water surface outline in the geographic scene to the navigation map. The intelligent lawn mowing robot performs path planning according to the actual location information of the water surface area in the navigation map, so as to plan a path that avoids the water surface area during the traveling process, and avoids the intelligent lawn mowing robot wading damage.

Scenario example 2:

When the intelligent disinfecting robot performs disinfecting operations in an indoor environment, it turns on the image acquisition device set at the front end of the intelligent disinfecting robot to collect images, and extracts the contours of the collected environmental images, as well as the internal areas of the extracted multiple contours Reflection detection is performed separately. A contour is a water contour when there is a reflection in the contoured interior area. Add the actual position information of the water surface outline in the geographic scene to the navigation map. The intelligent disinfecting robot performs path planning according to the actual position information of the water surface area in the navigation map, so as to plan a path that avoids the water surface area during the traveling process, and avoids the intelligent disinfecting robot wading damage.

It should be noted that the subject of execution of each step of the method provided in the foregoing embodiments may be the same device, or the method may also be executed by different devices. For example, the execution subject of steps 201 to 203 may be device A; for another example, the execution subject of steps 201 and 202 may be device A, and the execution subject of step 203 may be device B; and so on.

In addition, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear herein or executed in parallel , the serial numbers of the operations, such as 201, 202, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. Additionally, these processes can include more or fewer operations, and these operations can be performed sequentially or in parallel. It should be noted that the descriptions of "first" and "second" in this article are used to distinguish different messages, devices, modules, etc. are different types.

An embodiment of the present application also provides a self-moving device, and FIG. 8 is a schematic structural diagram of a self-moving device provided in an exemplary embodiment of the present application. As shown in FIG. 8 , the self-mobile device may at least include: an image acquisition device 80 , a memory 81 and a processor 82 , a vehicle frame 83 , and a walking device 84 .

The traveling device 84 is arranged on the vehicle frame 83 and is used to drive the self-moving equipment to walk. For example, the running gear may include driving wheels, driving motors, universal wheels and the like.

The image acquisition device 80 is arranged on the vehicle frame for image acquisition.

The memory is used to store computer programs; these computer programs can be executed by the processor 82, causing the processor 82 to control the self-mobile device to implement corresponding functions and complete corresponding actions or tasks. In addition to storing computer programs, memory 81 may also be configured to store various other data to support operations on the mobile device. Examples of such data include instructions for any application or method operating on the mobile device.

In this embodiment of the present application, the implementation form of the processor 82 is not limited, for example, it may be but not limited to a CPU, a GPU, or an MCU. The processor 82 can be regarded as the control system of the self-mobile device, and can be used to execute the computer program stored in the memory 81, so as to control the self-mobile device to realize corresponding functions and complete corresponding actions or tasks. It is worth noting that, depending on the form of the self-mobile device and the different scenarios, the required functions, completed actions or tasks will be different; correspondingly, the computer programs stored in the memory 81 will also be different. , and the processor 82 executes different computer programs to control the self-mobile device to realize different functions and complete different actions or tasks.

The processor 82 is coupled to the memory 81, and is used to: obtain the environmental image collected by the image acquisition device, the environmental image is an image in the direction of travel of the mobile device; perform contour extraction on the environmental image to obtain at least one contour to be detected; contour to be detected In order to determine whether there is a reflection in the contour to be detected; when there is a reflection in the contour to be detected, the actual position information of the contour to be detected with reflection in the geographical scene is obtained, so as to control the self-display according to the actual position information. The mobile device travels to avoid water surface areas under the geographic scene.

Further optionally, the processor 82 extracts the contour of the environment image in the execution step to obtain at least one contour to be detected, and performs reflection detection on the inner area of the contour to be detected in the step of performing reflection detection to determine whether there is a reflection in the contour to be detected. It is used for: filtering at least one contour to be detected to delete contours to be detected in the at least one contour to be detected that are below the horizon on the environment image, and/or to delete contours to be detected whose circumference is smaller than a first preset value.

Further optionally, the processor 82 is further configured to: acquire the horizon in the environment image according to the pose data of the self-mobile device when collecting the environment image and the optical center position of the image acquisition device.

Further optionally, when the processor 82 performs reflection detection in the inner area of the contour to be detected to determine whether there is a reflection in the contour to be detected, it is specifically used to: extract a plurality of first feature points in the contour to be detected; In the above, a plurality of first feature points are calculated one by one to determine whether each of the first feature points has a matching point that is symmetrical with respect to the horizon, and when the first feature point has a matching point that is symmetrical with respect to the horizon , then form a symmetrical point pair with the first feature point and the matching point; obtain the number of symmetrical point pairs, and when the number of symmetrical point pairs satisfies the second preset value, judge the contour memory to be detected in reflection.

Further optionally, when there is a reflection in the contour to be detected, the processor 82 is specifically used to obtain the two target environment images collected by the image acquisition device when acquiring the actual position information of the contour to be detected with the reflection in the geographical scene , wherein, each target environment image contains a contour to be detected with reflection; the second feature point corresponding to the contour to be detected with reflection in the two target environment images is obtained; the second feature point in the two target environment images is The feature points are matched to obtain the second feature point set successfully matched in the two target environment images; the three-dimensional space coordinates of the second feature point set in the geographical scene are constructed; according to the first set of the image acquisition device when shooting the two target environment images The geographical coordinate information and the three-dimensional space coordinates of the second feature point in the geographical scene obtain the geographical coordinate information of the contour to be detected with reflection, wherein the geographical coordinate information is the actual position information of the contour to be detected with reflection in the geographical scene.

Further optionally, when the processor 82 constructs the three-dimensional space coordinates of the second feature point set in the geographical scene, it is specifically used to: use the second feature point set to calculate the rotation matrix between two target environment images using epipolar constraints and a translation matrix; generating the three-dimensional space coordinates of the second feature point set in the geographical scene according to the rotation matrix and the translation matrix.

Further optionally, the second feature point is a feature point corresponding to a vertex on the contour to be detected with reflection.

Correspondingly, the embodiments of the present application also provide a computer-readable storage medium storing a computer program, and when the computer program is executed, the steps in the above method embodiments can be realized.

The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on communication standards, such as WiFi, 2G, 3G, 4G/LTE, 5G and other mobile communication networks, or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication assembly also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or a swipe action, but also detect duration and pressure associated with the touch or swipe operation.

The above-mentioned power supply component provides power for various components of the equipment where the power supply component is located. A power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

The aforementioned audio components may be configured to output and/or input audio signals. For example, the audio component includes a microphone (MIC), which is configured to receive an external audio signal when the device on which the audio component is located is in an operation mode, such as a calling mode, a recording mode, and a speech recognition mode. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. Memory is an example of computer readable media.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

The above are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (11)

1. A control method of a self-moving device having an image acquisition apparatus, the method comprising:

acquiring an environment image acquired by the image acquisition device, wherein the environment image is an image in the travelling direction of the self-mobile equipment;

extracting the contour of the environment image to obtain at least one contour to be detected;

performing reflection detection on the internal area of the outline to be detected to judge whether reflection exists in the outline to be detected;

when the back image exists in the contour to be detected, acquiring the actual position information of the contour to be detected with the back image under the geographic scene, so that the self-moving equipment can be controlled to travel according to the actual position information, and the water surface area under the geographic scene is avoided.

2. The method of claim 1, wherein the steps of extracting the contour of the environmental image to obtain at least one contour to be detected, and detecting the reflection of the internal region of the contour to be detected to determine whether the reflection exists in the contour to be detected, further comprise:

And screening at least one contour to be detected to delete the contour to be detected below the horizon on the environment image in at least one contour to be detected, and/or deleting the contour to be detected with the circumference smaller than a first preset value.

3. The method as recited in claim 2, further comprising:

and acquiring the horizon in the environment image according to the pose data of the self-mobile device when the environment image is acquired and the optical center position of the image acquisition device.

4. A method according to any one of claims 1 to 3, wherein the step of performing a ghost detection on an inner region of the outline to be detected to determine whether a ghost exists within the outline to be detected, comprises:

extracting a plurality of first characteristic points from the outline to be detected;

on the environment image, carrying out one-to-one calculation on a plurality of first feature points to judge whether each first feature point has a matching point symmetrical relative to a horizon, and when the first feature point has a matching point symmetrical relative to the horizon, forming a symmetrical point pair by the first feature point and the matching point;

And acquiring the number of the symmetrical point pairs, and judging that the contour to be detected has the reflection when the number of the symmetrical point pairs meets a second preset value.

5. The method according to claim 4, wherein the extracting of the first feature point may use any one of the following algorithms: ORB extraction algorithm, SIFT extraction algorithm, and HARRIS extraction algorithm.

6. A method according to any one of claims 1 to 3, wherein the step of performing a ghost detection on an inner region of the outline to be detected to determine whether a ghost exists within the outline to be detected, comprises:

step 2031: extracting a first characteristic point from the contour to be detected as a current first characteristic point;

step 2033: calculating the current first feature point on the environment image to judge whether the current first feature point has a current matching point symmetrical relative to the horizon, and when the current feature point has the current matching point symmetrical relative to the horizon, forming a current symmetrical point pair by the current first feature point and the current matching point;

step 2035: and taking the current symmetry point pair as a detected symmetry point pair, recording the current symmetry point pair as a detected symmetry point pair set, extracting any one of first characteristic points in different pairs of detected symmetry point pairs from the profile to be detected as the current first characteristic point, and repeating the steps 2033 to 2035 until the existence of the reflection in the profile to be detected is judged when the number of the detected symmetry point pairs meets a second preset value.

7. A method according to any one of claims 1 to 3, wherein when there is a reflection in the contour to be detected, the step of obtaining actual position information of the contour to be detected in which there is a reflection in a geographical scene, so as to control the self-mobile device to travel according to the actual position information, so as to avoid a water surface area in the geographical scene, specifically includes:

acquiring two target environment images acquired by the image acquisition device, wherein each target environment image comprises the contour to be detected with the reflection;

acquiring second feature points corresponding to the contours to be detected with the reflection in the two target environment images respectively;

matching the second characteristic points in the two target environment images to obtain a second characteristic point set successfully matched in the two target environment images;

constructing three-dimensional space coordinates of the second feature point set in the geographic scene;

and acquiring the geographic coordinate information of the contour to be detected with the reflection according to the first geographic coordinate information of the image acquisition device when two images of the target environment are shot and the three-dimensional space coordinates of the second characteristic points in the geographic scene, so that the self-moving equipment can be controlled to advance according to the actual position information to avoid a water surface area in the geographic scene, wherein the geographic coordinate information is the actual position information of the contour to be detected with the reflection in the geographic scene.

8. The method according to claim 7, wherein the step of constructing three-dimensional space coordinates of the second feature point set in the geographic scene specifically comprises:

calculating a rotation matrix and a translation matrix between the two target environment images by utilizing the second characteristic point set and adopting epipolar constraint;

and generating three-dimensional space coordinates of the second feature point set in the geographic scene according to the rotation matrix and the translation matrix.

9. The method of claim 7, wherein the second feature point is a feature point corresponding to a vertex on the contour to be detected where a reflection exists.

10. A self-moving device, comprising: the device comprises a frame, a traveling device, an image acquisition device, a memory and a processor, wherein,

the walking device is arranged on the frame and used for driving the self-moving equipment to walk;

the image acquisition device is arranged on the frame;

the memory is used for storing a computer program;

the processor is coupled to the memory for executing the computer program for performing the method of any of claims 1-9.

11. A storage medium storing a computer program which, when executed by a processor, causes the processor to implement the method of any one of claims 1-9.

Images