[go: up one dir, main page]

WO2025147881A1 - Projection d'image améliorée pour réalité augmentée - Google Patents

Projection d'image améliorée pour réalité augmentée

Info

Publication number
WO2025147881A1
WO2025147881A1 PCT/CN2024/071515 CN2024071515W WO2025147881A1 WO 2025147881 A1 WO2025147881 A1 WO 2025147881A1 CN 2024071515 W CN2024071515 W CN 2024071515W WO 2025147881 A1 WO2025147881 A1 WO 2025147881A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target
points
locations
reference image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/071515
Other languages
English (en)
Inventor
Xiaohui Wang
Junwei Chen
Shanshan Wang
Yucai Li
Yayun Deng
Jin Wang
Qian Sun
Zhijin Zhang
Xing Liu
Chaodong ZHU
Zhe TANG
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Telefonaktiebolaget LM Ericsson AB
Original Assignee
Telefonaktiebolaget LM Ericsson AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonaktiebolaget LM Ericsson AB filed Critical Telefonaktiebolaget LM Ericsson AB
Priority to PCT/CN2024/071515 priority Critical patent/WO2025147881A1/fr
Publication of WO2025147881A1 publication Critical patent/WO2025147881A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • H04N9/3185—Geometric adjustment, e.g. keystone or convergence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3191—Testing thereof
    • H04N9/3194—Testing thereof including sensor feedback

Definitions

  • the present disclosure is related to the field of Augmented Reality (AR) , and in particular, to an electronic device and a method for improved image projection for AR.
  • AR Augmented Reality
  • AR is the integration of digital information with the user′s environment in Real Time. Unlike virtual reality (VR) , which creates a totally artificial environment, AR users experience a real-world environment with generated perceptual information overlaid on top of it.
  • VR virtual reality
  • AR is widely used in smart manufacture. It can help operators to execute working process correctly and provide guidance for each step. Therefore, it can improve production efficiency and product quality dramatically. For example, in some modern factories, this technology is applied to guide the operators to locate the correct screw hole in a correct order when using a screw driver. Further, a quality inspection can be implemented for each step by using AR.
  • one of the smart manufacture systems may be a projector-camera system which can help operators by providing guidance and quality assurance.
  • another smart manufacture system may provide animated guidance on a display for operations. When each step on a product is finished, a camera may take a picture of the product and detect its assembly quality.
  • an operator needs to move a circular light spot 320 to a correct position where a corresponding screw hole (e.g., a screw hole 310) is located by using a mouse for example, as shown in Fig. 3.
  • a corresponding screw hole e.g., a screw hole 310
  • the light spot 320 is initially projected onto a product 300, and it is probably not projected at its correct position.
  • the operator has to move the light spot 320 to the location of the screw hole 310, for example, by using a mouse to manually drag and drop the light spot 320 onto the screw hole 310.
  • the light spot 320 can be automatically projected at the correct position (that is, the screw hole 310) for other products to be manufactured, which have a same product specification and is located at a same location as the product 300, as shown in (c) of Fig. 3.
  • a product to be manufactured is not always located at the expected location (e.g., as shown in (c) of Fig. 3) .
  • a translation and rotation of a product to be manufactured can be expected and probably inevitable, leading to for example incorrect positioning of the screw holes (e.g., as shown in (a) of Fig. 3) .
  • some other smart manufacture systems mainly focus on the quality inspection functionality, and it provides even less help in guidance than the systems mentioned above.
  • an operator needs to look up to a Personal Computer (PC) screen installed separately from a working area where a product is being manufactured, resulting in a frequent switching of line of sight between the working area and the PC screen, which is not convenient and efficient during the manufacturing of the product.
  • PC Personal Computer
  • some embodiments of the present disclosure provide an electronic device and a method for improved image projection for AR.
  • a method for image projection comprises: locating one or more first points on a first image of a target; locating one or more second points on the first image based on at least the one or more first points; and projecting a second image onto the target based on at least the one or more second points.
  • the system can determine the translation and/or rotation of the target and therefore accurately locate the second points where one or more operations need to be performed, and then light guidance can be projected accurately onto the second points.
  • the one or more first points correspond to one or more features of the target, respectively.
  • the one or more second points correspond to one or more locations on the target, respectively, at which one or more operations are to be performed by an operator.
  • at least one of the one or more first points is located by: searching for a first feature in a first Region of Interest (ROI) in the first image; and determining a point in the first feature as the first point.
  • the first ROI is centered at a location with coordinates same as those of a reference point in a first reference image associated with the target.
  • the reference point corresponds to the first point.
  • the step of searching for a first feature comprises: searching for the first feature by using a first U-Net convolutional network.
  • the step of determining a point in the first feature as the first point comprises: determining the center of the first feature as the first point by using the first U-Net convolutional network.
  • the step of locating one or more second points comprises: determining a first perspective transform from a first reference image associated with the target to the first image based on one or more first reference points in the first reference image and the one or more first points at least.
  • the first reference points correspond to the one or more first points, respectively.
  • the first perspective transform is defined by a first homography matrix that is determined based on the one or more first reference points and the one or more first points at least.
  • the one or more first points comprise four or more first points that are not collinear.
  • At least one of the one or more second points is located by: determining an expected location in the first image corresponding to a second point based on at least the first perspective transform and a second reference point in the first reference image, the second reference point corresponding to the second point; searching for a second feature corresponding to the second point in a second ROI in the first image, the second ROI being centered at the expected location; and determining a point in the second feature as the second point.
  • the first ROI has a larger area than that of the second ROI.
  • the step of searching for a second feature comprises: searching for the second feature by using a second U-Net convolution network.
  • the step of determining a point in the second feature as the second point comprises: determining the center of the second feature as the second point by using the second U-Net convolutional network.
  • the second image comprises one or more patterns which, when projected on the target, highlight one or more locations on the target, respectively.
  • the one or more highlighted locations on the target are locations where one or more operations are to be performed by an operator.
  • at least one of the one or more patterns is generated by: determining a location in a second reference image, the location corresponding to one of the one or more second points; determining a location in the second image, the determined location in the second image corresponding to the determined location in the second reference image; and generating the pattern in the second image based on at least the determined location in the second image.
  • the first reference image associated with the target is captured for a reference target that is placed on a supporting surface.
  • the second reference image is captured for the supporting surface without the reference target placed thereon.
  • the reference target has a same product specification as the target.
  • the step of determining a location in a second reference image comprises: calculating the location in the second reference image based on at least the corresponding second point and a second perspective transform from the first reference image to the second reference image.
  • the second perspective transform is defined by a second homography matrix that is calculated based on at least the first reference image and the second reference image.
  • the second perspective transform is determined by: capturing the first reference image when one or more first markers are projected onto a reference target placed on a supporting surface; determining, in the first reference image, the locations of the one or more first markers projected on the reference target; capturing the second reference image when one or more first markers are projected onto the supporting surface without the reference target placed thereon; determining, in the second reference image, the locations of the one or more first markers projected on the supporting surface; calculating the second perspective transform based on at least the locations of the one or more first markers projected on the reference target in the first reference image and the locations of the one or more first markers projected on the supporting surface in the second reference image.
  • the mapping from locations in the second reference image to locations in the second image is determined by: capturing a third reference image when a third image is projected on a supporting surface without any target placed thereon, the third image comprising one or more second markers; determining one or more locations in the third reference image corresponding to the one or more second markers; and determining a mapping based on at least the one or more determined locations in the third reference image and the locations of the one or more second markers in the third image, as the mapping from locations in the second reference image to locations in the second image.
  • the one or more second markers comprise an array of circles.
  • the one or more second markers projected on the supporting surface cover an area where the target is to be placed.
  • at least one of the one or more locations in the third reference image is determined by: labeling a point in a projected second marker in the third reference image; determining a third ROI comprising the projected second marker based on at least the labeled point; and searching for the projected second marker in the third ROI and determining the center of the projected second marker by using a third U-net convolutional network.
  • the step of determining a mapping comprises: training a Multi-Layer Perceptron (MLP) network by using the determined one or more locations in the third reference image as inputs and using the locations of the one or more second markers in the third image as expected outputs, such that the trained MLP network is able to be used as the mapping from locations in the second reference image to locations in the second image.
  • the step of generating the pattern in the second image comprises at least one of: generating a circle pattern in the second image that is centered at the determined location in the second image; and generating an arrow pattern in the second image that points to the determined location in the second image.
  • processing circuits may in some embodiments be embodied in one or more application-specific integrated circuits (ASICs) .
  • these processing circuits may comprise one or more microprocessors, microcontrollers, and/or digital signal processors programmed with appropriate software and/or firmware to carry out one or more of the operations described above, or variants thereof.
  • these processing circuits may comprise customized hardware to carry out one or more of the functions described above. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.
  • Step 2 Find the mapping relationship between the coordinates in a projected image and coordinates in a corresponding canvas image.
  • this uNet may be denoted as UNet_projection_circle;
  • this MLP network may be denoted as MLP_Projection_Canvas_Relationship.
  • Part 2 aims at finding the relationship between the product (e.g., the target 120 shown in Fig. 1A) and the desk (e.g., the supporting surface 105 shown in Fig. 1A) for the projection light.
  • the product e.g., the target 120 shown in Fig. 1A
  • the desk e.g., the supporting surface 105 shown in Fig. 1A
  • Step 1 Find the perspective transform between 2 scenarios with and without product for projection light.
  • the coordinates may be denoted as P_intersection_with_product.
  • this homography matrix may be denoted as PT_product.
  • Step 2 Use labelme to label key points of the product (e.g., K0 through K3 shown in Fig. 2) and the centers of the screw holes (e.g., S0 through S7 shown in Fig. 2) .
  • key points of the product e.g., K0 through K3 shown in Fig. 2
  • the centers of the screw holes e.g., S0 through S7 shown in Fig. 2
  • they may be denoted as P_key_points_reference_product and P_screw_hole_reference_product.
  • the product images captured in this part may be denoted as "reference images” .
  • reference images For example, before the system goes online for real production, a product prototype, which has a same product specification as those to be manufactured, may be used for generating the reference images.
  • Part 3 aims at implement light guidance on-line when the products (which have a same product specification as that of the product prototype) keep on going into the product line.
  • Step 1 Obtain the perspective transform between a real image and the reference image in part 2, for example, based on the 4 key points.
  • this homography matrix may be denoted as PT_key_points.
  • Step 2 Determine the centers of the screw holes.
  • a UNet may be used to detect the centers of the screw holes.
  • the coordinates may be denoted as P_screw_hole_real_product and the UNet may be denoted as UNet_screw_hole.
  • Step 3 Implement light guidance for the screw holes.
  • MLP_Projection_Canvas_Relationship in step 2 of part 1 to obtain the coordinates in the canvas.
  • the coordinates in the canvas may be denoted as P_screw_hole_canvas_real_product. Then color circles can be drawn on the canvas and projected onto the product to realize the light guidance.
  • calibration of camera and/or projector in the system can be done in an easy way, which leads to an easier deployment of the production line.
  • the system can be automatically adapted to the translations and/or rotations of the product by using a large ROI and perspective transformation for product location. Therefore, it can be used widely in smart manufacture. Furthermore, it enables a very fast introduction of a new product. Also, this system can be automatically adapted to variant kinds of products having different heights. The production efficiency can be improved dramatically.
  • automatic light guidance may be provided accurately even when translations and/or rotations of products to be manufactured occur. Further, it is easier and more convenient for an operator to introduce a new series of product, and no specific expertise knowledge is required. Further, the system is applicable to different series of products with different heights. Further, it is easier to determine a mapping relationship from a canvas, where an image to be projected is generated, to a projected image, such that a more accurate projection of the image onto a target may be achieved. Furthermore, downstream tasks, such as quality inspection, can be implemented based on this solution.
  • Fig. 1A and Fig. 1B are diagrams illustrating different perspective views of an exemplary system 10 for image projection according to an embodiment of the present disclosure.
  • the system 10 may comprise a production station (or a working table/desk or the like) 100 which has its top surface used as a supporting surface (or working surface) 105.
  • the supporting surface 105 may be used to provide a working area 115 and support a target 120 (e.g., a product to be manufactured) .
  • a frame 110 may be provided on the supporting surface 105, to which or in which a camera 111 and a projector 113 may be attached and fixed, such that images can be projected by the projector 113 onto the working area 115 and an image of anything in the working area 115 can be captured by the camera 111.
  • a computing system e.g., a PC, a laptop, a workstation, or the like
  • a computing system may be provided in the production station 100 or remote from and connected to the production station 100 (though not shown in Fig. 1A or Fig. 1B) , such that the images captured by the camera 111 and the images to be projected by the projector 113 can be processed by the computing system.
  • the computing system may be accommodated in the base compartment of the production station 100. In some embodiments, the computing system may be remotely located from the production station 100 and communicatively connected to the camera 111 and/or the projector 113 via a cable or wirelessly.
  • the present disclosure is not limited thereto, and the production station 100 and/or the system 10 may have more, less, or different components than those shown in Fig. 1A and Fig. 2B, and/or a different configuration of the components.
  • the camera 111 and/or the projector 113 may be located at different places other than those shown in Fig. 1B.
  • the supporting surface 105 and/or the working area 115 may have different sizes and/or geometries than those shown in Fig. 1A.
  • the camera 111 may capture an image of the target 120.
  • the computing system may detect the screw holes of the target 120 in the captured image and instruct the projector 113 to project an image onto the target 120.
  • the image projected onto the target 120 may provide light guidance (e.g., the circle patterns on the target 120 shown in Fig. 1A) for the detected screw holes.
  • Fig. 2 is a diagram illustrating an exemplary top view of a working area (e.g., the working area 115 shown in Fig. 1A) of a system (e.g., the system 10 shown in Fig. 1A and Fig. 1B) for image projection according to an embodiment of the present disclosure.
  • a target e.g., the target 120 shown in Fig. 1A
  • it may comprise one or more features 121 for target locating and one or more features 123 to which image projection is to be performed.
  • each of the features 121 may have a specific size and/or geometry that are relatively easy for the computing system to identify.
  • the features 121 may be circle patterns as shown in Fig. 2, or a more complex structure or even a printed label (e.g., a bar code, a QR code, or the like) .
  • a location (or coordinates) of a feature 121 may be determined as (coordinates of) any point in the feature (e.g., the center of the feature 121) , and the point may be used as the location of the feature 121 throughout the whole procedure.
  • the feature 121 is a centrally symmetrical feature (e.g., a circle pattern) , then its center may be used as the location of the feature 121.
  • each of the features 123 may also have a specific size and/or geometry that can be identified by the computing system.
  • the features 123 may correspond to the locations where one or more operations are to be performed.
  • a feature 123 may correspond to a screw hole into which a screw is to be driven.
  • a location (or coordinates) of a feature 123 may be determined as (coordinates of) any point in the feature (e.g., the center of the feature 123) , and the point may be used as the location of the feature 123 throughout the whole procedure.
  • the feature 123 is a centrally symmetrical feature (e.g., a circle pattern) , then its center may be used as the location of the feature 123.
  • a total of four features 121 and a total of eight features 123 are provided on the target 120.
  • the present disclosure is not limited thereto. Based on the actual needs, a different number of features 121 and/or a different number of features 123 with potentially different geometries and/or sizes may be provided at potentially different locations on the target 120.
  • a projector may project information for assisting an operator in manufacturing the target 120.
  • information for assisting an operator in manufacturing the target 120 For example, on the left side of the working area 115, progresses for various steps in the production procedure may be listed.
  • the information projected on the left side of the working area 115 may comprise:
  • EMC Electromagnetic Compatibility
  • the current step is to drive screws into the target 120 such that the EMC cover can be fixed to the main board.
  • information related to the production station e.g., the production station 100
  • the target 120 is provided, for example, the production line to which the production station belongs, the product model to be manufactured, the status of the projector 113, the status of the camera 111, and the industrial inspection result.
  • an icon for user interaction e.g., "the next step” icon shown in Fig. 2 may be provided for the operator.
  • exemplary information are shown in Fig. 2 and described above, the present disclosure is not limited thereto.
  • additional information and/or different information may be projected into the working area 115, such that the operator may be provided with useful information during his/her manufacture of the target 120.
  • an exemplary overall procedure for projecting light guidance onto a product in an accurate and robust manner may be divided into 3 parts: Part 1 for Calibration, Part 2 for new product introduction, and Part 3 for real production.
  • Part 1 for Calibration
  • Part 2 for new product introduction
  • Part 3 for real production.
  • this step may be optional since the camera (e.g., the camera 111) may already be well calibrated before its use. In some embodiments, following operations may be performed to calibrate the camera 111.
  • a target for calibration or a "calibration target” 400 may be placed at different places in a working area (e.g., the working area 115) , and one or more images thereof may be captured by using the camera 111 to be calibrated.
  • the calibration target 400 may be a chessboard like target or another type of target for calibration.
  • the calibration target 400 is placed in the upper-left corner of the working area 115 as shown in Fig. 4A and in the lower-right corner of the working area 115 as shown in Fig. 4B. Additionally or alternatively, the calibration target 400 may be placed at other locations.
  • the computing system may calculate distortions in the images, which are caused by the camera 111, and compensate the images for the calculated distortion. For example, when the calibration target 400 is placed as shown in Fig. 4B, a barrel distortion may be observed on the right side of the image captured by the camera 111 (for example, a part of the image captured by the camera 111 is shown in (a) of Fig. 5, in which a barrel distortion can be observed) , while it is known that the calibration target 400 has straight edges (for example, as shown in (b) of Fig. 5) .
  • the computing system may detect such a distortion in the image and compensate the image for the detected distortion, for example, by using a tool called OpenCV API.
  • the OpenCV API may be used to calculate a distortion related matrix to compensate the camera distortion. Since the OpenCV API is well known in the art, a detailed description thereof is omitted for simplicity. After the calibration, the part of the image may look like the original calibration target 400 shown in (b) of Fig. 5.
  • Step 2 Find the mapping relationship between the coordinates in the projected image and the coordinates in the corresponding canvas image.
  • a distortion in an image projected by the projector 113 may also be observed, for example, as shown in Fig. 6.
  • Fig. 6 when a canvas image 600 that has multiple circles 610 provided along its edges is projected onto the supporting surface 105, a part of the projected image including some of the projected circles 610′ cannot be fully observed, for example, those in the lower-left corner of the working area 115.
  • the edge of the projected image is not parallel to the edge of the desk and/or the edge of the image captured by the camera 111.
  • an image projected on the supporting surface 105 may cover a same area (i.e. same size and same geometry) as that of the working area 115 as shown in Fig. 6, the present disclosure is not limited thereto.
  • an image captured by the camera 111 may cover a same area (i.e., same size and geometry) as that of the working area 115, the present disclosure is not limited thereto.
  • the projected image and/or the captured image may cover an area (e.g., a different size and/or a different geometry) different from that of the working area 115 and/or different from each other.
  • the projected image and/or the captured image may have a larger area than that of the working area 115, such that the working area 115 can be fully covered by the captured image and/or the projected image.
  • a canvas image comprising one or more markers (e.g., an array of circles as shown in Fig. 7A) 700 may be generated in the canvas and projected onto the working area 115.
  • the one or more markers 700 may be a 15-by-15 array of circles as shown in Fig. 7A.
  • a number may be generated and projected together with each circle 700, and the numbers may be placed in or next to the circles, respectively.
  • the present disclosure is not limited thereto.
  • the one or more markers 700 may have a different geometry and/or a different size.
  • the canvas image may have a different arrangement of the one or more markers 700 than the circle array shown in Fig. 7A.
  • Fig. 7B shows a projected image corresponding to the canvas image shown in Fig. 7A, and a distortion may be observed in the projected markers 700′ due to one or more factors, such as projector lens distortion, an unexpected angle between the projector 113, the camera 111, and the supporting surface 105, or the like.
  • the program "labelme” as mentioned above may be used to label the centers of the circles in a rough/coarse manner.
  • a UNet as mentioned above may be used to determine, in the projected image, multiple ROIs based on at least the roughly labelled centers.
  • each of the multiple ROIs may be a square region and include a circle, but the present disclosure is not limited thereto.
  • the ROIs may have a different geometry than square.
  • an ROI may have 120 pixels in both of its width and height directions.
  • an ROI may have a different size and/or geometry as also mentioned earlier.
  • a UNet may be used to segment the circles and determine its center.
  • a UNet named UNet_projection_circle may be trained to do so.
  • a number of pairs of original image and its mask may be collected, and some examples of the pairs are shown in (b) of Fig. 8A.
  • the OpenCV API may be used to track the contours of the circles and determine the centers of the circles, as can be seen from (c) of Fig. 8A.
  • Fig. 8B shows an overall result for an exemplary center detection, which shows that the centers 715 of all the circles are identified successfully.
  • an MLP network may be built to map the coordinates of the centers of the circles in the projected image (e.g., that shown in Fig. 8B) to the coordinates of the centers of the circles in the corresponding canvas image (e.g., that shown in Fig. 7A) .
  • this MLP may be denoted as MLP_Projection_Canvas_Relationship.
  • Fig. 9 shows an exemplary MLP network that can be used to do such a mapping.
  • the pairs of the coordinates of the centers of the projected circles and the coordinates of the centers of the circles in the canvas image can be used as inputs and expected outputs, respectively.
  • the inputs or the coordinates in the projected image
  • the outputs or the coordinates in the canvas image
  • the mapping relation can be determined.
  • Step 1 Find the perspective transform between a scenario in which the product (e.g., the target 120 shown in Fig. 2) is placed in the working area (e.g., the working area 115) and a scenario in which the product is not placed in the working area, such that coordinates in these two scenarios can be converted to each other for later use for the projection light.
  • This step will be described in detail with reference to Fig. 10A through Fig. 10C.
  • a canvas image with one or more markers may be generated, for example, as shown in Fig. 10A.
  • the markers may be crosses.
  • the number of the markers may be greater than or equal to 4 to make sure the perspective transform determined based thereon is more accurate, robust, and reliable.
  • Fig. 10A As shown in Fig. 10A, four crosses P0 through P3 are generated in the canvas image. Then the canvas image may be projected onto the working area 115 as shown in Fig. 10B.
  • the projected markers P0′ through P3′ may have a slightly different space relationship than that of the original markers P0 through P3 in the canvas image. This difference may be caused by the projector lens distortion, an unexpected angle between the projector 113, the camera 111, and the working area 105 as mentioned earlier, and it will be eliminated or compensated at a later stage, for example, by using the MLP network mentioned above.
  • an image of projected markers P0′ through P3′ may be captured by the camera 111, and the centers of the projected markers P0′ through P3′ (e.g., the intersection points of the crosses) may be determined in the captured image, for example, by using labelme as indicated by the dashed line, and the coordinates of the intersection points may be recorded as, for example, P_intersection_without_product.
  • the target 120 may be placed in the working area 115 and the canvas image including the 4 crosses may be projected onto the target 120 and also onto the working area 115.
  • an image of projected markers P0" through P3" may be captured by the camera 111, and the centers of the projected markers P0" through P3" (e.g., the intersection points of the crosses) may be determined in the captured image, for example, by using labelme as indicated by the dashed line, and the coordinates of the intersection points may be recorded as, for example, P_intersection_with_product.
  • the perspective transform may be calculated or determined based on at least P_intersection_with_product and P_intersection_without_product.
  • this holography matrix may be denoted as PT_product. Please note that how to calculate a perspective transform based on coordinates before and after the perspective transform, and therefore a detailed description thereof is omitted for simplicity.
  • Step 2 Use labelme to label key points (e.g., K0 through K3) of the product and the centers (e.g., S0 through S7) of the screw holes.
  • key points e.g., K0 through K3
  • the centers e.g., S0 through S7
  • P_key_points_reference_product and P_screw_hole_reference_product respectively.
  • the key points cannot be located in a same line, and each key point is far away from other screw holes.
  • the captured product image in this part may be denoted as the reference image.
  • the key points, P_key_points_reference_product may consist of K0, K1, K2, and K3, for example, which may be identified by using labelme.
  • the centers of the screw holes, P_screw_hole_reference_product may consist of S0, S1, S2, S3, S4, S5, S6, and S7, for example, which may be identified by using labelme.
  • Step 1 Find the perspective transform between a real image in the production line and the reference image in part 2 based on the (4) key points.
  • these 4 points are not in the same line and they are located far away from other screw holes. With such a configuration, large ROIs may be used on them. From these 4 key points and algorithms, the screw holes can be positioned even when the translation and/or rotation happen.
  • large ROIs may be used to search for the key objects (e.g., the key points K0 through K3) .
  • a UNet may be used to detect the centers of the key objects.
  • the coordinates of the centers may be denoted as P_key_points_real_product.
  • this UNet may be denoted as UNet_key_point.
  • the training method may be performed in a same or similar way as that described in step 2 in part 1 above.
  • Fig. 12A shows an exemplary result of the key point detection.
  • dotted bounding boxes may be used as the large ROIs 1210 for the key point detection.
  • the centers of the ROIs 1210 may be determined as the coordinates named P_key_points_reference_product as that in Step 2 in part 2 above.
  • the size of the ROIs 1210 may be 320 *320 pixels.
  • the perspective transform from P_key_points_reference_product (as shown in Fig. 11A) to P_key_points_real_product (as shown in Fig. 12A) may be calculated.
  • this homography matrix may be denoted as PT_key_points.
  • this homography matrix can be used to calculate the coordinates in the real image based on the coordinates in the reference image.
  • Step 2 Determine the centers of the screw holes
  • screw holes may be located coarsely by using the matrix PT_key_points (which is calculated in Step 1, Part 3) and P_screw_hole_reference_product (the coordinates determined in Step 2, Part 2) as shown in Fig. 12B. That is, the expected locations of the screw holes in the real image may be determined based on the perspective transform and the coordinates of the screw holes in the reference image.
  • small ROIs 1220 may be used to search for the screw holes.
  • a UNet may be used to detect the centers of the screw holes in the real image.
  • this UNet may be denoted as UNet_screw_hole.
  • the training method may be same as that described in step 2 in part 1.
  • the coordinates may be denoted as P_screw_hole_real_product.
  • the dotted bounding box may be used as the ROIs 1220 for screw hole detection.
  • the centers of the ROIs 1220 may be the coordinates of the expected locations of the screw holes calculated above.
  • the size of the ROIs 1220 may be 108 *108 pixels, which is smaller than the ROIs 1210 for locating the key points.
  • Step 3 Implement light guidance for the screw holes
  • the homography matrix PT_product (which is determined in Step 1, Part 2) and P_screw_hole_real_product (which is determined in step 2, Part 3) may be used to determine the coordinates P_screw_hole_without_product. It means that if the product 120 is removed from the working area 115, the light will be projected onto the supporting surface 105 at the location with the determined coordinates. In this way, if the product 120 is placed on the working area 115, the light will be projected on the expected location (e.g., the locations of the screw holes) .
  • the height of a product of one kind may be different from that of a different kind.
  • this difference in heights may be automatically handled by the method described above, for example, by the perspective transform PT_Product.
  • this method can convert all different heights into the same height of the desk (i.e., the supporting surface 105 or the working area 115) . In this way, the system can work even if the product height is different for the new product introduction.
  • Fig. 13 is a diagram illustrating an exemplary working area with additional operations performed and additional information provided according to an embodiment of the present disclosure.
  • some subsequent tasks can be implemented after the screw hole detection and/or the light guidance projection.
  • screws may be driven into some of the screw holes S0 through S7 as shown in Fig. 13, and the system may detect/inspect the existence of the screws in the screw holes.
  • the system may detect whether there are screws in the screw holes based on at least the images captured by the camera 111. For example, the system may determine that there are screws in the screw holes S0, S1, S6, and S7 with a very high confidence (1 or close to 1) , and determine that the confidences that there are screws in the screw holes S2 through S5 are very low (close to 0) . In such a case, the system may provide the operator with the related information, for example, the information projected on the right side of the product 120 in the working area 115 as shown in Fig. 13.
  • Fig. 14 is a diagram illustrating an exemplary overall procedure 1400 for image projection for AR according to an embodiment of the present disclosure.
  • the procedure may begin with step S1410 where an image of a product (e.g., the target 120) to be manufactured may be captured by a camera (e.g., the camera 111) .
  • this step may be performed as described above in Step 1, Part 3.
  • the captured image may be calibrated for camera distortion.
  • the information used for calibration e.g., one or more distortion coefficients
  • the distortion coefficients may be obtained from images of a calibration target (e.g., the chessboard like target 400) captured by the camera. In some embodiments, this step may be performed as described above in Step 1, Part 1.
  • one or more (e.g., 4) key points (e.g., S0 through S3) on the product in the captured image may be determined by using large ROIs (compared to the ROIs for locating screw holes) .
  • this step may be performed as described above in Step 1, Part 3.
  • the coordinates of the screw holes in the image captured when the product is placed on the table may be converted into the coordinates in the image captured when the product is not placed on the table, such that the coordinates of the screw holes on different products with different heights, which means these coordinates are provided in different coordinate systems, may be converted into a same coordinate system.
  • coordinates of screw hole projection on the desk may be obtained assuming the product is removed from the desk.
  • this step may be performed as described above in Step 3, Part 3.
  • the coordinates of the screw holes in the same coordinate system may be mapped to the coordinates in a canvas image, for example, by using an MLP network, such that an image for light guidance may be generated.
  • this step may be performed as described above in Step 3, Part 3.
  • one or more second points on the first image may be located based on at least the one or more first points.
  • the mapping from locations in the second reference image to locations in the second image may be determined by: capturing a third reference image when a third image is projected on a supporting surface without any target placed thereon, the third image comprising one or more second markers; determining one or more locations in the third reference image corresponding to the one or more second markers; and determining a mapping based on at least the one or more determined locations in the third reference image and the locations of the one or more second markers in the third image, as the mapping from locations in the second reference image to locations in the second image.
  • the one or more second markers may comprise an array of circles.
  • the one or more second markers projected on the supporting surface may cover an area where the target is to be placed.
  • at least one of the one or more locations in the third reference image may be determined by: labeling a point in a projected second marker in the third reference image; determining a third ROI comprising the projected second marker based on at least the labeled point; and searching for the projected second marker in the third ROI and determining the center of the projected second marker by using a third U-net convolutional network.
  • the method 1500 may further comprise: calibrating a camera that is used for capturing at least one of the first image, the first reference image, the second reference image, and the third reference image.
  • the step of calibrating the camera may comprise: capturing one or more images of a calibration target by using the camera when the calibration target is placed at one or more places on the supporting surface; calculating a distortion of the camera based on at least the one or more images of the calibration target and one or more known characteristics of the calibration target; and compensating one or more images captured by the camera for the calculated distortion.
  • the second image when projected onto the target, may further indicate at least one of: information related to the target; information related to the operations to be performed on the target; information related to status of a camera used for capturing the first image and/or a projector used for projecting the second image; and one or more icons for interaction with an operator.
  • the arrangement 1600 may comprise at least one computer program product 1608 in the form of a non-volatile or volatile memory, e.g., an Electrically Erasable Programmable Read-Only Memory (EEPROM) , a flash memory and/or a hard drive.
  • the computer program product 1608 comprises a computer program 1610, which comprises code/computer readable instructions, which when executed by the processing unit 1606 in the arrangement 1600 causes the arrangement 1600 and/or the electronic device in which it is comprised to perform the actions, e.g., of the procedure described earlier in conjunction with Fig. 4A through Fig. 15 or any other variant.
  • EEPROM Electrically Erasable Programmable Read-Only Memory
  • the computer program 1610 may be configured as a computer program code structured in computer program modules 1610A, 1610B, and 1610C.
  • the code in the computer program of the arrangement 1600 includes: a module 1610A configured to locate one or more first points on a first image of a target; a module 1610B configured to locate one or more second points on the first image based on at least the one or more first points; and a module 1610C configured to project a second image onto the target based on at least the one or more second points.
  • the computer program modules could essentially perform the actions of the flow illustrated in Fig. 4A through Fig. 15, to emulate the electronic device.
  • the different computer program modules when executed in the processing unit 1606, they may correspond to different modules in the electronic device.
  • code means in the embodiments disclosed above in conjunction with Fig. 16 are implemented as computer program modules which when executed in the processing unit causes the arrangement to perform the actions described above in conjunction with the figures mentioned above, at least one of the code means may in alternative embodiments be implemented at least partly as hardware circuits.
  • the processor may be a single CPU (Central processing unit) , but could also comprise two or more processing units.
  • the processor may include general purpose microprocessors; instruction set processors and/or related chips sets and/or special purpose microprocessors such as Application Specific Integrated Circuit (ASICs) .
  • the processor may also comprise board memory for caching purposes.
  • the computer program may be carried by a computer program product connected to the processor.
  • the computer program product may comprise a computer readable medium on which the computer program is stored.
  • the computer program product may be a flash memory, a Random-access memory (RAM) , a Read-Only Memory (ROM) , or an EEPROM, and the computer program modules described above could in alternative embodiments be distributed on different computer program products in the form of memories within the electronic device.
  • RAM Random-access memory
  • ROM Read-Only Memory
  • EEPROM Electrically Erasable programmable read-only memory
  • FIG. 17 is a block diagram of an exemplary electronic device 1700 according to an embodiment of the present disclosure.
  • the electronic device 1700 can comprise e.g., the arrangement 1600.
  • the electronic device 1700 can be configured to perform the method 1500 as described above in connection with Fig. 15. As shown in Fig. 17, the electronic device 1700 may comprise a first locating module 1710 configured to locate one or more first points on a first image of a target; a second locating module 1720 configured to locate one or more second points on the first image based on at least the one or more first points; and a projecting module 1730 configured to project a second image onto the target based on at least the one or more second points.
  • a first locating module 1710 configured to locate one or more first points on a first image of a target
  • a second locating module 1720 configured to locate one or more second points on the first image based on at least the one or more first points
  • a projecting module 1730 configured to project a second image onto the target based on at least the one or more second points.
  • the above modules 1710, 1720, and/or 1730 can be implemented as a pure hardware solution or as a combination of software and hardware, e.g., by one or more of: a processor or a micro-processor and adequate software and memory for storing of the software, a Programmable Logic Device (PLD) or other electronic component (s) or processing circuitry configured to perform the actions described above, and illustrated, e.g., in Fig. 15. Further, the electronic device 1700 may comprise one or more further modules, each of which may perform any of the steps of the method 1500 described with reference to Fig. 15.
  • PLD Programmable Logic Device

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)

Abstract

La présente divulgation concerne un dispositif électronique et un procédé de projection d'image améliorée pour la réalité augmentée. Un procédé de projection d'image consiste à : localiser un ou plusieurs premiers points sur une première image d'une cible ; localiser un ou plusieurs seconds points sur la première image sur la base du et/ou des premiers points ; et projeter une seconde image sur la cible sur la base du et/ou des seconds points.
PCT/CN2024/071515 2024-01-10 2024-01-10 Projection d'image améliorée pour réalité augmentée Pending WO2025147881A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2024/071515 WO2025147881A1 (fr) 2024-01-10 2024-01-10 Projection d'image améliorée pour réalité augmentée

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2024/071515 WO2025147881A1 (fr) 2024-01-10 2024-01-10 Projection d'image améliorée pour réalité augmentée

Publications (1)

Publication Number Publication Date
WO2025147881A1 true WO2025147881A1 (fr) 2025-07-17

Family

ID=96386115

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/071515 Pending WO2025147881A1 (fr) 2024-01-10 2024-01-10 Projection d'image améliorée pour réalité augmentée

Country Status (1)

Country Link
WO (1) WO2025147881A1 (fr)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0942915A (ja) * 1995-07-31 1997-02-14 Mitsubishi Heavy Ind Ltd 穴位置検出装置
CN103226817A (zh) * 2013-04-12 2013-07-31 武汉大学 一种基于透视投影的体表静脉图像增强现实方法及装置
CN108074264A (zh) * 2017-11-30 2018-05-25 深圳市智能机器人研究院 一种分级多目视觉定位方法、系统及装置
CN110596997A (zh) * 2018-06-13 2019-12-20 苏州宝时得电动工具有限公司 安装定位方法、装置及光学投影设备
US20200035034A1 (en) * 2017-10-26 2020-01-30 Tencent Technology (Shenzhen) Company Limited Method, device, terminal device and storage medium for realizing augmented reality image
CN116051928A (zh) * 2023-02-01 2023-05-02 上海非夕机器人科技有限公司 螺孔定位方法、装置、计算机设备和存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0942915A (ja) * 1995-07-31 1997-02-14 Mitsubishi Heavy Ind Ltd 穴位置検出装置
CN103226817A (zh) * 2013-04-12 2013-07-31 武汉大学 一种基于透视投影的体表静脉图像增强现实方法及装置
US20200035034A1 (en) * 2017-10-26 2020-01-30 Tencent Technology (Shenzhen) Company Limited Method, device, terminal device and storage medium for realizing augmented reality image
CN108074264A (zh) * 2017-11-30 2018-05-25 深圳市智能机器人研究院 一种分级多目视觉定位方法、系统及装置
CN110596997A (zh) * 2018-06-13 2019-12-20 苏州宝时得电动工具有限公司 安装定位方法、装置及光学投影设备
CN116051928A (zh) * 2023-02-01 2023-05-02 上海非夕机器人科技有限公司 螺孔定位方法、装置、计算机设备和存储介质

Similar Documents

Publication Publication Date Title
US11164001B2 (en) Method, apparatus, and system for automatically annotating a target object in images
US9792731B2 (en) System and method for controlling a display
JP4868186B2 (ja) マーカ生成及びマーカ検出のシステム、方法とプログラム
CN107273888A (zh) 码识别装置及码识别方法
US10592536B2 (en) Systems and methods for determining a location of a user when using an imaging device in an indoor facility
CN109614983A (zh) 训练数据的生成方法、装置及系统
JPWO2008026722A1 (ja) ３次元モデルデータ生成方法及び３次元モデルデータ生成装置
US11301659B2 (en) Installation support device and installation support method for stationary code reader
CN115035032B (zh) 神经网络训练方法、相关方法、设备、终端及存储介质
CN114882483A (zh) 一种基于计算机视觉的图书盘点方法
JP2020179441A (ja) 制御システム、情報処理装置および制御方法
JP2022009474A (ja) ビジョンシステムでラインを検出するためのシステム及び方法
US9785839B2 (en) Technique for combining an image and marker without incongruity
JP7036874B2 (ja) コード認識装置
WO2025147881A1 (fr) Projection d'image améliorée pour réalité augmentée
JP6459613B2 (ja) プリント配線板作業支援方法及びプリント配線板作業支援システム
CN116266388B (zh) 标注信息生成方法、装置、系统、电子设备及存储介质
CN118429443A (zh) 激光雷达与相机联合的外参标定方法、装置、设备和介质
US20250005878A1 (en) Processing control method and apparatus, and device
US20220414916A1 (en) Systems and methods for assigning a symbol to an object
CN105975566A (zh) 基于图像的信息搜索方法及装置
JP7792176B1 (ja) 画像および点群を用いたアノテーション支援装置、方法、およびプログラム
JP7204504B2 (ja) 対象物確認装置
JP2017016169A (ja) 検査方法、検査装置、画像処理装置、プログラム及び記録媒体
CN116778466B (zh) 一种大角度车牌识别方法、装置、设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24915961

Country of ref document: EP

Kind code of ref document: A1