WO2022099225A1

WO2022099225A1 - Methods and systems for generating point clouds

Info

Publication number: WO2022099225A1
Application number: PCT/US2021/065541
Authority: WO
Inventors: Kevin Chon; Zhebin ZHANG; Hongyu Sun; Jian Sun
Original assignee: Innopeak Technology Inc
Current assignee: Innopeak Technology Inc
Priority date: 2021-12-29
Filing date: 2021-12-29
Publication date: 2022-05-12
Anticipated expiration: 2024-06-29

Abstract

The present invention is directed to point cloud generation. In a specific embodiment, the present invention provides a method for generating a cloud. Two-dimensional images captured by cameras are processed to generate initial point clouds. Using data received from a radar, an integrated point cloud is generated using the initial point clouds and the radar data. There are other embodiments as well.

Description

METHODS AND SYSTEMS FOR GENERATING POINT CLOUDS

BACKGROUND OF THE INVENTION

[0001] The present invention is directed to methods and systems for generating point clouds.

[0002] Point clouds have a wide range of applications. The rapid advancement of mobile devices and artificial intelligence has made advanced driver-assistance systems (ADAS) a highly desired feature of new vehicles. The rise of the need and integration of ADAS systems has grown significantly over the past decade. The implementation and use of ADAS have improved safety and reduced accidents. Usually, an ADAS is needed to generate a point cloud — which is constantly updated — of a vehicle’s surroundings. The ADAS uses the point cloud in many ways, including navigation and obstacle avoidance. Due to various factors, the adoption of the ADAS — especially for older vehicles — has been slower than desired.

Among other things, generating and updating point clouds for ADAS applications is complex and expensive.

[0003] Therefore, it is desirable to have new and improved methods and systems for point clouds generation.

BRIEF SUMMARY OF THE INVENTION

[0001] The present invention is directed to point cloud generation. In a specific embodiment, the present invention provides a method for generating a point cloud. Two- dimensional images captured by cameras are processed to generate initial point clouds.

Using data received from radar, an integrated point cloud is generated using the initial point clouds and the radar data. The integrated point cloud may be used in ADAS or other applications. There are other embodiments as well.

[0004] A general aspect includes a method for generating point cloud data. The method includes providing a mobile device including a first camera and a second camera. The first camera is characterized by a first field of view, the second camera is characterized by a second field of view. The method also includes establishing a communication link with a radar device. The radar device is characterized by a second field of view. The communication link is characterized by a latency. The method further includes generating a

SUBSTITUTE SHEET (RULE 26) set of calibration parameters based at least on angles of view and operating frequencies and the latency. The method additionally includes receiving a first radar data via the communication link from the radar device. The first radar data may be generated during a first time interval. The method also includes capturing a first image using the first camera at a first time. The method also includes generating a first point cloud based on the first image. The method also includes capturing a second image using the second camera at a second time. The method further includes generating a second point cloud based on the second image. The method also includes generating an integrated point cloud using the first point cloud and the second point cloud. The method additionally includes updating the integrated point cloud using the first radar data. The method also includes identifying at least a first object and a first distance using at least the integrated point cloud. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0005] Another general aspect of certain embodiments includes a device for generating point cloud data. The device includes a communication interface configured to receive radar data from a radar device. The device also includes a first camera housing positioned on a first surface. The device also includes a second camera housing positioned on the first surface. The device also includes a first camera that comprises a first optical element and a first sensor. The first camera is positioned in the first camera housing. The first optical element is characterized by a first field of view and the first sensor is characterized by a first resolution. The device also includes a second camera that includes a second optical element and a second sensor. The second camera is positioned in the second camera housing. The second optical element is characterized by a second field of view and the second sensor is characterized by a second resolution. The device also includes a memory. The device also includes a processor coupled to the first camera and the second camera and the memory. The first camera is configured to generate a first image at a first time. The second camera is configured to generate a second image at a second time. The processor is configured to generate a first point cloud based on the first image and to generate a second point cloud based on the second image. The processor is further configured to generate an integrated point cloud using the first point cloud and the second point cloud and the radar data. The memory is configured to store the first point cloud and the second point cloud and the integrated point cloud.

SUBSTITUTE SHEET (RULE 26) [0006] Yet another general aspect of certain embodiments includes a method for providing collision warnings. The method includes receiving a first radar data from a radar device. The first radar data may be generated during a first time interval. The method further includes capturing a first image using a first camera at a first time. The method also includes generating a first point cloud based on the first image. The method also includes capturing a second image using a second camera at a second time. The method also includes generating a second point cloud based on the second image. The method also includes generating an integrated point cloud using the first point cloud and the second point cloud. The method also includes updating the integrated point cloud using the first radar data. The method also includes identifying at least a first object and a first distance using at least the integrated point cloud. The method also includes receiving a second radar data from the radar device. The second radar data may be generated during a second time interval. The method also includes calculating a velocity associated with the first object using at least the first radar data and the second radar data. The method additionally includes generating a warning signal associated with the first object. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0007] It is to be appreciated that embodiments of the present disclosure may provide advantages over conventional techniques. New techniques according to the present invention offer effective and accurate three-dimensional (“3D”) object detection and classification based on the point cloud generated from two-dimensional (“2D”) images captured by cameras and a radar sensor device. The procedure is easy to implement and is readily available for real-time application. Methods and systems according to the present invention can be readily incorporated into existing systems, including older vehicles that do not have any ADAS mechanisms. Embodiments of the present invention can be readily incorporated into various types of systems and devices. There are other benefits as well.

[0008] The present invention achieves these benefits and others in the context of known technology. However, a further understanding of the nature and advantages of the present invention may be realized by reference to the latter portions of the specification and attached drawings.

SUBSTITUTE SHEET (RULE 26) BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The following diagrams are merely examples, which should not unduly limit the scope of the claims herein. One of ordinary skill in the art would recognize many other variations, modifications, and alternatives. It is also understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this process and scope of the appended claims.

[0010] Figure 1 is a simplified block diagram illustrating a system for generating point cloud data according to embodiments of the present invention.

[0011] Figure 2 is a simplified flow diagram illustrating a method for generating point cloud data according to embodiments of the present invention.

[0012] Figure 3 is a simplified diagram illustrating data generated by cameras and a radar according to embodiments of the present invention.

[0013] Figure 4 is a simplified diagram illustrating an integrated point cloud according to embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0002] The present invention is directed to point cloud generation. In a specific embodiment, the present invention provides a method for generating a point cloud. Two- dimensional images captured by cameras are processed to generate initial point clouds.

Using data received from a radar, an integrated point cloud is generated using the initial point clouds and the radar data. There are other embodiments as well. It is to be understood that the term “point cloud” is broadly defined, which refers to a set of data points in space, and when used in an ADAS, a point cloud may include additional information such as object identification, velocity, and/or other relevant information, and the term should not unduly limit the scope of the claims.

[0003] For an ADAS system to perceive the environment in real-time accurately, it usually relies on different sensors such as camera, radar, and LiDAR, to generate a point cloud.

Using the point cloud in real-time, an ADAS can detect, predict, and alert potential incoming vehicles and objects, thereby avoiding accidents. These sensors are normally interconnectedly communicating to an onboard CPU or GPU, where all the sensor data can

SUBSTITUTE SHEET (RULE 26) be aligned and processed for further tasks such as object detection. Among these sensors, LiDAR has often been relied upon as the key sensor for object detection and classification tasks for its robustness and accuracy. Some LiDARs include spinning lidar, solid-state lidar, and flash lidar depending upon the type of use case and price. Regarding processing point cloud data generated from LiDAR, there are mainly 3D voxel grid-based or 2D bird’s eye view (BEV) based processing methods. LiDAR has several disadvantages for the ADAS use case, including, but not limited to, its significantly high cost and struggles to measure distance under non-ideal, adverse weather conditions.

[0014] It is to be appreciated the embodiments of the present invention provide new methods and systems for generating point clouds without using LiDAR, and this point cloud can be readily used (and generated) by an ADAS.

[0015] The following description is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications, will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of embodiments. Thus, the present invention is not intended to be limited to the embodiments presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0016] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without necessarily being limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

[0017] The reader’ s attention is directed to all papers and documents which are filed concurrently with this specification and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification, (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

SUBSTITUTE SHEET (RULE 26) [0018] Furthermore, any element in a claim that does not explicitly state “means for” performing a specified function, or “step for” performing a specific function, is not to be interpreted as a “means” or “step” clause as specified in 35 U.S.C. Section 112, Paragraph 6. In particular, the use of “step of’ or “act of’ in the Claims herein is not intended to invoke the provisions of 35 U.S.C. 112, Paragraph 6.

[0019] Please note, if used, the labels left, right, front, back, top, bottom, forward, reverse, clockwise and counterclockwise have been used for convenience purposes only and are not intended to imply any particular fixed direction. Instead, they are used to reflect relative locations and/or directions between various portions of an object.

[0020] Figure 1 is a simplified block diagram illustrating a system for generating point cloud data according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. System 100 for generating point cloud data, as shown in Figure 1, includes a first camera 101, a second camera 102, a memory 103, a processor 104, and a communication interface 105. For example, system 100 may be a mobile phone, a tablet device, or a vehicle-based device. In addition to generating point cloud, system 100 may be an ADAS that provide accident avoidance, navigation, and/or other functions. In various embodiments, system 100 and/or radar 110 are secured on a mounting mechanism (e.g., car mount, clamp, etc.), the mounting mechanism aligns radar and the cameras of system 100 so that they point to the same general direction. Depending on the implementation, system 100 and radar 110 may be secured on different mounting mechanisms, and alignment and calibration processes may be performed to ensure that the radar device and the cameras point to a general direction.

[0021] Radar device 110, first camera 101, and second camera 102 are configured to detect one or more objects (e.g., first object 120, second object 130, and third object 140, etc.) and provide input data for system 100. In the ADAS driving scenario, these objects of interest usually include items such as vehicles, pedestrians, traffic signs. In a different environment such as indoor, these objects of interest can be different.

[0022] Radar devices are well suited for detecting distance, radial velocity, obstacle detection, and accuracy in variable environmental conditions. For example, radar device 110 is configured to provide range and azimuth data in two dimensions. In some cases, radar device 110 further includes a radar clock and processor 104 may be configured to synchronize with the radar clock. The radar device 110 additionally includes a wired or

SUBSTITUTE SHEET (RULE 26) wireless communication interface for sending radar data to system 100. Radar device 110 may comprise a 2D radar transmitter that emits a 2D radar signal along a plane that is orthogonal to the 2D radar transmitter. In some instances, the 2D radar transmitter comprises at least one antenna disposed on an integrated circuit ("IC") chip, wherein at least one antenna comprises one of a single IC -based antenna disposed on the IC chip, a plurality of IC-based antennas arranged as a one-dimensional ("ID") line of antennas disposed on the IC chip, or a 2D array of IC-based antennas disposed on the IC chip, and/or the like. The plane may be orthogonal to a surface of the IC chip on which at least one antenna is disposed. In some cases, the plane may be parallel to a ground surface below the radar sensor. In a nonlimiting application scenario of ADAS, for example, radar device 110 may be a millimeterwave radar. In some cases, radar device 110 may be configured for short-range radar (“SSR”) applications, where the objects can be localized up to around 80m within a range resolution of 35cm for objects. The antenna field of view may be +-60 degrees with an angular resolution of 15 degrees. For example, the data output of radar device 110 may be in a form of a 2D map containing a plurality of depth values.

[0023] Communication interface 105 is configured to receive radar data from radar device 110. For example, communication interface 105 may be wired or wireless. For example, communication interface 105 may be a wired connection, such as USB-C or Lightning connections. Communication interface 105 may also include wireless connections such as Bluetooth, WIFI, and/or others. Typically, wired connections are characterized by lower latencies than wireless connections, but either way the latency of receiving data from radar device 110 is taken into consideration when the radar data are used.

[0024] Video sensors such as cameras are well suited for detecting angles, boundaries of objects, object classification, etc. A plurality of camera/video sensors (e.g., first camera 101 and second camera 102) may be configured either in a compact format (e.g., enclosed in the same chamber) or a distributed format (e.g., placed within different chambers), and/or the like. It is to be noted that while camera 101 and camera 102 have different fields of view, they point to the same general direction, and the images they capture partially overlap.

[0025] For example, first camera 101 is configured in a first camera housing. First camera 101 includes a first optical element and a first sensor. The first optical element is characterized by a first field of view. The first sensor is characterized by a first resolution. The data output of first camera 101 may be in form of a first image. For example, first camera 101 is configured to generate the first image at a first time. Second camera 102 is

SUBSTITUTE SHEET (RULE 26) configured in a second camera housing. The second camera housing is positioned on the first surface of the first mounting mechanism. Second camera 102 includes a second optical element and a second sensor. The second optical element is characterized by a second field of view. The second sensor is characterized by a second resolution. The data output of second camera 102 may be in form of a second image. For example, second camera 102 is configured to generate the second image at a second time.

[0026] It is to be understood that the two cameras are configured to capture images at about the same time (e.g., synchronized), but due to various factors of implementation (e.g., device synchronization, clocking), the capture times of the first image and second image may not be exactly the same, but the difference in image capturing time is minimized (or at least taken into consideration) when the first image and second image are used to generate a point cloud. Cameras of a mobile device often have different sensor resolutions. For example, the “main” camera of a mobile device usually has a higher resolution than other cameras. For the purpose of point cloud generation, different sensor resolutions may be “matched” so objects captured by different cameras can be correlated and integrated in a point cloud.

[0027] The cameras of system 100 may be implemented in ways. For example, instead of two cameras with their own lenses and sensors, a stereo camera that includes two lenses — configured with the same field of view and positioned next to each other — that share the same sensor. For example, the camera may be a monocular camera, a stereo camera, and/or the like. For the data outputs of the sensors to be further processed and adopted by ADAS, the first mounting mechanism for the first camera and the second mounting mechanism for the second camera may be aligned according to a plurality of predefined mounting parameters. In some instances, the plurality of predefined mounting parameters may include a plurality of intrinsic parameters and a plurality of extrinsic parameters. For example, the plurality of intrinsic parameters is related to the conversion from camera to pixel coordinates and may include focal length, optical center, and others. The plurality of extrinsic parameters may be related to the conversion from world to camera coordinates and may include translation matrix, rotation matrix, and others. After the alignment between the first mounting mechanism and the second mounting mechanism, object detection may be performed to identify objects that are sensed or captured by radar device 110 and first camera 101 and second camera 102. The resultant object detection data may be sent to processor 104 to enable driver assistance.

SUBSTITUTE SHEET (RULE 26) [0028] Mobile devices such as smartphones typically include two or more cameras that have different fields of view: a “normal” camera with a field of view of about 70 degrees, an “ultrawide” camera with a field of greater than 100 degrees, and a “telephoto” camera with a field of view less than 40 degrees. In various implementations, cameras with a wide field of view are selected. In some embodiments, more than two cameras may be used for generating and updating a point cloud. As shown in Figure 1, camera 101 is characterized by a narrower field of view than that of camera 102 (i.e., 0i < 02). As a way of an example, radar device 110 is characterized by a field of view angle 03. Objects 120 and 130 are within the field of view of radar device 110; only object 130 is within the field of view of camera 101; and objects 120, 130, and 140 are within the field of view of camera 102. Depending on the implementation, the point cloud generated using both the radar data and camera images may include some or all of the objects. For example, object 130 is detected by cameras and the radar device, and point cloud would have the most precise information; both camera parallax and radar information can be used. On the other hand, object 120 is detected by camera 102 and radar device 110, and its point cloud information is likely to be less accurate compared to that of object 130. Object 140 as shown, is only captured by camera 102, and its point cloud information is likely to be the least accurate. In various embodiments, an ADAS is capable of using the information of all cameras and radar device with deep machine learning techniques.

[0029] According to some embodiments, processor 104 is coupled to first camera 101, second camera 102, memory 103, and communication interface 105. For example, processor 104 may be a microprocessor that is particularly suitable for object detection (e.g., a GPU). Communication interface 105 is configured to receive radar data from radar device 110 and communicate the radar data to processor 104. Based on the image data received from first camera 101 and second camera 102, and the radar data from radar device 110, processor 104 is configured to generate a depth map. For example, when the image data is collected from one or more monocular cameras (e.g., cameras 101 and 102), processor 104 may apply a monocular depth estimation model upon the image to generate a depth map, which may or may not include radar data. For example, camera focusing distance and edge detection information may be used to generate a depth map or an initial point cloud. When the image data is collected from one or more stereo cameras, a depth map may be generated based on an epipolar scheme and a disparity map. Processor 104 may convert the depth map and the first image into a first point cloud based on the first image. Processor 104 may also convert the depth map and the second image into a second point cloud. For example, each pixel

SUBSTITUTE SHEET (RULE 26) within the first point cloud and/or the second point cloud contains a depth value. Further, processor 104 is configured to generate an integrated point cloud using the first point cloud and the second point cloud and the radar data. In some cases, different fields of view of the cameras 101 and 102 are calibrated for generating the integrated point cloud. The integrated point cloud may be characterized by a third resolution that is different from the resolutions of the images captured by cameras 101 and 102; for ADAS, high image resolution that is intended for screen display is not needed, and images captured by cameras 101 and 102 are scaled down to point cloud resolution. This third may be the lower of, or a fraction of, the lower of the two camera resolutions.

[0030] Memory 103 stores the first point cloud and the second point cloud and the integrated point cloud. Processor 104 is coupled to memory 103, and it stores and updates point cloud data at memory 103. In various implementations, cameras and the communication interface 105 may be directly connected to memory 103 so that image data and radar data can be directly stored. For speed and availability, memory 103 may be implemented using synchronous dynamic random-access memory (SDRAM), but other types of memory devices may be used as well, such as flash memory, other non-volatile memory devices, random-access memory ("RAM"), static random-access memory ("SRAM"), dynamic random-access memory ("DRAM"), virtual memory, a RAM disk, or other volatile memory devices, non-volatile RAM devices, and/or the like.

[0031] Figure 2 is a simplified flow diagram illustrating a method for generating point cloud data according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. As an example, one or more of the steps may be added, removed, repeated, rearranged, overlapped, and modified, and should not limit the scope of claims.

[0032] At step 205, a mobile device including a first camera and a second camera is provided. For example, the mobile device may be system 100 in Figure 1. The first camera is characterized by a first field of view. The second camera is characterized by a second field of view. The first camera is configured to operate at a first frequency. The second camera is configured to operate at a second frequency. The first camera and the second camera may have their own sensors at the same or different resolutions. In some cases, the first camera and the second camera share an image sensor. Image resolution, image capturing frequency, camera latency, processing speeds, and/or other factors are all taken into consideration when

SUBSTITUTE SHEET (RULE 26) it comes to generating a point cloud that can be used by an ADAS. For example, if the first camera and the second camera capture images at different frequencies, a lower common frequency may be selected for the point cloud generation.

[0033] At step 210, a communication link with a radar device is established. The radar device is characterized by a third field of view. For example, the communication link may be wired or wireless, as explained above. The communication link is characterized by a latency that is taken into consideration when radar data are correlated with camera images when generating an integrated point cloud.

[0034] At step 215, a set of calibration parameters is generated based on at least angles of view and operating frequencies and the latency. Typically, a radar device has a higher operating frequency than cameras, and the limiting factor is usually not the speed for the radar device, but the quality of the communication link. For ADAS-related applications, the latency attribute to the radar device is minimized and compensated. Additionally, other operating parameters of the radar device are calibrated. For example, the radar device may have a field of view that is different from that of the cameras, and the calibration parameters are generated according to account for the differences.

[0035] At step 220, a first radar data is received via the communication link from the radar device. As explained above, a radar device typically operates at a higher frequency compared to that of the cameras. In various implementations, the radar device sends radar data via the communication interface at predefined intervals. During these intervals, the radar device may have performed scanning multiple times, and the first radar data are generated based on one or more of these scannings.

[0036] At step 225, a first image is captured using the first camera at a first time. The first image is characterized by a first resolution. For example, the first image is captured by camera 101 or 102.

[0037] At step 230, a first point cloud is generated based on the first image. For example, the first point cloud comprises a two-dimensional point cloud or a three-dimensional point cloud. In various embodiments, edge detection and/or other algorithms may be used to identify objects, thereby allowing distance estimation and the generation of a three- dimensional point cloud. Information such as focusing distance and relative objective size (e.g., the expected height of a person in an image with the known field of view) can also be used for distance estimation, thus making three-dimensional point cloud possible.

SUBSTITUTE SHEET (RULE 26) [0038] Figure 3 is a simplified diagram illustrating data generated by cameras and a radar according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. As an example, point cloud 301 includes information for objects 310, 320, and 330. In various implementations, these objects are detected using edge detection and/or other image processing techniques. As an example, object 320 partially covers object 310, and it may be inferred that object 320 is closer to the camera than object 310.

[0039] Now referring back to Figure 2. At step 235, a second image is captured using the second camera at a second time. The second image is characterized by a second resolution. For example, the second image is captured by camera 101 or 102.

[0040] At step 240, a second point cloud is generated based on the second image. For example, the second point cloud comprises a two-dimensional point cloud or a three- dimensional point cloud. As explained above, edge detection and/or other algorithms may be used to identify objects, thereby allowing distance estimation and the generation of the three- dimensional point cloud. Information such as focusing distance and relative objective size (e.g., the expected height of a person in an image with the known field of view) can also be used for distance estimation, thus making three-dimensional point cloud possible.

[0041] Figure 3 illustrated point cloud 302 that is generated based on the second image captured by the second camera. As an example, the second camera has a wider field of view than that of the first camera, and as a result, point cloud 302 includes more edges of objects 310 and 330 than point cloud 301. And due to camera parallax, the relative positions of these objects are different in point clouds 301 and 302. The difference in relative positions can be used in generating the integrated point cloud.

[0042] Now referring back to Figure 2. At step 245, an integrated point cloud is generated using the first point cloud and the second point cloud. The integrated point cloud is three- dimensional. The integrated point cloud is generated by interpolating the first point cloud and the second point cloud using the known parallax between the first camera and the second camera. Additionally, factors such as camera angles, fields of view, resolutions, capturing frequencies, and others may be taken into considerations when generating the integrated point cloud. For example, the integrated point cloud is stored on a memory (e.g., memory 103 in Figure 1).

SUBSTITUTE SHEET (RULE 26) [0043] At step 250, the integrated point cloud is updated using at least radar data.

[0044] The generation of the integrated point cloud involves using the first radar data and its calibration parameters. Radar data typically do not have as many “pixels” compared to the images captured by cameras, but radar data points are richer in the distance information they contain. During the process of generating the integrated point cloud, distance information from the radar data is mapped to the point clouds that were generated from camera images. In various embodiments, object identification and distance estimation were performed at steps 230 and 240, and the radar data are used to accurize distance data.

[0045] Data points in the radar data are illustrated in Figure 3. As an example, point 403c is associated with object 310; distance and/or velocity information of point 403c is assigned to object 310. Similarly, points 403a and 403b information is assigned to object 320; points 403e and 403f information is assigned to object 330. Point 403d is not associated with any objects, and it may be used to identify an “open space” or sky in the point cloud. During the process of generating the integrated point cloud, data points from the radar data are used in conjunction with point clouds generated by images (e.g., point clouds 301 and 302). In various embodiments, machine learning algorithms are utilized in generating the integrated point cloud.

[0046] Figure 4 is a simplified diagram illustrating an integrated point cloud according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. For example, integrated point 400 shows both objects and radar data points. Frame 401 shows data associated with camera 1 (with the field of view angle 0i) and its point cloud; frame 402 shows data associated with camera 2 (with the field of view angle 62) and its point cloud; and frame 403 shows data associated with the radar device (with the field of view angle 63). As an example, point cloud 400 contains a large spatial volume (e.g., for headroom) than the first and the second point clouds.

[0047] At step 255, at least a first object and a first distance are identified using at least the integrated point cloud. For example, edge detection techniques and radar data may be used in combination for object identification and distance determination.

[0048] At step 260, the velocity of the first object is calculated. For example, a set of second radar data is used to determine the velocity of the first object. In various embodiments, the second radar data are captured at a time interval that is later than the first

SUBSTITUTE SHEET (RULE 26) time interval, with updated distance and/or velocity information. For example, the second radar data are used to calculate velocity (e.g., change in distance vs. difference in time). Images from the first camera and the second camera may be used to update the point cloud as well. In some implementations, velocity and other information of objects are stored in the integrated point cloud. As shown in Figure 2, steps 220-260 are repeated for the integrated point cloud to be updated (and in real-time for ADAS applications).

[0049] In ADAS-related applications, the integrated point cloud data may be used to provide collision warnings according to some embodiments of the present invention. For example, velocity and/or distance information of the first object may be compared to one or more predefined threshold values. For example, a warning signal may be generated if the distance from an object is below a threshold value. The point integrated point cloud may also be used to improve real-time navigation. In various implementations, the latency — attributed to radar, cameras, and processing delay — associated with the integrated point cloud are taken into consideration in ADAS, navigation, and other applications.

[0050] While the above is a full description of the specific embodiments, various modifications, alternative constructions and equivalents may be used. Therefore, the above description and illustrations should not be taken as limiting the scope of the present invention which is defined by the appended claims.

SUBSTITUTE SHEET (RULE 26)

Claims

WHAT IS CLAIMED IS:

1. A method for generating point cloud data, the method comprising: providing a mobile device including a first camera and a second camera, the first camera being characterized by a first field of view, the second camera being characterized by a second field of view; establishing a communication link with a radar device, the radar device being characterized by a third field of view, the communication link being characterized by a latency; generating a set of calibration parameters based at least on angles of view and operating frequencies and the latency; receiving a first radar data via the communication link from the radar device, the first radar data being generated during a first time interval; capturing a first image using the first camera at a first time; generating a first point cloud based on the first image; capturing a second image using the second camera at a second time; generating a second point cloud based on the second image; generating an integrated point cloud using the first point cloud and the second point cloud; updating the integrated point cloud using the first radar data; and identifying at least a first object and a first distance using at least the integrated point cloud.

2. The method of claim 1 further comprising performing edge detection to identify the first object.

3. The method of claim 1 further comprising calculating a velocity associated with the first object.

4. The method of claim 1 further comprising interpolating the first point cloud and the second point cloud using a known parallax between the first camera and the second camera.

5. The method of claim 1 further comprising interpolating the first point cloud and the second point cloud based on a difference between the first field of view and the second field of view, the first field of view being different from the second field of view.

SUBSTITUTE SHEET (RULE 26)

6. The method of claim 1 wherein the first point cloud comprises a two- dimensional point cloud and the integrated point cloud comprises a three-dimensional point cloud.

7. The method of claim 1 further comprising: storing the integrated point cloud on a memory; receiving a second radar data; and updating the integrated point cloud based on at least on the second radar data.

8. The method of claim 1 wherein the first camera and the second camera share an image sensor.

9. The method of claim 1, comprising correlating a first resolution and a second resolution and a third resolution, wherein: the first image is characterized by the first resolution; the second image is characterized by the second resolution; and the first radar data is characterized by the third resolution.

10. The method of claim 1, further comprising correlating a first frequency and a second frequency and a third frequency, wherein: the first camera is configured to operate at the first frequency; the second camera is configured to operate at the second frequency; the radar device is configured to operate at the third frequency.

11. A device for generating point cloud data, the device comprising: a communication interface configured to receive radar data from a radar device; a first camera housing positioned on a first surface; a second camera housing positioned on the first surface; a first camera comprising a first optical element and a first sensor, first camera being positioned in the first camera housing, the first optical element being characterized by a first field of view, the first sensor being characterized by a first resolution; a second camera comprising a second optical element and a second sensor, second camera being positioned in the second camera housing, the second optical element being characterized by a second field of view, the second sensor being characterized by a second resolution;

SUBSTITUTE SHEET (RULE 26) 17 a memory; and a processor coupled to the first camera and the second camera and the memory; wherein: the first camera is configured to generate a first image at a first time; the second camera is configured to generate a second image at a second time; the processor is configured to generate a first point cloud based on the first image and to generate a second point cloud based on the second image; the processing is further configured to generate an integrated point cloud using the first point cloud and the second point cloud and the radar data; and the memory is configured to store the first point cloud and the second point cloud and the integrated point cloud.

12. The device of claim 11 , wherein the integrated point cloud is characterized by a third resolution, the third resolution being defined based on the first resolution and the second resolution.

13. The device of claim 11 , wherein: the radar device comprises a radar clock; the processor is configured to synchronize with the radar clock.

14. The device of claim 11 , wherein the first field of view and the second field of view are calibrated for generating the integrated point cloud.

15. The device of claim 11 , further comprising a first mounting mechanism, the radar device being secured on a second mounting mechanism, the first mounting mechanism and the second mounting mechanism being aligned according to a plurality of predefined mounting parameters.

16. A method for providing collision warnings, the method comprising: receiving a first radar data from a radar device, the first radar data being generated during a first time interval; capturing a first image using a first camera at a first time; generating a first point cloud based on the first image; capturing a second image using a second camera at a second time; generating a second point cloud based on the second image;

SUBSTITUTE SHEET (RULE 26) 18 generating an integrated point cloud using the first point cloud and the second point cloud; updating the integrated point cloud using the first radar data; identifying at least a first object and a first distance using at least the integrated point cloud; receiving a second radar data from the radar device, the second radar data being generated during a second time interval; calculating a velocity associated with the first object using at least the first radar data and the second radar data; and generating a warning signal associated with the first object.

17. The method of claim 16, further comprising updating the integrated point cloud using the second radar data.

18. The method of claim 16, further comprising comparing the first distance to a first threshold value.

19. The method of claim 16, further comprising comparing the velocity to a second threshold value.

20. The method of claim 16, further comprising determining a latency associated with the integrated point cloud.

SUBSTITUTE SHEET (RULE 26)