RU2836700C1 - Method and processor for classification of points on boundary of polygonal section - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: group of inventions relates to image processing. Method of classifying points of boundary of a polygonal portion of an object on an image of a neighbourhood map includes obtaining image data for a training area comprising a training object, generating a segmentation map containing a plurality of points based on the image data and determining the training boundary of the polygonal section. Plurality of points is associated with a feature vector formed by the segmentation model. Method includes obtaining the actual boundary of the polygonal section, determining the deviation of the position from the nearest edge from the sequence of edges indicating the mark of the boundary point, and training the second model. Training iteration includes use of a feature vector associated with a boundary point and a set of adjacent point feature vectors to generate a predicted estimate of the position of the boundary point, as well as adjusting the second model based on the result of comparing the predicted estimate and the mark.

EFFECT: high accuracy of navigation and safety of controlling autonomous vehicles.

24 cl, 11 dwg

Description

Field of technology to which the invention relates

[001] The present technology generally relates to image processing, and in particular to methods and processors for classifying points on the boundary of a polygonal region.

State of the art

[002] In known solutions, computer navigation systems are proposed and implemented that facilitate navigation and/or control of vehicles. Various systems of this kind are known: from simple solutions based on determining the location on a map and using a computer system to assist the driver in navigating along a route from the departure point to the destination, to more complex solutions, such as computerized and/or autonomous driving systems.

[003] Some of these systems are implemented in the form of the well-known cruise control system. In this case, a computer system located on the vehicle maintains the vehicle's speed set by the user. Some cruise control systems implement an intelligent distance control system, allowing the user to set the distance to the vehicle in front (for example, to select a value expressed in the number of vehicles). In this case, the computer system regulates the vehicle's speed, at least in part, depending on its approach to the vehicle in front at a predetermined distance. Some of the cruise control systems are additionally equipped with a collision warning system, which, upon detecting a vehicle (or other obstacles) in front of the moving vehicle, slows down or stops it.

[004] Some of the most advanced systems provide fully autonomous vehicle operation without direct operator (i.e. driver) involvement in the control. Such autonomous vehicles contain systems capable of accelerating, decelerating, stopping, changing lanes, and parking the vehicle independently.

[005] Autonomous vehicles may use map data to process navigation and control information in addition to or instead of sensor data. However, traditional maps have a number of drawbacks that make them difficult to use in autonomous vehicles such as self-driving cars and delivery robots. In particular, maps do not provide the accuracy required for safe navigation (e.g., 10 cm or less). GPS systems provide an accuracy of approximately 3-5 m, but due to certain conditions, significant errors occur and the error can exceed 100 m. This makes it difficult to accurately determine the location of the vehicle. Traditional methods of maintaining digital maps and identifying objects on maps do not provide sufficiently accurate and up-to-date data for safe navigation and control of autonomous vehicles.

[006] Patent application US20180188059 describes a system for generating dividing lines that indicate traffic lanes of roads, for example, based on displaying pixels of a camera image that correspond with a high probability to dividing lines.

Disclosure of invention

[007] In connection with the above, systems and methods are needed that allow one to avoid, reduce or overcome the limitations of known technologies.

[008] In the context of the present technology, a segmentation model is used to segment image data with respect to map regions to determine objects and to form boundaries for these objects in the corresponding map regions. It is assumed that these objects may include road sections accessible to road vehicles (e.g., cars, trucks, buses, and motorcycles), sidewalk sections accessible to pedestrians and relatively small robotic vehicles (e.g., delivery robots), path sections accessible to pedestrians, road vehicles, and relatively small robotic vehicles, etc.

[009] The segmentation model can be applied at the use stage to areas of the map that have not been assessed by experts. For example, a cartographer may be tasked with tracking the boundaries of objects of one or more classes on a map. However, obtaining such expert-assessed data requires significant time and money. Since obtaining data on the actual boundaries of objects on a very large map is practically impossible, the developers of this technology have developed a system where information on the actual boundaries of objects is used in expert-assessed areas of the map, and the results of the segmentation model can be used for areas of the map that have not been assessed in this way.

[010] The developers of the present technology have found that, while the segmentation model is useful in unrated areas, the model does not provide satisfactory prediction quality in some of these unrated areas. In this regard, the developers have developed methods and systems that use a second model to indicate the reliability of the boundary information generated by the segmentation model.

[011] In general, the second model may be trained to predict, for one or more points along the boundary formed by the segmentation model, confidence scores indicating whether these points are correctly placed on the map. In some embodiments of the invention, the confidence scores may be said to indicate the probability of correctly placing these points on the map. In other embodiments of the invention, the confidence scores may be said to indicate the level of confidence that these points are correctly placed on the map.

[012] The estimated map areas may be used to train a second model based on actual input from experts and segmentation data from the segmentation model. When the prediction model determines during its use phase that one or more points of the boundary generated by the segmentation model for an unestimated area may be incorrectly located, then this unestimated area may be marked for evaluation by an expert. In other words, the segmentation model may be used to generate boundaries of polygonal areas in some unestimated areas without additional input from an expert, while in other unestimated areas, the output of the second model may indicate that the determination of actual boundaries should be entrusted to experts instead of using only the output of the segmentation model.

[013] According to a first aspect of the present technology, a method for classifying points on a boundary of a polygonal region is implemented. The method is performed by a processor. The method includes receiving, by the processor, image data for a training region containing a training object. The method includes forming, by the processor, a segmentation map using a segmentation model based on the image data. The segmentation map contains a plurality of points and defines a boundary of the polygonal region. The plurality of points is associated with a feature vector generated by the segmentation model. The training boundary of the polygonal region defines a training object in the training region. The training boundary of the polygonal region contains a sequence of boundary points. The position of the boundary point in the sequence is defined as a change in position relative to the position of the previous boundary point from this sequence. The method includes receiving, by the processor, an actual boundary of the polygonal region defining the training object on the segmentation map. The actual boundary of the polygonal region contains a sequence of boundary edges determined by an expert. The method includes determining, for a boundary point from the sequence of boundary points, a deviation of its position from the nearest edge from the sequence of edges. The deviation indicates a label of the boundary point. The method includes training the processor of the second model to predict the reliability estimate for the boundary point of the stage of use of the boundary of the polygonal section of the stage of use. Training during the training iteration includes using the second model to form the predicted estimate: the position of the boundary point, the feature vector associated with the boundary point, and a set of feature vectors of neighboring points on the segmentation map representing the context of the boundary point. Training during the training iteration includes adjusting the second model based on a comparison of the predicted estimate and the label.

[014] In some embodiments of the method, it further includes receiving, by the processor, an image data of the stage of use for a region of the stage of use containing an object of the stage of use. The object of the stage of use has not been evaluated by the expert to form the actual boundary of the polygonal region defining the object of the stage of use. The method further includes generating, by the processor, a segmentation map of the stage of use defining the boundary of the polygonal region of the stage of use, containing a sequence of points of the boundary of the stage of use, using a segmentation model. The method further includes generating, by the processor, a confidence estimate for a point of the boundary of the stage of use from the sequence of points of the boundary of the stage of use using a second model. The method further includes automatically determining at least a part of the region of the stage of use for evaluation by the expert if the confidence estimate indicates low confidence. The method includes receiving, by the processor, the actual boundary of the polygonal region of the stage of use object from the expert.

[015] In some embodiments of the method, the training object is a section of a road or a section of a sidewalk.

[016] In some embodiments of the method, the image data comprises a bird's eye view representation of the training region.

[017] In some embodiments of the method, the image data comprises a two-dimensional image of the training region formed from a three-dimensional point cloud representing the training region.

[018] In some embodiments of the method, the image data comprises one or more raster images.

[019] In some embodiments of the method, the segmentation model is a fully convolutional neural network.

[020] In some embodiments of the method, the second model is a classification model, and the label indicates a first class corresponding to a high confidence class if the deviation is less than a predetermined threshold, or indicates a second class corresponding to a low confidence class if the deviation is greater than a predetermined threshold.

[021] In some embodiments of the method, the second model comprises two heads, wherein the first head is trained based on a first label indicating a deviation from the nearest edge, and the second head is trained based on a second label indicating that the deviation is less than or greater than a predetermined threshold.

[022] In some embodiments of the method, the boundary of the polygonal region at the use stage is stored in combination with the region of the use stage and transmitted to a second processor associated with the vehicle for navigating the vehicle in the region of the use stage.

[023] In some embodiments of the method, the vehicle is a self-driving car (SDC) operated on road sections.

[024] In some embodiments of the method, the vehicle is a delivery robot operated on sections of sidewalk.

[025] According to a second aspect of the present technology, a processor is implemented for classifying points on the boundary of a polygonal region. The processor is capable of receiving image data for a training region containing a training object. The processor is capable of forming a segmentation map based on the image data using a segmentation model. The segmentation map contains a plurality of points associated with a feature vector formed by the segmentation model and defines a training boundary of the polygonal region. The training boundary of the polygonal region specifies a training object in the training region and contains a sequence of boundary points, wherein the position of the boundary point in the sequence is specified as a change in position relative to the position of the previous boundary point from this sequence. The processor is capable of receiving an actual boundary of the polygonal region specifying the training object on the segmentation map and containing a sequence of boundary edges determined by an expert. For a boundary point from the sequence of boundary points, the processor is capable of determining a position deviation from the nearest edge from the sequence of edges indicating a label of the boundary point. The processor is capable of training a second model for predicting a reliability assessment for a boundary point of the use stage of the boundary of the polygonal region of the use stage. Training during a training iteration involves the second model using the feature vector associated with the boundary point and a set of feature vectors of neighboring points on the segmentation map representing the context of the boundary point to form a predicted estimate of the boundary point's position. Training during a training iteration involves adjusting the second model based on a comparison of the predicted estimate and the label.

[026] In some embodiments, the processor is further configured to obtain image data of the use stage for a region of the use stage containing a use stage object that has not been evaluated by the expert, in order to generate an actual boundary of a polygonal region defining the use stage object. The processor is further configured to generate a segmentation map of the use stage defining a boundary of the polygonal region of the use stage, comprising a sequence of points of the use stage boundary, using a segmentation model. The processor is further configured to generate a confidence estimate for a point of the use stage boundary from the sequence of points of the use stage boundary using a second model. The processor is further configured to automatically determine at least a portion of the region of the use stage for evaluation by the expert if the confidence estimate indicates low confidence. The processor is further configured to obtain the actual boundary of the polygonal region of the use stage object from the expert.

[027] In some embodiments of the processor, the training object is a section of a road or a section of a sidewalk.

[028] In some embodiments of the processor, the image data comprises a bird's eye view representation of the training region.

[029] In some embodiments of the processor, the image data comprises a two-dimensional image of the training region generated from a three-dimensional point cloud representing the training region.

[030] In some embodiments of the processor, the image data comprises one or more raster images.

[031] In some embodiments of the processor, the segmentation model is a fully convolutional neural network.

[032] In some embodiments of the processor, the second model is a classification model, and the label indicates a first class corresponding to a high confidence class if the deviation is less than a predetermined threshold, or indicates a second class corresponding to a low confidence class if the deviation is greater than a predetermined threshold.

[033] In some embodiments of the processor, the second model comprises two heads, wherein the first head is trained based on a first label indicating a deviation from the nearest edge, and the second head is trained based on a second label indicating that the deviation is less than or greater than a predetermined threshold.

[034] In some embodiments of the processor, the boundary of the polygonal region in the use stage is stored in combination with the region of the use stage and transmitted to a second processor associated with the vehicle for navigating the vehicle in the region of the use stage.

[035] In some embodiments of the processor, the vehicle is an SDC vehicle operated on road sections.

[036] In some embodiments of the processor, the vehicle is a delivery robot operated on sections of sidewalk.

[037] As used herein, the term "light source" refers to any device capable of providing radiation in the form of a beam, such as, but not limited to, a beam of light comprising radiation of one or more wavelengths of the electromagnetic spectrum. In one example, the light source may be a laser. Thus, the light source may comprise a laser such as a solid-state laser, a laser diode, a high-power laser, or an alternative light source such as a light-emitting diode (LED) light source. In some (non-limiting) examples, the laser comprises a Fabry-Perot laser diode, a quantum well laser, a Distributed Bragg Reflector (DBR) laser, a Distributed FeedBack (DFB) laser, a fiber laser, or a VCSEL Vertical-Cavity Surface-Emitting Laser. In addition, the laser may emit light beams of various types, such as light pulses, continuous oscillations (CW, Continuous Wave), quasi-continuous oscillations, etc. In some non-limiting examples, the laser may comprise a laser diode capable of emitting light with a wavelength in the range of approximately 650-1150 nm. Alternatively, the light source may comprise a laser diode capable of emitting light beams with a wavelength in the ranges of approximately 800-1000 nm, approximately 850-950 nm, approximately 1300-1600 nm, or any other suitable range. Unless otherwise specified, the term "approximately" with respect to a numerical value means a deviation of up to 10% from the specified value.

[038] In the context of the present description, the expression "output beam" may correspond to a radiation beam generated by a radiation source and directed towards a region of interest (ROI). The output beam may have one or more parameters such as pulse duration, beam angular dispersion, wavelength, instantaneous power, photon density at different distances from the light source, average power, beam intensity, beam width, beam repetition rate, beam sequence, pulse duty cycle, phase, etc. The output beam may be unpolarized or randomly polarized, may not have a specific or fixed polarization (e.g., polarization may change over time) or may have a specific polarization (e.g., linear polarization, elliptical polarization, or circular polarization).

[039] As used herein, the term "input beam" refers to radiation or light that enters the system, typically after being reflected from one or more objects in the ROI. An input beam may also be referred to as a radiation beam or a light beam. By reflection, it is meant that at least a portion of the output beam incident on one or more objects in the ROI is reflected from one or more objects. An input beam may have one or more parameters such as a time of flight (i.e., the time from emission to detection), an instantaneous power (e.g., a power signature), an average power over the entire return pulse, a spectrum or signal over the period of the return pulse, etc. Depending on the particular use case, a portion of the radiation or light in the input beam may come from sources other than the reflected output beam. For example, at least a portion of the input beam may comprise noise light from the environment (including scattered sunlight) or from other light sources external to the system.

[040] In the context of the present description, the term "surrounding area" or "environment" of a vehicle corresponds to an area or space around the vehicle that contains part of its surrounding environment at a given time that is available for scanning using one or more sensors mounted on the vehicle, for example, to form a 3D map (three-dimensional map) of such surrounding area or to detect objects.

[041] In the context of the present description, the ROI region may comprise a portion of the observable environment of the lidar system, where one or more objects can be detected. It should be noted that the ROI region of the lidar system may depend on various conditions, such as the orientation of the lidar system (e.g., the direction of the optical axis of the lidar system), the position of the lidar system relative to the environment (e.g., the height relative to ground level and the distance to adjacent terrain elements and obstacles), the operating parameters of the lidar system (e.g., the radiation power, the computing settings, the specified operating angles), etc. For example, the ROI region of the lidar system can be defined as a plane angle or a solid angle. In one example, the ROI region can also be defined by a specific range of distances (e.g., up to 200 m, etc.).

[042] In the context of the present description, the term "server" means a computer program executed by appropriate hardware and capable of receiving requests (e.g., from electronic devices) via a network and executing or causing these requests to be executed. The hardware may be one physical computer or one computer system, which is not essential for the present technology. In this context, the expression "server" does not mean that every task (e.g., a received command or request) or some specific task is received, executed or started by the same server (i.e., the same software and/or hardware). This expression means that any number of software or hardware can receive, send, execute or cause to be executed any task or request or the results of any tasks or requests. All of these software and hardware may represent one server or several servers, and both of these cases are implied in the expression "at least one server".

[043] In the context of the present description, the term "electronic device" means any computer hardware capable of executing programs suitable for solving a given task. In the context of the present description, the term "electronic device" implies that the device can function as a server for other electronic devices, however, this is not necessary for the present technology. Thus, some (non-limiting) examples of electronic devices include an autonomous driving unit, personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, as well as network equipment such as routers, switches and gateways. It should be understood that in the present context, the functioning of a device as an electronic device does not mean that it cannot function as a server for other electronic devices.

[044] In the context of the present description, the term "information" includes information of any kind or type capable of being stored in a database. Thus, information includes, but is not limited to, visual works (e.g., maps), audiovisual works (e.g., images, films, sound recordings, presentations, etc.), data (e.g., location data, weather data, traffic data, numerical data, etc.), text (e.g., opinions, comments, questions, messages, etc.), documents, spreadsheets, etc.

[045] As used herein, the term "database" means any structured collection of data, regardless of its specific structure, database management software, or computer hardware for storing, using, or otherwise enabling the use of that data. A database may be located on the same hardware as the process that enables the storage or use of the information stored in the database, or the database may be located on separate hardware, such as a dedicated server or multiple servers.

[046] In the context of the present description, the numerals "first," "second," "third," etc. are used merely to indicate a distinction between the nouns to which they refer, but not to describe any specific relationship between those nouns. As occurs in other contexts in the present description, reference to a "first" element and a "second" element does not exclude the possibility that these two elements may in fact be the same element.

[047] Each embodiment of the present technology relates to at least one of the above-mentioned objectives and/or to one of the above-mentioned aspects, but not necessarily to all of them. It should be understood that some aspects of the present technology associated with an attempt to achieve the above-mentioned objective may not correspond to this objective and/or may correspond to other objectives not explicitly mentioned herein.

[048] Additional and/or alternative features, aspects and advantages of embodiments of the present technology are contained in the following description, in the attached drawings and in the claims.

Brief description of the drawings

[049] These and other features, aspects and advantages of the present technology are explained in the following description, in the appended claims and in the following drawings.

[050] Fig. 1 is a diagram of an illustrative computer system for implementing certain non-limiting embodiments of the present technology.

[051] Fig. 2 is a diagram of a network computing environment suitable for use with certain non-limiting embodiments of the present technology.

[052] Fig. 3 is a diagram of an illustrative lidar system implemented in accordance with certain non-limiting embodiments of the present technology.

[053] Fig. 4 shows data stored in a data storage device of the network computing environment according to Fig. 2.

[054] Fig. 5 shows a segmentation map generated by the server shown in Fig. 2 and containing the boundary of a polygonal region, according to some non-limiting embodiments of the present technology.

[055] Fig. 6 illustrates a process for generating a segmentation map using a training boundary of a polygonal region and an actual boundary of a polygonal region stored in the data storage device shown in Fig. 4, according to some non-limiting embodiments of the present technology.

[056] Fig. 7 shows the deviations of the boundary points from the training boundary of the polygonal region y from the edge of the actual boundary of the polygonal region shown in Fig. 6, according to some non-limiting embodiments of the present technology.

[057] Fig. 8 shows the process of training a second model based on the training boundary of the polygonal region and the actual boundary of the polygonal region shown in Fig. 6, according to some non-limiting embodiments of the present technology.

[058] Fig. 9 shows a process of the step of using the second model shown in Fig. 8, according to some non-limiting embodiments of the present technology.

[059] Fig. 10 is a flow chart of a method performed according to some non-limiting embodiments of the present technology.

[060] Fig. 11 is a diagram of a machine learning architecture used to implement the second model shown in Fig. 8, according to some non-limiting embodiments of the present technology.

Implementation of the invention

[061] The examples and conventional language provided herein are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specifically provided examples and conditions. It is obvious that those skilled in the art are capable of devising various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its spirit and scope.

[062] Furthermore, in order to facilitate better understanding, the following description may contain simplified embodiments of the present technology. Those skilled in the art will understand that other embodiments of the present technology may be significantly more complex.

[063] In some cases, useful examples of modifications of the present technology are given. They facilitate understanding, but also do not define the scope or boundaries of the present technology. The list of modifications provided is not exhaustive and a person skilled in the art can develop other modifications within the scope of the present technology. In addition, if in some cases modifications are not described, this does not mean that they are impossible and/or that the description contains the only possible embodiment of a particular element of the present technology.

[064] The description of the principles, aspects and embodiments of the present technology, as well as specific examples thereof, are intended to cover their structural and functional equivalents, regardless of whether they are currently known or will be developed in the future. For example, those skilled in the art should understand that any block diagrams described herein correspond to conceptual representations of illustrative circuit diagrams implementing the principles of the present technology. It should also be understood that any block diagrams, process diagrams, state transition diagrams, pseudocodes, etc. correspond to various processes that can be represented on a machine-readable physical storage medium and can be executed by a computer or processor, regardless of whether such a computer or processor is explicitly shown or not.

[065] The functions of the various elements shown in the drawings, including any functional unit designated as a "processor", may be implemented using dedicated hardware as well as using hardware capable of executing associated software. If a processor is used, these functions may be performed by a single dedicated processor, a single shared processor, or a plurality of separate processors, some of which may be shared. Furthermore, the explicit use of the term "processor" or "controller" shall not be construed as referring exclusively to hardware capable of executing software, and may include, but are not limited to, a digital signal processor (DSP), a network processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a read-only memory (ROM) for storing software, a random access memory (RAM), and/or a non-volatile memory. Other general-purpose and/or custom hardware may also be included.

[066] Software modules or simply modules, the implementation of which is intended to be in the form of software, may be represented here as any combination of flow chart elements or other elements indicating the execution of process steps and/or containing a text description. Such modules may be executed by hardware, shown explicitly or implied.

[067] Next, taking into account the above principles, some non-limiting examples are considered, illustrating various options for implementing aspects of this technology.

Computer system

[068] Fig. 1 is a diagram of a computer system 100 suitable for use with some embodiments of the present technology. The computer system 100 includes various hardware elements, including one or more single-core or multi-core processors, generally represented by a processor 110, a solid-state drive 120, and memory 130, which may be RAM or any other type of memory.

[069] Communication between elements of computer system 100 may be accomplished via one or more internal and/or external buses (not shown), such as a PCI bus, a USB bus, an IEEE 1394 FireWire bus, a SCSI bus, a Serial-ATA bus, etc., to which various hardware elements are electronically connected. According to embodiments of the present technology, solid-state drive 120 stores software instructions suitable for loading into memory 130 and executing by processor 110 to determine the presence of an object. For example, software instructions may be part of a vehicle control application executed by processor 110. It should be noted that computer system 100 may include additional and/or optional elements (not shown), such as network data communication modules, location determination modules, etc.

Network computing environment

[070] Fig. 2 illustrates a network computing environment 200 suitable for use with certain non-limiting embodiments of the present technology. The network computing environment 200 includes an electronic device 210 in communication with a vehicle 220 and/or with a user (not shown) in communication with the vehicle 220 (such as an operator of the vehicle 220). The network computing environment 200 also includes a server 235 in communication with the electronic device 210 via a communications network 240 (such as the Internet, etc., as described in more detail below).

[071] In some non-limiting embodiments of the present technology, the network computing environment 200 may comprise a GPS satellite (not shown) that transmits a GPS signal to the electronic device 210 or/and receives a GPS signal from it. It should be understood that the present technology is not limited to the GPS system and may use a positioning technology other than the GPS system. It should be noted that a GPS satellite may be completely absent.

[072] The vehicle 220 to which the electronic device 210 is associated may be any recreational or other activity vehicle, such as a personal or commercial automobile, a truck, a motorcycle, and the like. Although the vehicle 220 is depicted as a land vehicle, this is not necessarily the case for all non-limiting embodiments of the present technology. For example, in some non-limiting embodiments of the present technology, the vehicle 220 may be a watercraft, such as a boat, or an aircraft, such as an unmanned aerial vehicle.

[073] The vehicle 220 may be controlled by a user or may be a self-driving vehicle. In some non-limiting embodiments of the present technology, it is contemplated that the vehicle 220 may be implemented as an SDC car. It should be noted that there is no limitation on the specific parameters of the vehicle 220, such as the manufacturer of the vehicle, the model of the vehicle, the year of manufacture of the vehicle, the mass of the vehicle, the dimensions of the vehicle, the distribution of the mass of the vehicle, the surface area of the vehicle, the height of the vehicle, the type of transmission (e.g., two-wheel drive or four-wheel drive), the type of tires, the braking system, the fuel system, the mileage, the identification number of the vehicle, the engine displacement, etc.

[074] In other embodiments of the present technology, the vehicle 220 may be implemented as a delivery robot capable of making deliveries. Unlike SDC vehicles that operate on the road, delivery robots are typically operated on sidewalks or other parts of a geographic area where people may walk. Compared to an SDC vehicle, they are small in size and can navigate through an environment to deliver items from a sender to a recipient.

[075] According to the present technology, there are no particular limitations on the implementation of the electronic device 210. For example, the electronic device 210 can be implemented as an engine control unit of a vehicle, a central processor of a vehicle, a navigator (e.g., TomTom™, Garmin™), a tablet, a personal computer built into the vehicle 220, etc. It should be noted that the electronic device 210 may or may not be permanently connected to the vehicle 220. Additionally or alternatively, the electronic device 210 can be implemented in a wireless communication device, such as a mobile phone (e.g., a smartphone or a radiotelephone). In some embodiments of the invention, the electronic device 210 comprises a display 270.

[076] Depending on the particular embodiment of the invention, the electronic device 210 may include some or all of the elements of the computer system 100 shown in Fig. 1. In some embodiments of the invention, the electronic device 210 is an on-board computer device and includes a processor 110, a solid-state drive 120, and a memory 130. In other words, the electronic device 210 includes hardware and/or application software and/or firmware, or a combination thereof, for processing data, as described in more detail below.

[077] In some non-limiting embodiments of the present technology, the communication network 240 is the Internet. In alternative non-limiting embodiments of the present technology, the communication network 240 may be implemented as any suitable local area network (LAN), wide area network (WAN), private communication network, etc. It will be appreciated that the embodiments of the communication network 240 are provided for illustrative purposes only. The implementation of the communication link (not specifically indicated) between the electronic device 210 and the communication network 240 depends, among other things, on the implementation of the electronic device 210. As a non-limiting example, in those non-limiting embodiments of the present technology where the electronic device 210 is implemented as a wireless communication device, such as a smartphone or a navigation device, the communication link may be implemented as a wireless communication link. Examples of wireless communication links include a 3G communication network channel, a 4G communication network channel, etc. The communication network 240 may also use a wireless connection to the server 235.

[078] In some embodiments of the present technology, the server 235 is implemented as a computer server and may include some or all of the elements of the computer system 100 shown in Fig. 1. In one non-limiting example, the server 235 is implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system, but it may also be implemented using any other suitable hardware, application software, and/or firmware, or a combination thereof. In the illustrated non-limiting embodiments of the present technology, the server 235 is a single server. In other non-limiting embodiments of the present technology, the functions of the server 235 may be distributed among multiple servers (not shown).

[079] In some non-limiting embodiments of the present technology, the processor 110 of the electronic device 210 may be connected to the server 235 to receive one or more updates. Such updates may include software updates, map updates, route updates, weather data updates, and the like. In some non-limiting embodiments of the present technology, the processor 110 may also send some operational data to the server 235, such as routes traveled, traffic data, performance characteristics, and the like. Some or all of such data transmitted between the vehicle 220 and the server 235 may be encrypted and/or anonymized.

[080] It should be noted that the server 235 can be connected to the data storage device 400. The data storage device 400 can store various data for use by the server 235 and/or the electronic device 210. In some embodiments of the invention, one or more databases can be located in the data storage device. In general, a database is a structured collection of data and is accessible by electronic means. Small databases can be stored in a file system, and large databases are located in computer clusters or in a cloud data warehouse. The structure of the databases depends on various technologies and practical considerations, including data modeling, efficient representation and storage of data, query languages, security and protection of confidential data, as well as distributed computing issues, including concurrent access and fault tolerance. At least some non-limiting examples of data stored in the data storage device 400 are described below with reference to Fig. 4.

[081] In other embodiments of the present technology, the server 235 may execute one or more Machine Learned Algorithms (MLA). In general, MLA algorithms are capable of learning from training samples and providing predictions based on new (unknown) data. MLA algorithms are typically used to initially build a model based on training input data, and then make predictions or decisions based on the data, expressed as output data, rather than executing static machine-readable commands.

[082] MLA algorithms are typically used as evaluation models, classification models, regression models, segmentation models, etc. It should be understood that different types of MLA algorithms with different structures or topologies can be used for different tasks. The data used in training can be stored and made available in the data storage device 400, and can also be further processed to form training samples.

[083] One particular type of MLA algorithm involves neural networks (NN). In general, an NN consists of interconnected groups of artificial neurons that process information using a connectionist approach to computing. NNs are used to model complex relationships between inputs and outputs (without actual knowledge of these relationships) or to find patterns in data. NNs are first adapted during a training phase, where they are provided with a known set of inputs and information to adapt the NN to produce appropriate outputs (for the situation being attempted to be modeled). During this training phase, the NN adapts to the situation being learned and changes its structure so that it is able to produce adequate predictions of the output for the inputs in the new situation (based on what has been learned). Thus, instead of trying to define complex statistical distributions or mathematical algorithms for a situation, the NN tries to provide an "intuitive" answer based on its "perception" of the situation.

[084] NN networks are widely used in many such situations where it is important to obtain output data based on input data and it is less important or not important at all how the output data is obtained. For example, NN networks are widely used to optimize the distribution of web traffic between servers, automatically translate text into different languages, and in data processing, including filtering, clustering, vectorization, etc.

[085] The implementation of the MLA algorithm can be divided into two main stages: a training stage and a usage stage. In general, the MLA algorithm is first "built" (or trained) using training data and training targets. In a training iteration, the MLA algorithm receives the training input data as input and generates a corresponding prediction. The server 235 is then, in a sense, able to adjust the MLA algorithm based on a comparison of the prediction with the training target for the training input data. For example, the adjustment can be performed by the server 235 using one or more machine learning methods, such as back propagation, etc. After a large number of training iterations, the MLA algorithm is adjusted so as to generate predictions based on the input data that are close to the corresponding training targets.

[086] As shown in FIG. 2, server 235 may execute segmentation model 650 and prediction model 850. Segmentation model 650 may be an MLA algorithm capable of segmenting image data. In general, image segmentation is the process of dividing a digital image into a plurality of image segments and may be used to determine the location of objects and/or boundaries (lines, curves, etc.) in images. In some embodiments of the invention, it is contemplated that segmentation model 650 may, among other things, assign labels to pixels of the input image such that pixels with the same label have the same characteristic (or several characteristics). In one example, pixels associated with a label may represent the same object in the input image. It is also contemplated that a plurality of algorithms may be applied to the labeled pixels in order to determine the outline or boundary of an object in the input image.

[087] In some embodiments of the present technology, the segmentation model 650 may be implemented as a convolutional neural network (CNN) trained to segment images. In one embodiment of the invention, the CNN may have a U-Net architecture. In general, such a CNN consists of a convolutional part and a development part that form a U-shaped architecture. The convolutional part is a CNN consisting of repeatedly applied convolutions, each of which is followed by a truncated linear unit (ReLU) and a maximum pooling operation. During the convolutions, the amount of spatial information is reduced and the amount of feature information is increased. In the development part, the feature information and spatial information are combined by a series of up-convolutions and concatenations with high-resolution features from the convolutional part. The U-Net architecture can accommodate a large number of feature channels in its high-resolution portion, allowing the network to pass contextual information to higher-resolution layers.

[088] In some embodiments of the invention, the segmentation model 650 may be implemented similar to the CNN network described in the paper “U-net: Convolutional Networks for Biomedical Image Segmentation» by Olaf Ronneberger et al., published May 18, 2015, the contents of which are incorporated herein by reference in their entirety.

[089] As described in more detail below, the segmentation model 650 may be used to segment image data about regions of a map in order to determine objects and to form boundaries of these objects in the corresponding regions of the map. It is contemplated that these objects may include sections of a road accessible to road vehicles (e.g., cars, trucks, buses, and motorcycles), sections of a sidewalk accessible to pedestrians and relatively small robotic vehicles (e.g., delivery robots), sections of a path accessible to pedestrians, road vehicles, and relatively small robotic vehicles, and the like.

[090] It should be noted that the segmentation model 650 can be applied at the stage of its use in areas of the map that have not been assessed by experts. For example, a cartographer may be tasked with tracking the boundaries of objects of one or more classes on a map. However, obtaining such expert-assessed data requires significant time and money. Since determining the actual boundaries of objects on a very large map is practically impossible, the developers of the present technology have developed a system where actual information about the boundaries of objects is used in expert-assessed areas of the map, and the results of the segmentation model 650 can be used for areas of the map that have not been assessed in this way.

[091] The developers of the present technology have determined that, despite the benefit of using the segmentation model 650 in unscored areas, the model 650 does not provide satisfactory prediction quality in some of these unscored areas. In this regard, the developers have developed methods and systems that use the prediction model 850 to generate an indication of the reliability of the boundary information generated by the segmentation model 650.

[092] In general, the prediction model 850 may be trained to predict, for one or more points along the boundary generated by the segmentation model 650, confidence scores that indicate whether these points are correctly placed on the map. In some embodiments of the invention, the confidence scores may be said to indicate the likelihood of these points being correctly placed on the map. In other embodiments of the invention, the confidence scores may be said to indicate the level of confidence that these points are correctly placed on the map. The estimated areas of the map may be used to train the prediction of the prediction model 850 based on actual data from experts and segmentation data from the segmentation model 650. The training of the prediction model 850 is described in more detail below. It should be noted that when the prediction model 850 determines during its use that one or more points of the boundary generated by the segmentation model 650 for an unestimated area may be incorrectly placed, then this unestimated area may be marked for evaluation by an expert. In other words, the segmentation model 650 may be used to generate polygonal region boundaries in some unscored areas without additional expert input, while in other unscored areas, the output of the prediction model 850 may indicate that experts should be tasked with determining the actual boundaries instead of using only the output of the segmentation model 650.

[093] It should be noted that the electronic device 210 may use a plurality of sensors and systems to collect information about the vicinity 250 of the vehicle 220 (see Fig. 2). As shown, the vehicle 220 may be equipped with a plurality of sensor systems 280. It should be noted that different sensor systems from the plurality of sensor systems 280 may be used to collect different types of data about the vicinity 250 of the vehicle 220.

[094] In one example, the plurality of sensor systems 280 may comprise various optical systems, including, but not limited to, one or more camera-type sensor systems mounted on the vehicle 220 and coupled to the processor 110 of the electronic device 210. In general, the one or more camera-type sensor systems may collect image data about various portions of the vicinity 250 of the vehicle 220. In some cases, the image data provided by the one or more camera-type sensor systems may be used by the electronic device 210 to perform object detection procedures. For example, the electronic device 210 may input the image data provided by the one or more camera-type sensor systems into an Object Detection Neural Network (ODNN) trained to determine the location and classify potential objects in the vicinity 250 of the vehicle 220.

[095] In another example, the plurality of sensor systems 280 may comprise one or more radar sensor systems mounted on the vehicle 220 and coupled to the processor 110. In general, the one or more radar sensor systems may use radio waves to collect image data for various portions of the vicinity 250 of the vehicle 220. For example, the one or more radar sensor systems may collect radar data about potential objects in the vicinity 250 of the vehicle 220, wherein such data may potentially represent the distance of the objects from the radar sensor system, the orientation of the objects, the speed of the objects, etc.

[096] It should be noted that the plurality of sensor systems 280 may include sensor systems of other types in addition to those non-exhaustively described above without departing from the scope of the present technology.

[097] In some embodiments of the present technology, the vehicle 220 may be implemented as a delivery robot. Delivery robots or autonomous delivery robots are relatively small vehicles used for delivery. For example, they may navigate on sidewalks and are adapted to overcome various obstacles along the route to the destination. Advantages of autonomous delivery robots include a reduction in the need for large trucks with potentially large emissions when driving in densely populated cities and a reduction in the number of drivers required to quickly deliver items ordered online to users.

Lidar system

[098] According to non-limiting embodiments of the present technology, the vehicle 220 (see FIG. 2) is equipped with at least one lidar system, such as the lidar system 300, for collecting information about the surroundings 250 of the vehicle 220. Although described herein only for the case of being mounted on the vehicle 220, it is also contemplated that the lidar system 300 may operate autonomously or may be linked to another system.

[099] Depending on the embodiment of the invention, the vehicle 220 may include more or fewer lidar systems 300 than shown. Depending on the embodiment of the invention, the use of specific systems of the plurality of sensor systems 280 may depend on the embodiment of the lidar system 300. The lidar system 300 may be installed on the vehicle 220 initially or as a retrofit in various locations and/or in various configurations.

[0100] For example, depending on the embodiment of the vehicle 220 and the lidar system 300, the lidar system 300 may be mounted on the inside of the upper portion of the windshield of the vehicle 220. However, other mounting locations for the lidar system 300 are also possible within the scope of the present technology, including the rear window, side windows, front hood, roof, front grille, front bumper, or side surface of the vehicle 220 (see Fig. 2). In some cases, the lidar system 300 may be mounted in a special housing mounted on top of the vehicle 220.

[0101] In some non-limiting embodiments of the invention, the system of multiple lidar systems 300 is mounted on the roof of the vehicle 220 in a rotatable configuration (see Fig. 2). For example, the lidar system 300 mounted on the vehicle 220 with the ability to rotate may include at least some elements that are capable of rotating 360 degrees about the axis of rotation of the lidar system 300. When mounted in a rotatable configuration, the lidar system 300 can collect data about most parts of the vicinity 250 of the vehicle 220.

[0102] In some non-limiting embodiments of the present technology, the lidar system 300 is mounted, for example, on a side surface or on a front grille, in a non-rotatable configuration (see Fig. 2). In particular, the lidar system 300 mounted on the vehicle 220 without the possibility of rotation may include at least some elements that do not rotate 360 degrees, but provide for collecting data about predetermined portions of the vicinity 250 of the vehicle 220.

[0103] Regardless of the specific location and/or specific configuration, the lidar system 300 is capable of capturing data about the vicinity 250 of the vehicle 220, used, for example, to construct a multidimensional map of objects in the vicinity 250 of the vehicle 220. The configuration of the lidar systems 300 for capturing data about the vicinity 250 of the vehicle 220 is described in detail below.

[0104] It should be noted that, although the lidar system 300 described herein is implemented as a Time of Flight lidar system and therefore contains the corresponding elements suitable for its implementation, other embodiments of the lidar system 300 are also possible without going beyond the scope of the present technology. For example, in some non-limiting embodiments of the present technology, the lidar system 300 can also be implemented as a Frequency-Modulated Continuous Wave (FMCW) lidar system according to one or more embodiments and be based on the corresponding elements as described in patent application RU2020117983 “Methods and Systems for Detecting Lidars”, filed on June 01, 2020, the contents of which are fully incorporated herein by reference.

[0105] Fig. 3 is a diagram of one specific embodiment of a lidar system 300 implemented in accordance with certain non-limiting embodiments of the present technology.

[0106] In general, the lidar system 300 includes a plurality of internal elements, including, but not limited to, (a) a light source 302 (also called a laser or a radiation source), (b) a beam splitter 304, (c) a scanning unit 308 (also called a scanner or a scanner unit), (d) a detection unit 306 (also called a detection system, a receiving unit, or a detector), and (e) a controller 310. It is contemplated that, in addition to the elements of the above open list, the lidar system 300 may include a plurality of sensors (such as a temperature sensor, a moisture sensor, etc.), which are omitted from Fig. 3 for simplicity.

[0107] In some non-limiting embodiments of the present technology, one or more internal elements of the lidar system 300 are located in a common housing 330, as shown in Fig. 3. In some embodiments of the present technology, the controller 310 may be located outside the common housing 330 and communicate with the elements located inside it. As described below, a window 380 is provided in the housing 330 on the side of the vicinity of the vehicle 220, through which light rays can exit the housing 330 and enter the housing 330.

[0108] In general, the lidar system 300 operates as follows: the light source 302 of the lidar system 300 emits light pulses that form an output beam 314, the scanning unit 308 scans the output beam 314 through the window 380 the vicinity 250 of the vehicle 220 in order to determine the location or record data about previously unknown objects (such as the object 320), for example, to form a multidimensional map of the vicinity 250, in which the objects (including the object 320) are represented in the form of one or more data points. The light source 302 and the scanning unit 308 are described in more detail below.

[0109] In some non-limiting examples, object 320 may include, in whole or in part, a person, a vehicle, a motorcycle, a truck, a train, a bicycle, a wheelchair, a baby carriage, a pedestrian, an animal, a road sign, a traffic light, a median line, a road marking, a parking space, a pylon, a guardrail, a road barrier, a pothole, a railroad crossing, an obstruction on or near a road, a curb, a vehicle stopped on or near a road, a pole, a house, a building, a trash can, a mailbox, a tree, any other suitable object, or any suitable combination of two or more objects or parts thereof.

[0110] In addition, it can be assumed that the object 320 is located at a distance 318 from the lidar system 300. When the output beam 314 reaches the object 320, the object 320 typically reflects at least a portion of the light from the output beam 314 and some of the reflected light rays may return to the lidar system 300, where they are received as an input beam 316. By reflection is meant that at least a portion of the light beam from the output beam 314 is reflected from the object 320. A portion of the light beam from the output beam 314 may be absorbed or scattered by the object 320.

[0111] The input beam 316 is detected and detected by the lidar system 300 using the detection unit 306. In this case, the detection unit 306 is capable of generating one or more representative data signals. For example, the detection unit 306 can generate an output electrical signal (not shown) representing the input beam 316. The detection unit 306 can also output the electrical signal generated in this way to the controller 310 for further processing. The controller 310 calculates the distance 318 to the object 320 by measuring the time between the emission of the output beam 314 and the reception of the input beam 316.

[0112] As described in more detail below, beam splitter 304 is used to direct output beam 314 from light source 302 to scanning unit 308 and to direct input beam 316 from scanning unit to detection unit 306.

[0113] The use and implementation of these elements of the lidar system 300 according to some non-limiting embodiments of the present technology are described below.

Light source

[ 0114] The light source 302 is connected to the controller 310 and is capable of emitting light with a certain operating wavelength. For this purpose, in some non-limiting embodiments of the present technology, the light source 302 may comprise at least one laser preset to operate with a certain operating wavelength. The operating wavelength of the light source 302 may correspond to the infrared, visible and/or ultraviolet parts of the electromagnetic spectrum. For example, the light source 302 may comprise at least one laser with an operating wavelength in the range of approximately 650-1150 nm. Alternatively, the light source 302 may comprise a laser diode capable of emitting light with a wavelength in the ranges of approximately 800-1000 nm, approximately 850-950 nm or approximately 1300-1600 nm. In some other embodiments of the invention, the light source 302 may comprise a light-emitting diode.

[0115] The light source 302 of the lidar system 300 is typically an eye-safe laser, or in other words, the lidar system 300 can be classified as an eye-safe laser system or a laser product. In general, an eye-safe laser, laser system, or laser product can be a system in which some or all of the parameters of the emission wavelength, average power, peak power, peak intensity, pulse energy, beam size, beam divergence, exposure duration, or output beam trajectory are such that the risk of damage to human eyes from the light emitted from this system is insignificant or completely absent.

[0116] According to some non-limiting embodiments of the present technology, the operating wavelength of the light source 302 may be within portions of the electromagnetic spectrum corresponding to sunlight. Therefore, in some cases, sunlight may represent background noise that may drown out the light signal detected by the lidar system 300. This solar background noise may lead to false positive detections and/or may otherwise distort the measurements of the lidar system 300. Although it is possible in some cases to increase the signal-to-noise ratio (SNR) of the lidar system 300 by increasing the power level of the output beam 314, this may be undesirable in at least some situations. For example, in some embodiments of the invention, it may be undesirable to increase the power of the output beam 314 to levels exceeding eye-safe thresholds.

[0117] The light source 302 comprises a pulsed laser capable of generating and emitting light pulses of a certain duration. For example, in some non-limiting embodiments of the present technology, the light source 302 can emit pulses with a pulse duration (pulse width) from 10 ps to 100 ns. In other non-limiting embodiments of the present technology, the light source 302 can emit pulses with a pulse repetition rate of approximately 100 kHz to 5 MHz or with a pulse period (time between successive pulses) from approximately 200 ns to 10 μs. In general, the light source 302 can generate an output beam 314 with any suitable average optical power. In addition, the output beam 314 can comprise optical pulses with any pulse energy or peak optical power suitable for a given application.

[0118] In some non-limiting embodiments of the present technology, the light source 302 may comprise one or more laser diodes, including, but not limited to, a Fabry-Perot laser diode, a quantum well laser, a DBR reflector laser, a DFB feedback laser, or a VCSEL laser. For example, the laser diode operating in the light source 302 may be an aluminum gallium arsenide (AlGaAs) laser diode, an indium gallium arsenide (InGaAs) laser diode, an indium gallium arsenide phosphide (InGaAsP) laser diode, or any other suitable laser diode. It is also contemplated that the light source 302 may comprise one or more current-modulated laser diodes for generating optical pulses.

[0119] In some non-limiting embodiments of the present technology, the light source 302 is generally capable of emitting a collimated output beam 314. However, it is contemplated that the generated beam may have any beam divergence suitable for a given application. In general, the divergence of the output beam 314 is an angular measure of the increase in the cross-sectional size of the beam (e.g., the radius or diameter of the beam) as the output beam 314 moves away from the light source 302 or from the lidar system 300. In some non-limiting embodiments of the present technology, the output beam 314 may have a substantially circular cross-section.

[0120] It is also contemplated that the output beam 314 emitted by the light source 302 may be unpolarized or randomly polarized, may not have a specific or fixed polarization (e.g., the polarization may change over time), or may have a specific polarization (e.g., the output beam 314 may have linear polarization, elliptical polarization, or circular polarization).

[0121] In at least some non-limiting embodiments of the present technology, the output beam 314 and the input beam 316 may be substantially coaxial. In other words, the output beam 314 and the input beam 316 may at least partially overlap or have a common propagation axis such that the input beam 316 and the output beam 314 propagate along substantially the same optical path (but in opposite directions). However, in other non-limiting embodiments of the present technology, the output beam 314 and the input beam 316 may be non-coaxial or, in other words, may not overlap or have a common propagation axis within the lidar system 300 without departing from the scope of the present technology.

[0122] It should be noted that in at least some non-limiting embodiments of the present technology, the light source 302 may rotate, for example, by 360 degrees or less, about the rotation axis (not shown) of the lidar system 300, if the lidar system 300 is implemented in a rotary configuration. In other embodiments of the invention, the light source 302 may be stationary, even when the lidar system 300 is implemented in a rotary configuration, without departing from the scope of the present technology.

Beam splitter element

[ 0123] Fig. 3 also shows a beam splitter 304 located in the housing 330. For example, as described above, the beam splitter 304 is capable of directing the output beam 314 from the light source 302 to the scanning unit 308. The beam splitter 304 is also capable of directing the input beam 316 reflected from the object 320 to the detection unit 306 for further processing by the controller 310.

[0124] However, according to other non-limiting embodiments of the present technology, the beam splitter 304 may split the output beam 314 into at least two lower intensity components, including a scanning beam (not separately shown) used to scan the vicinity 250 of the lidar system 300, and a reference beam (not separately shown) directed to the detection unit 306.

[0125] In other words, it can be said that in these embodiments of the invention, the beam splitter 304 can split the intensity (optical power) of the output beam 314 between the scanning beam and the reference beam. In some non-limiting embodiments of the present technology, the beam splitter 304 can split the intensity of the output beam 314 between the scanning beam and the reference beam equally. In other non-limiting embodiments of the present technology, the beam splitter 304 can split the intensity of the output beam 314 in any predetermined ratio. For example, the beam splitter 304 can use up to 80% of the intensity of the output beam 314 to form the scanning beam and the remaining up to 20% of the intensity of the output beam 314 to form the reference beam. In other non-limiting embodiments of the present technology, the beam splitter 304 may vary the split ratio for forming the scanning beam (e.g., from 1% to 95% of the intensity of the output beam 314).

[0126] It should also be noted that some portion (for example, up to 10%) of the intensity of the output beam 314 may be absorbed by the material of the beam splitter element 304, which depends on its specific design.

[0127] Depending on the embodiment of the lidar system 300, the beam splitter 304 may be of various types, including, among other things, a glass prism-based beam splitter, a half-silvered mirror-based beam splitter, a dichroic mirror prism-based beam splitter, a fiber optic beam splitter, etc.

[0128] Thus, according to non-limiting embodiments of the present technology, an open list of adjustable parameters associated with the beam splitter element 304 may, for example, depending on the specific embodiment of its application, include a range of operating wavelengths, which can vary from a finite number of wavelengths to a broader spectrum of light (e.g., 1200-1600 nm), an angle of incidence of the input beam, the presence or absence of polarization, etc.

[0129] In a specific non-limiting example, the beam splitter 304 may be implemented as a fiber optic beam splitter, such as that available from OZ Optics Ltd. (219 Westbrook Rd Ottawa, Ontario K0A 1L0, Canada). It is understood that the beam splitter 304 may be implemented as any other suitable equipment.

Internal Ray Paths

[0130] As schematically shown in Fig. 3, the lidar system 300 is provided with a plurality of internal ray paths 312 along which the output ray 314 (generated by the light source 302) and the input ray 316 (obtained from the vicinity 250) propagate. In particular, the light propagates along the internal ray paths 312 as follows: the light from the light source 302 passes through the beam splitter 304 into the scanning unit 308, which directs the output ray 314 outward toward the vicinity 250.

[0131] In a similar manner, the input beam 316 propagates along a plurality of internal beam paths 312 into the detection unit 306. In particular, the input beam 316 is directed by the scanning unit 308 into the lidar system 300 - through the beam splitter 304 into the detection unit 306. In some embodiments of the invention, the lidar system 300 may be provided with beam paths that direct the input beam 316 from the vicinity 250 directly into the detection unit 306 (without the input beam 316 passing through the scanning unit 308).

[0132] It should be noted that in various non-limiting embodiments of the present technology, the plurality of internal beam paths 312 may comprise a variety of optical elements. For example, the lidar system 300 may comprise one or more optical elements capable of processing, shaping, filtering, modifying, or directing the output beam 314 and/or the input beam 316. For example, the lidar system 300 may comprise one or more elements from among lenses, mirrors, filters (e.g., bandpass or interference filters), optical fibers, circulators, beam splitters, polarizers, polarizing beam splitters, wave plates (e.g., half-wave or quarter-wave plates), diffractive elements, microelectromechanical elements, collimating elements, or holographic elements.

[0133] It is contemplated that in at least some non-limiting embodiments of the present technology, at least some common optical elements may be shared between the inner beam path and another inner beam path of the plurality of inner beam paths 312. However, this is not necessarily the case for all embodiments of the present technology.

Scanning unit

[0134] In general, the scanning unit 308 directs the output beam 314 in one or more directions toward the vicinity 250. The scanning unit 308 is connected to a controller 310. The controller 310 is capable of controlling the scanning unit 308 so as to direct the output beam 314 in a desired direction and/or according to a predetermined scanning pattern. In general, in the context of the present description, the scanning pattern may correspond to a pattern or trajectory according to which the output beam 314 is directed by the scanning unit 308 during its operation.

[0135] In some non-limiting embodiments of the present technology, the controller 310 may cause the scanning unit 308 to sweep the output beam 314 over a plurality of horizontal angular ranges and/or vertical angular ranges. The total angular range over which the scanning unit 308 scans the output beam 314 is sometimes referred to as a field of view (FOV). It is contemplated that the specific design, orientation, and/or angular ranges may depend on a specific embodiment of the lidar system 300. The field of view typically comprises a plurality of ROIs defined as portions of the FOV, which may, for example, contain objects of interest. In some embodiments of the invention, the scanning unit 308 may further explore a selected ROI 325. The ROI region 325 of the lidar system 300 may correspond to an area, volume, zone, angular range, and/or portion (or multiple portions) of the neighborhood 250 where the lidar system 300 is capable of scanning and/or capturing data.

[0136] It should be noted that the location of the object 320 in the vicinity 250 of the vehicle 220 may overlap, be contained, or be covered at least partially by the ROI 325 of the lidar system 300.

[0137] According to some non-limiting embodiments of the present technology, the scanning unit 308 can scan the output beam 314 horizontally and/or vertically, and therefore the ROI 325 of the lidar system 300 can have a horizontal direction and a vertical direction. For example, the ROI 325 can be defined by an angle of 45 degrees horizontally and an angle of 45 degrees vertically. In some embodiments of the invention, different scanning axes can have different orientations.

[0138] The scanning unit 308 may comprise a first reflective element 350 and a second reflective element 360. The first reflective element 350 may redirect the output beam 314 from the beam splitter toward the second reflective element 360, ensuring that the output beam is distributed along the first axis. The second reflective element 360 may redirect the output beam 314 from the first reflective element 350 toward the vicinity 250 (through the window 380 of the housing 330), ensuring that the output beam is distributed along the second axis. The second axis may be perpendicular and/or orthogonal to the first axis. Due to the fact that the output beam 314 is thus redirected and distributed using a combination of the first reflective element 350 and the second reflective element 360, it is possible to scan the vicinity 250 of the vehicle 220 along at least two perpendicular or orthogonal axes.

[0139] It can be said that the scanning unit can scan the surrounding area along a pair of axes. For example, the output beam 314 can be distributed by the first reflective element 350 along the first axis and the second reflective element 360 along the second axis. In one embodiment of the invention, the first axis can be a vertical axis, and the second axis can be a horizontal axis. In another embodiment of the invention, the first axis can be a horizontal axis, and the second axis can be a vertical axis.

[0140] In some non-limiting embodiments of the present technology, the scanning unit may further comprise a plurality of other optical elements and/or mechanical elements for scanning with the output beam. For example, the scanning unit may comprise one or more mirrors, prisms, lenses, microelectromechanical elements, piezoelectric elements, optical fibers, splitters, diffraction elements, collimation elements, etc. It should be noted that the scanning unit may also comprise one or more additional drives (not separately shown) that provide, for example, rotation, tilt, pivoting, or angular movement around one or more axes of at least some other optical elements.

[0141] The lidar system 300 shown in Fig. 3 may use a predetermined scanning pattern to generate a point cloud that substantially covers the ROI 325 of the lidar system 300. This point cloud of the lidar system 300 may be used to generate a multidimensional map of objects in the vicinity 250 of the vehicle 220.

Detection unit

[0142] According to some non-limiting embodiments of the present technology, the detection unit 306 is coupled to the controller 310 and can be implemented in a variety of ways. According to the present technology, the detection unit 306 comprises a photodetector, but may, among other things, comprise a photodetector, an optical receiver, an optical sensor, a detector, an optical detector, optical fibers, and the like. As described above, in some non-limiting embodiments of the present technology, the detection unit 306 can receive or detect at least a portion of the input beam 316 and generate an electrical signal corresponding to the input beam 316. For example, if the input beam 316 comprises an optical pulse, the detection unit 306 can generate an electrical current or voltage pulse corresponding to the optical pulse detected by the detection unit 306.

[0143] It is contemplated that in various non-limiting embodiments of the present technology, the detection unit 306 may be implemented using one or more avalanche photodiodes (APD), one or more single-photon avalanche diodes (SPAD), one or more PN photodiodes (e.g., a photodiode structure formed by a p-type semiconductor and an n-type semiconductor), one or more PIN photodiodes (e.g., a photodiode structure formed by a region of undoped semiconductor located between p-type and n-type regions), and the like.

[0144] In some non-limiting embodiments of the invention, the detection unit 306 may also comprise a circuit that performs signal amplification, sampling, filtering, signal conditioning, analog-to-digital conversion, time-to-code conversion, pulse detection, threshold detection, rising edge detection, falling edge detection, and the like. For example, the detection unit 306 may comprise electronic elements capable of converting a received photocurrent (e.g., a current generated by an avalanche photodiode in response to a received optical signal) into a voltage signal. The detection unit 306 may also comprise additional circuitry for generating an analog or digital output signal corresponding to one or more characteristics (e.g., a rising edge, a falling edge, an amplitude, a duration, and the like) of the received optical pulse.

Controller

[ 0145] Depending on the embodiment, the controller 310 may comprise one or more processors, an ASIC circuit, an FPGA array, and/or other suitable circuitry. The controller 310 may also comprise a machine-readable physical memory for storing instructions executed by the controller 310 and data that the controller 310 may generate based on signals received from other internal elements of the lidar system 300 and/or signals output to other internal elements of the lidar system 300. The memory may comprise volatile elements (e.g., RAM) and/or non-volatile elements (e.g., flash memory, hard disk). The controller 310 may generate data during operation and store it in the memory. For example, the data generated by the controller 310 may indicate data points in a point cloud of the lidar system 300.

[0146] It is contemplated that in at least some non-limiting embodiments of the present technology, the controller 310 may be implemented similarly to the electronic device 210 and/or the computer system 100 without departing from the scope of the present technology. In addition to collecting data from the detection unit 306, the controller 310 may output control signals to the light source 302 and to the scanning unit 308 and potentially receive diagnostic data from them.

[0147] As described above, the controller 310 is connected to the light source 302, to the scanning unit 308 and to the detection unit 306. In some non-limiting embodiments of the present technology, the controller 310 can receive electrical trigger pulses from the light source 302, each of which corresponds to the emission of an optical pulse by the light source 302. The controller 310 can also output to the light source 302 commands, a control signal and/or a trigger signal indicating when the light source 302 should generate optical pulses, for example, forming the output beam 314.

[0148] For example, the controller 310 may send an electrical triggering signal comprising electrical pulses so that the light source 302 emits an optical pulse, which may be represented by the output beam 314, in response to each electrical pulse of the electrical triggering signal. It is also contemplated that the controller 310 may cause the light source 302 to adjust one or more characteristics of the output beam 314 generated by the light source 302, such as frequency, period, duration, pulse energy, peak power, average power, wavelength of the optical pulses, etc.

[0149] According to the present technology, the controller 310 is configured to determine a "time of flight" value for an optical pulse to determine a distance between the lidar system 300 and one or more objects in the field of view, as described below. The time of flight is based on time information associated with (a) a first time of emission of the optical pulse (e.g., from the output beam 314) by the light source 302 and (b) a second time of detection or reception by the detection unit 306 of a portion of the optical pulse (e.g., from the input beam 316). In some non-limiting embodiments of the present technology, the first time may indicate a time of output by the controller 310 of a corresponding electrical pulse associated with the optical pulse, and the second time may indicate a time of receipt by the controller 310 from the detection unit 306 of an electrical signal generated in response to receiving the portion of the optical pulse from the input beam 316.

[0150] In other non-limiting embodiments of the present technology, where the beam splitter 304 can split the output beam 314 into a scanning beam (not shown) and a reference beam (not shown), the first time point can correspond to the time point of receipt from the detection unit 306 of a first electrical signal generated in response to receiving a portion of the reference beam. Accordingly, in these embodiments of the invention, the second time point can be determined as the time point of receipt by the controller 310 from the detection unit 306 of a second electrical signal generated in response to receiving another portion of the optical pulse from the input beam 316.

[0151] According to the present technology, the controller 310 is capable of determining, based on the first time instant and the second time instant, the time-of-flight value and/or the phase modulation value for the emitted pulse from the output beam 314. The time-of-flight value T , in a sense, corresponds to the time of passage of the emitted pulse in both directions: from the lidar system 300 to the object 320 and back to the lidar system 300. Thus, the controller 310 can determine the distance 318 according to the following equation:

, (1)

where D is the distance 318, T is the time of flight value, c is the speed of light (approximately 3.0×10 ⁸ m/s).

[0152] As described above, the lidar system 300 may be used to determine a distance 318 to one or more other potential objects located in the vicinity 250. By scanning the output beam 314 of the ROI 325 of the lidar system 300 according to a predetermined scanning pattern, the controller 310 is able to match distances (such as the distance 318) with corresponding data points in the ROI 325 of the lidar system 300. As a result, the controller 310 is able to display these sequentially recorded data points (e.g., a point cloud) in the form of a multidimensional map. In some embodiments of the invention, the measured time of flight and/or distance to objects data may be displayed in various information formats.

[0153] For example, such a multidimensional map can be used by the electronic device 210 to detect objects or otherwise identify them, or to determine the shape of or distance to potential objects in the ROI 325 of the lidar system 300. It is contemplated that the lidar system 300 can cyclically or iteratively capture and/or generate point clouds at any rate suitable for a given application.

[0154] It should be noted that such multi-dimensional maps can be recorded and stored as part of the log data associated with the vehicle 220. As a result, the point cloud data recorded by the lidar systems in the fleet of vehicles can be stored for subsequent use. In addition, the multi-dimensional maps recorded by the lidar system can be used to determine the location of the SDC vehicle during its operation. The use of the point cloud data recorded by the lidar system during operation to determine the location of the vehicle 220 is described below.

Data storage device

[0155] Fig. 4 shows a data storage device 400. The data storage device 400 may be connected to the server 235 and to the electronic device 210 of the vehicle 220 (see Fig. 2). The data storage device 400 may be used by the server 235 and/or the electronic device 210 to store data collected during the operation of the vehicle 220. The data storage device 400 may be used by the server 235 and/or the electronic device 210 to determine at least a portion of the stored data to be updated. The data storage device 400 may be used by the server 235 and/or the electronic device 210 to obtain data to be used during the operation of the vehicle 220.

[0156] In some embodiments of the present technology, the data storage device 400 may receive data collected by sensors of a plurality 404 of vehicles, such as hundreds or thousands of vehicles implemented similarly to the vehicle 220. The vehicles collect sensor data recorded while driving along various routes and send it to the data storage device 400. As a result, it can be said that the data storage device 400 may store log data 406 for a plurality 404 of vehicles (for a fleet of vehicles).

[0157] In other embodiments of the present technology, the server 235 may use the log data 406 to create and/or update a map representation 402 of a geographic area where the plurality 404 vehicles operate and/or have operated. It is contemplated that the server 235 may create a map representation 402 for maps of a geographic area based on the aggregate information received from the plurality 404 vehicles and store the map representation 402 in the data storage device 400.

[0158] The server 235 may send data corresponding to the map representation 402 to the electronic devices associated with the vehicles of the plurality of 404 vehicles. For example, if the vehicle 220 is to travel along a route, the electronic device 210 may provide information describing the route of travel to the server 235. In response, the server 235 provides the map representation 402 (or a portion thereof) for travel along this route. It is contemplated that the map representation 402 may be periodically provided for local storage by the electronic device 210 for use without going beyond the scope of the present technology.

[0159] In one embodiment of the invention, the server 235 sends portions of the map representation 402 to the vehicles in a compressed form so that the transmitted data requires less bandwidth of the communication channel. The server 235 can receive information from various vehicles that describes data locally stored by the electronic device 210. If the server 235 determines that the vehicle 220 does not contain a certain locally stored portion of the map representation 402, then the server 235 can send this portion of the map representation 402. If the server 235 determines that the vehicle 220 has previously received this particular portion of the map representation, but the corresponding data has been updated by the server 235 since the last time the data was received by the vehicle 220, then the server 235 can send an update to the electronic device 210 for this portion of the map representation 402.

[0160] It should be noted that the server 235 can generate the map representation 402 based on the log data 406. The server 235 updates the previously calculated map data by receiving newer information from vehicles that have recently traveled along the routes for which the map information has been changed. In one embodiment of the invention, methods for combining maps using iterative closest point (ICP) algorithms (and their variants) can be used by the server 235 for corresponding points with the smallest distance on different cloud maps from the log data 406 to combine them and expand or increase the map representation of the geographic area in which the cloud data from the log data 406 was recorded.

[0161] In at least some embodiments of the present technology, the server 235 may generate a two-dimensional representation of the region based on the three-dimensional representation of the region. For example, the server 235 may project a plurality of vertically arranged points onto an element of a two-dimensional grid. Various characteristics of the plurality of vertically arranged points may be used to determine a value for this element. In one non-limiting example, intensity values from the plurality of vertically arranged points may be combined to determine a value to be associated with the element of the two-dimensional grid. It may be said that by projecting 3D points onto a two-dimensional grid, image data may be generated that represents various areas of a map. It may be said that in at least some embodiments of the invention, one or more images representing a map are stored in the data storage device 400.

[0162] In addition, additional data 410 may be stored in the data storage device 400. In some embodiments of the invention, the additional data 410 comprises information indicating the actual boundaries of the polygonal region provided by experts for at least some regions of the map representation 402. In other embodiments of the invention, a plurality of graph structures may be stored in the data storage device 400. For example, a graph structure representing sidewalks may be stored in the data storage device 400. In another example, a graph structure representing the location of roads and the direction of travel along these roads may be stored in the data storage device 400. In yet another example, a graph structure representing paths that can be used by pedestrians and vehicles may be stored in the data storage device 400. In a further example, a graph structure representing crosswalks that can be used by pedestrians and relatively small robotic vehicles may be stored in the data storage device 400.

[0163] In some embodiments of the invention, it is contemplated that information indicating one or more graph structures may be stored in conjunction with corresponding regions of the map representation. It is also contemplated that one or more graph structures may be rasterized and the resulting image data may be stored in conjunction with corresponding regions of the map representation.

[0164] In some embodiments of the present technology, at least some image data may be stored as GeoTIFF files. Briefly, GeoTIFF is an open metadata standard that allows georeferencing information to be embedded in an image file. Using the GeoTIFF format, geospatial metadata is embedded in image files, such as aerial photographs, satellite images, and digitized maps, so that they can be used in geographic information systems.

[0165] In particular, the geographic metadata of the GeoTIFF format describes the actual location in space represented by each pixel of the image. When a GeoTIFF file is generated, spatial information is included in the ".tif" file in the form of embedded tags, which may contain raster image metadata such as the horizontal and vertical origin, spatial extent (i.e., the area covered by the data set), the coordinate reference system (CRS) used to store the data, spatial resolution (the number of independent pixels per unit length), the number of layers in the ".tif" file, ellipsoids and geoids (e.g., calculated models of the shape of the Earth), mathematical map projection rules for transforming three-dimensional space data into a two-dimensional image, etc.

[0166] Fig. 5 shows a polygonal region 500 in an image 510 of a map region 520. As shown, the polygonal region 500 may contain stepped boundaries. Also shown is a polygonal region 502 in an image 512 of a map region 522. As shown, the polygon 502 may contain rounded corners and/or slightly undershot corners and boundaries. Also shown is a polygonal region 504 in an image 514 of a map region 524. As shown, the polygonal region 504 may have erroneous boundaries that do not correspond to reality. In this example, images 510, 512, and 514 are GeoTIFF files.

[0167] Fig. 6 shows a representation 600 for generating a segmentation map 630 by a segmentation model 650 based on image data 620. In some embodiments of the invention, the image data 620 may comprise a two-dimensional representation of a map region. Such an image may be generated by projecting points from a three-dimensional point cloud onto a two-dimensional grid. In other embodiments of the invention, one or more additional channels 610 of image data may be fed to the segmentation model to generate the segmentation map 630. For example, one or more graph structures may be rasterized and input as images into the segmentation model 650.

[0168] The segmentation map 630 comprises a plurality of points (without a numerical designation). Each point of the plurality of points is associated with a feature vector generated by the segmentation model 650. For example, the feature vectors may comprise features obtained from GeoTiff images by the segmentation model 650. It should be noted that the feature vector generated for a pixel may be at least partially based on features generated for neighboring pixels.

[0169] The neighboring pixels for a given pixel may be determined or depend on a particular embodiment of the segmentation model 650. In at least some embodiments of the invention, the plurality of points may correspond to a sequence of points that form a boundary of a polygonal region, where the neighboring points for a point from the sequence may include one or more preceding points and/or one or more subsequent points for this point from the sequence.

[0170] The segmentation map 630 also contains an indication of the boundary of the polygonal region 640. The boundary of the polygonal region 640 can be said to be an estimate of the portion of the segmentation map 630 that contains the object. The boundary of the polygonal region 640 can be said to be an estimate of the perimeter that defines the object in the region. The boundary of the polygonal region 640 contains a sequence of boundary points (without a numerical designation).

[0171] The server 235 is capable of obtaining the actual boundary of the polygonal area 660 associated with the object stored in the data storage device 400 in combination with the region. As described above, the actual boundary of the polygonal area 660 is provided by an expert (for example, a cartographer). In particular, the expert may be instructed to manually determine a sequence of edges of the boundary representing the actual location of the object in the region.

[0172] The server 235 may generate labels for points from a sequence of boundary points for the boundary of the polygonal region 640. These labels may be determined by mapping the actual boundary of the polygonal region 660 to the segmentation map 630 and comparing the boundary points from the boundary of the polygonal region 640 with the edges of the actual boundary of the polygonal region 660.

[0173] Fig. 7 shows a representation 700 for determining by the server 235 labels for corresponding points of the boundary of the polygonal area 640. Here, an edge 710 from the sequence of edges of the actual boundary of the polygonal area 660 is shown. Also shown is a subsequence 750 of boundary points from the sequence of boundary points, containing a first boundary point 720, a second boundary point 730, and a third boundary point 740.

[0174] It is contemplated that the position of a boundary point in a sequence of boundary points may be defined as a change in position relative to the position of the previous boundary point of this sequence of boundary points. For example, the position of the second boundary point 730 may be defined as a change in position relative to the position of the first boundary point 720, the position of the third boundary point 740 may be defined as a change in position relative to the position of the second boundary point 730, the next boundary point of the sequence of boundary points may be defined by a change in position relative to the position of the third boundary point 740, etc.

[0175] The server 235 may determine a position deviation from the closest edge of the sequence of edges. It should be noted that in the non-limiting example shown in Fig. 7, the closest edge to the first boundary point 720, the second boundary point 730, and the third boundary point 740 corresponds to the edge 710, but this is not necessary for all embodiments of the present technology. For example, each point from the first boundary point 720, the second boundary point 730, and the third boundary point 740 may correspond to different closest edges.

[0176] The server 235 may determine a first deviation 721 for the first boundary point 720 and a second deviation 722 for the second boundary point 730, and may also determine that the third boundary point 740 is on the edge 710 (i.e., determine a zero deviation and/or that the edge 710 passes through a point on the segmentation map whose position corresponds to the third boundary point 740).

[0177] The server 235 may determine labels for the points from the sequence 750 of boundary points based on the corresponding deviations. In some embodiments of the invention, where the second model is capable of performing the classification task, the server 235 may determine that the deviation is less than or greater than a predetermined threshold. For example, the server 235 may determine a label indicating a first class for the first boundary point 720 based on the first deviation 721 if the first deviation 721 is less than the predetermined threshold, and may determine a label indicating a second class for the second boundary point 730 based on the second deviation 722 if the second deviation 722 is greater than the predetermined threshold. For example, the predetermined deviation threshold may be 20 cm.

[0178] In other embodiments of the invention, where the second model is capable of performing a regression task, the server 235 may use the deviation as a label for the corresponding boundary point. For example, the server 235 may determine labels for the boundary points that indicate an offset value (e.g., deviations) for the corresponding boundary points. As described in more detail below, in at least some embodiments of the invention, the second model may comprise multiple output heads, where the first classification head is capable of solving a classification task, the second regression head is capable of solving a regression task, and the labels for both heads are generated based on deviations calculated for the corresponding boundary points.

[0179] It is contemplated that the output of the regression head may be used in the usage stage separately and/or in combination with the output of the classification head. For example, the output of the regression head may be used in the usage stage to determine an offset value for a boundary point. In this way, the server may automatically adjust the boundary point to compensate for this offset value.

[0180] In a non-limiting example, the second model may employ convolutional networks based on one-dimensional convolutions, recurrent networks, and/or transformers with attention-based machine learning (ML) architectures to perform the classification task. Convolutional networks based on one-dimensional convolution ML architectures may be implemented as described in the paper “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation” by Liang-Chieh Chen et al., published in 2018, the contents of which are incorporated herein by reference in their entirety. ML recurrent network architectures may be implemented as described in the paper “Sequence to Sequence Learning with Neural Networks” by Ilya Sutskever et al., published in 2014, the contents of which are incorporated herein by reference in their entirety. Transformers with attention-based ML architectures can be implemented as described in “Attention is All you Need” by Ashish Vaswani et al., published in 2017, the contents of which are incorporated herein by reference in their entirety.

[0181] In other embodiments of the invention, it is contemplated that the second model may comprise different parts for performing a classification task and a regression task. In these embodiments of the invention, the server 235 may determine multiple labels for the boundary point. The multiple labels may include a first label indicating a deviation from the closest edge, and a second label indicating that this deviation is less than or greater than a predetermined threshold.

[0182] Fig. 11 shows a block diagram of an ML architecture that can be used to implement the second model, where two label channels can be used, namely, a first channel for first labels and a second channel for second labels.

[0183] In particular, the ML architecture may comprise a neural network backbone that is fed with input features. In some embodiments of the invention, the input features comprise a tensor whose size depends on the number of boundary points and the number of image features. For example, the size of the tensor may be equal to [ N× (2+ K )], where N is the number of boundary points and K is the number of raw features from the first model, where each boundary point may be encoded using the difference in position relative to the immediately preceding boundary point from the sequence of boundary points and a vector of raw features from the first network for the pixel corresponding to this boundary point.

[0184] The neural network for feature extraction generates boundary point features that can be displayed using a tensor whose size depends on the number of boundary points and the number of image features. For example, the tensor size can be equal to [ N × M ], where N is the number of boundary points and M is the number of features from the second model. The boundary point features can include raw features for each boundary point, based on which deviations and/or classes are subsequently predicted.

[0185] As shown in Fig. 11, the ML architecture comprises a classification head and a regression head, which may be two different neural networks used in predicting classes and deviations. In some embodiments of the invention, the classification head and the regression head may form a multilayer perceptron.

[0186] The classification head generates the classes of the boundary points as a tensor of size [ N ×1], and the regression head generates the biases of the boundary points as a tensor of size [ N ×2]. The developers found that the classification head can be trained based on the first channel of class labels (indicating the classes), and the regression head can be trained based on the second channel of regression labels (indicating the bias value). It is assumed that the server 235 can apply the regression head depending on the class predicted by the classification head of the use stage data set.

[0187] In some embodiments of the present technology, the second model may be implemented according to the CatBoost algorithm. Briefly, CatBoost is a gradient boosting algorithm designed to process categorical features. The CatBoost algorithm operates within the gradient boosting framework (a popular ensemble learning tool). The model iteratively builds an ensemble of decision trees to create a strong predictive model. Each tree corrects the errors of its predecessor, resulting in improved overall predictions. In these embodiments of the invention, it is contemplated that the first CatBoost model can be trained and used to solve a classification problem using a first channel of class labels, and the second CatBoost model can be trained and used to solve a regression problem using a second channel of regression labels, as described above. In these embodiments of the invention, the input data may comprise data indicating neighboring points determined by server 235 for the input points.

[0188] Fig. 8 shows a representation 800 of a training procedure that server 235 may perform for a second model 850 based on the boundary of polygonal region 640 and the actual boundary of polygonal region 660.

[0189] Server 235 may generate training data based on (a) the positions of boundary points from a sequence of boundary points, (b) feature vectors associated with the boundary points from the sequence of boundary points, (c) feature vectors of other points from the segmentation map that are neighbors of the boundary points from the sequence of boundary points, and (d) labels associated with the boundary points from the sequence of boundary points.

[0190] Fig. 8 shows a sequence 810 of positions corresponding to the positions of a sequence of boundary points. It can be assumed that the positions 811, 812 and 813 correspond to the positions of the first boundary point 720, the second boundary point 730 and the third boundary point 740, respectively.

[0191] It should be noted that the position 812 of the second boundary point 730 is associated with a feature vector 815 associated with the second boundary point 730 and with a plurality of 818 neighboring feature vectors associated with neighboring points of the second boundary point 730. It can be said that the plurality of 818 neighboring feature vectors represents the context of the second boundary point 730. It is assumed that the model can determine the plurality of 818 neighboring feature vectors based on the neighboring points.

[0192] The selection of a neighbor may depend, among other things, on the specific implementation options of the UNet network (which processes the contour points): the number of layers, parameters, and connections between layers may be used to control the size of the neighborhood. For example, points may be uniformly selected on the contour, and then a fully convolutional network with one-dimensional convolutions may be applied to them. Depending on the number of convolutions and their parameters, the point, so to speak, "observes" a neighborhood of some size. For example, as a result of one one-dimensional convolution with a kernel configuration of size 3, the point may have information about a neighborhood with radius 1, as a result of one convolution with a kernel configuration of size 5, the point may have information about a neighborhood with radius 2, as a result of two consecutive convolutions with a kernel configuration of size 5, the point may have information about a neighborhood with radius 4.

[0193] The server 235 may input the corresponding positions from the sequence 810 of positions with feature vectors into the second model 850, which generates predicted scores from the sequence 830 of predicted scores. For example, the second model 850 may generate a predicted score 832 for the position 812 and the corresponding feature vectors.

[0194] The server 235 is capable of comparing the predicted scores from the sequence 830 of predicted scores with the sequence 820 of labels. For example, the sequence 820 of labels may comprise labels 821, 822 and 823 for corresponding points from among the first boundary point 720, the second boundary point 730 and the third boundary point 740. Thus, the server 235 can compare the predicted score 832 with the label 822. The server 235 can adjust the second model 850 based on the results of these comparisons of the predicted values and labels.

[0195] Fig. 9 shows a representation 900 for the operation of the second model 850 in the use stage. The second model 850 can receive a sequence 910 of positions corresponding to the positions of a sequence of boundary points from a polygonal region in the use stage generated by the segmentation model 650 in the unestimated region of the map. The sequence 910 of positions comprises positions 911, 912 and 913.

[0196] It should be noted that the position 912 corresponding to the boundary point of the polygonal section of the use stage is associated with the feature vector 915 associated with the second boundary point and with a plurality of 918 neighboring feature vectors associated with neighboring points of this boundary point. It can be said that the plurality of 918 neighboring feature vectors represents the context of the boundary point.

[0197] The server 235 may input the positions from the sequence 910 of the positions of the use stage with the feature vectors into the second model 850, which generates predicted estimates (confidence estimates) of the use stage from the sequence 930 of predicted estimates. For example, the second model 850 may generate a predicted estimate 932 for the position 912 and the corresponding feature vectors.

[0198] In some embodiments of the invention, if one or more predicted estimates from the sequence 930 of predicted estimates indicate that the boundary points of the use stage are incorrectly located, the server 235 may determine the polygonal region in the use stage (and/or a portion thereof) and/or the unrated region of the map for subsequent evaluation by an expert.

[0199] Fig. 10 is a flow chart of a method 1000 performed by a processor (e.g., a processor of server 235). The steps of the method 1000 are described in more detail below.

Step 1002: Obtain image data for the training region containing the training object.

[0200] The method 1000 begins with step 1002, in which the processor receives image data for a training region containing a training object. In some embodiments of the invention, the training object is a section of a road or a section of a sidewalk. It should be noted that the image data for the training region and the data about the training object can be used when operating a vehicle in the training region. In some embodiments of the invention, the vehicle can be an SDC vehicle operated on sections of a road. In other embodiments of the invention, the vehicle can be a delivery robot operated on sections of a sidewalk.

[0201] It is contemplated that the image data may comprise a plurality of image data files. In some embodiments of the invention, the image data may comprise a bird's eye view representation of the training region. In other embodiments of the invention, the image data may comprise a two-dimensional image of the training region generated by a processor from a three-dimensional point cloud representing the training region. In other embodiments of the invention, the image data may comprise one or more raster images.

Step 1004: Generate a segmentation map using the segmentation model based on the image data.

[0202] The method 1000 continues at step 1004, where the processor is configured to generate a segmentation map based on the image data using a segmentation model 650. The segmentation model 650 may be an MLA algorithm that is configured to perform segmentation of the image data. In some embodiments of the invention, it is contemplated that the segmentation model 650 may, among other things, assign labels to pixels of the input image so that pixels with the same label have the same characteristic (or several characteristics). In one example, pixels associated with a label may represent the same object in the input image. It is also contemplated that a plurality of algorithms may be applied to the labeled pixels in order to determine the contour or boundary of an object in the input image.

[0203] In some embodiments of the present technology, the segmentation model 650 may be implemented as a CNN network trained for image segmentation. In one embodiment of the invention, the CNN network may have a U-Net architecture.

[0204] It can be said that the output data of the segmentation model 650 (segmentation map) may comprise a plurality of points defining a training boundary of a polygonal region. The output data of the segmentation model 650 may comprise correspondingly associated feature vectors for the plurality of points.

[0205] The training polygonal boundary generated by the segmentation model 650 may be a computer-generated boundary corresponding to an attempt to determine a training object in the training region. The training boundary of the polygonal region comprises a sequence of boundary points, wherein the position of the boundary point in the sequence is defined as a change in position relative to the position of the previous boundary point from this sequence.

Step 1006: Get the actual boundary of the polygonal region that defines the training object in the segmentation map.

[0206] The method 1000 continues at step 1006, where the processor is configured to obtain an actual boundary of the polygonal region defining the training object on the segmentation map. The actual boundary of the polygonal region comprises a sequence of boundary edges determined by an expert (e.g., a cartographer). Data indicating the actual boundary of the polygonal region may be stored in the data storage device 400 in conjunction with the training region. In some embodiments of the invention, the expert may be instructed to manually determine the sequence of boundary edges representing the actual location of the training object in the training region.

Step 1008: Determine the position deviation from the nearest edge from the edge sequence that points to the boundary point label for the boundary point from the boundary point sequence.

[0207] The method 1000 continues at step 1008, where the processor is configured to determine, for a boundary point from a sequence of points of the training boundary of the polygonal region, a deviation of its position from the nearest edge from the sequence of edges of the actual boundary of the polygonal region. It should be noted that the deviation determined in this way indicates a label of the boundary point to be used for training the second model.

Step 1010: Train the second model to predict the confidence score for the boundary point of the use stage boundary of the polygonal region of the use stage.

[0208] The method 1000 continues at step 1010, wherein for a boundary point of the sequence of boundary points, the processor is configured to train a second model 850 to predict a confidence score for the boundary point of the use stage boundary of the polygonal portion of the use stage.

[0209] Training during the training iteration includes using a second model of a feature vector associated with the boundary point and a set of feature vectors of neighboring points on the segmentation map representing the context of the boundary point to form a predicted estimate of the position of the boundary point. Training during the training iteration includes adjusting the second model 850 based on a comparison of the predicted estimate and the label.

[0210] In some embodiments of the invention, it is contemplated that a set of feature vectors of neighboring points may correspond to feature vectors of neighboring points along a boundary of a polygonal region (e.g., a sequence of boundary points). In particular, neighboring points may include one or more preceding boundary points and/or one or more subsequent boundary points from a sequence of boundary points forming a boundary of a polygonal region.

[0211] In some embodiments of the invention, the second model 850 is a classification model, and the label indicates a first class corresponding to a high confidence class if the deviation is less than a predetermined threshold, or indicates a second class corresponding to a low confidence class if the deviation is greater than a predetermined threshold. For example, the predetermined threshold may be 20 cm.

[0212] In other embodiments of the invention, it is contemplated that the second model 850 may comprise different parts or different heads for performing the classification task and the regression task. In these embodiments of the invention, the processor may determine multiple labels for the boundary point. The multiple labels may include a first label indicating a deviation from the closest edge, and a second label indicating that this deviation is less than or greater than a predetermined threshold.

[0213] It is contemplated that the processor may receive the image data of the use stage for the region of the use stage containing the use stage object that has not been evaluated by the expert. The processor, using the segmentation model 650, may generate a segmentation map of the use stage that defines the boundary of the polygonal region of the use stage, similar to how the segmentation model 650 generates the training polygonal region. The processor may then use the trained second model 850 to generate a confidence estimate of the use stage for a point of the boundary of the use stage from a sequence of points of the boundary of the use stage. If the confidence estimate indicates low confidence (i.e., indicates an appropriate class and/or is less than a threshold), then the processor may automatically assign at least a portion of the region of the use stage for evaluation by the expert and receive the actual boundary of the polygonal region of the use stage object from the expert.

[0214] It should be noted that the polygonal sections can be sent from the server 235 to the electronic device 210 associated with the vehicle for navigation purposes. For example, one or more navigation and/or control algorithms performed by the electronic device 210 can use the polygonal sections of one or more elements from the map region for navigation and/or for controlling the operation of the vehicle in this map region.

[0215] Possible changes and improvements to the above-described embodiments of the present technology may be obvious to a person skilled in the art. The preceding description is provided for illustrative purposes only and is not intended to limit the scope of the invention. The scope of protection of the present technology is determined solely by the scope of the appended claims.

Claims (50)

1. A method for classifying the boundary points of a polygonal section of an object on an image of a neighborhood map, performed by a processor and including: receiving, by a processor, image data for a training region containing a training object from at least one sensor; forming by the processor, based on the image data, a segmentation map using a segmentation model, wherein the segmentation map comprises a plurality of points associated with a corresponding feature vector formed by the segmentation model, and defines a training boundary of a polygonal area defining a training object in the training region and containing a sequence of boundary points, wherein the position of the boundary point in the sequence is specified as a change in position relative to the position of the previous boundary point from this sequence; obtaining by the processor the actual boundary of the polygonal area, defining the training object on the segmentation map and containing a sequence of boundary edges determined by the expert; determining for a boundary point from a sequence of boundary points the deviation of its position from the nearest edge from the sequence of edges pointing to the boundary point label; training the processor of the second model to predict the reliability estimate for a point of the boundary of the use stage of the boundary of the polygonal section of the use stage, wherein training at the training iteration includes: - using a second model of a feature vector associated with a boundary point and a set of feature vectors of neighboring points on the segmentation map representing the context of the boundary point to generate a predictive estimate of the position of the boundary point; - adjustment of the second model based on the result of comparison of the predicted estimate and the label; classification by the processor based on the obtained image data for the area of the stage of using the boundary points of the polygonal area using the trained second model. 2. The method according to paragraph 1, characterized in that it additionally includes: obtaining by the processor of the image data at the use stage for the area of the use stage containing the object of the use stage that has not been assessed by the expert, to form the actual boundary of the polygonal area defining the object of the use stage; the formation by the processor of a segmentation map at the use stage, defining the boundary of the polygonal section of the use stage, containing a sequence of points of the boundary of the use stage, using the segmentation model; generating by the processor a reliability estimate for a point of the boundary of the stage of use from a sequence of points of the boundary of the stage of use using the second model; in response to determining a low confidence score, automatically determining at least a portion of the area of the use stage for evaluation by an expert; receiving by the processor the actual boundary of the polygonal section of the object at the stage of use from the expert. 3. The method according to paragraph 1, characterized in that the training object is a section of a road or a section of a sidewalk. 4. The method according to claim 1, characterized in that the image data contains a representation of a bird's eye view of the training area. 5. The method according to claim 1, characterized in that the image data contains a two-dimensional image of the training region, formed from a three-dimensional point cloud representing the training region. 6. The method according to paragraph 1, characterized in that the image data contains one or more raster images. 7. The method according to claim 1, characterized in that the segmentation model is a fully convolutional neural network. 8. The method according to claim 1, characterized in that the second model is a classification model, and the label indicates the first class corresponding to the high confidence class if the deviation is less than a predetermined threshold, or indicates the second class corresponding to the low confidence class if the deviation is greater than a predetermined threshold. 9. The method according to claim 1, characterized in that the second model contains two heads, wherein the first head is trained on the basis of a first mark indicating a deviation from the nearest edge, and the second head is trained on the basis of a second mark indicating that this deviation is less than or greater than a predetermined threshold. 10. The method according to claim 1, characterized in that the boundary of the polygonal section at the use stage is stored in combination with the area of the use stage and is transmitted to a second processor associated with the vehicle for navigating the vehicle in the area of the use stage. 11. The method according to paragraph 10, characterized in that the vehicle is a self-driving car operated on road sections. 12. The method according to paragraph 10, characterized in that the vehicle is a delivery robot operated on sections of sidewalk. 13. A processor for classifying the boundary points of a polygonal section of an object on an image of a map of the vicinity, configured to: obtaining image data for a training region containing a training object from at least one sensor; forming, based on the image data, a segmentation map using a segmentation model, wherein the segmentation map comprises a plurality of points associated with a corresponding feature vector generated by the segmentation model and defines a training boundary of a polygonal area defining a training object in the training region and containing a sequence of boundary points, wherein the position of the boundary point in the sequence is specified as a change in position relative to the position of the previous boundary point from this sequence; obtaining the actual boundary of a polygonal area, defining a training object on the segmentation map and containing a sequence of boundary edges determined by an expert; determining for a boundary point from a sequence of boundary points the deviation of its position from the nearest edge from the sequence of edges pointing to the boundary point label; training the second model to predict the reliability score for a point of the boundary of the use stage of the boundary of the polygonal section of the use stage, whereby training at the training iteration includes: - using a second model of a feature vector associated with a boundary point and a set of feature vectors of neighboring points on the segmentation map representing the context of the boundary point to generate a predictive estimate of the position of the boundary point; - adjustment of the second model based on the result of comparison of the predicted estimate and the label; classification based on the obtained image data for the area of the stage of using the boundary points of the polygonal area using the trained second model. 14. The processor according to item 13, characterized in that it is additionally designed with the ability to: obtaining the image data of the use stage for the area of the use stage containing the use stage object that has not been assessed by the expert to form the actual boundary of the polygonal area defining the use stage object; forming a segmentation map of the use stage, defining the boundary of the polygonal section of the use stage, containing a sequence of points of the boundary of the use stage, using the segmentation model; forming a reliability estimate for a point in the use stage boundary from a sequence of points in the use stage boundary using a second model; automatically determining at least a portion of the area of the use stage for evaluation by an expert if the reliability assessment indicates low reliability; obtaining the actual boundary of a polygonal section of an object at the stage of use from an expert. 15. The processor according to paragraph 13, characterized in that the training object is a section of a road or a section of a sidewalk. 16. The processor according to claim 13, characterized in that the image data contains a representation of a bird's eye view of the training area. 17. The processor according to claim 13, characterized in that the image data contains a two-dimensional image of the training region, formed from a three-dimensional point cloud representing the training region. 18. The processor according to claim 13, characterized in that the image data contains one or more raster images. 19. The processor according to claim 13, characterized in that the segmentation model is a fully convolutional neural network. 20. The processor according to claim 13, characterized in that the second model is a classification model, and the label indicates a first class corresponding to a high confidence class if the deviation is less than a predetermined threshold, or indicates a second class corresponding to a low confidence class if the deviation is greater than a predetermined threshold. 21. The processor according to claim 13, characterized in that the second model contains two heads, wherein the first head is trained on the basis of a first mark indicating a deviation from the nearest edge, and the second head is trained on the basis of a second mark indicating that this deviation is less than or greater than a predetermined threshold. 22. The processor according to claim 13, characterized in that the boundary of the polygonal section at the use stage is stored in combination with the area of the use stage and is transmitted to a second processor associated with the vehicle for navigation of the vehicle in the area of the use stage. 23. The processor according to paragraph 22, characterized in that the vehicle is a self-driving car operated on road sections. 24. The processor according to paragraph 22, characterized in that the vehicle is a delivery robot operated on sections of sidewalk.

Images