WO2024140749A1 - Ultrasonic scanning method, apparatus and system, and electronic device and storage medium - Google Patents

Abstract

An ultrasonic scanning method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring at least one image to be subjected to detection, wherein among the at least one image to be subjected to detection, a first portion of images to be subjected to detection each contain an ultrasonic probe, and a second portion of images to be subjected to detection each contain a target checking region of an object to be detected; performing target detection on the basis of the first portion of images to be subjected to detection, so as to determine initial probe coordinates of the ultrasonic probe; performing pose estimation on the basis of the second portion of images to be subjected to detection, so as to determine key point coordinates of at least one key point in the target checking region; on the basis of the initial probe coordinates of the ultrasonic probe and the key point coordinates of the at least one key point, planning a motion trajectory of the ultrasonic probe scanning the at least one key point; and according to the planned motion trajectory, controlling the ultrasonic probe to sequentially move to the location of the at least one key point, so as to scan a part to which the at least one key point belongs. By means of the solution, complex user operations are not required, such that the hands of a user can be freed to a certain extent.

Description

Ultrasonic scanning method, device and system, electronic device and storage medium Technical Field

The present invention relates to the technical field of ultrasonic imaging, and in particular to an ultrasonic scanning method, an ultrasonic scanning device, an electronic device, an ultrasonic scanning system and a storage medium.

Background technique

Ultrasound scanning is a simple and effective medical examination method, and has been widely used in medical diagnosis in recent years. At present, ultrasound scanning mainly relies on doctors holding ultrasound probes to operate. This is a repetitive and tedious job for doctors. At the same time, long hours of work may cause diseases such as arthritis, affecting the doctor's health and work efficiency. In addition, since different doctors have different scanning techniques, it is difficult to ensure the standardization of ultrasound images obtained by ultrasound scanning.

Therefore, a new ultrasonic scanning method is urgently needed to solve the above problems.

Summary of the invention

In order to at least partially solve the problems existing in the prior art, an ultrasonic scanning method, an ultrasonic scanning device, an electronic device, an ultrasonic scanning system and a storage medium are provided.

According to one aspect of the present invention, there is provided an ultrasonic scanning method, comprising: acquiring at least one image to be tested, wherein a first portion of the image to be tested in the at least one image to be tested includes an ultrasonic probe, and a second portion of the image to be tested in the at least one image to be tested includes a target inspection area of the object to be tested, the first portion of the image to be tested is at least a portion of the image to be tested in the at least one image to be tested, and the second portion of the image to be tested is at least a portion of the image to be tested in the at least one image to be tested; performing target detection based on the first portion of the image to be tested to determine initial probe coordinates of the ultrasonic probe, wherein the initial probe coordinates are two-dimensional coordinates or three-dimensional coordinates; performing posture estimation based on the second portion of the image to be tested to determine key point coordinates of at least one key point in the target inspection area, wherein the key point coordinates are two-dimensional coordinates or three-dimensional coordinates; planning a motion trajectory of the ultrasonic probe scanning at least one key point based on the initial probe coordinates of the ultrasonic probe and the key point coordinates of at least one key point; and controlling the ultrasonic probe according to the planned motion trajectory. The ultrasonic probe is moved to the position of at least one key point in sequence to scan the part to which the at least one key point belongs.

Exemplarily, in the process of controlling the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method also includes: when the ultrasonic probe moves to the position of any key point, performing a corresponding scanning feedback operation based on the part to which the key point belongs, the scanning feedback operation includes one or more of the following operations: displaying a diagnostic measurement interface, the diagnostic measurement interface including at least one first measurement item corresponding to the part to which the key point belongs; measuring at least one second measurement item corresponding to the part to which the key point belongs; measuring at least one third measurement item in response to a measurement instruction of at least one third measurement item corresponding to the part to which the key point belongs input by the user; displaying a standard human body model, and highlighting the human body area corresponding to the part to which the key point belongs on the standard human body model.

Exemplarily, at least one second measurement item includes: identifying and measuring standard sections based on an ultrasound image scanned for the part to which the key point belongs, identifying and measuring standard sections based on an ultrasound image scanned for the part to which the key point belongs, including: real-time standard section identification based on the ultrasound image to determine whether the ultrasound image belongs to at least one of the preset types of standard sections; when the ultrasound image belongs to a specific preset type of standard section, performing image segmentation on the ultrasound image to segment at least one target structure from the ultrasound image, the at least one target structure belonging to a structure related to the specific preset type of standard section; based on the segmentation result of at least one target structure, measuring at least one sub-measurement item, the at least one sub-measurement item being a sub-measurement item related to at least one target structure.

Exemplarily, in response to a measurement instruction input by a user for at least one third measurement item corresponding to the part to which the key point belongs, measuring the at least one third measurement item includes: in response to a selection operation instruction input by a user for one or more first measurement items included on the diagnostic measurement interface, measuring the at least one third measurement item, wherein the at least one third measurement item is one or more first measurement items, and the measurement instruction is a selection operation instruction.

Exemplarily, in the process of controlling the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method further includes: obtaining resistance information detected by a force sensor, the force sensor being arranged at one end of the ultrasonic probe in contact with the object to be measured; when the ultrasonic probe moves to the position of any key point, according to The corresponding resistance information adjusts the motion trajectory of the ultrasonic probe.

Exemplarily, when the ultrasonic probe moves to the location of any key point, the motion trajectory of the ultrasonic probe is adjusted according to the corresponding resistance information, including: when the resistance information is greater than a preset resistance threshold, controlling the ultrasonic probe to move in a direction opposite to the current motion direction to adjust the motion trajectory of the ultrasonic probe.

Exemplarily, one of the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point is located in the image coordinate system, and the other is located in the world coordinate system. Based on the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point, a motion trajectory of the ultrasound probe scanning at least one key point is planned, including: based on the conversion relationship between the image coordinate system and the world coordinate system, performing coordinate transformation on the initial probe coordinates of the ultrasound probe and/or the key point coordinates of at least one key point, so as to unify the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point into the world coordinate system or the image coordinate system; when unified into the world coordinate system, based on the conversion relationship between the image coordinate system and the world coordinate system, The distance between the ultrasound probe and the at least one key point in the world coordinate system is determined based on the three-dimensional coordinates of the head in the world coordinate system and the three-dimensional coordinates of at least one key point in the world coordinate system; when unified to the image coordinate system, the distance between the ultrasound probe and the at least one key point in the image coordinate system is determined based on the two-dimensional coordinates of the ultrasound probe in the image coordinate system and the two-dimensional coordinates of at least one key point in the image coordinate system, and the distance between the ultrasound probe and the at least one key point in the world coordinate system is determined based on the conversion relationship and the distance between the ultrasound probe and the at least one key point in the image coordinate system; the motion trajectory is planned based on the distance between the ultrasound probe and the at least one key point in the world coordinate system.

Exemplarily, the first part of the images to be tested includes at least one two-dimensional image, wherein target detection is performed based on the first part of the images to be tested to determine the initial probe coordinates of the ultrasound probe, including: inputting any two-dimensional image of the at least one two-dimensional image into the target detection model to obtain probe position information of the ultrasound probe; determining the initial probe coordinates of the ultrasound probe based on the probe position information, the initial probe coordinates being two-dimensional coordinates.

Exemplarily, the probe position information includes a target detection frame for indicating the position of the ultrasound probe, and determining the initial probe coordinates of the ultrasound probe based on the probe position information includes: determining the coordinates of the center of mass of the ultrasound probe as the initial probe coordinates based on the target detection frame.

Exemplarily, the second portion of the image to be tested includes at least one three-dimensional depth image, wherein the posture estimation is performed based on the second portion of the image to be tested to determine at least one key position in the target inspection area. The key point coordinates of the key point include: inputting any three-dimensional depth image of at least one three-dimensional depth image into a three-dimensional posture estimation model to obtain the key point coordinates of at least one key point, where the key point coordinates are three-dimensional coordinates.

Exemplarily, in the process of controlling the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method also includes: acquiring mask information of the object to be measured and real-time image acquisition in real time, the real-time image acquisition is an image acquired in real time by the ultrasonic probe during the movement of the ultrasonic probe; performing target detection based on the real-time image acquisition to determine the real-time probe coordinates of the ultrasonic probe, the real-time probe coordinates are two-dimensional coordinates or three-dimensional coordinates; determining whether the ultrasonic probe falls on the object to be measured based on the real-time probe coordinates and mask information of the ultrasonic probe; if the ultrasonic probe does not fall on the object to be measured, and/or, if the ultrasonic probe falls on the object to be measured, outputting corresponding prompt information.

According to another aspect of the present invention, an ultrasonic scanning device is also provided, comprising: an acquisition module, used to acquire at least one image to be tested, a first part of the image to be tested in the at least one image to be tested includes an ultrasonic probe, and a second part of the image to be tested in the at least one image to be tested includes a target inspection area of the object to be tested, the first part of the image to be tested is at least a part of the image to be tested in the at least one image to be tested, and the second part of the image to be tested is at least a part of the image to be tested in the at least one image to be tested; a detection module, used to perform target detection based on the first part of the image to be tested to determine the initial probe coordinates of the ultrasonic probe, the initial probe coordinates are two-dimensional coordinates or three-dimensional coordinates; an estimation module, used to perform posture estimation based on the second part of the image to be tested to determine the key point coordinates of at least one key point in the target inspection area, the key point coordinates are two-dimensional coordinates or three-dimensional coordinates; a planning module, used to plan a motion trajectory of the ultrasonic probe scanning at least one key point based on the initial probe coordinates of the ultrasonic probe and the key point coordinates of at least one key point; and a control module, used to control the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs.

According to another aspect of the present invention, there is provided an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the above-mentioned ultrasonic scanning method.

Exemplarily, the electronic device is an ultrasound diagnostic device or an ultrasound workstation.

According to another aspect of the present invention, there is also provided an ultrasonic scanning system, comprising: a mechanical arm, at the end of which an ultrasonic probe is provided; at least one image acquisition device for acquiring at least one an image to be tested; the above-mentioned electronic device, the processor in the electronic device is connected to at least one image acquisition device and a robotic arm, and is used to perform the above-mentioned ultrasonic scanning method based on at least one image to be tested, wherein the processor controls the robotic arm to drive the ultrasonic probe to move by sending a control instruction to the robotic arm.

Exemplarily, at least one image acquisition device includes a depth camera and/or a color camera, and at least one image to be measured includes at least one three-dimensional depth image acquired by the depth camera and/or at least one two-dimensional image acquired by the color camera.

According to yet another aspect of the present invention, a storage medium is provided, storing a computer program/instruction, and the computer program/instruction implements the above-mentioned ultrasonic scanning method when executed by a processor.

According to the ultrasonic scanning method, ultrasonic scanning device, electronic device, ultrasonic scanning system and storage medium of the embodiments of the present invention, the initial probe coordinates of the ultrasonic probe and the key point coordinates of at least one key point can be automatically determined based on the image to be measured, and the motion trajectory of the ultrasonic probe can be planned based on these coordinates and the ultrasonic probe can be controlled to scan according to the motion trajectory. Since the solution adds detection of the probe and key points, the positioning accuracy is high and the planned motion trajectory is more accurate. In addition, by controlling the ultrasonic probe to perform ultrasonic scanning through the planned motion trajectory, the standard of the ultrasonic image obtained by the scan can be improved. In addition, this solution does not require the user to perform complex operations and can free the user's hands to a certain extent.

The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

FIG1 is a schematic flow chart of an ultrasonic scanning method according to an embodiment of the present invention;

FIG2 is a schematic diagram showing a highlighted human body area according to an embodiment of the present invention;

FIG3 shows a schematic block diagram of an ultrasonic scanning device according to an embodiment of the present invention; and

FIG. 4 shows a schematic block diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

In order to at least partially solve the above technical problems, an embodiment of the present invention provides an ultrasonic scanning method. FIG1 shows a schematic diagram of an ultrasonic scanning method 100 according to an embodiment of the present invention. As shown in FIG1 , the ultrasonic scanning method 100 may include the following steps S110, S120, S130, S140, and S150.

Step S110, acquiring at least one image to be tested, wherein a first part of the image to be tested in at least one image to be tested includes an ultrasonic probe, and a second part of the image to be tested in at least one image to be tested includes a target inspection area of the object to be tested, the first part of the image to be tested is at least a part of the image to be tested in at least one image to be tested, and the second part of the image to be tested is at least a part of the image to be tested in at least one image to be tested.

The object to be tested may be any object, including but not limited to humans or animals, etc. The target inspection area of the object to be tested may be at least a part of the object to be tested, such as the whole body of the object to be tested, or one or more parts of the object to be tested, such as the uterus, etc.

Exemplarily, the ultrasound scanning system may include an ultrasound probe and at least one image acquisition device. The at least one image acquisition device may include a depth camera and/or a color camera. At least one image acquisition device may be used to acquire at least one image to be tested that includes a target inspection area of the object to be tested and the ultrasound probe. In any image to be tested, the ultrasound probe and the target inspection area may exist at the same time, or only one of them may exist. That is, the first part of the image to be tested and the second part of the image to be tested described herein may be all the same, partially the same, or all different.

In one embodiment, a color camera may be used alone to capture the image to be tested. In this case, the image to be tested is a two-dimensional image. In this case, the number of the images to be tested may be one or more. By way of example and not limitation, for the first part of the images to be tested among the multiple images to be tested, each image to be tested may be a two-dimensional image. The image may include an ultrasonic probe, and the position of the ultrasonic probe can be identified from each image to be tested. For the second part of the images to be tested in the multiple images to be tested, each image to be tested may include a target inspection area, and the position of the key points in the target inspection area can be identified from each image to be tested. In another embodiment, a depth camera can be used alone to capture the image to be tested. In this case, the image to be tested is a three-dimensional depth image. In this case, the number of images to be tested can be one or more. Exemplarily but not restrictively, for the first part of the images to be tested in the multiple images to be tested, each image to be tested may include an ultrasonic probe, and the position of the ultrasonic probe can be identified from each image to be tested. For the second part of the images to be tested in the multiple images to be tested, each image to be tested may include a target inspection area, and the position of the key points in the target inspection area can be identified from each image to be tested. In another embodiment, the images to be tested can be captured separately using a color camera and a depth camera to obtain multiple images to be tested. Exemplarily but not restrictively, one or more images to be tested captured by the color camera may at least include an ultrasonic probe, and one or more images to be tested captured by the depth camera may at least include a target inspection area. In this case, the position of the ultrasound probe (for example, two-dimensional coordinates) can be determined based on the first part of the image to be tested captured by the color camera, and the position of at least one key point in the target inspection area (for example, three-dimensional coordinates) can be determined based on the second part of the image to be tested captured by the depth camera.

Step S120 , performing target detection based on the first part of the image to be tested to determine the initial probe coordinates of the ultrasound probe, where the initial probe coordinates are two-dimensional coordinates or three-dimensional coordinates.

Exemplarily, the target detection model can be used to perform target detection on the first part of the images to be tested. In the case where at least one image to be tested is all two-dimensional images or three-dimensional depth images, target detection can be performed directly based on all the images to be tested in at least one image to be tested (that is, all the images to be tested in at least one image to be tested are the first part of the images to be tested). Of course, in this case, it is also feasible to perform target detection based on part of the images to be tested in at least one image to be tested (that is, part of the images to be tested in at least one image to be tested is the first part of the images to be tested). The first part of the images to be tested can optionally be an image obtained after filtering at least one image to be tested and removing abnormal images such as reflections and blurs. Exemplarily, the target detection model can be implemented using any suitable neural network model, for example, using any suitable existing or future target detection network model. For example, the target detection model can include one or more of the following: You Only Look Once (YOLO) series, Region-Convolutional Neural Networks (RCNN) series, Retina-Net and other network models. Of course, the above-mentioned target detection network models are only examples, and the target detection model can also use any suitable existing Or it may be implemented by an image segmentation network model that may appear in the future. For example, the target detection model may include one or more of the following: Fully Convolutional Networks (FCN), U-type network (Unet), DeepLab series, V-type network (Vnet) and other network models. It can be understood that the image segmentation network model can also identify the location of the target object (such as the above-mentioned ultrasound probe). The target detection network model can obtain a target detection box (bounding box) for indicating the location of the ultrasound probe. The target detection box can be of any suitable shape, preferably a rectangle. The image segmentation network model can obtain a mask or envelope for indicating the location of the ultrasound probe.

Based on the target detection results, the initial probe coordinates of the ultrasound probe can be determined. For the image to be tested acquired by the depth camera, the three-dimensional coordinates of the ultrasound probe can be obtained based on the corresponding target detection results. For the image to be tested acquired by the color camera, the two-dimensional coordinates of the ultrasound probe can be obtained based on the corresponding target detection results.

The target detection model used in step S120 can be obtained by training with a first training data set. The first training data set may include a plurality of first sample images and first annotation information (ground truth) corresponding to the plurality of first sample images. The first sample image may include an ultrasound probe. The first annotation information may include an annotation target detection box for indicating the location of the ultrasound probe. The plurality of first sample images are respectively input into the initial target detection model, and the corresponding predicted target detection results can be obtained, and the predicted target detection results include a predicted target detection box for indicating the location of the ultrasound probe. The network structure of the initial target detection model is consistent with that of the target detection model used in step S120, but the parameters may be inconsistent. After training the parameters in the initial target detection model, the target detection model used in step S120 is obtained. The predicted target detection results and the first annotation information of the plurality of first sample images can be substituted into the first preset loss function for loss calculation to obtain a first loss value. Subsequently, the parameters in the initial target detection model can be optimized according to the first loss value using the back propagation and gradient descent algorithms. The optimization of the parameters can be iteratively performed until the target detection model reaches a convergence state. After the training is completed, the obtained target detection model can be used for subsequent target detection. This stage can be called the inference stage of the model.

Step S130 , performing posture estimation based on the second part of the image to be tested to determine the key point coordinates of at least one key point in the target inspection area, where the key point coordinates are two-dimensional coordinates or three-dimensional coordinates.

For example, for the second part of the image to be measured obtained above, the pose estimation model can be used to estimate the pose of the image to determine the position of at least one key point. In the case where all images are two-dimensional images or three-dimensional depth images, posture estimation can be performed directly based on all images to be tested in at least one image to be tested (i.e., all images to be tested in at least one image to be tested are the second part of images to be tested). Of course, in this case, it is also feasible to perform posture estimation based on part of the images to be tested in at least one image to be tested (i.e., part of the images to be tested in at least one image to be tested is the second part of images to be tested). The second part of the images to be tested can optionally be an image obtained after filtering at least one image to be tested and removing abnormal images such as reflections and blurs. The posture estimation model can be implemented using any suitable neural network model, such as any suitable existing or future posture estimation network model. For example, the posture estimation model can include one or more of the following: hourglass network structure (Hourglass), C2F-Vol and other network models. Based on the posture estimation result of the posture estimation model, the two-dimensional or three-dimensional coordinates of at least one key point in the target inspection area can be obtained. In the case where the object to be tested is a person, the at least one key point can include human body key points such as nose, eyes, ears, knees, etc. The number of at least one key point can be set as needed, and the present invention is not limited thereto. For example, the number of at least one key point can be 12, 20, 32, and so on. For the image to be tested acquired by the depth camera, the three-dimensional coordinates of the key points can be obtained. For the image to be tested acquired by the color camera, the two-dimensional coordinates of the key points can be obtained.

The above-mentioned posture estimation model can be obtained by training the second training data set. The second training data set may include multiple second sample images and second annotation information corresponding to the multiple second sample images one by one. The second annotation information may include annotated two-dimensional coordinates or annotated three-dimensional coordinates of each human key point in the second sample image. Multiple second sample images are respectively input into the initial posture estimation model to obtain the corresponding predicted posture estimation results. In the case where the sample image is a two-dimensional image, the predicted posture estimation result is the predicted two-dimensional coordinate of the key point. It can be understood that in this case, the posture estimation model is a model for processing two-dimensional images, so the predicted image input in the reasoning stage is also a two-dimensional image. Similarly, in the case where the sample image is a three-dimensional depth image, the predicted posture estimation result is the predicted three-dimensional coordinate of the key point. It can be understood that in this case, the posture estimation model is a model for processing three-dimensional depth images, so the predicted image input in the reasoning stage is also a three-dimensional depth image. The predicted posture estimation result and the second annotation information of the multiple second sample images can be substituted into the second preset loss function for loss calculation to obtain a second loss value. Subsequently, according to the second loss value, the parameters in the initial posture estimation model are optimized using back propagation and gradient descent algorithms. The optimization of parameters can be performed iteratively until the pose estimation model reaches a convergence state. When the training is completed, the obtained pose estimation model can be used for subsequent pose estimation.

Step S140, based on the initial probe coordinates of the ultrasound probe and the key point of at least one key point Coordinates, plan the motion trajectory of the ultrasound probe scanning at least one key point.

Exemplarily, according to the initial probe coordinates of the ultrasound probe and the key point coordinates corresponding to at least one key point, the distance between the ultrasound probe and each key point in the world coordinate system can be determined. For example, for key points A, B, and C, they are sorted from near to far in order of distance from the ultrasound probe as key point B, key point C, and key point A. Thus, the motion trajectory (i.e., path) of the robot arm carrying the ultrasound probe can be planned to be from the starting position of the ultrasound probe to the position of key point B to the position of key point C and then to the position of key point A.

Step S150: Control the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory, so as to scan the part to which the at least one key point belongs.

For example, based on the planning result of the motion trajectory, the ultrasonic probe can be controlled to move to the positions of the three key points B, C, and A in sequence according to the planned motion trajectory to scan the parts to which the three key points belong.

According to the ultrasonic scanning method of an embodiment of the present invention, the initial probe coordinates of the ultrasonic probe and the key point coordinates of at least one key point can be automatically determined based on the image to be tested, and the motion trajectory of the ultrasonic probe can be planned based on these coordinates and the ultrasonic probe can be controlled to scan according to the motion trajectory. Since the solution adds detection of the probe and key points, the positioning accuracy is high and the planned motion trajectory is more accurate. In addition, by controlling the ultrasonic probe to perform ultrasonic scanning through the planned motion trajectory, the standard of the ultrasonic image obtained by the scan can be improved. In addition, this solution does not require the user to perform complex operations, and can free the user's hands to a certain extent.

Exemplarily, in the process of controlling the ultrasound probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method may also include: when the ultrasound probe moves to the position of any key point, performing a corresponding scanning feedback operation based on the part to which the key point belongs, the scanning feedback operation includes one or more of the following operations: displaying a diagnostic measurement interface, the diagnostic measurement interface may include at least one first measurement item corresponding to the part to which the key point belongs; measuring at least one second measurement item corresponding to the part to which the key point belongs; measuring at least one third measurement item in response to a measurement instruction of at least one third measurement item corresponding to the part to which the key point belongs input by the user; displaying a standard human body model, and highlighting the human body area corresponding to the part to which the key point belongs on the standard human body model.

In one embodiment, when the ultrasound probe moves to the position of any key point, The corresponding feedback operation is performed based on the part to which the key point belongs. The part to which any key point belongs can be one of a plurality of preset parts. The preset parts can include but are not limited to the following parts: thyroid part, heart part, uterus part, kidney part, etc. The scanning feedback operation can include one or more of any of the following operations.

First operation: display the diagnostic measurement interface. The device for performing the ultrasound scanning method 100 may include a display device or the device for performing the ultrasound scanning method 100 may be communicatively connected to the display device. On the display device, the diagnostic measurement interface may be displayed. The diagnostic measurement interface may include at least one first measurement item corresponding to the part to which the current key point belongs. For example, when the part to which the key point belongs is the thyroid gland, the at least one first measurement item may include one or more of the following: measuring the length of the lesion based on the scanned ultrasound image; measuring the area of the lesion based on the scanned ultrasound image; measuring the length of the thyroid gland based on the scanned ultrasound image; measuring the area of the thyroid gland based on the scanned ultrasound image, etc.

For another example, when the key point belongs to the uterus, at least one first measurement item may include: identifying a standard section based on the scanned ultrasound image and measuring the standard section. The identification of the standard section and the measurement of the standard section may include: identifying the standard section of the ultrasound image to determine whether the ultrasound image belongs to at least one of the preset types of standard sections; when the ultrasound image belongs to a specific preset type of standard section, segmenting the ultrasound image to segment at least one target structure from the ultrasound image, and at least one target structure belongs to a structure related to the specific preset type of standard section; based on the segmentation result of at least one target structure, measuring at least one sub-measurement item, and at least one sub-measurement item is a sub-measurement item related to at least one target structure. The above-mentioned at least one preset type of standard section may include but is not limited to one or more of the following: head-to-hip length standard section, thalamus horizontal cross section, umbilical blood flow section, upper abdominal cross section, femoral long axis section, placental section, amniotic fluid section, cervical canal sagittal section, etc. The at least one target structure may include but is not limited to one or more of the following: amniotic fluid, placenta, thalamus, four-chamber heart, etc. At least one sub-measurement item may include, but is not limited to, one or more of the following: crown-rump length, biparietal diameter length, placental thickness, maximum depth of amniotic fluid, etc.

On the diagnostic measurement interface, at least one first measurement item may be displayed, and the user may select a specific first measurement item to be measured. After the user selects the specific first measurement item, the device for performing the ultrasound scanning method 100 may automatically perform a corresponding operation based on the user's selection. In the prior art, when a doctor scans different parts of a patient, whether it is a traditional manual measurement method or an intelligent measurement method, the user needs to manually switch to the diagnostic measurement corresponding to the part. The measurement interface has increased the workload of users to a certain extent. However, the present solution can automatically switch to the diagnosis and measurement interface of the corresponding part according to the scanning position of the ultrasound probe, which can effectively reduce the user's operation process and further improve work efficiency.

The second operation: measure at least one second measurement item corresponding to the part to which the current key point belongs. The device for executing the ultrasonic scanning method 100 can pre-store second measurement items corresponding to each part. When the ultrasonic probe moves to any key point, the second measurement item corresponding to the part to which the key point belongs can be automatically executed. For example, when the ultrasonic probe moves to the key point included in the thyroid part, the length or area of the lesion can be automatically measured based on the scanned ultrasonic image. For another example, when the ultrasonic probe moves to the key point included in the uterus part, the above-mentioned standard section can be automatically identified and the standard section can be measured. Through this scheme, the measurement of the part to which the key point belongs can be automatically started, which can further improve work efficiency.

The third operation: in response to the measurement instruction of at least one third measurement item corresponding to the part to which the current key point belongs input by the user, measure at least one third measurement item. The device for performing the ultrasonic scanning method 100 may include an input device or the device for performing the ultrasonic scanning method 100 may be communicatively connected to the input device. The input device may include but is not limited to one or more of a mouse, a keyboard, a touch screen, a microphone, etc. The user can input the measurement instruction of the third measurement item through the above-mentioned input device. In one example, the input device and the above-mentioned display device are the same touch screen, and the user can select any first measurement item displayed in the above-mentioned diagnostic measurement interface, for example, by clicking a button control related to the measurement item on the touch screen, to input the measurement instruction of the first measurement item, and the third measurement item is the first measurement item. Based on the third measurement item selected by the user, the measurement can be started. Through this scheme, the user is allowed to determine the measurement items to be measured by himself, with a high degree of autonomy, which can better meet the personalized needs of the user.

Fourth operation: display the standard human body model, and highlight the human body area corresponding to the part to which the current key point belongs on the standard human body model. Figure 2 shows a schematic diagram of a highlighted human body area according to an embodiment of the present invention. As shown in Figure 2, the highlighted part is the human body area on the human body model corresponding to the thyroid gland part of the human body. The standard human body model can be pre-stored in a storage device. The storage device can be included in a device for performing the ultrasound scanning method 100 or can be communicatively connected to the device for performing the ultrasound scanning method 100. On the standard human body model, the human body area where the key point is located can be highlighted based on the position of the current key point. Exemplarily, the human body area can be surrounded by edges where at least two key points are respectively located. An area or an area surrounded by key points where at least two key points are located and other edges, wherein at least two key points include the key point to which the ultrasound probe is currently moved. For example, in the highlighted thyroid area shown in FIG2 , there are two upper and lower edges, and these two edges may each contain a key point, wherein one key point is the key point to which the ultrasound probe is currently moved, and the other is the key point closest to the key point. In this way, a human body area can be determined by the key point currently moved to and the key point closest to it. In addition, referring to FIG2 , it can be seen that the thyroid area also includes two left and right edges, which may optionally not contain key points, and may be formed by a section of the contour of the human body itself. Through this solution, the currently scanned human body area can be highlighted, which is convenient for users to view the scanning progress, which can effectively improve the user experience.

Among the at least one first measurement item and the at least one second measurement item, at least some of the first measurement items and at least some of the second measurement items may be the same, or all of the first measurement items and all of the second measurement items may be different. The at least one second measurement item and the at least one third measurement item, as well as the at least one first measurement item and the at least one third measurement item, may be similar, and will not be described in detail.

According to the above technical solution, based on a variety of scanning feedback operations, the measurement needs of different users in different application scenarios can be met.

Exemplarily, the scanning feedback operation may also include a fifth operation: displaying the currently scanned ultrasound image, that is, an ultrasound image of the part to which the current key point belongs. This makes it convenient for the user to view the results of the ultrasound image. Exemplarily but not restrictively, the fifth operation may be performed while executing any one or more of the first operation, the second operation, the third operation, and the fourth operation. Exemplarily but not restrictively, the diagnostic measurement interface may be displayed in the first area of the display interface of the display device, and the currently scanned ultrasound image may be displayed in the second area.

Exemplarily, at least one second measurement item may include: identifying and measuring standard sections based on an ultrasound image scanned for the part to which the key point belongs; identifying and measuring standard sections based on an ultrasound image scanned for the part to which the key point belongs may include: real-time standard section identification based on the ultrasound image to determine whether the ultrasound image belongs to at least one of the preset types of standard sections; when the ultrasound image belongs to a specific preset type of standard section, performing image segmentation on the ultrasound image to segment at least one target structure from the ultrasound image, the at least one target structure belonging to a structure related to the specific preset type of standard section; and measuring at least one sub-measurement item based on the segmentation result of at least one target structure, the at least one sub-measurement item being related to at least one target structure. Sub-measurement items.

Those skilled in the art can understand the implementation of the technical solution and its beneficial effects by reading the embodiments described above, and for the sake of brevity, they will not be described in detail here.

Exemplarily, in response to a measurement instruction input by a user for at least one third measurement item corresponding to the part to which the key point belongs, measuring the at least one third measurement item includes: in response to a selection operation instruction input by a user for one or more first measurement items included on the diagnostic measurement interface, measuring the at least one third measurement item, wherein the at least one third measurement item is one or more first measurement items, and the measurement instruction is a selection operation instruction.

As described above, the user can directly click on the diagnostic measurement interface to select a first measurement item for measurement, and the first measurement item is the third measurement item. Of course, optionally, the input device can also be other devices other than the display device, such as a mouse, a keyboard, etc. The user can input a selection operation instruction for any first measurement item through the mouse and/or the keyboard to measure the selected measurement item based on the currently scanned ultrasound image (i.e., the ultrasound image containing the part to which the current key point belongs).

Exemplarily, in the process of controlling the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method may also include: obtaining resistance information detected by a force sensor, wherein the force sensor is arranged at the end where the ultrasonic probe contacts the object to be measured; when the ultrasonic probe moves to the position of any key point, adjusting the motion trajectory of the ultrasonic probe according to the corresponding resistance information.

In one embodiment, a force sensor may be further provided at the end of the ultrasonic probe that contacts the object to be measured. The force sensor may be implemented using any type of force sensor. In the process of controlling the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory, the resistance information detected by the force sensor may be obtained in real time. The resistance information may indicate the magnitude of the resistance. For example, for the motion trajectory of the ultrasonic probe planned above, when the ultrasonic probe moves to key point B, key point C, and key point A respectively, the corresponding resistance information may be obtained. Taking key point B as an example, after obtaining the resistance information corresponding to the current key point, the direction and/or speed of the ultrasonic probe movement may be adjusted according to the magnitude of the resistance information, thereby replanning the motion trajectory.

According to the above technical solution, based on the force sensor, the motion trajectory can be adjusted in time according to the acquired resistance information, and this trajectory adjustment solution has a high degree of intelligence.

For example, when the ultrasonic probe is moved to the position of any key point, the corresponding resistance Adjusting the motion trajectory of the ultrasonic probe using the force information may include: when the resistance information is greater than a preset resistance threshold, controlling the ultrasonic probe to move in a direction opposite to the current motion direction to adjust the motion trajectory of the ultrasonic probe.

In one embodiment, the user can pre-set a resistance threshold. The resistance threshold can be any value greater than 0, for example, the resistance threshold can be 3 Newtons (N). When the resistance information is greater than the preset resistance threshold 3N, the ultrasonic probe can be controlled to move in the direction opposite to the current direction of movement. For example, the current direction of movement of the ultrasonic probe is to move in the positive direction of the Z axis in the world coordinate system. After the acquired resistance information is greater than 3N, the ultrasonic probe can be controlled to stop moving forward and move in the negative direction of the X axis in the world coordinate system. In one example, the ultrasonic probe can be controlled to move in the direction opposite to the current direction of movement until the acquired resistance information is less than the preset resistance threshold.

According to the above technical solution, when the resistance is too large, the ultrasonic probe can be controlled to move in a direction opposite to the current movement direction, which can effectively prevent the ultrasonic probe from exerting excessive pressure on the object to be measured, causing discomfort to the object to be measured.

Exemplarily, one of the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point is located in the image coordinate system, and the other is located in the world coordinate system. Based on the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point, planning the motion trajectory of the ultrasound probe scanning at least one key point may include: based on the conversion relationship between the image coordinate system and the world coordinate system, performing coordinate transformation on the initial probe coordinates of the ultrasound probe and/or the key point coordinates of at least one key point, so as to unify the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point into the world coordinate system or the image coordinate system; when unified into the world coordinate system, based on the ultrasound The distance between the ultrasound probe and the at least one key point in the world coordinate system is determined based on the three-dimensional coordinates of the probe in the world coordinate system and the three-dimensional coordinates of at least one key point in the world coordinate system; when unified to the image coordinate system, the distance between the ultrasound probe and the at least one key point in the image coordinate system is determined based on the two-dimensional coordinates of the ultrasound probe in the image coordinate system and the two-dimensional coordinates of at least one key point in the image coordinate system, and the distance between the ultrasound probe and the at least one key point in the world coordinate system is determined based on the conversion relationship and the distance between the ultrasound probe and the at least one key point in the image coordinate system; the motion trajectory is planned based on the distance between the ultrasound probe and the at least one key point in the world coordinate system.

In one embodiment, one of the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point is located in the image coordinate system, and the other is located in the world coordinate system. When the image acquisition device is determined, its internal and external parameters are determined, that is, the image acquired by the image acquisition device The conversion relationship between the image coordinate system and the world coordinate system is known. The conversion relationship between the image coordinate system and the world coordinate system can be pre-stored. Exemplarily but not restrictively, the upper left corner vertex of the image to be measured can be used as the origin o of the image coordinate system, and the side passing through the origin o and parallel to the bottom side of the image to be measured can be used as the x-axis (i.e., the width direction of the image); the side passing through the origin o and perpendicular to the x-axis can be used as the y-axis (i.e., the height direction of the image) to establish an image coordinate system. Based on the position of the ultrasound probe in the image to be measured, the image coordinates of any point in the area to which the position belongs can be selected as the initial probe coordinates. For example, the image coordinates corresponding to the center point of the area to which the position belongs can be selected as the initial probe coordinates. Based on the conversion relationship between the image coordinate system and the world coordinate system, the initial probe coordinates of the ultrasound probe and/or the key point coordinates of at least one key point are converted, and the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point can be unified to the world coordinate system or the image coordinate system. For the case of unification to the world coordinate system, the distance between the ultrasound probe and the at least one key point in the world coordinate system can be determined based on the coordinates of the ultrasound probe in the world coordinate system and the coordinates of at least one key point in the world coordinate system. For the case of unification to the image coordinate system, the distance between the ultrasound probe and the at least one key point in the image coordinate system can be determined based on the two-dimensional coordinates of the ultrasound probe in the image coordinate system and the two-dimensional coordinates of at least one key point in the image coordinate system, and the distance between the ultrasound probe and the at least one key point in the world coordinate system can be determined based on the conversion relationship and the distance between the ultrasound probe and the at least one key point in the image coordinate system. According to the distance between the ultrasound probe and the at least one key point in the world coordinate system, the motion trajectory of the ultrasound probe can be planned.

In the case where both the initial probe coordinates and the key point coordinates are two-dimensional coordinates, both coordinates can be converted into three-dimensional coordinates based on the conversion relationship between the coordinate systems, and then the distance between the ultrasound probe and the key point in the world coordinate system is determined, and then the trajectory planning is performed based on the distance. In addition, the coordinate difference between the two-dimensional coordinates of the ultrasound probe and the two-dimensional coordinates of the key point can be calculated first, and then the two-dimensional coordinate difference is converted into a three-dimensional coordinate difference based on the conversion relationship between the coordinate systems, and the distance between the ultrasound probe and the key point in the world coordinate system is determined, and then the trajectory planning is performed based on the distance.

When the initial probe coordinates and the key point coordinates are both three-dimensional coordinates, the difference between the two can be directly calculated to determine the distance between the ultrasound probe and the key point in the world coordinate system, and trajectory planning can be performed based on the distance.

According to the above technical solution, based on the conversion relationship between the image coordinate system and the world coordinate system, the initial probe coordinates of the ultrasound probe and/or the key point coordinates of at least one key point are converted to plan the motion trajectory of the ultrasound probe. It can improve the efficiency of motion trajectory planning.

Exemplarily, the first part of the images to be tested may include at least one two-dimensional image, wherein target detection is performed based on the first part of the images to be tested to determine the initial probe coordinates of the ultrasound probe, which may include: inputting any two-dimensional image of the at least one two-dimensional image into a target detection model to obtain probe position information of the ultrasound probe; determining the initial probe coordinates of the ultrasound probe based on the probe position information, wherein the initial probe coordinates are two-dimensional coordinates.

In one embodiment, the first part of the image to be tested may include at least one two-dimensional image. The two-dimensional image may be an image acquired using a color camera. By inputting any two-dimensional image into the target detection model described above, the probe position information where the ultrasonic probe is located can be obtained. The probe position information can be represented by a target detection frame of any shape, or by the envelope of the ultrasonic probe. Based on the probe position information, the initial probe coordinates of the ultrasonic probe can be determined, for example, by selecting the center point coordinates of the target detection frame as the initial probe coordinates. The initial probe coordinates may be two-dimensional coordinates in the image coordinate system. The method for establishing the image coordinate system has been described in detail in the previous embodiments, and for the sake of brevity, it will not be repeated here.

According to the above technical solution, the two-dimensional image of the ultrasound probe is relatively easy to obtain, so the solution of determining the two-dimensional coordinates of the ultrasound probe based on the two-dimensional image is relatively simple to implement.

Exemplarily, the probe position information includes a target detection frame for indicating the position of the ultrasound probe, and determining the initial probe coordinates of the ultrasound probe based on the probe position information may include: determining the coordinates of the center of mass of the ultrasound probe as the initial probe coordinates based on the target detection frame.

In one embodiment, the probe position information may include a target detection frame indicating the location of the ultrasound probe. The form of the target detection frame is described above and will not be repeated here. Exemplarily, the center point, any corner point, or any other suitable point of the target detection frame may be determined as the centroid of the ultrasound probe. The coordinates corresponding to the centroid may be used as the initial probe coordinates of the ultrasound probe.

According to the above technical solution, the coordinates of the center of mass of the ultrasonic probe are determined based on the target detection frame as the initial probe coordinates. The initial probe coordinates thus determined are more accurate, thereby ensuring accurate planning of the motion trajectory.

Exemplarily, the second part of the image to be tested may include at least one three-dimensional depth image, wherein the posture estimation based on the second part of the image to be tested to determine the key point coordinates of at least one key point in the target inspection area may include: inputting any three-dimensional depth image of the at least one three-dimensional depth image into the three-dimensional posture estimation model to obtain the key point coordinates of at least one key point, The key point coordinates are three-dimensional coordinates.

The implementation scheme of the posture estimation model is described above and will not be repeated here. When processing a three-dimensional depth image, the posture estimation model can be referred to as a three-dimensional posture estimation model. In one embodiment, the second part of the image to be tested may include at least one three-dimensional depth image. The three-dimensional depth image may be an image acquired using a depth camera. By inputting any three-dimensional depth image into the posture estimation model described above, the key point coordinates of at least one key point can be obtained. The key point coordinates are three-dimensional coordinates in the world coordinate system.

According to the above technical solution, it is currently easier to obtain three-dimensional depth images of key points of the human body. The three-dimensional depth images can be directly obtained and the three-dimensional coordinates of the key points can be determined, which helps to improve planning efficiency when planning the motion trajectory of the ultrasound probe in the world coordinate system.

Exemplarily, in the process of controlling the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method may also include: acquiring mask information of the object to be measured and real-time image acquisition in real time, the real-time image acquisition is an image acquired in real time by the ultrasonic probe during the movement of the ultrasonic probe; performing target detection based on the real-time image acquisition to determine the real-time probe coordinates of the ultrasonic probe, the real-time probe coordinates are two-dimensional coordinates or three-dimensional coordinates; determining whether the ultrasonic probe falls on the object to be measured based on the real-time probe coordinates and mask information of the ultrasonic probe; if the ultrasonic probe does not fall on the object to be measured, and/or, if the ultrasonic probe falls on the object to be measured, outputting corresponding prompt information.

In the process of controlling the ultrasound probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the real-time probe coordinates of the ultrasound probe at each current moment and the mask information at each current moment can be obtained in real time.

The mask information can be used to indicate the location of the object to be measured. Exemplarily, the real-time acquisition image can be a two-dimensional image, and the mask information can be a mask image. The size of the mask image is consistent with the real-time acquisition image, and the pixel value of each pixel in the mask image is used to indicate whether the pixel at the same position on the real-time acquisition image belongs to the object to be measured. For example, the pixel value of any pixel of the mask image is a first value, indicating that the corresponding pixel on the real-time acquisition image belongs to the image to be measured, and the pixel value of any pixel of the mask image is a second value, indicating that the corresponding pixel on the real-time acquisition image does not belong to the image to be measured. One of the first value and the second value can be 1, and the other can be 0. Of course, the mask information can also be the three-dimensional coordinates of all points corresponding to the object to be measured (i.e., the three-dimensional point cloud of the object to be measured).

Optionally, in one embodiment, the depth camera may have a detection function for a human body mask, which may directly output mask information. The device for performing the ultrasonic scanning method 100 may directly obtain the mask information output by the depth camera. In another embodiment, the mask of the object to be tested may be identified from any image to be tested containing the object to be tested to obtain mask information. For example, the image to be tested may be a two-dimensional image, and identifying the mask of the object to be tested may optionally be implemented by the above-mentioned image segmentation model. In the case where the depth camera outputs mask information and the mask information is a mask image, the resolution of the mask image is consistent with the resolution of the three-dimensional depth image output by the depth camera.

In the case where the mask information and the real-time probe coordinates are located in different coordinate systems (one is the image coordinate system and the other is the world coordinate system), the two can be converted to the same coordinate system through the conversion relationship of the above coordinate systems, and then it is determined whether the ultrasonic probe falls on the object to be measured. In the case where the ultrasonic probe does not fall on the object to be measured, the first prompt information can be optionally output. In the case where the ultrasonic probe falls on the object to be measured, the second prompt information can be optionally output.

Outputting any prompt information can be achieved through an output device. The output device can be included in the device for performing the ultrasonic scanning method 100, or can be communicatively connected to the device for performing the ultrasonic scanning method 100. The output device can include, but is not limited to, one or more of a display device, a speaker, a light-emitting device, and a communication device. The communication device can be any wired and/or wireless communication device. Through the display device, the prompt information can be output in the form of video, image, text, etc. Through the speaker, the prompt information can be output in the form of audio. Through the light-emitting device, the prompt information can be output in the form of an optical signal. Through the communication device, the prompt information can be output to any associated device, such as a personal computer, a server, a mobile terminal, etc. In one example, the first prompt information can be a text message such as "the probe is moving". The second prompt information can be a text message such as "the probe is scanning".

Through the above scheme, it is possible to monitor in real time whether the ultrasonic probe falls on the object to be measured, and output corresponding prompt information, which makes it convenient for users to understand the current scanning progress in a timely manner.

Exemplarily, in the process of controlling the ultrasonic probe to move to the position of at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method may also include: acquiring mask information of the object to be measured and real-time acquisition images in real time, the mask information is a mask image, and the real-time acquisition image is an image acquired in real time by the ultrasonic probe during the movement of the ultrasonic probe; performing target detection based on the real-time acquisition image to determine the real-time probe coordinates of the ultrasonic probe, the real-time probe coordinates are two-dimensional coordinates or three-dimensional coordinates; synthesizing at least one key point on the mask image based on the key point coordinates of the at least one key point and the real-time probe coordinates of the ultrasonic probe. point and identification information of the ultrasound probe; and output a synthesized mask image.

The method for obtaining the mask information and the real-time probe coordinates can refer to the above embodiment and will not be repeated here. The object to be tested is generally stationary during the inspection process, so the key point coordinates of at least one key point can use the key point coordinates determined in the above step S130 (which can be called the initial key point coordinates). Of course, optionally, the real-time key point coordinates of at least one key point can also be determined from the real-time ultrasound image in a manner similar to step S130. According to the key point coordinates of at least one key point (which can be the initial key point coordinates or the real-time key point coordinates) and the real-time probe coordinates of the ultrasound probe, the identification information of the corresponding key points and the ultrasound probe can be synthesized on the mask image. The identification information can be in any form of identification. For example, points or rectangular boxes can be used to represent the respective corresponding positions of the key points and the ultrasound probe. After obtaining the identification information synthesized on the mask image, the synthesized mask image can be displayed on the display device.

According to the above technical solution, the synthesized mask image is obtained and output, which can facilitate users to view the positions of the ultrasound probe and key points.

According to another aspect of the present invention, an ultrasonic scanning device is also provided. FIG3 shows a schematic block diagram of an ultrasonic image processing device 300 according to an embodiment of the present invention. As shown in FIG3 , the device 300 may include: an acquisition module 310 , a detection module 320 , an estimation module 330 , a planning module 340 and a control module 350 .

The acquisition module 310 can be used to acquire at least one image to be tested, wherein the first part of the image to be tested in the at least one image to be tested includes an ultrasound probe, and the second part of the image to be tested in the at least one image to be tested includes a target inspection area of the object to be tested, the first part of the image to be tested is at least a part of the image to be tested in the at least one image to be tested, and the second part of the image to be tested is at least a part of the image to be tested in the at least one image to be tested.

The detection module 320 may be used to perform target detection based on the first part of the image to be detected to determine the initial probe coordinates of the ultrasound probe, where the initial probe coordinates are two-dimensional coordinates or three-dimensional coordinates.

The estimation module 330 may be used to perform posture estimation based on the second part of the image to be tested, so as to determine the key point coordinates of at least one key point in the target inspection area, where the key point coordinates are two-dimensional coordinates or three-dimensional coordinates.

The planning module 340 may be used to plan a motion trajectory of the ultrasound probe scanning at least one key point based on the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point.

The control module 350 can be used to control the ultrasound probe to move in sequence according to the planned motion trajectory. The method further comprises moving to the position of at least one key point to scan the part to which the at least one key point belongs.

According to another aspect of the present invention, an electronic device is also provided. FIG4 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. As shown in the figure, the electronic device 400 includes a processor 410 and a memory 420, wherein the memory 420 stores computer program instructions, and the computer program instructions are used to execute the above-mentioned ultrasound scanning method 100 when the processor 410 is running.

Exemplarily, the electronic device 400 may be an ultrasonic diagnostic device or an ultrasonic workstation.

The ultrasonic diagnostic device may include an ultrasonic probe, a first processor, a first memory, and a first display, etc. In the case where the electronic device 400 is an ultrasonic diagnostic device, the processor 410 may be the first processor, and the memory 420 may be the first memory. The ultrasonic probe may be used to transmit ultrasonic waves to a target inspection area (e.g., the uterine part of a human body), and receive ultrasonic echoes returned from the target inspection area, thereby obtaining an ultrasonic echo signal. The ultrasonic probe transmits the ultrasonic echo signal to the first processor. The first processor may process the ultrasonic echo signal to obtain an ultrasonic image of the target inspection area. The ultrasonic image obtained by the first processor may be stored in the first memory. These ultrasonic images may be optionally displayed on the first display for viewing by the user. In addition, the first processor may be used to perform the ultrasonic scanning method 100 described herein, and may optionally store some processing information generated during the processing in the first memory. The above-mentioned processing information may include intermediate data and/or final results, such as mask information described herein, detection results of a target detection model, posture estimation results of a posture estimation model, first measurement items, and the like. Optionally, the ultrasonic diagnostic equipment may further include a first input device, such as one or more of a mouse, a keyboard, and a touch screen, for a user to input information, instructions, and the like.

An ultrasound workstation can also be called an ultrasound imaging workstation. An ultrasound workstation is a device that integrates functional modules such as patient registration, image acquisition, diagnosis editing, report printing, image post-processing, medical record query, and statistical analysis. An ultrasound workstation can be communicatively connected to an ultrasound diagnostic device, for example, through any wired or wireless communication method. The ultrasound diagnostic device can transmit information such as the collected ultrasound echo signal and/or the image data of the ultrasound image to the ultrasound workstation.

The ultrasound workstation may include a second processor, a second memory, a second display, etc. In the case where the electronic device 400 is an ultrasound workstation, the processor 410 may be the second processor, and the memory 420 may be the second memory. The second processor may receive an ultrasound echo signal and/or image data of an ultrasound image from an ultrasound diagnostic device through a communication interface, and may obtain an ultrasound image based on the ultrasound echo signal or directly obtain an ultrasound image based on the image data of the ultrasound image. The ultrasonic image can be stored in the second memory. These ultrasonic images can be optionally displayed on the second display for the user to view. In addition, the second processor can be used to perform the ultrasonic scanning method 100 described herein, and some processing information generated during the processing can be optionally stored in the second memory. Examples of processing information can refer to the description above. Optionally, the ultrasonic diagnostic equipment can also include a second input device, such as one or more of a mouse, a keyboard, and a touch screen, for the user to input information, instructions, etc.

In addition, the ultrasound workstation may also include other functional modules such as a printing device, etc. Through the second processor, the second memory and various functional modules of the ultrasound workstation, functions such as processing, storing, replaying, printing, statistics, and retrieval of ultrasound images can be completed.

According to another aspect of the present invention, there is also provided an ultrasonic scanning system, comprising: a robotic arm, at the end of which is provided an ultrasonic probe; at least one image acquisition device, for acquiring at least one image to be measured; the above-mentioned electronic device 400, wherein a processor 410 in the electronic device is connected to the at least one image acquisition device and the robotic arm, for executing the above-mentioned ultrasonic scanning method 100 based on at least one image to be measured, wherein the processor 410 controls the robotic arm to drive the ultrasonic probe to move by sending a control instruction to the robotic arm.

In one example, at least one image acquisition device may be a kinect camera, which includes a color camera and a depth camera. At least one image acquisition device may be fixed directly above the object to be measured. The ultrasonic probe may be fixed at the end of the robotic arm, and the robotic arm may drive the ultrasonic probe to move. The processor 410 is connected to the image acquisition device and the robotic arm, respectively. Before scanning, the internal and external parameters of each image acquisition device may be calibrated first, the conversion relationship between the above coordinate systems may be obtained (expressed by a conversion matrix), and the conversion relationship may be stored in the memory 420. The processor 410 may receive at least one image to be measured acquired by at least one image acquisition device, determine the key point coordinates of each key point and the initial probe coordinates of the ultrasonic probe based on the image to be measured, perform trajectory planning based on these coordinates, and send corresponding control instructions to the robotic arm according to the planned motion trajectory, control the movement of the robotic arm, and then drive the ultrasonic probe to move to the location of at least one key point in sequence according to the planned motion trajectory.

Exemplarily, at least one image acquisition device may include a depth camera and/or a color camera, and at least one image to be measured may include at least one three-dimensional depth image acquired by the depth camera and/or at least one two-dimensional image acquired by the color camera.

According to another aspect of the present invention, there is also provided a storage medium on which is stored Program instructions, which are used to execute the above-mentioned ultrasound scanning method 100 when running. The storage medium may include, for example, a storage component of a tablet computer, a hard disk of a personal computer, an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

A person skilled in the art may understand the specific implementation schemes of the ultrasonic scanning device, electronic device, ultrasonic scanning system and storage medium by reading the above descriptions of the ultrasonic scanning method, which will not be described in detail here for the sake of brevity.

Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

Similarly, it should be understood that in order to simplify the present invention and help understand one or more of the various inventive aspects, in the description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of the present invention should not be interpreted as reflecting the following intention: that the claimed invention requires more than each claim. More features than those explicitly recited in the claims. More specifically, as the respective claims reflect, the inventive point is that the respective technical problems can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention.

It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in an ultrasonic scanning device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be are embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims (17)

An ultrasonic scanning method, comprising: Acquire at least one image to be tested, wherein a first portion of the image to be tested in the at least one image to be tested includes an ultrasound probe, and a second portion of the image to be tested in the at least one image to be tested includes a target inspection area of the object to be tested, the first portion of the image to be tested is at least a portion of the image to be tested in the at least one image to be tested, and the second portion of the image to be tested is at least a portion of the image to be tested in the at least one image to be tested; Performing target detection based on the first part of the image to be tested to determine the initial probe coordinates of the ultrasound probe, where the initial probe coordinates are two-dimensional coordinates or three-dimensional coordinates; Performing posture estimation based on the second part of the image to be tested to determine the key point coordinates of at least one key point in the target inspection area, wherein the key point coordinates are two-dimensional coordinates or three-dimensional coordinates; Planning a motion trajectory of the ultrasound probe scanning the at least one key point based on the initial probe coordinates of the ultrasound probe and the key point coordinates of the at least one key point; The ultrasonic probe is controlled to move to the position of the at least one key point in sequence according to the planned motion trajectory, so as to scan the part to which the at least one key point belongs. The method according to claim 1, wherein, in the process of controlling the ultrasonic probe to move to the position of the at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method further comprises: When the ultrasonic probe moves to the position of any key point, a corresponding scanning feedback operation is performed based on the part to which the key point belongs, and the scanning feedback operation includes one or more of the following operations: Displaying a diagnostic measurement interface, wherein the diagnostic measurement interface includes at least one first measurement item corresponding to the part to which the key point belongs; Measuring at least one second measurement item corresponding to the part to which the key point belongs; In response to a measurement instruction of at least one third measurement item corresponding to the part to which the key point belongs input by a user, measuring the at least one third measurement item; A standard human body model is displayed, and a human body region corresponding to the part to which the key point belongs is highlighted on the standard human body model. The method of claim 2, wherein the at least one second measurement item comprises: identifying a standard section based on an ultrasound image scanned for the part to which the key point belongs and measuring the standard section, wherein the identifying a standard section based on an ultrasound image scanned for the part to which the key point belongs and measuring the standard section comprises: Performing standard section recognition based on the ultrasound image in real time to determine whether the ultrasound image belongs to one of at least one preset type of standard section; In a case where the ultrasound image belongs to the standard section of the specific preset type, performing image segmentation on the ultrasound image to segment at least one target structure from the ultrasound image, wherein the at least one target structure belongs to a structure related to the standard section of the specific preset type; Based on the segmentation result of the at least one target structure, the at least one sub-measurement item is measured, and the at least one sub-measurement item is a sub-measurement item related to the at least one target structure. The method of claim 2, wherein, in response to a measurement instruction of at least one third measurement item corresponding to the part to which the key point belongs input by the user, measuring the at least one third measurement item comprises: In response to a selection operation instruction input by the user for one or more first measurement items included on the diagnostic measurement interface, the at least one third measurement item is measured, wherein the at least one third measurement item is the one or more first measurement items, and the measurement instruction is the selection operation instruction. The method according to claim 1, wherein, in the process of controlling the ultrasonic probe to move to the position of the at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method further comprises: Acquiring resistance information detected by a force sensor, wherein the force sensor is arranged at an end of the ultrasonic probe in contact with the object to be measured; When the ultrasonic probe moves to the position of any key point, the motion trajectory of the ultrasonic probe is adjusted according to the corresponding resistance information. The method of claim 5, wherein when the ultrasonic probe moves to the position of any key point, adjusting the motion trajectory of the ultrasonic probe according to the corresponding resistance information comprises: When the resistance information is greater than a preset resistance threshold, the ultrasonic probe is controlled to move in a direction opposite to the current moving direction to adjust the movement trajectory of the ultrasonic probe. The method according to any one of claims 1 to 6, wherein one of the initial probe coordinates of the ultrasound probe and the key point coordinates of at least one key point is located in an image coordinate system, and the other is located in a world coordinate system, and the step of planning the motion trajectory of the ultrasound probe scanning the at least one key point based on the initial probe coordinates of the ultrasound probe and the key point coordinates of the at least one key point comprises: Based on the conversion relationship between the image coordinate system and the world coordinate system, coordinate conversion is performed on the initial probe coordinates of the ultrasound probe and/or the key point coordinates of the at least one key point, so as to unify the initial probe coordinates of the ultrasound probe and the key point coordinates of the at least one key point into the world coordinate system or the image coordinate system; In a case where the coordinates are unified to the world coordinate system, determining a distance between the ultrasound probe and the at least one key point in the world coordinate system based on the three-dimensional coordinates of the ultrasound probe in the world coordinate system and the three-dimensional coordinates of the at least one key point in the world coordinate system; In the case of unification to the image coordinate system, based on the two-dimensional coordinates of the ultrasound probe in the image coordinate system and the two-dimensional coordinates of the at least one key point in the image coordinate system, determine the distance between the ultrasound probe and the at least one key point in the image coordinate system, and according to the conversion relationship and the distance between the ultrasound probe and the at least one key point in the image coordinate system, determine the distance between the ultrasound probe and the at least one key point in the world coordinate system; The motion trajectory is planned based on the distance between the ultrasound probe and the at least one key point in the world coordinate system. The method according to any one of claims 1 to 6, wherein the first portion of the image to be measured comprises at least one two-dimensional image, wherein: The performing target detection based on the first part of the image to be tested to determine the initial probe coordinates of the ultrasound probe includes: Inputting any two-dimensional image of the at least one two-dimensional image into a target detection model to obtain probe position information where the ultrasound probe is located; The initial probe coordinates of the ultrasound probe are determined based on the probe position information. The probe coordinates are two-dimensional coordinates. The method of claim 8, wherein the probe position information includes a target detection frame for indicating a position of the ultrasound probe, and determining the initial probe coordinates of the ultrasound probe based on the probe position information comprises: The coordinates of the center of mass of the ultrasound probe are determined based on the target detection frame as the initial probe coordinates. The method according to any one of claims 1 to 6, wherein the second portion of the image to be measured includes at least one three-dimensional depth image, wherein: The performing posture estimation based on the second part of the image to be tested to determine the key point coordinates of at least one key point in the target inspection area includes: Any one of the at least one three-dimensional depth image is input into a three-dimensional posture estimation model to obtain key point coordinates of the at least one key point, where the key point coordinates are three-dimensional coordinates. The method according to any one of claims 1 to 6, wherein, in the process of controlling the ultrasonic probe to move to the position of the at least one key point in sequence according to the planned motion trajectory to scan the part to which the at least one key point belongs, the method further comprises: Acquire mask information of the object to be measured and collect images in real time, wherein the images collected in real time by the ultrasound probe during the movement of the ultrasound probe; Performing target detection based on the real-time acquired image to determine the real-time probe coordinates of the ultrasound probe, wherein the real-time probe coordinates are two-dimensional coordinates or three-dimensional coordinates; Determining whether the ultrasonic probe falls on the object to be measured based on the real-time probe coordinates of the ultrasonic probe and the mask information; In the case that the ultrasonic probe does not fall on the object to be measured, and/or in the case that the ultrasonic probe falls on the object to be measured, corresponding prompt information is output. An ultrasonic scanning device, comprising: an acquisition module, configured to acquire at least one image to be tested, wherein a first portion of the image to be tested in the at least one image to be tested includes an ultrasound probe, and a second portion of the image to be tested in the at least one image to be tested includes a target inspection area of the object to be tested, the first portion of the image to be tested is at least a portion of the image to be tested in the at least one image to be tested, and the second portion of the image to be tested is the target inspection area of the object to be tested. At least a portion of at least one image to be tested; A detection module, configured to perform target detection based on the first portion of the image to be detected, so as to determine initial probe coordinates of the ultrasound probe, wherein the initial probe coordinates are two-dimensional coordinates or three-dimensional coordinates; an estimation module, configured to perform posture estimation based on the second part of the image to be tested, so as to determine key point coordinates of at least one key point in the target inspection area, wherein the key point coordinates are two-dimensional coordinates or three-dimensional coordinates; A planning module, configured to plan a motion trajectory of the ultrasound probe scanning the at least one key point based on the initial probe coordinates of the ultrasound probe and the key point coordinates of the at least one key point; The control module is used to control the ultrasonic probe to move to the position of the at least one key point in sequence according to the planned motion trajectory, so as to scan the part to which the at least one key point belongs. An electronic device comprises a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the ultrasonic scanning method according to any one of claims 1 to 11. The electronic device according to claim 13, wherein the electronic device is an ultrasonic diagnostic device or an ultrasonic workstation. An ultrasonic scanning system, comprising: A mechanical arm, wherein an ultrasonic probe is disposed at the end of the mechanical arm; At least one image acquisition device, used to acquire at least one image to be measured; The electronic device according to claim 13, wherein the processor in the electronic device is connected to the at least one image acquisition device and the robotic arm, and is used to perform the ultrasonic scanning method according to any one of claims 1 to 11 based on the at least one image to be measured, The processor controls the robotic arm to drive the ultrasonic probe to move by sending a control instruction to the robotic arm. The ultrasonic scanning system as described in claim 15, wherein the at least one image acquisition device includes a depth camera and/or a color camera, and the at least one image to be measured includes at least one three-dimensional depth image acquired by the depth camera and/or at least one two-dimensional image acquired by the color camera. A storage medium storing a computer program/instruction, wherein the computer program/instruction, when executed by a processor, implements the ultrasonic scanning method according to any one of claims 1 to 11.