[go: up one dir, main page]

WO2025190181A1 - Procédé de commande de robot - Google Patents

Procédé de commande de robot

Info

Publication number
WO2025190181A1
WO2025190181A1 PCT/CN2025/081364 CN2025081364W WO2025190181A1 WO 2025190181 A1 WO2025190181 A1 WO 2025190181A1 CN 2025081364 W CN2025081364 W CN 2025081364W WO 2025190181 A1 WO2025190181 A1 WO 2025190181A1
Authority
WO
WIPO (PCT)
Prior art keywords
control instruction
robot
control
target
server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/081364
Other languages
English (en)
Chinese (zh)
Inventor
程冉
刘力格
孙涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Midea Robozone Technology Co Ltd
Original Assignee
Midea Robozone Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Midea Robozone Technology Co Ltd filed Critical Midea Robozone Technology Co Ltd
Publication of WO2025190181A1 publication Critical patent/WO2025190181A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1628—Programme controls characterised by the control loop
    • B25J9/163—Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J11/00—Manipulators not otherwise provided for
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B25J13/087—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices for sensing other physical parameters, e.g. electrical or chemical properties
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/08—Programme-controlled manipulators characterised by modular constructions
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1602—Programme controls characterised by the control system, structure, architecture

Definitions

  • the present application relates to the field of robot control technology, and in particular to a robot control method.
  • This application aims to solve at least one of the technical problems existing in the prior art. To this end, this application proposes a robot control method that can convert complex instructions into instructions that the robot can recognize, thereby remotely controlling the robot to execute complex and flexible operator instructions, thereby improving the robot's generalization, flexibility, and functionality.
  • the present application provides a robot control method, applied to a server, comprising:
  • the second control instruction is used to control the target robot;
  • the action translation model is trained using sample user instructions as samples and sample robot instructions corresponding to the sample user instructions and recognizable by multiple categories of robots as sample labels.
  • a motion translation model is trained by using sample robot instructions and sample user instructions that can be recognized by multiple categories of robots.
  • the motion translation model obtained through training maps the first control instruction sent by the receiving operating terminal into a second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction that can be recognized by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the generalization ability of the robot in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions.
  • the translation of control instructions through the motion translation model can also improve the translation speed and significantly improve the translation efficiency, thereby enhancing the user experience.
  • inputting the first control instruction into a motion translation model and obtaining a second control instruction corresponding to the target robot output by the motion translation model includes:
  • extracting features from the video frame to obtain posture information corresponding to the first object includes:
  • Feature extraction is performed on joint points corresponding to the first object in the foreground image to obtain the posture information.
  • the method after sending the second control instruction to the target robot, the method further includes:
  • the first object feedback information is used to represent a situation in which the target robot executes the second control instruction; the first object feedback information is sent by the operating terminal;
  • the action translation model is optimized based on the first object feedback information.
  • the sending of the second control instruction to the target robot includes:
  • the encrypted information is sent to the target robot.
  • the present application provides a robot control device, which is applied to a server side and includes:
  • a first processing module is used to receive a first control instruction sent by an operating terminal
  • a second processing module configured to input the first control instruction into a motion translation model, and obtain a second control instruction corresponding to the target robot output by the motion translation model;
  • the third processing module is used to send the second control instruction to the target robot; the second control instruction is used to control the target robot; wherein,
  • the action translation model is trained using sample user instructions as samples and sample robot instructions corresponding to the sample user instructions and recognizable by multiple categories of robots as sample labels.
  • a motion translation model is trained by adopting sample robot instructions and sample user instructions that can be recognized by multiple categories of robots.
  • the motion translation model obtained through training maps the first control instruction sent by the receiving operating end into the second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction that can be recognized by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the generalization ability of the robot in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions.
  • the translation of control instructions through the motion translation model can also improve the translation speed and significantly improve the translation efficiency, thereby enhancing the user experience.
  • the present application provides a robot control method, applied to an operating end, the method comprising:
  • a first control instruction is obtained; the first control instruction is used for mapping by the server to convert it into a second control instruction that can be recognized by the target robot; the second control instruction is used to control the target robot;
  • the robot control method of the present application by receiving the first input of the first object of the operating end, in response to the first input, the first input or the collected video frame corresponding to the first object of the operating end is determined as the first control instruction, the control instruction of the first object is effectively obtained, and the robot is controlled by the first control instruction.
  • the method after sending the first control instruction to the server, the method further includes:
  • First object feedback information is obtained based on the action execution status of the target robot performing the action based on the second control instruction; the first object feedback information is used to optimize the action translation model, and the action translation model is used to map the first control instruction to the second control instruction.
  • the present application provides a robot control device, applied to an operating end, the device comprising:
  • a fourth processing module configured to receive a first input from a first object
  • a fifth processing module configured to obtain a first control instruction in response to the first input; the first control instruction is used for mapping by the server to convert it into a second control instruction recognizable by the target robot; the second control instruction is used to control the target robot;
  • a sixth processing module configured to collect a video frame corresponding to the first object, and determine the video frame as the first control instruction
  • a seventh processing module is used to send the first control instruction to the server.
  • the robot control device of the present application by receiving the first input of the first object of the operating end, in response to the first input, the first input or the collected video frame corresponding to the first object of the operating end is determined as the first control instruction, thereby effectively obtaining the control instruction of the first object, thereby controlling the robot through the first control instruction.
  • the present application provides a robot control method, applied to a robot side, the method comprising:
  • the robot side by receiving the second control instruction sent by the server side, the robot side executes the first control instruction sent by the operator side based on the recognizable second control instruction, effectively executing the target action, thereby realizing the operator side controlling the robot side, which is suitable for different application scenarios and improves the user experience.
  • the present application provides a robot control device, applied to a robot end, the device comprising:
  • an eighth processing module configured to receive a second control instruction sent by the server, where the second control instruction is generated by the server by mapping the received first control instruction
  • a ninth processing module is configured to execute a target action based on the second control instruction.
  • the robot side by receiving the second control instruction sent by the server side, the robot side executes the first control instruction sent by the operation side based on the recognizable second control instruction, effectively executes the target action, thereby realizing the operation side controlling the robot side, which is suitable for different application scenarios and improves the user experience.
  • the present application provides a robot, comprising:
  • the at least one mechanical arm is disposed on the mobile device
  • processor being electrically connected to the at least one robotic arm, the mobile device, and the plurality of sensors, respectively;
  • the robot performs actions based on a robot control method.
  • the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot control method as described in the first aspect above.
  • the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the robot control method as described in the first aspect above.
  • the motion translation model obtained through training maps the first control instruction sent by the receiving operation end into the second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction that can be recognized by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the robot's generalization ability in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions.
  • the translation of control instructions through the motion translation model can also increase the translation speed and significantly improve the translation efficiency, thereby enhancing the user experience.
  • the motion translation model can be continuously optimized through large-scale first object feedback data, thereby improving the generalization ability of the motion translation model as well as the precision and accuracy of the motion translation model, so that the second control instruction output by the motion translation model is closer to the true intention of the first object, thereby improving the user experience.
  • the encoded information is obtained, and the encoded information is encrypted based on the target encryption algorithm to obtain encrypted information, which is then sent to the target robot to ensure data security and thus protect user privacy.
  • FIG1 is a flow chart of a robot control method according to an embodiment of the present application.
  • FIG2 is a second flow chart of the robot control method provided in an embodiment of the present application.
  • FIG3 is a third flow chart of the robot control method provided in an embodiment of the present application.
  • FIG4 is a schematic diagram of a robot control method according to an embodiment of the present application.
  • FIG5 is a schematic diagram of a structure of a robot control device according to an embodiment of the present application.
  • FIG6 is a second structural diagram of the robot control device provided in an embodiment of the present application.
  • FIG7 is a third structural diagram of the robot control device provided in an embodiment of the present application.
  • FIG8 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
  • first,” “second,” and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first,” “second,” and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more.
  • the term “and/or” in the specification and claims refers to at least one of the connected objects, and the character “/" generally indicates that the objects connected are in an "or” relationship.
  • the robot control method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.
  • the terminal includes but is not limited to portable communication devices such as mobile phones or tablet computers. It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer.
  • a terminal including a display and a touch-sensitive surface is described.
  • the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.
  • the robot control method provided in the embodiment of the present application can be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the robot control method.
  • the electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras and wearable devices, etc.
  • the robot control method provided in the embodiment of the present application is explained below using an electronic device as an example of the execution subject.
  • the robot control method of the present application can be applied to the field of robot remote control.
  • a dedicated recruitment platform can be used to equip each robot with a remote operator so that the operator can remotely control the robot to perform tasks such as navigation, robotic arm operation and walking.
  • the server side is the terminal for processing data; the operation side is the terminal used by the operator, and the robot side is the terminal for performing tasks.
  • the server side is connected between the operation side and the robot side, and is used to receive, send and process data.
  • the operating end and the robot end may not be in the same area, for example, the operating end may be in position A and the robot end may be in position B, etc.
  • the robot control method is applied to the server side and includes: step 110 , step 120 and step 130 .
  • Step 110 The server receives a first control instruction sent by the operator
  • the operating terminal is used to obtain a first control instruction input by a first object.
  • the first control instruction is an instruction proposed by the first object for controlling the robot to perform an action; the first control instruction may include: a remote control instruction and an on-site control instruction.
  • the first object of the operating end is at position A, and the robot is at the home of a second object at position B.
  • the operating end sends the first control instruction to the server end, and the server end receives the first control instruction.
  • the first object is an operator or a specific user who controls the robot to perform actions.
  • the first object can send a first control instruction through the operation terminal.
  • the second object may be a general user such as the owner or user of the target robot.
  • the target robot can be placed in the scene where the second subject needs to use the robot.
  • the first control instruction may be expressed as an action instruction, a voice instruction, a character instruction, or any other instruction.
  • the first control instruction can be input through a camera, a microphone, a screen button, etc.
  • the server end receives the first control instruction.
  • the first control instruction may be obtained based on the following steps:
  • the operating end receives a first input of a first object
  • a first control instruction is obtained; the first control instruction is used for mapping by the server to convert into a second control instruction that can be recognized by the target robot; the second control instruction is used to control the target robot;
  • the first input is used to receive a control instruction input by a first object.
  • the first input may be in at least one of the following ways:
  • the first input may be a touch operation, including but not limited to a click operation, a slide operation, and a press operation.
  • receiving the first input of the first object may be receiving a touch operation of the first object in the display area of the terminal display screen.
  • the effective area of the first input can be limited to a specific area, such as the upper middle area of the control instruction interface; or when the control instruction interface is displayed, the target control is displayed on the current interface, and the first input can be achieved by touching the target control; or the first input can be set to a continuous multiple tapping operation on the display area within a target time interval.
  • the first input may be a physical key input.
  • a physical button corresponding to the control instruction is provided on the terminal body, and receiving the first input of the first object can be the operation of receiving the first object pressing the corresponding physical button; the first input can also be a combined operation of pressing multiple physical buttons at the same time.
  • the first input may be voice input.
  • the terminal when receiving a voice such as "grab”, the terminal may use "grab” as the first control instruction and send the first control instruction "grab" to the server.
  • the first input may also be in other forms, including but not limited to character input, etc., which can be determined according to actual needs and is not limited in this embodiment of the present application.
  • the first control instruction is used for mapping by the server to be converted into a second control instruction that can be recognized by the target robot.
  • the second control instruction is used to control the target robot.
  • the operating terminal when the operating terminal receives the first input of the first object at position A, the operating terminal responds to the first input and determines the first input as the first control instruction.
  • the control instruction issued by the operating end is effectively obtained by receiving the first input from the operating end and determining the first input as the first control instruction.
  • the first control instruction may also be obtained based on the following steps:
  • a video frame corresponding to the first object is collected, and the video frame is determined as a first control instruction.
  • the server receives the video frame and determines the video frame as the first control instruction.
  • the robot control method by collecting the video frame corresponding to the first object and determining the video frame as the first control instruction, the first control instruction corresponding to the video frame input by the operating end is effectively obtained.
  • Step 120 The server inputs the first control instruction into the action translation model, and obtains the second control instruction corresponding to the target robot output by the action translation model.
  • the target robot is the robot that executes the action corresponding to the first control instruction.
  • the second control instruction is an instruction that can be recognized by the target robot corresponding to the first control instruction.
  • the action translation model is trained using sample user instructions as samples and sample robot instructions that can be recognized by multiple categories of robots corresponding to the sample user instructions as sample labels.
  • the action translation model may be an artificial intelligence model.
  • the action translation model can map different types of first control instructions into corresponding second control instructions.
  • the first control instruction type includes: video frames, voice instructions, and screen input instructions.
  • the action translation model can be a neural network model or a machine learning model.
  • the action translation model may also be any other model that can map the first control instruction to the second control instruction.
  • a dedicated motion translator can be deployed on the server.
  • the motion translation model in the motion translator can map the operator's motion data into the control instructions of the robot.
  • the action translator can also automatically adapt to different robot models and forms to ensure the universality of instruction mapping.
  • the sample user instructions are multiple first control instructions sent by the operator and stored on the server.
  • the sample robot instruction is a second control instruction stored on the server side corresponding to the sample user instruction.
  • sample robot instructions that can be recognized by robots of different categories may be different.
  • sample robot instructions that can be recognized by different categories of robots may also be different.
  • the action translation model can be continuously trained and optimized through a large number of sample user instructions and sample robot instructions.
  • the first control instruction is input into the motion translation model, and the motion translation model outputs a second control instruction that can be recognized by the target robot.
  • the video frame is mapped to a second control instruction that can be recognized by the target robot.
  • the voice instruction is mapped into a second control instruction that can be recognized by the target robot.
  • the screen input instruction is mapped to a second control instruction that can be recognized by the target robot.
  • any other feasible method can also be used, for example, by pre-building a mapping relationship table between the first control instruction and the second control instruction, and translating based on the mapping relationship table to translate the first control instruction into a second control instruction that can be recognized by the target robot.
  • the first control instruction may be mapped to a second control instruction that can be recognized by the target robot through an artificial intelligence model that is different from the motion translation model.
  • Step 130 The server sends a second control instruction to the target robot.
  • the second control instruction is used to control the target robot.
  • the second control instruction may be sent to the target robot through a wireless communication module provided in the target robot.
  • the robot performs actions based on the following:
  • the robot receives the second control instruction sent by the server
  • the target action is executed.
  • the robot side can receive the second control instruction sent by the server side through the wireless communication module.
  • the target action is the action that the second control instruction requires the robot to perform.
  • the operating end sends the first control instruction to the server end.
  • the server end translates the first control instruction into a second control instruction that can be recognized by the target robot through the action translation model, and then sends the second control instruction to the target robot in the home of a second object at position B.
  • the target robot receives the second control instruction through the wireless communication module, it can collect image data of the surrounding environment through the image sensor, and process it through the image processor to obtain information about the current surrounding environment, so as to execute the target action based on the second control instruction and the information of the current surrounding environment.
  • navigation sensors can also be used to control the robot's navigation, such as enabling the robot to follow an optimal path or drive along an edge while executing a target action.
  • the robot control method by receiving a second control instruction sent by a server, the robot executes the first control instruction sent by an operator based on the recognizable second control instruction, effectively executing the target action, thereby enabling the operator to control the robot. This method is applicable to different application scenarios and improves the user experience.
  • step 130 may further include:
  • the coded information is information obtained by pairing and coding the first control instruction and the second control instruction corresponding to the first control instruction.
  • pairing coding can be determined based on actual conditions and is not limited in this application.
  • the target encryption algorithm is the algorithm used to encrypt the encoded information.
  • the target encryption algorithm can be determined based on actual conditions and is not limited in this application; for example, the target encryption algorithm can be the Advanced Encryption Standard (AES) 256 encryption algorithm or the RSA encryption algorithm.
  • AES Advanced Encryption Standard
  • the encrypted information is the corresponding information after the coded information is encrypted using the target encryption algorithm.
  • the first control instruction is paired with the second control instruction corresponding to the first control instruction to obtain the coded information, and the coded information is encrypted based on the target encryption algorithm to obtain the encrypted information, which is then sent to the target robot through the wireless transmission module.
  • an encryptor may be provided in the wireless transmission module, and the first control instruction and the second control instruction corresponding to the first control instruction may be encrypted based on the encryptor.
  • users can choose whether to share their data to further protect privacy.
  • the server when the server receives the first control instruction sent by the operating terminal at location A, it translates the first control instruction into a second control instruction, and then encrypts the second control instruction through the encryptor in the wireless transmission module, and sends the encrypted second control instruction to the robot in the home of a second object at location B to ensure data security.
  • the robot control method by pairing and encoding a first control instruction with a second control instruction corresponding to the first control instruction, obtaining coded information, encrypting the coded information based on a target encryption algorithm, and obtaining encrypted information, the encrypted information is sent to the target robot, which can improve data security, thereby protecting user privacy and ensuring that it is not accessed by unauthorized third parties.
  • the first control instruction and the second control instruction corresponding to the first control instruction may be stored in a memory on the server side, and the storage method may be encrypted storage.
  • This application sets up an action translation model to map different types of first control instructions received from the operating end into second control instructions that can be recognized by the target robot, effectively converting complex instructions into instructions that can be recognized by the robot, and sending the second control instruction to the target robot, so that the target robot executes the action corresponding to the second control instruction, thereby realizing the needs of remote control of the robot and improving the flexibility of the robot in executing actions.
  • a motion translation model is trained by using sample robot instructions and sample user instructions that can be recognized by multiple categories of robots.
  • the motion translation model obtained through training maps the first control instruction sent by the received operating terminal into a second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction that can be recognized by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the generalization ability of the robot in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions.
  • the translation speed can be improved, the translation efficiency can be significantly improved, and the user experience can be enhanced.
  • step 120 may further include:
  • the posture information is mapped to obtain a second control instruction.
  • feature extraction is extracting motion features of the first object in the video frame.
  • the posture information is the position and posture information of the first object.
  • features of the video frame can be extracted through deep learning models and machine learning models to obtain the posture information corresponding to the first object in the video frame, and the posture information can be mapped into a second control instruction that can be recognized by the target robot.
  • features of the video frames can also be extracted in any other feasible manner.
  • advanced computer vision algorithms are used to capture and analyze the operator's body movements in real time, and deep learning and other AI technologies are used to improve the accuracy and speed of motion recognition.
  • mapping is performed based on the actual situation of the target robot.
  • the posture information of the first object is mapped, and the obtained second control instruction may be an instruction that the target robot can recognize, which is for the target robot to raise its robotic arm and extend all four fingers.
  • the robot control method by performing feature extraction on the video frame corresponding to the first control instruction, the posture information of the operator in the video frame is obtained, and the posture information is mapped to the second control instruction, so that the robot can capture the operator's body movements in real time, reduce delays, improve the real-time and accuracy of the operation, enable the robot to perform the same actions as the operator, improve its functionality, and achieve the effect of real-time translation.
  • performing feature extraction on the video frame to obtain pose information corresponding to the first object includes:
  • Feature extraction is performed on the joint points corresponding to the first object in the foreground image to obtain pose information.
  • the foreground image is a body movement image of the first subject.
  • the joint points corresponding to the first object are positions of the joint points corresponding to the arm, palm, and fingers of the first object.
  • deep learning models and machine learning models can be used to identify joint points and obstacles corresponding to the first object in the video frame, obtain a foreground image in the video frame, segment the foreground image from the background image in the video frame, obtain the joint points corresponding to the first object in the foreground image, perform feature extraction on the joint points corresponding to the first object, determine the action of the first object based on the information corresponding to multiple joint points, and obtain the posture information corresponding to each joint point.
  • the operator at position A raises his hand
  • the camera at the operating end captures the operator's hand-raising action
  • the operating end obtains a video frame corresponding to the hand-raising action
  • the video frame includes: the operator and the environmental background in which the operator is located.
  • the foreground image is the operator;
  • the background image is the environment in which the operator is located, which may be a classroom or living room.
  • the specific joint points of the operator in the foreground image can be identified and extracted, and the corresponding posture information of the operator can be determined based on the specific joint points.
  • this method can identify other important elements in the environment, such as obstacles and target objects, to enhance the robot's environmental perception ability.
  • the robot control method by extracting the foreground image in the video frame, the foreground image is segmented from the background image in the video frame, and the interference of the background image is reduced, the joint points corresponding to the first object in the foreground image are accurately obtained, and the features of the joint points corresponding to the first object are extracted to obtain accurate posture information, and instruction mapping is performed based on the posture information to improve the accuracy of the second control instruction finally mapped, thereby further improving the precision and accuracy of the robot control; in addition, it can also reduce the amount of data involved in the calculation, increase the speed of action translation, and improve the efficiency of controlling the robot.
  • the method may further include:
  • the action translation model is optimized based on the first object feedback information.
  • the first object feedback information is an evaluation of the first object on the execution result of the second control instruction executed by the target robot.
  • the first object feedback information is used to represent a situation in which the target robot executes the second control instruction.
  • the first object feedback information is sent by the operating terminal.
  • the first object feedback information may be obtained in the following manner:
  • first object feedback information is obtained.
  • the second input is used to receive feedback information of the first object.
  • the second input may be the same as the first input, such as touch input, physical key input, voice input, character input, or any other feasible input, which will not be described in detail in this application.
  • the first object feedback information includes: incentive information and loss information.
  • the action translation model may be optimized based on the specific content of the feedback information of the first object received by the server.
  • the action translation model maps the new first control instruction to a second control instruction corresponding to the original first control instruction when receiving a new first control instruction that is identical to the original first control instruction.
  • the action translation model when the action translation model receives a new first control instruction that is the same as the original first control instruction, the second control instruction corresponding to the new first control instruction will be different from the second control instruction corresponding to the original first control instruction.
  • the action translation model can be optimized by setting the activation function and loss function in the large language model.
  • the motion translation model can also be optimized in any other feasible way, such as optimizing motion control and the motion translation model for remote control mapping translation through proximal policy optimization (PPO).
  • PPO proximal policy optimization
  • a large number of robots are used to operate in the home of the second object, while collecting control data, original perception data and feedback information of the first object, so as to optimize the AI model in turn based on the feedback information, and train a more generalized AI model so that it can perform well in a variety of environments and tasks, thereby improving the generalization ability of the motion translation model as well as the precision and accuracy of the motion translation model.
  • the action translator can be continuously optimized through machine learning technology to improve the accuracy and efficiency of its mapping.
  • the AI model is optimized in turn by using the feedback data of the first object obtained after the robot is controlled to perform the relevant task.
  • the motion translation model can be continuously optimized through large-scale first object feedback data, thereby improving the generalization ability of the motion translation model as well as the precision and accuracy of the motion translation model, so that the second control instruction output by the motion translation model is closer to the true intention of the first object, improving the accuracy and efficiency of the mapping, and realizing precise control of the robot, thereby improving the user experience.
  • the server includes an image processor and a memory
  • optimizing the motion translation model based on the first object feedback information includes:
  • the image processor is used to train the reinforcement learning model to optimize the action translation model.
  • the image processor is a processor used to train a reinforcement learning model.
  • the data stored in the memory includes: a pre-trained model, first object feedback information, a first control instruction, and a second control instruction.
  • a data storage unit can be set up and a large-capacity storage device can be equipped on the server to save the robot's operation history and original perception data.
  • a processing unit can also be set up, and an efficient data processing unit can be set up inside the server to analyze and process the data in real time.
  • pre-trained models may include: neural network models, machine learning models and other models.
  • Image processors can be used to train reinforcement learning models.
  • Reinforcement learning models can be constructed through horizontal federated reinforcement learning algorithms in large language models and hierarchical reinforcement learning strategies.
  • HFRL high-performance reinforcement learning
  • training equipment can be configured on the server.
  • a set of dedicated graphics processing units (GPUs) can be installed in the server for reinforcement learning model training.
  • multiple pre-trained neural network models can be stored on the server to form a model library, which can be continuously optimized based on the collected data.
  • the reinforcement learning model is trained through the large amount of data stored on the server side, the generalization degree of the reinforcement learning model is improved, and different application scenarios and different tasks are effectively handled.
  • the reinforcement learning model is effectively trained and the accuracy of the reinforcement learning model is improved, so that the motion translation model can not only translate the operator's actions in real time, but also has a learning function. Over time, it can automatically identify and adapt to the operator's operating habits, thereby further improving the efficiency and accuracy of the operation, thereby optimizing the motion translation model and improving the generalization ability of the model.
  • the robot control method provided in the embodiment of the present application can be executed by a robot control device.
  • the robot control device provided in the embodiment of the present application is described by taking the robot control method executed by the robot control device as an example.
  • An embodiment of the present application also provides a robot control device, which is applied to a server side.
  • the robot control device includes a first processing module 510 , a second processing module 520 and a third processing module 530 .
  • a first processing module 510 is configured to receive a first control instruction sent by an operator
  • the second processing module 520 is configured to input the first control instruction into the motion translation model and obtain a second control instruction corresponding to the target robot output by the motion translation model;
  • the third processing module 530 is used to send a second control instruction to the target robot; the second control instruction is used to control the target robot; wherein,
  • the action translation model is trained using sample user instructions as samples and sample robot instructions that can be recognized by multiple categories of robots corresponding to the sample user instructions as sample labels.
  • a motion translation model is trained by using sample robot instructions and sample user instructions that can be recognized by multiple categories of robots.
  • the trained motion translation model maps the first control instruction sent by the received operating terminal into a second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction that can be recognized by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the generalization ability of the robot in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions.
  • the translation speed can be improved, the translation efficiency can be significantly improved, and the user experience can be enhanced.
  • the second processing module 520 may also be configured to:
  • the posture information is mapped to obtain a second control instruction.
  • the apparatus may further include a tenth processing module configured to:
  • Feature extraction is performed on the joint points corresponding to the first object in the foreground image to obtain pose information.
  • the apparatus may further include an eleventh processing module configured to:
  • the first object feedback information is used to represent a situation in which the target robot executes the second control instruction; the first object feedback information is sent by the operating terminal;
  • the action translation model is optimized based on the feedback information of the first object.
  • the apparatus may further include a twelfth processing module configured to:
  • the robot control method is applied to the operating end, including: step 210 , step 220 , step 230 and step 240 .
  • the first control instruction acquired in step 210 and step 220 and the first control instruction determined in step 230 are two parallel execution modes.
  • Step 210 The operating terminal receives a first input from a first object
  • the first object is an operator or a specific user who controls the robot to perform actions.
  • the first input is used to receive an instruction input by a first object.
  • the first input may be in at least one of the following ways:
  • the first input may be a touch operation, including but not limited to a click operation, a slide operation, and a press operation.
  • receiving the first input of the first object may be receiving a touch operation of the first object in the display area of the terminal display screen.
  • the effective area of the first input can be limited to a specific area, such as the upper middle area of the control instruction interface; or when the control instruction interface is displayed, the target control is displayed on the current interface, and the first input can be achieved by touching the target control; or the first input can be set to a continuous multiple tapping operation on the display area within a target time interval.
  • the first input may be a physical key input.
  • a physical button corresponding to the control instruction is provided on the terminal body, and receiving the first input of the first object can be the operation of receiving the first object pressing the corresponding physical button; the first input can also be a combined operation of pressing multiple physical buttons at the same time.
  • the first input may be voice input.
  • the terminal when receiving a voice such as "grab”, the terminal may use "grab” as the first control instruction and send the first control instruction "grab" to the server.
  • the first input may also be in other forms, including but not limited to character input, etc., which can be determined according to actual needs and is not limited in this embodiment of the present application.
  • operator control is by verbal command, this may be accomplished through a miniature microphone or other sound capturing device.
  • the operator can also operate directly via the touch screen, for example, by swiping or clicking to give instructions.
  • a user console can be set up, and the operator can remotely operate the robot through a dedicated user console.
  • This console can be a computer or mobile device equipped with a camera and microphone.
  • console coupled with tactile and auditory feedback, provides operators with a more immersive and intuitive operating experience; this not only enhances the operator's operating confidence, but also greatly improves user satisfaction.
  • Step 220 The operating terminal obtains a first control instruction in response to the first input
  • the first control instruction is used for mapping by the server to be converted into a second control instruction that can be recognized by the target robot.
  • the second control instruction is used to control the target robot.
  • the operating terminal when the operating terminal receives the first input of the first object, the operating terminal responds to the first input and determines the first input as the first control instruction.
  • Step 230 The operating terminal collects a video frame corresponding to the first object and determines the video frame as a first control instruction
  • the video frame corresponding to the first object is an action video frame of the first object captured by a camera or a depth sensor.
  • the first control instruction is an instruction corresponding to the information in the video frame.
  • a video frame corresponding to the action of the first object may be collected through a camera or a depth sensor, and the video frame may be determined as the first control instruction.
  • Step 240 The operation end sends a first control instruction to the server end.
  • the operating end sends the acquired first control instruction to the server end through a wireless communication method.
  • a modular general control framework can be designed, which can be easily integrated with different robot models and task types.
  • the framework supports multiple input sources, such as visual recognition, user voice commands, etc., to ensure flexibility and adaptability.
  • the robot control method by providing a variety of different input methods for the first object of the operating end to input the first control instruction, it has high flexibility, so that the first object can select the best input method based on actual conditions, thereby broadening the application scenarios, and can ensure the efficient transmission of control instructions, further improving the user experience.
  • the method may further include:
  • the operating end obtains the first object feedback information according to the action execution status of the target robot performing the action based on the second control instruction.
  • the action execution status refers to the degree of matching between the result of the target robot executing the action and the first control instruction.
  • the first object feedback information may be motivational information.
  • the first object feedback information may be loss information.
  • Feedback information is used to optimize the action translation model.
  • the action translation model may be used to map the first control instruction into the second control instruction.
  • a feedback system can be set up on a dedicated console, through which the operator can provide comments and suggestions on the robot operation.
  • the first subject's evaluation of the target robot i.e., first subject feedback information
  • the target robot's performance of the action based on the second control instruction is obtained based on the target robot's performance of the action based on the second control instruction.
  • This first subject feedback information can then be used to optimize the robot's performance and improve control accuracy.
  • the robot control method provided in the embodiment of the present application can be executed by a robot control device.
  • the robot control device provided in the embodiment of the present application is described by taking the robot control method executed by the robot control device as an example.
  • An embodiment of the present application also provides a robot control device, which is applied to an operating end.
  • the robot control device includes: a fourth processing module 610 , a fifth processing module 620 , a sixth processing module 630 and a seventh processing module 640 .
  • a fourth processing module 610 is configured to receive a first input from a first object
  • a fifth processing module 620 is configured to obtain a first control instruction in response to the first input; the first control instruction is used for mapping by the server to convert it into a second control instruction recognizable by the target robot; the second control instruction is used to control the target robot;
  • a sixth processing module 630 is configured to collect a video frame corresponding to the first object and determine the video frame as a first control instruction
  • the seventh processing module 640 is configured to send a first control instruction to the server.
  • the robot control device by receiving the first input of the first object of the operating end, in response to the first input, the first input or the collected video frame corresponding to the first object of the operating end is determined as the first control instruction, thereby effectively obtaining the control instruction of the first object, thereby controlling the robot through the first control instruction.
  • the apparatus may further include a thirteenth processing module configured to:
  • the first object feedback information is obtained according to the action execution status of the target robot performing the action based on the second control instruction.
  • the robot control method is applied to the robot side, including: step 310 and step 320 .
  • the robot side includes different categories of robots.
  • the robot is equipped with multiple joints, a robotic arm, and an end effector.
  • the robotic arm imitates the structure of the human arm.
  • the end effector is a gripping tool similar to the palm and fingers of a hand.
  • the bottom of the robot is provided with multiple motion wheels.
  • the robot is also equipped with navigation sensors.
  • the top and front of the robot are equipped with high-resolution image sensors and image processors.
  • the image sensor can be a camera; high-resolution cameras are configured on the top and front of the robot to capture the environment and the operator's movements in real time.
  • the image processor is used to analyze and process the data collected by the image sensor.
  • a powerful graphics processor can be integrated to quickly analyze the data captured by the camera and perform real-time action recognition through a pre-trained deep learning model.
  • the robot can be equipped with a wireless communication module.
  • the wireless communication module can be used to communicate with the server and the operation end.
  • the low power consumption design of the wireless module ensures that the robot can continue to work for a long time without frequent charging due to excessive power consumption. This continuous working ability further enhances the robot's working efficiency and use value.
  • a high-speed wireless communication module can be configured inside the robot to ensure real-time communication between the robot and the remote server and operator.
  • An encryptor may be provided in the wireless communication module.
  • the cipher can be used to encrypt the transmitted data.
  • Step 310 The robot receives the second control instruction sent by the server.
  • the second control instruction is generated by the server side by mapping the received first control instruction.
  • the second control instruction may be an encrypted control instruction.
  • the robot side can receive the second control instruction sent by the server side through the wireless communication module.
  • Step 320 The robot performs the target action based on the second control instruction.
  • the target action is the action that the second control instruction requires the robot to perform.
  • the operating end sends the first control instruction to the server end.
  • the server end translates the first control instruction into a second control instruction that can be recognized by the target robot through the action translation model, and then sends the second control instruction to the target robot in the home of a second object at position B.
  • the target robot receives the second control instruction through the wireless communication module, it can collect image data of the surrounding environment through the image sensor, and process it through the image processor to obtain information about the current surrounding environment, so as to execute the target action based on the second control instruction.
  • the camera captures the video frame of the operating end and obtains the joint point information of the first object action in the video frame.
  • the server side performs detection and fusion based on the information in the video frame and repositions based on the joint information to generate instructions that can be recognized by the robot side, so that the simulator or robot on the robot side performs the corresponding action.
  • the robots can be mass-produced and placed in the home of the second subject, and the first subject at the operating end can control the robots through the control strategy.
  • the server continuously collects and processes the robot's operating data and environmental information, the robot's autonomous control algorithm is continuously optimized.
  • the robot side by receiving the second control instruction sent by the server side, the robot side executes the first control instruction sent by the operator side based on the recognizable second control instruction, effectively executing the target action, thereby realizing the operator side controlling the robot side, which is suitable for different application scenarios and improves the user experience.
  • the robot control method provided in the embodiment of the present application can be executed by a robot control device.
  • the robot control device provided in the embodiment of the present application is described by taking the robot control method executed by the robot control device as an example.
  • An embodiment of the present application also provides a robot control device, which is applied to a robot side.
  • the robot control device includes an eighth processing module 710 and a ninth processing module 720 .
  • an eighth processing module configured to receive a second control instruction sent by the server, where the second control instruction is generated by the server by mapping the received first control instruction
  • a ninth processing module is configured to execute a target action based on the second control instruction.
  • the robot side by receiving the second control instruction sent by the server side, the robot side executes the first control instruction sent by the operator side based on the recognizable second control instruction, effectively executing the target action, thereby realizing the operator side controlling the robot side, which is suitable for different application scenarios and improves the user experience.
  • the robot control device in the embodiment of the present application can be a robot, or an electronic device that can communicate with the robot, or a component in the robot or electronic device, such as an integrated circuit or chip.
  • the electronic device can be a terminal, or it can be other devices other than a terminal.
  • the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (MID), an augmented reality (AR)/virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc.
  • It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.
  • the robot control device in the embodiment of the present application may be a device having an operating system.
  • the operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
  • the robot control device provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 1 to 4. To avoid repetition, they will not be described here.
  • An embodiment of the present application also provides a robot, comprising: at least one robotic arm, a moving device, a processor, and multiple sensors.
  • At least one robotic arm includes a plurality of joints.
  • the robotic arm is similar in structure to the human arm.
  • the robotic arm is provided with an end effector, which may be a structure similar to a palm and fingers.
  • the terminal effector can be rotated flexibly.
  • this structure when combined with an advanced visual recognition system to capture the operator's body movements, this structure ensures that the translation of movements from humans to machines is more accurate and natural. This natural movement response not only enhances the smoothness of the operation, but also reduces the possibility of misoperation.
  • the robotic arm is arranged on the mobile device.
  • the moving means may comprise at least one roller.
  • the moving device can be arranged at the bottom of the robot.
  • the robot can move freely in complex environments through the mobile device.
  • Sensors are used to collect data required by the target robot.
  • the plurality of sensors includes an image sensor and a navigation sensor.
  • the image sensor is used to collect environmental image information of the target robot's location.
  • Navigation sensors are used to locate the target robot and assist in path planning.
  • the processor is electrically connected to the robot arm, the sensor and the moving device respectively.
  • the processor is used to receive and process the data collected by each sensor and control the robotic arm or mobile device to execute relevant instructions.
  • the target robot can integrate environmental image information and positioning information, and perform task strategy planning based on the second control instruction to better perform the target action.
  • the processor may include an image processor to quickly analyze data captured by the camera and perform real-time action recognition through a pre-trained deep learning model.
  • the processor may further include a robotic control device as described in any of the above embodiments.
  • the robot is used to execute the robot control method described in any of the above embodiments.
  • the robot side by receiving the second control instruction sent by the server, the robot side executes the first control instruction sent by the operation side based on the recognizable second control instruction, effectively executes the target action, thereby realizing remote control, being suitable for different application scenarios, and improving the user experience.
  • the robot further comprises: an encryptor.
  • the encryptor is used to encrypt the transmitted data.
  • the specific implementation method has been described in the above embodiment and will not be repeated here.
  • the security of data transmission can be effectively improved and the privacy of the user can be protected.
  • an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801.
  • the program is executed by the processor 801
  • the various processes of the above-mentioned robot control method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
  • the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
  • An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored.
  • a computer program is stored.
  • the various processes of the above-mentioned robot control method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
  • the processor is the processor in the electronic device described in the above embodiment.
  • the readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
  • An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned robot control method when executed by a processor.
  • the processor is the processor in the electronic device described in the above embodiment.
  • the readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
  • An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned robot control method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
  • the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
  • the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
  • a storage medium such as ROM/RAM, magnetic disk, optical disk

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • Automation & Control Theory (AREA)
  • Manipulator (AREA)

Abstract

L'invention concerne un procédé de commande de robot, consistant à : recevoir une première instruction de commande envoyée par une extrémité d'actionnement (110) ; entrer la première instruction de commande dans un modèle de traduction d'action, et obtenir une seconde instruction de commande correspondant à un robot cible délivré par le modèle de traduction d'action (120) ; et envoyer la seconde instruction de commande au robot cible, la seconde instruction de commande étant utilisée pour commander le robot cible, le modèle de traduction d'action étant obtenu par apprentissage en utilisant des instructions d'utilisateur d'échantillon en tant qu'échantillons et instructions de robot d'échantillon qui peuvent être reconnues par de multiples catégories de robots correspondant aux instructions d'utilisateur d'échantillon en tant qu'étiquettes d'échantillon (130).
PCT/CN2025/081364 2024-03-13 2025-03-07 Procédé de commande de robot Pending WO2025190181A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410286911.9 2024-03-13
CN202410286911.9A CN118927232A (zh) 2024-03-13 2024-03-13 机器人控制方法、机器人控制装置和机器人

Publications (1)

Publication Number Publication Date
WO2025190181A1 true WO2025190181A1 (fr) 2025-09-18

Family

ID=93362513

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/081364 Pending WO2025190181A1 (fr) 2024-03-13 2025-03-07 Procédé de commande de robot

Country Status (2)

Country Link
CN (1) CN118927232A (fr)
WO (1) WO2025190181A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118927232A (zh) * 2024-03-13 2024-11-12 美智纵横科技有限责任公司 机器人控制方法、机器人控制装置和机器人

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108803874A (zh) * 2018-05-30 2018-11-13 广东省智能制造研究所 一种基于机器视觉的人机行为交互方法
CN108986801A (zh) * 2017-06-02 2018-12-11 腾讯科技（深圳）有限公司 一种人机交互方法、装置及人机交互终端
CN109551476A (zh) * 2017-09-26 2019-04-02 丰田研究所股份有限公司 结合云服务系统的机器人系统
CN110039545A (zh) * 2019-04-30 2019-07-23 齐鲁工业大学 一种基于可穿戴设备的机器人远程控制系统及控制方法
US20200019237A1 (en) * 2018-07-12 2020-01-16 Toyota Research Institute, Inc. Methods and systems for determining teleoperating user intent via eye tracking
CN114721509A (zh) * 2022-03-08 2022-07-08 五邑大学 基于人体动作识别的人机交互方法及系统
CN116098536A (zh) * 2021-11-08 2023-05-12 青岛海尔科技有限公司 一种机器人控制方法及装置
CN116901110A (zh) * 2023-06-15 2023-10-20 达闼机器人股份有限公司 移动机器人控制方法、装置、电子设备及存储介质
CN118927231A (zh) * 2024-03-13 2024-11-12 美智纵横科技有限责任公司 机器人控制方法、装置和机器人
CN118927232A (zh) * 2024-03-13 2024-11-12 美智纵横科技有限责任公司 机器人控制方法、机器人控制装置和机器人

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108986801A (zh) * 2017-06-02 2018-12-11 腾讯科技（深圳）有限公司 一种人机交互方法、装置及人机交互终端
CN109551476A (zh) * 2017-09-26 2019-04-02 丰田研究所股份有限公司 结合云服务系统的机器人系统
CN108803874A (zh) * 2018-05-30 2018-11-13 广东省智能制造研究所 一种基于机器视觉的人机行为交互方法
US20200019237A1 (en) * 2018-07-12 2020-01-16 Toyota Research Institute, Inc. Methods and systems for determining teleoperating user intent via eye tracking
CN110039545A (zh) * 2019-04-30 2019-07-23 齐鲁工业大学 一种基于可穿戴设备的机器人远程控制系统及控制方法
CN116098536A (zh) * 2021-11-08 2023-05-12 青岛海尔科技有限公司 一种机器人控制方法及装置
CN114721509A (zh) * 2022-03-08 2022-07-08 五邑大学 基于人体动作识别的人机交互方法及系统
CN116901110A (zh) * 2023-06-15 2023-10-20 达闼机器人股份有限公司 移动机器人控制方法、装置、电子设备及存储介质
CN118927231A (zh) * 2024-03-13 2024-11-12 美智纵横科技有限责任公司 机器人控制方法、装置和机器人
CN118927232A (zh) * 2024-03-13 2024-11-12 美智纵横科技有限责任公司 机器人控制方法、机器人控制装置和机器人

Also Published As

Publication number Publication date
CN118927232A (zh) 2024-11-12

Similar Documents

Publication Publication Date Title
Mahmud et al. Interface for human machine interaction for assistant devices: A review
Delmerico et al. Spatial computing and intuitive interaction: Bringing mixed reality and robotics together
CN102789313B (zh) 一种用户交互系统和方法
US10179407B2 (en) Dynamic multi-sensor and multi-robot interface system
CN110568929B (zh) 一种基于虚拟键盘的虚拟场景交互方法、装置和电子设备
CN113829357B (zh) 机器人手臂遥操作方法、装置、系统及介质
CN102789312A (zh) 一种用户交互系统和方法
CN106468917B (zh) 一种可触摸现场实时视频图像的远程呈现交互方法和系统
CN102830798A (zh) 单臂机器人基于Kinect的无标记人手跟踪方法
CN106272409A (zh) 基于手势识别的机械臂控制方法及系统
Meng et al. Virtual reality based robot teleoperation via human-scene interaction
CN107030692A (zh) 一种基于感知增强的机械手遥操作方法及系统
WO2025190181A1 (fr) Procédé de commande de robot
Chen et al. A human–robot interface for mobile manipulator
CN118832580A (zh) 基于视觉语言模型的机器人控制方法、装置及机器人
CN118466805A (zh) 机器视觉和手势识别的非接触式三维模型人机交互方法
Kurian et al. Visual Gesture-Based Home Automation
Christian et al. Hand gesture recognition and infrared information system
Tsuji et al. A Survey on Imitation Learning for Contact-Rich Tasks in Robotics
Singh et al. Digitized Interaction: A Gesture-Controlled Whiteboard System with OpenCV, MediaPipe and NumPy
CN119347801A (zh) 智能体灵巧手的动作模仿优化方法、装置、设备、介质及产品
Dhamanskar et al. Human computer interaction using hand gestures and voice
TK et al. Real-Time Virtual Mouse using Hand Gestures for Unconventional Environment
CN118802992A (zh) 一种机器人远程控制方法及系统
CN116442218A (zh) 一种遥操作跟踪方法、装置、设备、系统及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25769352

Country of ref document: EP

Kind code of ref document: A1