CN120371145A

CN120371145A - Positioning method of remote controller and remote controller

Info

Publication number: CN120371145A
Application number: CN202510552597.9A
Authority: CN
Inventors: 史灿灿; 黄志明
Original assignee: Hisense Electronic Technology Shenzhen Co ltd
Current assignee: Hisense Electronic Technology Shenzhen Co ltd
Priority date: 2025-04-28
Filing date: 2025-04-28
Publication date: 2025-07-25

Abstract

The present application discloses a remote control positioning method and remote control, the method comprising: if IMU data is detected at the first moment, the first state information of the remote control at the first moment is adjusted by applying the IMU data and the first adjustment strategy to obtain the second state information at the second moment; if UWB data is detected at the second moment, the second state information is adjusted by applying the UWB data and the second adjustment strategy to obtain the target state information at the second moment; the target state information includes the target position, target attitude, target speed and target bias information at the second moment; the first adjustment strategy and the second adjustment strategy are determined according to a predetermined extended Kalman filter; the focus of the remote control on the display device is calculated by applying the target attitude and the actual position of the remote control at the second moment, where the actual position is determined according to the UWB data detected at the second moment. The embodiment of the present application improves the positioning accuracy of the remote control.

Description

Positioning method of remote controller and remote controller

Technical Field

The present application relates to the field of positioning technologies, and in particular, to a positioning method of a remote controller and a remote controller.

Background

In the field of remote control positioning, the position of the remote control can be calculated through UWB data or the position of the remote control can be calculated through IMU data, and then the position of a cursor of display equipment is calculated according to the position of the remote control, so that the positioning of the remote control is realized.

However, according to the scheme of positioning by UWB data, a plurality of UWB base stations are required to be distributed in a positioning area, the cost is high, and the positioning accuracy is easily influenced by the UWB signal propagation range and metal. According to the scheme of positioning by IMU data, positioning errors can be accumulated along with time, positioning accuracy can be obviously reduced after long-time use, and the measurement accuracy is easily affected by external vibration, impact and other interference. And, the absolute position of the remote control cannot be determined by only relying on IMU data.

Disclosure of Invention

The application provides a remote controller positioning method and a remote controller, which are used for improving the positioning accuracy of the remote controller under the condition of ensuring low cost.

According to a first aspect in an exemplary embodiment, there is provided a positioning method of a remote controller, applied to the remote controller, the method including:

if the IMU data is detected at the first moment, the first state information of the remote controller at the first moment is adjusted by applying the IMU data and a first adjustment strategy to obtain second state information at the second moment, wherein the second state information comprises a reference position, a reference gesture and a reference speed at the second moment and bias information at the first moment;

if UWB data is detected at the second moment, the UWB data and a second adjustment strategy are applied to adjust the second state information to obtain target state information at the second moment, wherein the target state information comprises target position, target gesture, target speed and target bias information at the second moment;

And calculating the focus of the remote controller on the display device by applying the target gesture and the actual position of the remote controller at the second moment, wherein the actual position of the remote controller at the second moment is determined according to UWB data detected at the second moment.

According to the embodiment of the application, the IMU data and the UWB data are fused by utilizing the extended Kalman filter, when the IMU data are detected, the state pushing operation is executed, and when the UWB data are detected, the state observing operation is executed. And calculating the focus of the remote controller on the display device by using the target gesture in the state information obtained by the state observation and the actual position of the remote controller. By using the mode to position the remote controller, the defects of only using IMU data for positioning and only using UWB data for positioning are overcome, and the positioning accuracy of the remote controller is improved.

In an alternative embodiment, if no UWB data is detected at the second time, the method further comprises:

And adjusting the second state information by applying a third adjustment strategy to obtain target state information at the second moment, wherein the actual position of the remote controller at the second moment is determined according to the position calculated by the remote controller according to the UWB data at the first moment, the IMU data at the first moment and the IMU data at the second moment.

In the above embodiment, if no UWB data is detected at the second time, the adjustment policy applied at this time is distinguished from the adjustment policy at the time of detection of UWB data, and in this case, the actual position of the remote controller applied at this time is calculated from the position calculated from UWB data at the first time and IMU data of each of the two times due to lack of UWB data. By the design, the positioning accuracy can be further improved according to whether UWB data is received or not.

In an alternative embodiment, the first adjustment strategy is an adjustment strategy indicated by a state advance equation in an extended kalman filter;

The method for adjusting the first state information of the remote controller at the first moment by applying the IMU data and the first adjustment strategy to obtain the second state information at the second moment comprises the following steps:

calculating a state transition matrix and a process noise driving matrix of a state propulsion equation according to first state information of the remote controller at a first moment;

And applying the angular velocity and the acceleration in the IMU data, and executing state propulsion operation on the position, the gesture and the velocity in the first state information based on the state transition matrix and the process noise driving matrix to obtain second state information.

In the above embodiment, during the state propulsion process, the state transition matrix and the process noise driving matrix may be calculated first, and then, based on the state transition matrix and the process noise driving matrix, the state propulsion operation may be performed on the position, the posture and the speed in the first state information according to the angular velocity and the acceleration in the applied IMU data, so as to obtain the second state information. The state advancing process fully utilizes IMU data and improves positioning accuracy.

In an alternative embodiment, the state propulsion equation includes:

wherein t is a first moment, t+τ is a second moment, b is a coordinate system where the remote controller is located, and w is a world coordinate system;

For the rotation of the remote control at the second instant, For the rotation of the remote controller at the first moment, the angular speed of the omega ^b remote controller, b _ω is angular speed bias, and n _ω is noise corresponding to the angular speed;

for the speed of the remote control at the second instant, For the speed of the remote control at the first moment,In order to set a rotation matrix, a ^b is the acceleration of the remote controller, n _a is the noise corresponding to the acceleration, and g ^w is the gravity direction;

for the reference position of the remote control at the second moment, The reference position of the remote controller at the first moment;

b _a,t+τ is the acceleration bias at the second time, b _a,t is the acceleration bias at the first time, Biasing corresponding noise for acceleration;

b _ω,t+τ is the angular velocity offset at the second time, b _ω,t is the angular velocity offset at the first time, The corresponding noise is offset for the angular velocity.

According to the embodiment, the state propulsion operation is executed by applying the state propulsion equation, so that the accuracy of state propulsion is improved, the precision of extended Kalman filtering is further improved, and the positioning precision is further improved.

In an alternative embodiment, the second adjustment strategy is an adjustment strategy indicated by a state observation equation in extended kalman filtering;

the UWB data and the second adjustment strategy are applied to adjust the second state information to obtain target state information at the second moment, and the method comprises the following steps:

calculating an observation matrix and a process noise covariance matrix of a state observation equation according to the second state information;

and performing state observation operation on the second state information based on the observation matrix and the process noise covariance matrix by applying the distance, the pitch angle and the horizontal angle in the UWB data to obtain target state information.

In the above embodiment, in the state observation process, the observation matrix and the process noise covariance matrix are calculated first, and then the distance, the pitch angle and the horizontal angle in the UWB data are applied, and based on the observation matrix and the process noise covariance matrix, the state observation operation is performed on the second state information, so as to obtain the target state information. The state advancing process fully utilizes UWB data, further improves the precision of extended Kalman filtering, and improves the positioning precision.

In an alternative embodiment, the state observation equation includes:

Wherein, the Is a transformation matrix of a world coordinate system to an UWB coordinate system of an external device,For the reference position of the remote controller at the first moment, d is the distance between the UWB unit of the external device and the UWB unit of the remote controller, n _d is the noise corresponding to the distance, theta ^R is the first horizontal angle of the remote controller relative to the external device,For the noise corresponding to the first horizontal angle,For a first elevation angle of the remote control relative to the external device,For noise corresponding to the first elevation angle,The direction of the remote controller relative to the external equipment is shown, wherein the external equipment is display equipment or Dongle equipment communicated with the display equipment;

for the transformation matrix of the remote control IMU coordinate system to the display device coordinate system, For the transformation matrix from the IMU coordinate system of the remote controller to the world coordinate system, theta ^I is the second horizontal angle of the external equipment relative to the remote controller,The noise corresponding to the second horizontal angle is,For a second elevation angle of the external device relative to the remote control,For noise corresponding to the second elevation angle,Indicating the orientation of the external device relative to the remote control.

According to the embodiment, the state observation operation is performed by applying the state observation equation, so that the accuracy of state observation is improved, the precision of extended Kalman filtering is further improved, and the positioning precision is further improved.

In an alternative embodiment, the third adjustment strategy is an adjustment strategy indicated by an updated state observation equation in the extended kalman filter;

The updated state observation equation is a state observation equation falsified based on noise characteristics.

In the above embodiment, when UWB data is not detected, in order to ensure smooth progress of the state observation process, the state observation equation may be falsified according to noise characteristics. By the design, different conditions are considered, the precision of the extended Kalman filtering is further improved, and the positioning precision is further improved.

In an alternative embodiment, calculating the focus of the remote control on the display device using the target pose and the actual position of the remote control at the second time, includes:

Calculating the coordinates of the remote controller on the display equipment according to the target gesture and the actual position of the remote controller at the second moment;

and determining the pixel coordinates of the remote controller on the display device according to the coordinates of the remote controller on the display device, wherein the position indicated by the pixel coordinates on the display device is a focus.

In the above embodiment, after the coordinates of the remote controller on the display device are obtained, the pixel coordinates of the remote controller on the display device may be calculated, so that the cursor of the remote controller on the display device may be more accurately determined according to the pixel coordinates.

In an alternative embodiment, the method further comprises:

And smoothing the pixel coordinates by applying a first-order filter, wherein the filter coefficient in the first-order filter is determined according to the rotation speed in the speed of the remote controller.

In the above embodiment, after the smoothing filtering process is performed, the jitter condition can be effectively removed.

According to a second aspect in an exemplary embodiment, there is provided a remote control applied to a remote control, the remote control comprising a data transmission unit and a processor;

A data transmission unit configured to perform:

Acquiring IMU data and UWB data;

a processor configured to perform:

According to a third aspect in an exemplary embodiment, there is provided a positioning device of a remote controller, applied to the remote controller, the device comprising:

the state updating unit is used for adjusting the first state information of the remote controller at the first moment by applying the IMU data and the first adjustment strategy if the IMU data is detected at the first moment to obtain second state information at the second moment, wherein the second state information comprises a reference position, a reference gesture and a reference speed at the second moment and bias information at the first moment;

The state updating unit is further used for adjusting the second state information by applying UWB data and a second adjustment strategy if UWB data is detected at the second moment to obtain target state information at the second moment, wherein the target state information comprises target position, target gesture, target speed and target bias information at the second moment;

And the positioning unit is used for calculating the focus of the remote controller on the display device by applying the target gesture and the actual position of the remote controller at the second moment, wherein the actual position of the remote controller at the second moment is determined according to UWB data detected at the second moment.

According to a fourth aspect in an exemplary embodiment, a computer storage medium is provided, in which computer program instructions are stored which, when run on a computer, cause the computer to perform the positioning method of the remote control as in the first aspect.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it will be apparent that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1a schematically illustrates UWB data provided by an embodiment of the application;

Fig. 1b illustrates an application scenario diagram for positioning a remote controller according to an embodiment of the present application;

fig. 1c is a schematic diagram schematically illustrating a positioning principle of a remote controller according to an embodiment of the present application;

FIG. 1d illustrates a schematic diagram of a fused positioning scheme for UWB and IMU pointing to remote controls provided by embodiments of the present application;

Fig. 2 is a flowchart illustrating a positioning method of a remote controller according to an embodiment of the present application;

FIG. 3 is a flow chart illustrating a method for determining second status information according to an embodiment of the present application;

FIG. 4 is a flowchart illustrating a method for determining target state information according to an embodiment of the present application;

FIG. 5 schematically illustrates a method for calculating pixel coordinates according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a positioning device of a remote controller according to an embodiment of the present application;

Fig. 7 is a schematic structural diagram of a remote controller according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

For convenience of understanding, names and symbols involved in the embodiments of the present application are explained as follows:

(1) An Ultra Wide (UWB) directional remote controller (hereinafter referred to as remote controller) is a remote controller which realizes accurate pointing and control functions by using UWB technology.

The positioning principle of UWB is as follows:

UWB technology transmits data by transmitting ultra wideband pulse signals. In the remote controller, the UWB chip is capable of measuring a distance between the remote controller and the display device with accuracy of millimeter level using an angle (AoA) technique, and determining a pointing direction of the remote controller through angle measurement. For example, some types of UWB chips support UWB channels 5 and 9, the frequency ranges from 6250MHz to 8250MHz, by which the UWB function can achieve accurate positioning of the remote controller, support 3 antenna specifications, enable measurement of 2D AoA and 3DAoA, and provide powerful hardware support for the remote controller, so that a user can achieve mouse-like operation experience on a display device by pointing to the remote controller.

Therefore, the UWB pointing remote controller can realize a high-precision pointing positioning function, brings brand-new control experience to users, easily realizes clicking, selecting, sliding, dragging and other operations by using the remote controller, and greatly expands the use scene of equipment.

(2) An inertial measurement unit (Inertial Measurement Unit, IMU), which is a device for measuring the state of motion of an object, is composed of an accelerometer and a gyroscope. The accelerometer is used to measure acceleration of an object in three axes. The working principle of the accelerometer is based on Newton's second law, and the acceleration value is calculated by detecting the force generated by the mass block under the action of acceleration. The gyroscope mainly measures the angular velocity of an object around three coordinate axes. The gyroscope utilizes the principle of conservation of angular momentum, when an object rotates, a rotor in the gyroscope generates corresponding angular momentum change, and angular velocity information is obtained by detecting the change.

The IMU works on the principle that acceleration and angular velocity information of an object are measured in real time through an accelerometer and a gyroscope. The acceleration data measured by the accelerometer can be integrated to obtain speed and displacement information, but the error can be larger when the accelerometer is used alone for measuring displacement for a long time because the error can be accumulated in the integration process. The angular velocity information measured by the gyroscope can be integrated to obtain the attitude angle of the object, but the problem of error accumulation exists. Therefore, a fusion algorithm is generally adopted to fuse the data of the accelerometer and the gyroscope, so as to improve the measurement accuracy and reliability. For example, the kalman filtering algorithm is a common fusion algorithm, and can perform optimal estimation on the data of the accelerometer and the gyroscope according to the measurement error of the sensor and the dynamic model of the system, so as to obtain more accurate object motion state information.

(3) Dongle device, a small external device, is also known as a Dongle, or adaptor, among others. In some smart televisions, game machines and other devices, dongle can be used for expanding functions of the devices, such as adding bluetooth functions, wi-Fi functions and the like, so that performance and use experience of the devices are improved. In the embodiment of the application, the UWB chip can be arranged in the Dongle device or the UWB chip can be arranged in the display device and used for communicating with the UWB chip in the remote controller so as to position the remote controller.

(4) In the embodiment of the application, the reference position is the position of the remote controller obtained during finger state propulsion, the target position is the position of the remote controller obtained during state observation, and the actual position is the actual position of the remote controller.

(5) Scalar is lower case, vector is lower case, matrix is upper case.

Q represents a quaternion rotation;

v represents a velocity vector;

p represents a position vector;

b represents a bias vector;

n represents noise or a directional vector;

r represents a 3x3 rotation matrix;

p represents a covariance matrix;

Subscript s represents a display device coordinate system, an origin is at the upper left corner of the display device, the horizontal right is the x-axis, and the vertical downward is the y-axis;

the subscript w represents a world coordinate system, an origin is a UWB unit of Dongle, and a coordinate axis is parallel to s;

Subscript b represents an IMU coordinate system of the remote controller;

Subscript R represents a Dongle UWB coordinate system;

subscript I represents the remote control UWB coordinate system;

the subscript T represents the time, the subscript tau represents the time interval, and the superscript T represents the matrix transposition;

exp, log represent exponential and logarithmic functions;

representing a quaternion multiplication;

The lambda superscript indicates deflection operation, and a 3x1 vector is converted into a 3x3 antisymmetric matrix;

the v superscript indicates the inverse of the skew operation;

The FromTwoVector (v ₁,v₂) function returns one rotation, coinciding the vector v ₁ rotation with another vector v ₂;

g ^w＝[0,-9.81,0]^T denotes the direction of gravity;

rotation of UWB R-terminal coordinate system to display device coordinate system And translation

Rotation from IMU coordinate system to UWB I end coordinate system of remote controller

The current intelligent television interface is complex, the traditional remote controller is controlled by keys, the operation is complex, the learning cost is high, and the intelligent television interface is not friendly to users, especially old users. Even an intelligent remote control added with voice cannot completely replace keys.

In the projector field, a pointing remote controller appears, the remote controller emits infrared light to irradiate on a curtain to form light spots, the projector camera recognizes the light spots and calculates the positions of the light spots on the curtain through an algorithm, and therefore UI interaction is achieved, and the operation is direct, simple and clear. Therefore, the problem can be well solved by adding the pointing function on the television remote controller, but the scheme can not be directly transplanted to the television remote controller, so that the UWB ultra-wideband positioning scheme is generated.

The UWB ultra-wideband positioning is composed of two base stations, each base station is composed of 1 to 3 antennas, coded radio pulses with high frequency and short time are sent between the base stations, the distance and azimuth angle of the opposite base station can be measured, 1 antenna can only measure the distance, 2 antennas can measure the distance and 1 azimuth angle, and 3 days can measure the distance and 2 azimuth angles. A base station is arranged on the television, a remote controller is provided with the base station, and the pointing direction of the remote controller can be calculated by knowing the positions of the base station and the remote controller. However, UWB positioning has fatal defects that firstly, the measurement frequency is difficult to improve, a certain company can achieve 50Hz, the frequency is still lower for smooth operation, secondly, the power consumption is large, the power consumption is in direct proportion to the measurement frequency, the required frequency is not more than 10Hz in order to meet the endurance time of a remote controller, and thirdly, the frequency is easy to be interfered by external environment. There is no mature commercially available UWB position pointing remote control solution for a long time. In the embodiment of the application, a positioning scheme of 3 antennas (UWB units in remote controllers) +3 antennas (UWB units in display equipment) can be adopted to measure the frequency of 10Hz, and an IMU sensor is fused, so that a commercially available pointing remote controller scheme is realized, and better precision and range are provided.

FIG. 1a is a schematic diagram of UWB data according to an embodiment of the present application, wherein d is the distance between two UWB units, θ is the horizontal angle,Is a height angle.

After the design idea of the embodiment of the present application is introduced, some simple descriptions are made below for application scenarios applicable to the technical solution of the embodiment of the present application, and it should be noted that the application scenarios described below are only used for illustrating the embodiment of the present application and are not limiting. In the specific implementation, the technical scheme provided by the embodiment of the application can be flexibly applied according to actual needs.

Referring to fig. 1b, an application scenario diagram of remote controller positioning is shown, where the application scenario diagram includes a dongle device, and in an actual application process, the application scenario diagram may also be implemented by directly applying a display device without applying the dongle device. This is by way of illustration only and is not intended to be limiting.

Referring to fig. 1c, there is shown a schematic diagram of a positioning principle of a remote controller, in which a dongle device includes a UWB unit (referred to as an R-terminal), a main control unit, and bluetooth, a remote controller includes a UWB unit (referred to as an I-terminal), a main control unit, bluetooth, an IMU, an infrared transmitter, and a key, and a display device (television) includes an infrared receiver.

The method comprises the steps that a UWB R end and a UWB I end mutually measure the azimuth of the other side to obtain a distance, 4 azimuth angles (two groups of horizontal angles and pitch angles) and confidence degrees, the distance, the 4 azimuth angles and the confidence degrees are sent to a main control unit of a remote controller, an IMU sensor measures acceleration and angular velocity of the remote controller and sends the acceleration and the angular velocity to the main control unit of the remote controller, a fusion algorithm calculates the pointing direction of the remote controller and a focus of display equipment according to the measured data, the pointing direction and the focus of the display equipment are sent to dongle equipment through Bluetooth, and then the focus is sent to an operating system of a television through USB, and finally a cursor is displayed.

In order to further explain the technical solution provided by the embodiments of the present application, the following details are described with reference to the accompanying drawings and the detailed description. Although embodiments of the present application provide the method operational steps shown in the following embodiments or figures, more or fewer operational steps may be included in the method based on routine or non-inventive labor. In steps where there is logically no necessary causal relationship, the execution order of the steps is not limited to the execution order provided by the embodiments of the present application.

Before the technical scheme of the application is introduced, the principle of the fusion positioning of UWB and IMU of the pointing remote controller is described.

Fig. 1D is a schematic diagram of a fusion positioning scheme of a UWB and an IMU of a pointing remote controller according to an embodiment of the present application, referring to fig. 1D, it can be known that, an Extended kalman filter (Extended KALMAN FILTER, EKF) is applied to IMU data and UWB data to perform fusion filtering, 3DoF pose fusion is performed on the result obtained by jointly filtering pose data in the result with acceleration and angular velocity in IMU data, 6DoF pose fusion is performed on the fused pose information and an anchor position determined according to UWB data, so as to obtain 2D plane coordinates, and then the 2D plane coordinates are converted into pixel coordinates on a display device.

Next, the process of EKF fusion filtering and the positioning process will be described with reference to fig. 1 d.

Referring to the flowchart of a method for positioning a remote controller shown in fig. 2, the method is applied to the remote controller in conjunction with the application scenario shown in fig. 1a-1 d. The technical scheme provided by the embodiment of the application is described with reference to fig. 2.

And S201, if the IMU data is detected at the first moment, the first state information of the remote controller at the first moment is adjusted by applying the IMU data and a first adjustment strategy to obtain second state information at the second moment.

The second state information comprises a reference position, a reference gesture and a reference speed at a second moment and bias information at a first moment, wherein the bias information comprises accelerometer bias information and gyroscope bias information.

And S202, if the UWB data is detected at the second moment, the UWB data and a second adjustment strategy are applied to adjust the second state information, and the target state information at the second moment is obtained.

The target state information comprises target position, target gesture, target speed and target bias information at a second moment, and the first adjustment strategy and the second adjustment strategy are determined according to a predetermined extended Kalman filter.

And S203, calculating the focus of the remote controller on the display device by applying the target gesture and the actual position of the remote controller at the second moment.

Wherein the actual position of the remote control at the second instant is determined from the UWB data detected at the second instant.

First, three elements of the extended kalman filter, namely, a state variable, a state advancing model and a state observing model are explained. In the embodiment of the application, the state variables are:

the 5 vectors therein represent the rotation (angular change), speed, position and accelerometer bias and gyroscope bias, respectively, of the remote control in the world coordinate system. Alternatively, the size of the state variable is 4+3+3+3+3=16.

Optionally, the extended kalman filtering process is a process of estimating and optimizing a state variable at the next moment according to the state information at the current moment.

Next, a description is given of an initialization process of the kalman filter:

the initial values of the state s and covariance P need to be provided to the EKF as follows

For rotationIs performed in the initialization process:

Rotating Two constraints should be satisfied, wherein in the first constraint condition, assuming that the remote controller is stationary during initialization, the acceleration direction of the IMU is the gravity direction under the world coordinate system, so that a period t ₀～t₁ (for example, 0.5 seconds) can be integrated, and the speed direction is the gravity direction, and the formula is written as follows:

Wherein, the first constraint conditions are:

From the constraints, it can be determined that:

In the second constraint, the UWB R and I ends measure the direction of each other The overlapping and opposite directions can be deduced that the plane projection is also overlapped, and the opposite directions can be obtained:

Position of Directly using UWB R-side measurements, namely:

the remaining states are set to 0:

Covariance (covariance) The value set to a fixed diagonal matrix may be empirically preset.

The meaning of each variable may be determined by referring to the definition of each symbol, which is not described herein.

In summary, in the initialization, IMU integration is performed, UWB data is recorded, if valid UWB data exists in 0.5s, rotation and position are initialized, a 6DoF state is entered, if valid UWB data does not exist after 0.5s, rotation is initialized, the position is fixed, and 3DoF is entered. Under the 6DoF state, there is continuous UWB angle constraint or static, if the accumulated static time 30s has no UWB angle constraint, the device enters the 3DoF state, the device has no UWB constraint in the 3DoF state, the direction is not credible, and if there is UWB angle constraint, the device enters the 6DoF state, and the device re-initializes the position.

Referring to S201, the IMU unit (which may also be referred to as an IMU sensor) has a sampling frequency of 200Hz and the uwb unit has a sampling frequency of 10Hz. Therefore, UWB data cannot be directly detected at some time, and at this time, the estimated UWB data may be applied to perform calculation according to no UWB processing or according to another data interpolation scheme. The description is not limited herein.

In the embodiment of the application, the first time is denoted by t, and if the IMU data is detected at the t time, the IMU data and the first adjustment strategy can be applied to adjust the state information at the first time to obtain the second state information.

The first adjustment policy is an adjustment policy indicated by a state advance equation in the extended kalman filter, and the process of applying IMU data and the first adjustment policy to adjust the first state information of the remote controller at the first moment to obtain the second state information may be implemented through steps S201-1 to S201-2 in fig. 3:

S201-1, calculating a state transition matrix and a process noise driving matrix of a state propulsion equation according to first state information of the remote controller at a first moment.

And S201-2, applying the angular speed and the acceleration in the IMU data, and executing state propulsion operation on the position, the gesture and the speed in the first state information based on the state transition matrix and the process noise driving matrix to obtain second state information.

The state advance equation and the state advance process are described below with a specific example:

alternatively, the state propulsion equation is as follows:

wherein t is a first moment, t+τ is a second moment, b is a coordinate system where the remote controller is located, and w is a world coordinate system; For the rotation of the remote control at the second instant, For the rotation of the remote controller at the first moment, the angular speed of the omega ^b remote controller, b _ω is angular speed bias, and n _ω is noise corresponding to the angular speed; for the speed of the remote control at the second instant, For the speed of the remote control at the first moment,In order to set a rotation matrix, a ^b is the acceleration of the remote controller, n _a is the noise corresponding to the acceleration, and g ^w is the gravity direction; for the reference position of the remote control at the second moment, B _a,t+τ is the acceleration bias of the second moment, b _a,t is the acceleration bias of the first moment,Noise corresponding to acceleration bias, b _ω,t+τ angular velocity bias at the second time, b _ω,t angular velocity bias at the first time,The corresponding noise is offset for the angular velocity.

The 5 equations in the above state propulsion equation are collectively expressed as:

s_t+τ＝f(s_t,n_imu);

It needs to be converted into an error state advance equation, namely:

δs_t+τ＝Fδs_t+Gn_imu;

Because of rotation of The degree of freedom is 3, and the error state size is 3, so the size of the error state δs _t is 3+3+3+3+3=15.

In the case where the state and the error state are different in size, the manifold addition and subtraction is defined as follows:

Manifold addition:

Manifold subtraction:

Wherein, manifold addition and subtraction satisfies the relation:

Thus, the state transition matrix F and the process noise driving matrix G can be calculated as follows:

Wherein F is a matrix of 15x15 size and G is a matrix of 15x12 size.

In this way, the state propulsion equation may be applied to perform a state propulsion process.

In a specific example, t represents a first time, t+τ represents a second time, and IMU data is received at the first time to perform state promotion, and the following operations are performed

P_t+τ＝FP_tF^T+GP_imuG^T

Where P _imu is the IMU covariance, which is a known fixed value.

Referring to S202, if UWB data is detected at the second time, UWB data and a second adjustment policy are applied to adjust the second state information.

The second adjustment strategy is an adjustment strategy indicated by a state observation equation in extended Kalman filtering. Thus, the process of adjusting the second state information by applying UWB data and the second adjustment policy to obtain the second state information at the second time may be implemented through steps S202-1 to S202-2 in fig. 4:

And S202-1, calculating an observation matrix and a process noise covariance matrix of the state observation equation according to the second state information.

S202-2, applying the distance, pitch angle and horizontal angle in UWB data, and performing state observation operation on the second state information based on the observation matrix and the process noise covariance matrix to obtain target state information.

The following describes the state observation equation and the observation process with a specific example:

According to whether UWB data can be detected at the second moment, the state observation equation comprises the following two types:

in the first case, the UWB data is detected at the second time.

This case, i.e., the 6DoF case, the UWB data includes distance d, horizontal angle θ, and altitude angleThe direction of the object under test (remote control) is indicated as follows:

The remote control observes at the location of Dongle:

the observation of the position of the Dongle on the remote controller is as follows:

Unified writing is in residual form, namely:

The residual is nonlinear, and the estimated value is linearized to obtain:

thus, the observation matrix H and the process noise covariance matrix Q can be calculated as follows:

in this way, the state observation process can be performed using the state observation equation.

In the second case, no UWB data is detected at the second time.

In this case, if no UWB data is detected at the second time, a third adjustment policy is applied to adjust the second state information, thereby obtaining target state information at the second time. The third policy here refers to a policy represented by a falsified state observer.

This is the case of 3DoF, where no UWB data is detected, and no observation is made, so that a reasonable observation (state observation equation forged from noise characteristics) can be forged, and the remote controller is located not too far from the original location, i.e.:

written as residual form, namely:

residual is linear, and derivative can be obtained:

when UWB data is received at the time t+tau, the state is updated, the following operation is carried out, and the gain K and the updated state are obtained And updated covariance P _t+τ:

K=P_t+τH^T(HP_t+τH^T+QP_uwb/pQ^T)^-1

P_t+τ←(I-KH)P_t+τ

referring to S203, for the result of the extended kalman filtering, the process of calculating the focus of the remote controller on the display device by applying the obtained target pose (i.e. the first dimension information of the state variable) in the state variable and the actual position of the remote controller at the second moment may be implemented by steps A1-A2:

A1, calculating the coordinates of the remote controller on the display device according to the target gesture and the actual position of the remote controller at the second moment.

In one specific example, the calculated coordinates of the remote control on the display device are (x _p,y_p,z_p).

A2, determining the pixel coordinates of the remote controller on the display device according to the coordinates of the remote controller on the display device.

From the geometrical relationship shown in fig. 5, it can be determined that the pixel coordinates u and v of the focus of the remote control on the display device are:

Where w _p,h_p is the width and height of the pixel resolution of the display device, w, h is the physical size of the display device, and x _a,y_a,z_a is the position of the remote control in the display device coordinate system.

After obtaining the pixel coordinates, a first order filter may be further applied to perform smoothing filtering processing on the pixel coordinates. Wherein the filter coefficients in the first order filter are determined based on a rotational speed of the remote control.

Optionally, the filtering process using the first order filter is as follows:

The filter coefficient alpha epsilon (0, 1) can be changed according to the change of the motion speed, and the smaller the value is, the more obvious the filter effect is. One of the policies can be written as:

Alpha is adjusted according to the rotation speed of the remote controller, omega ^b =0, alpha=0 when the remote controller is stationary, and parameters beta and gamma are preset according to experience.

In order to make the technical solution of the present application more complete, the following describes a complete process of positioning by giving a complete example, which can be implemented by steps B1-B7:

b1, turning on the directional switch/wake-up.

The side of the remote controller is provided with a dial switch, a user can control whether the pointing function is to be used or not, after the user picks up the remote controller, the user can wake up the main control, and when the pointing function is started, the pointing function is initialized.

B2, initializing UWB and IMU.

And waking up the UWB chip, starting the IMU chip, and sending the measurement data to the main control through the SPI interface.

B3 fusion algorithm initialization

And initializing the remote controller to rotate and translate to a preset value to perform 3DoF initialization if the trusted UWB data does not exist in 0.5 seconds.

B4, EKF filter.

The EKF is divided into two stages of state pushing and updating, wherein the state pushing is carried out once when one piece of IMU data is received, the pose and covariance of the remote controller are updated, the measurement error can be accumulated, and the updating is carried out once when one piece of UWB data is received, so that the pose error of the remote controller is reduced, and the operation is repeatedly carried out.

B5, smoothing filtering.

And calculating cursor coordinates according to the position of the remote controller, and applying a smoothing filter to reduce jitter.

And B6, bluetooth transmits cursor coordinates.

The cursor coordinates are sent to the Dongle through the private Bluetooth protocol at the frequency of 200Hz, and finally, the cursor coordinates are sent to the television UI system and the cursor is displayed;

b7 stopping algorithm and UWB and IMU

When a user toggles the pointing switch to close the pointing function or the remote controller is dormant after being stationary for 15 seconds, the pointing algorithm is stopped, and UWB and IMU chips are closed to reduce power consumption

In summary, the embodiment of the application adopts a UWB 3 antenna+3 antenna positioning scheme, wherein the 3 antenna has better positioning precision and measuring range than the 2 antenna, and an EKF filter algorithm is used to fuse an IMU sensor, so that the output of 200Hz can be provided, and the UWB measuring frequency only needs 10Hz, thereby meeting the endurance requirement of a remote controller. The fusion algorithm is realized by an EKF (Kalman) filter, operates on the master control of the remote controller, fuses UWB and IMU data, estimates the pose (translation+rotation) of the remote controller, calculates the cursor position, and transmits the cursor position to the display equipment through Bluetooth after smooth filtering.

As shown in fig. 6, based on the same inventive concept, an embodiment of the present application provides a positioning device of a remote controller, which is applied to the remote controller, the device including a status updating unit 61 and a positioning unit 62.

A state updating unit 61, configured to, if IMU data is detected at a first time, apply IMU data and a first adjustment policy to adjust first state information of the remote controller at the first time to obtain second state information at a second time, where the second state information includes a reference position, a reference posture, a reference speed, and bias information at the first time;

the state updating unit 61 is further configured to, if UWB data is detected at the second time, apply UWB data and a second adjustment policy to adjust the second state information to obtain target state information at the second time, where the target state information includes a target position, a target posture, a target speed, and target bias information at the second time;

And a positioning unit 62 for calculating the focus of the remote controller on the display device by applying the target gesture and the actual position of the remote controller at the second time, wherein the actual position of the remote controller at the second time is determined according to the UWB data detected at the second time.

In an alternative embodiment, the status updating unit 61 is further configured to:

And if the UWB data is not detected at the second moment, the second state information is adjusted by applying a third adjustment strategy to obtain target state information at the second moment, wherein the actual position of the remote controller at the second moment is determined according to the position of the remote controller calculated according to the UWB data at the first moment, the IMU data at the first moment and the IMU data at the second moment.

the state updating unit 61 is specifically configured to:

In an alternative embodiment, the state propulsion equation includes:

the state updating unit 61 is specifically configured to:

In an alternative embodiment, the state observation equation includes:

In an alternative embodiment, the positioning unit 62 is specifically configured to:

In an alternative embodiment, the method further comprises a processing unit, wherein the processing unit is used for:

Since the device is the device in the method according to the embodiment of the present application, and the principle of the device for solving the problem is similar to that of the method, the implementation of the device may refer to the implementation of the method, and the repetition is omitted.

As shown in fig. 7, based on the same inventive concept, an embodiment of the present application provides a remote controller applied to a remote controller including a data transmission unit 71 and a processor 72.

A data transmission unit 71 configured to perform:

Acquiring IMU data and UWB data;

a processor 72 configured to perform:

Calculating a focus of the remote control on the display device using the target pose and an actual position of the remote control at a second time, wherein the actual position of the remote control at the second time is determined from UWB data detected at the second time, and in an alternative embodiment, the processor 72 is further configured to:

The processor 72 is specifically configured to:

In an alternative embodiment, the state propulsion equation includes:

The processor 72 is specifically configured to:

In an alternative embodiment, the state observation equation includes:

In an alternative embodiment, processor 72 is specifically configured to:

In an alternative embodiment, the apparatus further comprises a processing unit, and the processor 72 is configured to:

The embodiment of the application also provides a computer storage medium, wherein the computer storage medium stores computer program instructions which, when run on a computer, cause the computer to execute the steps of the positioning method of the remote controller.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A method for positioning a remote control, the method comprising:

If IMU data is detected at a first moment, the IMU data and a first adjustment strategy are applied to adjust first state information of the remote controller at the first moment to obtain second state information at a second moment, wherein the second state information comprises a reference position, a reference gesture and a reference speed at the second moment and bias information at the first moment, and the bias information comprises accelerometer bias information and gyroscope bias information;

If UWB data is detected at a second moment, the second state information is adjusted by applying the UWB data and a second adjustment strategy to obtain target state information at the second moment, wherein the target state information comprises target position, target gesture, target speed and target bias information at the second moment;

2. The method of claim 1, wherein if UWB data is not detected at the second time, the method further comprises:

And adjusting the second state information by applying a third adjustment strategy to obtain target state information of the second moment, wherein the actual position of the remote controller at the second moment is determined according to the position of the remote controller calculated according to UWB data of the first moment, the IMU data of the first moment and the IMU data of the second moment.

3. The method of claim 1, wherein the first adjustment strategy is an adjustment strategy indicated by a state advance equation in the extended kalman filter;

The step of applying the IMU data and the first adjustment policy to adjust the first state information of the remote controller at the first moment to obtain second state information at the second moment includes:

calculating a state transition matrix and a process noise driving matrix of the state propulsion equation according to the first state information of the remote controller at the first moment;

And applying the angular velocity and the acceleration in the IMU data, and executing state propulsion operation on the position, the gesture and the speed in the first state information based on the state transition matrix and the process noise driving matrix to obtain the second state information.

4. A method according to claim 3, wherein the state propulsion equation comprises:

For rotation of the remote control at the second moment, For the rotation of the remote controller at the first moment, omega ^b is the angular velocity of the remote controller, b _ω is the angular velocity bias, and n _ω is the noise corresponding to the angular velocity;

For the speed of the remote control at the second moment, For the speed of the remote control at the first moment,In order to set a rotation matrix, a ^b is the acceleration of the remote controller, n _a is the noise corresponding to the acceleration, and g ^w is the gravity direction;

for a reference position of the remote control at the second moment, A reference position of the remote controller at the first moment;

b _a,t+τ is the acceleration bias at the second time, b _a,t is the acceleration bias at the first time, Biasing the corresponding noise for the acceleration;

b _ω,t+τ is the angular velocity offset at the second time, b _ω,t is the angular velocity offset at the first time, And biasing the corresponding noise for the angular velocity.

5. The method of claim 1, wherein the second adjustment strategy is an adjustment strategy indicated by a state observation equation in the extended kalman filter;

the step of adjusting the second state information by applying the UWB data and a second adjustment policy to obtain target state information at the second time, including:

calculating an observation matrix and a process noise covariance matrix of the state observation equation according to the second state information;

And applying the distance, pitch angle and horizontal angle in the UWB data, and performing state observation operation on the second state information based on the observation matrix and the process noise covariance matrix to obtain the target state information.

6. The method of claim 5, wherein the state observation equation comprises:

Wherein, the Is a transformation matrix of a world coordinate system to an UWB coordinate system of an external device,For the reference position of the remote controller at the first moment, d is the distance between the UWB unit of the external device and the UWB unit of the remote controller, n _d is the noise corresponding to the distance, theta ^R is the first horizontal angle of the remote controller relative to the external device,For noise corresponding to the first horizontal angle,For a first elevation angle of the remote control relative to the external device,For noise corresponding to the first elevation angle, The direction of the remote controller relative to the external equipment is represented, wherein the external equipment is the display equipment or a Dongle device communicated with the display equipment;

for the transformation matrix of the remote control IMU coordinate system to the display device coordinate system, For the transformation matrix from the IMU coordinate system to the world coordinate system of the remote controller, θ ^I is the second horizontal angle of the external device relative to the remote controller,The noise corresponding to the second horizontal angle,For a second elevation angle of the external device relative to the remote control,For noise corresponding to the second elevation angle, ) And indicating the direction of the external equipment relative to the remote controller.

7. The method of claim 2, wherein the third adjustment strategy is an adjustment strategy indicated by an updated state observation equation in the extended kalman filter;

8. The method of any of claims 1, wherein the applying the target pose and the actual position of the remote control at the second time, calculating a focus of the remote control on a display device, comprises:

calculating coordinates of the remote controller on the display device according to the target gesture and the actual position of the remote controller at the second moment;

And determining pixel coordinates of the remote controller on the display equipment according to the coordinates of the remote controller on the display equipment, wherein the position indicated by the pixel coordinates on the display equipment is the focus.

9. The method according to any one of claims 1-8, further comprising:

And performing smoothing filtering processing on the pixel coordinates by using a first-order filter, wherein a filter coefficient in the first-order filter is determined according to a rotation speed in the speed of the remote controller.

10. A remote controller, characterized in that it is applied to a remote controller, the remote controller includes a data transmission unit and a processor;

the data transmission unit is configured to perform:

Acquiring IMU data and UWB data;

The processor is configured to perform: