CN115917597A - Boosting 2D Representations to 3D Using Attention Models - Google Patents

Abstract

A processing device (200) is described for forming a system for estimating a three-dimensional state of one or more joint objects represented by a plurality of data sets (101), each data set being indicative of a projection of a joint of an object on a two-dimensional space, the processing device comprising one or more processors (201) configured to: receiving (301) a data set; processing (302) a data set using an encoder architecture (104) having a plurality of encoder layers, each encoder layer comprising a respective self-attention mechanism (107) and a respective feed-forward network (109), each self-attention mechanism implementing a plurality of attention heads; and training (303) the encoder layers from the data set to improve the accuracy of the encoder architecture in estimating the three-dimensional state (106) of the one or more objects represented by the data set. This may allow for elevation of 2D body keypoints to 3D relative to the root (e.g. pelvic joint) by extrapolating the input data to estimate depth at real-time runtime using a small, efficient model.

Description

Boosting 2D Representations to 3D Using Attention Models

technical field

The invention relates to estimating a three-dimensional position of an object from a two-dimensional data set at a processing device.

Background technique

In recent years, 3D position estimation of human joints has been the subject of extensive research. Particular attention has been given to defining methods for extrapolating 2D data (in the form of x, y keypoints) to 3D to predict the root-relative coordinates of joints relative to the human skeleton. There are 17 keypoints describing the human skeleton, including the head, shoulders, elbows, wrists, pelvis, knees, and ankles.

Initially, the work was mainly based on pre-designed models with a large number of constraints to account for the high dependencies between different human joints.

With the development of Convolutional Neural Network (CNN), pose estimators have been developed, which reconstruct 3D pose end-to-end directly from RGB images or intermediate 2D predictions. This approach has rapidly surpassed the accuracy of previous hand-crafted estimators.

Current state-of-the-art methods generally fall into two main approaches. Some methods directly predict 3D keypoints end-to-end from monocular images. This usually produces good results, but requires very large models. Other methods perform lifting, where a 2D predictor is used to predict the human pose from an image, and then the 2D keypoints (relative to the image) are upscaled to 3D. This two-step approach typically uses Temporal Convolution Layers to aggregate information from poses derived from videos.

There is a need to develop a method for lifting 2D projections of a subject's joints to 3D that can overcome the limitations of existing methods by extrapolating input data to estimate depth in real-time using a small efficient model.

Contents of the invention

According to one aspect, there is provided a processing device for forming a system for estimating the three-dimensional state of one or more articulated objects represented by a plurality of data sets, each data set indicating the position of the object's joints in two-dimensional space projection, the processing device comprising one or more processors configured to: receive a data set; process the data set using an encoder architecture having a plurality of encoder layers, each encoder layer comprising a respective self-attention mechanism and Respective feed-forward networks, each self-attention mechanism implementing multiple attention heads; and encoder layers trained on the dataset to improve the accuracy of the encoder architecture in estimating the 3D state of one or more objects represented by the dataset .

This can allow the processing device to estimate depth by extrapolating input data in real-time using a small efficient model, forming a method for lifting a 2D dataset including, for example, 2D body keypoints (x,y coordinates) relative to the root (e.g. 3D (x, y, z coordinates) system of the pelvic joint).

The or each object may be a person and each data set may be indicative of a human pose. Each dataset can include multiple keypoints (each with 2D coordinates) of the human body. Typically, there are 17 keypoints describing the human skeleton, including the head, shoulders, elbows, wrists, pelvis, knees, and ankles. Each dataset indicating human pose can define 2D coordinates for each of multiple keypoints. This could allow estimating the 3D positions of human joints, which could be useful in video games or virtual reality applications.

The encoder architecture can implement the Transformer (change) architecture. In the task of lifting 2D keypoints to 3D, using the Transformer encoder model yields good accuracy.

The operation of each self-attention mechanism may be defined by a set of parameters, and the device may be configured to share one or more of such parameters among multiple self-attention mechanisms. Weight sharing can make the model more efficient while maintaining (or in some cases improving) the accuracy of predicting the 3D state of the object.

The device may be configured to adjust one or more of said parameters in a self-attention mechanism in response to the continuous data set, and after adjusting the parameter, propagate the parameter to one or more other self-attention mechanisms. Therefore, the parameters of one or some attention layers can be shared with other attention layers. This can further improve the efficiency of the model.

The operation of each encoder layer may be defined by a set of parameters, the device may be configured to adjust one or more of these parameters in an encoder layer in response to successive data sets, and the device may be configured not to The parameters of are propagated to any other encoder layers. Therefore, in some embodiments, parameters of an encoder layer may not be shared with other encoder layers. This can further improve the model.

The one or more processors may be configured to perform a one-dimensional convolution on the dataset to form convolved data and process the convolved data using the encoder architecture. This can allow the dimensions of the dataset to be adjusted to match the dimensions of the model inputs and outputs.

One or more processors may be configured to perform one-dimensional convolutions on series of continuous data sets. This can allow estimating the 3D state of an object for a 2D input sequence. For example, a 3D motion sequence of a human body can be estimated.

A series of datasets can be a series of an odd number of datasets. This can allow datasets on either side of the central dataset that need to obtain a 3D state to be considered during training.

The device may be configured to estimate, from the series of data sets, the three-dimensional state of an intermediate one of the series of data sets. The middle one in the series of datasets may correspond to the center of the original receptive field (the total number of poses used to predict a 3D pose). During training, half of the receptive field corresponds to past poses and the other half to future poses. An intermediate pose in the perceived field may be a pose that is currently being promoted from 2D to 3D.

One or more processors can be configured to train the encoder architecture to lift the dataset into three dimensions. Training the encoder can allow the model to more accurately predict the 3D state of an object.

Each data set may represent the position of a human joint relative to a predetermined joint or structure of the human body. The structure may be the pelvis. The position of the pelvis can serve as the root. A separate model can then be used to determine the distance from the camera to the pelvis. This can allow determining the position of each joint relative to the camera.

The processing device may be configured to: receive a plurality of images, each image representing an articulated object; and for each image, detect the joint position of the object in the image, thereby forming one of said datasets. Accurate 2D poses of human bodies in multiple images can be obtained using a 2D pose estimator. This could allow 2D poses to be predicted from multiple images, and these 2D poses could then be used as input datasets to the aforementioned devices to lift 2D datasets to 3D.

Once trained, the system can be used to estimate the 3D state of one or more articulated objects represented by multiple such datasets during the inference phase.

According to another aspect, there is provided a system for estimating a three-dimensional state of one or more articulated objects, the system being formed by the above-mentioned processing device.

According to yet another aspect, there is provided a method for estimating the three-dimensional state of one or more articulated objects represented by a plurality of data sets, each data set indicating a projection of the object's joints onto a two-dimensional space, the method comprising : Receive a dataset; process the dataset using an encoder architecture with multiple encoder layers, each encoder layer including its own self-attention mechanism and its own feed-forward network, each self-attention mechanism implementing multiple attentions head; and training an encoder layer on the dataset to improve the accuracy of the encoder architecture in estimating the three-dimensional state of one or more objects represented by the dataset.

The method may also include estimating the three-dimensional state of one or more articulated objects represented by a plurality of such datasets.

Use of this method can allow estimation of depth by extrapolating input data at runtime using a small efficient model, lifting a 2D dataset including for example 2D human body keypoints (x,y coordinates) relative to the root (eg pelvic joints) 3D(x, y, z).

Description of drawings

The invention will now be described by way of example with reference to the accompanying drawings.

In the attached picture:

Figure 1 shows an overview of the model architecture, which takes as input a dataset sequence comprising 2D keypoints of human poses, and generates 3D pose estimates using self-attention on long-term information.

Figure 2 shows a processing device for forming a system for estimating the three-dimensional state of one or more articulated objects represented by a plurality of data sets.

Fig. 3 shows an exemplary flowchart detailing the method steps performed by the processing device.

Figure 4 shows qualitative results for several actions of human poses using the Humans3.6M dataset: (a) raw RGB images using CPN for 2D keypoint prediction; (b) 3D reconstruction using the method described in this paper (n = 243, where n is the receptive field of poses in the input sequence); (c) ground truth 3D keypoints.

Detailed ways

The method described in this paper is exemplified by processing datasets, each indicating a human pose. However, it should be understood that the method can also be applied to other datasets and objects where data needs to be converted from 2D to 3D.

Typically, there are 17 key points for depicting the human skeleton, including the head, shoulders, elbows, wrists, pelvis, knees, and ankles. In the example described here, each dataset indicating human pose defines 2D coordinates for each of these keypoints. Preferably, each data set indicative of a pose of the human body represents the position of a joint of the human body relative to a predetermined joint or structure of the human body. In a preferred embodiment, the structure is a human pelvis.

In the examples described herein, each dataset input to the model (eg, individual human poses) may be derived from a larger motion capture dataset that includes images depicting different human poses. Such larger datasets can include motion capture datasets such as Human3.6M (see C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, "Human3.6M: Large-scale 3D Human Perception in Natural Environments") Datasets and predictive methods (Large scale datasets and predictive methods for 3d human sensing in natural environments), IEEE Transactions on Pattern Analysis and Machine Intelligence (Pattern Analysis and Machine Intelligence, PAMI), 7 (2013), pp. 1325-1339) or HumanEva (see L.Sigal, A.O.Balan, and M.J.Black, "HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion) ", International Journal of Computer Vision (IJCV) (2010), 87(1-2): 4) Motion Capture Dataset. Human3.6M includes 3.6 million frames from 11 different subjects, but only 7 of them are annotated. The subjects performed up to 15 different types of movements, which were recorded from four different angles. In contrast, HumanEva is a smaller motion capture dataset with only three subjects recorded from three angles.

The methods described herein may include a two-step pose estimation approach. A dataset indicative of human pose input to the model can first be obtained by using a 2D pose estimator to obtain accurate 2D poses of the human body from images. This can be done in a top-down manner. These joints can then be lifted by predicting their depth relative to the root (eg pelvis).

Accordingly, the processing device may be configured to receive a plurality of images, each image representing a joint object. For each image, the processing device may then detect the positions of the joints of the human body (or other object) in the image, forming one of the datasets indicative of a single pose.

For example, the 2D pose can be obtained by using the ground truth human bounding box followed by a 2D pose estimator. Some common 2D estimation models that can be used to obtain 2D pose sequences are: Stacked Hourglass (SH), as described in A. Newell, K. Yang and J. Deng in "Stacked hourglass networks for human pose estimation (Stacked hourglass networks for human pose estimation)" (IEEE European Conference on Computer Vision (ECCV) Proceedings (2016), p. 483–499); mask RCNN (Mask-RCNN), such as K.He, G.Gkioxari, P.Dollar, and R.Girshick in "Mask-RCNN" (Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017), pp. 2961–2969) described; or Cascaded Pyramid Network (CPN), such as Y.Chen, Z.Wang, Y.Peng, Z.Zhang, G.Yu and J.Sun in "For multi-person pose estimation Cascaded pyramid network (Cascaded pyramid network for multi-person pose estimation)" (IEEE Computer Vision and Pattern Recognition Conference (Computer Vision and Pattern Recognition, CVPR) Proceedings (2018), pp. 7103–7112).

Alternatively, 2D detectors that do not rely on ground truth can be used. For example, SH and CPN can be used as detectors for the Human3.6M motion capture dataset, and Mask-RCNN is used as a detector for the HumanEva motion capture dataset. However, it is also possible to use ground truth 2D poses for training. In a specific example, follow D.Pavllo, C.Feichtenhofer, D.Grangier, and M.Auli in "3D human pose estimation in video with temporal convolutions and Semi-supervised training)" (IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Proceedings (2019), pp. 7753–7762), SH can be pretrained on the MPII motion capture dataset (L.Pishchulin , E.Insafutdinov, S.Tang, B.Andres, M.Andriluka, P.Gehler, and B.Schiele, "DeepCut: Joint Subset Partition and Labeling for MultiPerson Pose Estimation ", IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Proceedings (2016), pp. 4929–4937), and fine-tuned on the Human3.6M motion capture dataset. Both Mask-RCNN and CPN can be pre-trained on COCO (T.Lin, M.Maire, S.Belongie, J.Hays, P.Perona, D.Ramanan, P.Dollar, and C.L.Zitnick, "Microsoft coco: Context Common objects in context", IEEE European Conference on Computer Vision (ECCV) Proceedings (2014), pp. 740–755), followed by fine-tuning on the 2D pose of Human3.6M , since keypoints are defined differently in each motion capture dataset. More specifically, Mask-RCNN uses ResNet-101 with FPN. Since CPN requires a bounding box, Mask-RCNN can be used first to detect the human body. It can then determine keypoints from the image using a ResNet-50 with an input resolution of 384×384.

In the example described in this paper, the open source Transformer model is used to boost the keypoint dataset. In this example, a sequence of 2D keypoints is passed through several Transformer encoder blocks to generate a 3D pose prediction (corresponding to the pose at the center of the input sequence/receptive field). Further 3D estimates can be computed in a sliding window manner.

Figure 1 shows an overview of the model architecture, which takes as input a dataset sequence comprising 2D keypoints of human poses, and generates 3D pose estimates using self-attention on long-term information.

As shown at 101, the device takes as input a sequence of datasets including 2D keypoints (the sequence of inputs is collectively referred to as a receptive field). The method can be used with different numbers of datasets. The series of data sets is preferably a series of an odd number of data sets. For example, receptive fields including 27, 81 and 243 keypoint datasets can be used. Preferably, the middle dataset of the sequence is chosen for boosting since it corresponds to the center of the original receptive field.

As mentioned above, the dataset corresponding to the input sequence can come from a 2D predictor that estimates 2D pose from image frames, or directly from the 2D ground truth. Therefore, these input datasets lie in image space.

As shown in Figure 1, in some embodiments certain modifications may be performed to match the dimensions of the input and output of the model. In this example, a sequence of input datasets is passed through a convolutional layer 102 to change dimensions. In this example, the Transformer's input is reprojected from input dimension [B, N, 34] to [B, N, 512], where B is the batch size and N is the receptive field (i.e., the input to the model at each processing step number of human poses), 34 corresponds to 17 joints times 2 (number of coordinates, i.e. x and y).

Then, a temporal encoding is added to embed the order of the input sequence of the dataset (pose). As shown at 103, temporal encodings are aggregated by adding vectors to the input embeddings. This allows the model to exploit the order of pose sequences. This can also be called positional encoding and is used to inject information about the relative or absolute position of tokens in the sequence. These temporal embeddings can be created using sine and cosine functions, which are then summed as input for reprojection. In this example, the injected temporal embedding has the same dimensionality as the input.

The temporally embedded datasets are then fed into a Transformer encoder model that processes them. The Transformer is shown at 104 in FIG. 1 .

Self-attention models are used in Natural Language Processing (NLP) to embed poses over time, rather than using fully convolutional approaches. The encoder architecture has multiple encoder layers, each comprising a respective self-attention mechanism 107 and a respective feed-forward network 109 . Each self-attention mechanism implements multiple attention heads. The outputs from blocks 107 and 109 can be summed and normalized as shown at 108 and 110 .

.Kaiser和I.Polosukhin在“注意力就是你所需要的一切(Attention is all you need)”(IEEE神经信息处理系统(Neural Information Processing System，NeurIPS)进展论文集(2017)，第5998–6008页)中描述的基本Transformer编码器可作为基线。在此示例中，有512维隐藏层、8个多注意力头和6个编码器块。A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, ANGomez,

.Kaiser and I. Polosukhin in "Attention is all you need" (IEEE Neural Information Processing System (Neural Information Processing System, NeurIPS) Progress Papers (2017), pp. 5998–6008 The basic Transformer encoder described in ) can be used as a baseline. In this example, there are 512-dimensional hidden layers, 8 multi-attention heads, and 6 encoder blocks.

Since the decoder part of the Transformer is not used in this implementation, and due to the residual connections within self-attention, the dimensions of the output are the same as those of the input. In this example, that is [B, N, 512].

The 1D convolutional layer 105 changes dimensions such that the output of the model (shown at 106 ) is the x, y and z coordinates of all joints relative to the pelvis for the current pose being lifted.

Use a 1D convolutional layer 105 to reproject the output embedding to the desired dimension, from [B, 1, 512] to [B, 1, 51], where 51 corresponds to 17 joints times 3 (for x, y and z-coordinate). The loss is then computed, e.g. using the mean-per-joint position error (MPJPE) against the 3D ground truth of the dataset. The error is then backpropagated and the next training iteration begins.

Preferably, the model output label (the current pose is lifted from 2D to 3D within the pose input sequence) corresponds to the intermediate data set of the receptive field (intermediate pose). This is because during training, half of the receptive field corresponds to past poses and the other half to future poses. Thus, during training, the model utilizes temporal data from past and future frames in order to be able to create temporally consistent predictions.

During inference, the model architecture remains the same as during training, but only past pose frames are used in the receptive field, rather than the frames on either side of the current pose being lifted that are used during training. The model can work in a sliding window manner so that eventually 3D reconstructions can be obtained for all 2D representations in the input sequence.

A benefit of this architecture is that the number of receptive fields and multi-attention heads can be modified without affecting the model size. In addition, the hyperparameters of the Transformer can also be modified.

In some embodiments, weight sharing can be used to maintain or improve final accuracy while significantly reducing the total number of parameters, resulting in more efficient models.

Optionally, parameters can be shared within each encoder block. Specifically, attention layer parameters can be shared.

In one embodiment of shared attention layer parameters, the operation of each self-attention mechanism is defined by a set of parameters, and one or more of these parameters are shared among multiple self-attention mechanisms. One or more of said parameters in a self-attention mechanism may be adjusted in response to successive data sets, and after adjusting a parameter, the parameter may be propagated to one or more other self-attention mechanisms.

Alternatively or additionally, the operation of each encoder layer may be defined by a set of parameters. The device may be configured to adjust one or more of these parameters in an encoder layer in response to successive data sets. In one embodiment, the device may be configured not to propagate such parameters to any other encoder layers.

In particular, sharing only attention layer parameters (and not encoder block parameters) improves the final accuracy while significantly reducing the total number of parameters.

In some embodiments, additional data augmentation may be applied to the dataset during training and testing. For example, each pose can be flipped horizontally.

During the training of the attention layer of the Transformer model, an optimizer can be used, such as the Adam optimizer (as S.J.Reddi, S.Kale and S.Kumar in "On the convergence of Adam and beyond" (Described in Proceedings of the International Conference on Learning Representation (ICLR) (2018)). For example, a training run for the Human3.6M motion capture dataset can last for 80 epochs, and the HumanEva motion capture dataset can last for 1000 epochs.

The learning rate can be increased linearly for the first training step (e.g., 1000 iterations with a learning rate factor of 12) and then decreased proportional to the inverse square root of the number of steps. This is commonly referred to as NoamOpt.

The batch size can be proportional to the receptive wild value. For example, when n=27, 81, and 243, b=5120, 3072, and 1536, respectively.

On the hardware side, the system can be trained and evaluated using eight NVIDIA V100 GPUs, and optimized in parallel. Typically, the training time for each receptive field can be around 8, 14 and 40 hours, respectively, taking into account the batch size.

FIG. 2 is a schematic representation of a system 200 configured to perform the methods described herein. System 200 can be implemented on a device such as a laptop, tablet, smartphone, or television (TV).

System 200 includes a processor 201 configured to process data sets in the manner described herein. For example, the processor 201 may be implemented as a computer program running on a programmable device such as a GPU or a central processing unit (Central Processing Unit, CPU). System 200 includes a memory 202 arranged in communication with processor 201 . Memory 202 may be a non-volatile memory. Processor 201 may also include a cache (not shown in FIG. 2 ), which may be used to temporarily store data from memory 202 . The system may include more than one processor and more than one memory. The memory may store data executable by the processor. The processor may be configured to operate according to a computer program stored in a non-transitory form on a machine-readable storage medium. A computer program may store instructions for causing a processor to perform its method in the manner described herein.

Figure 3 shows a flowchart summarizing an example of a method for estimating the three-dimensional state of one or more articulated objects represented by multiple datasets (e.g., multiple human poses), each dataset indicating an object (e.g., a human body) The projection of the joints on the two-dimensional space.

At step 301, the method includes receiving a data set. At step 302, the method includes processing the dataset using an encoder architecture having multiple encoder layers, each encoder layer comprising a respective self-attention mechanism and a respective feed-forward network, each self-attention mechanism implementing multiple attention head. At step 303, the method includes training an encoder layer based on the dataset to improve the accuracy of the encoder architecture in estimating the three-dimensional state of one or more objects represented by the dataset.

This paper describes the use of a self-attention Transformer model to estimate the depth of 2D keypoints. The encoder's self-attention architecture allows the model to produce temporally consistent poses by exploiting long-range temporal information across frames/poses.

It was found in some embodiments that the methods described herein can provide better results and allow for smaller model sizes than previous methods.

For input 2D predictions (Mask-RCNN and CPN), the method described in this paper is found to outperform previous boosting methods and perform comparable to methods using keypoints and features extracted from raw RGB images. For ground-truth inputs, the model is found to outperform previous models, achieving comparable results to Skinned Multi-Person Linear Model (SMPL) or multi-view methods that simultaneously predict body shape and pose. The number of parameters in the model is easily tuned and is smaller (e.g., 9.5 million) than current methods (which may have around 11-17 million parameters), while still achieving better performance. Therefore, compared with the state-of-the-art, this method can achieve better results with a smaller model size.

Figure 4 shows qualitative results for several actions using human poses in the Humans3.6M dataset. Column (a) shows the original RGB image using CPN for 2D keypoint prediction. Column (b) shows 3D reconstructions (with n=243 receptive fields) using the method described here. Column (c) shows the ground truth 3D keypoints. It can be seen that the obtained 3D reconstructions closely match the ground truth 3D keypoints.

Thus, the method described in this paper allows estimating depth by extrapolating input data at runtime using a small efficient model, lifting 2D human keypoints (x, y coordinates) to 3D (x, y coordinates) relative to roots (such as pelvic joints). y, z).

The applicant hereby separately discloses each individual feature described herein as well as any combination of two or more such features, provided that these features or combinations can be understood based on the present description as a whole according to the common general knowledge of a person skilled in the art. implementation, regardless of whether these features or combinations of features solve any of the problems disclosed herein, and are not limited to the scope of the claims. The applicant indicates that aspects of the invention may comprise any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims (16)

1. A processing device (200) for forming a system for estimating the three-dimensional state of one or more articulated objects represented by a plurality of data sets (101), each data set indicating the joints of said object in two projection on a three-dimensional space, the processing device includes one or more processors (201), which are configured to: receiving (301) said data set; The dataset is processed (302) using an encoder architecture (104) with multiple encoder layers, each encoder layer comprising a respective self-attention mechanism (107) and a respective feed-forward network (109), each The self-attention mechanism implements multiple attention heads; and The encoder layer is trained (303) based on the dataset to improve the accuracy of the encoder architecture in estimating a three-dimensional state (106) of one or more objects represented by the dataset. 2. The processing device (200) according to claim 1, wherein each object is a person and each data set is indicative of a human pose. 3. The processing device (200) according to claim 1 or 2, wherein the encoder architecture (104) implements a Transformer architecture. 4. The processing device (200) according to any one of the preceding claims, wherein the operation of each self-attention mechanism (107) is defined by a set of parameters, and the device is configured to operate on multiple self-attention mechanisms One or more of the parameters are shared between. 5. The processing device (200) according to claim 4, wherein said device is configured to adjust one or more of said parameters in a self-attention mechanism (107) in response to a continuous data set, and to adjust parameters After that, the parameters are propagated to one or more other self-attention mechanisms. 6. The processing device (200) according to claim 5, wherein the operation of each encoder layer is defined by a set of parameters, said device being configured to adjust one or more of the encoder layers in response to successive data sets. and the device is configured not to propagate the parameters to any other encoder layers. 7. The processing device (200) according to any one of the preceding claims, wherein the one or more processors (201 ) are configured to: A one-dimensional convolution is performed (102) on the data set to form convolutional data and the convolutional data is processed using the encoder architecture (104). 8. The processing device (200) according to claim 7, wherein said one or more processors (201) are configured to perform said one-dimensional convolution (102) on a series of continuous data sets (101). 9. The processing device (200) according to claim 8, wherein the series of data sets (101 ) is a series of an odd number of data sets. 10. The processing device (200) according to claim 9, wherein the device is configured to estimate, from the series of data sets (101), the three-dimensional state of a middle one of the series of data sets. 11. The processing device (200) according to any one of the preceding claims, wherein the one or more processors (201) are configured to train the encoder architecture (104) to upscale the dataset to three dimensions. 12. The processing device (200) according to claim 11, wherein each data set represents the position of a human body joint relative to a predetermined joint or structure of the human body. 13. The treatment device (200) according to claim 12, wherein said structure is a pelvis. 14. The processing device (200) according to any one of the preceding claims, wherein the processing device is configured to: receives a plurality of images, each representing a joint object; and For each image, joint positions of said object in said image are detected to form one of said datasets (101). 15. A system for estimating a three-dimensional state of one or more articulated objects, said system being formed by a processing device (200) as claimed in any one of the preceding claims. 16. A method (300) for estimating a three-dimensional state of one or more articulated objects represented by a plurality of data sets (101), each data set indicating a projection of a joint of said object onto a two-dimensional space, The methods include: receiving (301) said data set; The dataset is processed (302) using an encoder architecture (104) with multiple encoder layers, each encoder layer comprising a respective self-attention mechanism (107) and a respective feed-forward network (109), each The self-attention mechanism implements multiple attention heads; and The encoder layer is trained (303) based on the dataset to improve the accuracy of the encoder architecture in estimating a three-dimensional state (106) of one or more objects represented by the dataset.

Images