WO1998032100A1 - Method and installation for producing a film sequence of a sport activity enhanced by inserting virtual elements in real time - Google Patents

Abstract

The invention concerns a method for producing a film sequence of a sport activity characterised in that it consists in filming in real time a sequence of the evolution of the sport activity using a network of cameras each of which is equipped with a system generating data representing its position in space, its movements, its focusing and a clock signal; in acquiring in real time the instantaneous position of each person performing on the playing field and of the mobile object by analysing the filmed video images; in producing recurrently a matrix of digital data representing the position of each person and of the mobile object; in constructing at least a three-dimensional visual virtual element (C) of information and/or commentary; and in making a video image from the filmed video image and said constructed virtual element by inserting it in the filmed video image for the live or delayed broadcasting of a film sequence of the sport activity comprising said visual virtual element of information and/or commentary.

Description

 "Process and installation for the production of a filmed sequence of a sporting activity embellished with real-time insertions of virtual elements"

The present invention relates to the field of television sports broadcasts.

The invention relates more particularly to a method of producing a filmed sequence of a sporting activity comprising a playing field on which at least one natural person, in particular a player or a competitor, and a moving object, in particular a ball .

The invention finds more particularly to apply to retransmissions of meetings of collective sports, in particular football, but it is not in any way limited to this type of meetings and can also relate to other individual sports such as for example tennis, or athletics.

The televised retransmissions of sports events, whether national or international, are meeting with growing success with viewers and are in this respect the subject of important commercial negotiations relating to retransmission rights and the advertising broadcasts which accompany them.

Paradoxically, the content of the retransmissions, beyond the simple course of the sporting action in progress, has changed very little, in particular with regard to the complements of the images, that is to say what can be qualified generally speaking of comment (s).

Indeed, the classic retransmission of a football match, whether it be a direct or deferred retransmission, boils down to the broadcasting of real filmed images accompanied by a generally sound commentary. pronounced and recorded live by one or more commentators.

Whatever the qualities of commentators, professionals in television, sport and / or journalism, their services are similar to those of radio broadcasting, that is to say that they are generally not in a situation technique to provide comments other than those resulting from their own vision, analysis and understanding of the game phase in progress, their perception of this game phase is generally double, that is to say that they can take Instantly live re knowledge, if they are present in the sports venue in which the meeting takes place, and in parallel on a screen or video monitor which restores the real filmed images which are broadcast for the viewers.

In order to improve the commentary, and in particular to enrich its technical content relating to the analysis of the game, it has already been proposed to add to the commentators assistants who carry out statistical analyzes of certain aspects of the game and who put their results available to the director responsible for broadcasting the meeting and / or the commentators.

These complements can be put in images, that is to say that they are the subject of the production of informative "panels" which are inserted during broadcasting, generally during idle times during which these panels are in turn commented orally.

In addition, the complexity of the rules of play, in particular collective sports such as football, rugby, basketball, etc. , makes it necessary to enrich televised retransmissions with additional illustrations that illustrate the implementation of the rules by one or more referees, and this in order to improve the education of spectators and the public with a view in particular to promoting a fair play and serene behavior of the public during the matches. We also know that the training of sportspeople with a view to improving their performance and the quality of sports play, increasingly requires the use of television recordings, whether they are meetings previous official or training sequences, which are in turn commented on by those responsible for training and in particular by the coach.

In this task, and despite his skill and his technical and tactical culture, the coach is faced with the same difficulties as the commentator of public broadcasts, that is to say that they do not have tools for visual commentary of real images filmed other than panels and tables independent of the image broadcast.

The object of the invention is to remedy these drawbacks by proposing a new method for producing a filmed sequence which makes it possible to match its distribution with a visual commentary in three dimensions and in real time.

To this end, the invention provides a method of producing a filmed sequence of a sporting activity comprising an area of development on which at least one physical person, in particular a player or a competitor, and a moving object, in particular a balloon, characterized in that it consists of:

- film in real time a sequence of sports activity using a network of cameras, each of which is fitted with a system generating data representative of its position in space relative to a fixed reference point linked to the terrain, its movements, its optical focus and a clock signal; - acquire in real time the instantaneous position of each person moving on the ground and of the mobile object by analysis, by means of a computer unit, of the video images filmed by said network of cameras; - cyclically produce a matrix of digital data representative of the position of each person and of the mobile object in said frame and store the successive matrices in a computer processing unit;

- build at least one virtual visual element, in particular in three dimensions, information and / or commentary by means of a computer unit for the production of visual elements as a function of the given data;

- synchronize the network of cameras and computer units; and - fabricating a video image from a filmed video image and said virtual element constructed by inserting it into the filmed video image so as to allow the broadcasting, in real time or delayed, of a filmed sequence of the sport activity including led it virtual visual element of information and / or comment.

According to other characteristics of the process according to the invention:

- the stage of construction of the virtual element consists of:

- perform an analysis and processing of said data to calculate information representative of the evolution over time of a person or of the moving object (speed-acceleration-trajectory-mobility-etc.); and

- build a virtual representation of this information intended to be inserted; - the stage of construction of the virtual element consists of: - perform an analysis and processing of said data to calculate the relative positions of people and / or the moving object relative to the terrain of evolution; and

- construct a "fixed" virtual visual object associated with said relative positions (shooting cone - position of a wall - viewing angle) and intended to be inserted:

- the stage of construction of the virtual element consists of:

- perform an analysis and processing of said data to determine the evolution over time of at least one person and / or of the mobile object; and

- associate a virtual element of known structure with said person or with the mobile element to construct a virtual object to be inserted mobile during the course of the sequence;

the step of constructing the virtual element consists of: performing an analysis and processing of said data in order to calculate statistical information relating to the progress of the sporting activity and / or to the development and movement of people; the moving object; and

- build a virtual representation of this information intended to be inserted;

the step of acquisition in real time of the instantaneous position of each person evolving on the ground consists in acquiring the position of these people in a benchmark with at least two dimensions corresponding to the dimensions of the terrain of evolution;

the step of acquiring the instantaneous position of the mobile object in real time consists in acquiring the position of this object in a three-dimensional reference frame, two of which correspond to the dimensions of the terrain of evolution; - q ue the step of acq uisition in real time of the instantaneous position of the moving object consists in filming the moving object by means of two cameras belonging to said network of cameras; the method comprises an additional step consisting in fabricating a sequence of virtual video images from the virtual elements constructed so as to allow the delayed broadcasting of this sequence of virtual video images in association with the broadcasting of said filmed sequence.

The invention also proposes an installation for implementing the method according to the invention, characterized in that it comprises: - a network of cameras, each of which is equipped with a system generating data representative of its position in the space in relation to a fixed reference linked to the ground, its movements, its optical focusing and a clock signal; - a computer unit for real-time acquisition of the instantaneous position of each person moving on the ground and of the object;

a computer unit for cyclically producing a matrix of digital data representative of the position of each person and of the mobile object in the led it locates and memorizes the matrices;

- a computer unit for constructing at least one virtual visual element of information and / or comment as a function of said data; - means for synchronizing the network of cameras and the computer units; and

- Means for manufacturing a video image from a filmed video image and from said constructed virtual element.

Other characteristics and advantages of the invention will appear on reading the description which follows for the understanding of which reference may be made to the accompanying drawings in which FIGS. 1 to 7 are diagrams illustrating examples of virtual elements inserted in application of the method according to the invention.

The description which follows, given by way of nonlimiting example, relates to the televised retransmission of a football match.

Such a meeting takes place on a playing field, in the open air or in a closed enclosure, on which play two teams of eleven players each, a field referee and two assistant referees called touch judges, or twenty five physical persons in movement, and a movable ball which constitutes the movable object within the meaning of the invention.

The twenty-five natural person actors evolve globally in two dimensions, that is to say in the plane of the terrain which is defined, in a Cartesian coordinate system orthonormal by two perpendicular horizontal axes X and Y, corresponding for example to a lateral line delimiting the pitch, called touchline, and a transverse line called goal line. The evolutions reduced in height of the players can be negated in the case of football or for example of rugby, but be important and taken into account within the framework of the method according to the invention for other sports such as basketball or volleyball .

On the contrary, the ball constantly evolves in three dimensions on the ground, that is to say that knowing its position requires knowing its coordinates [x, y, z] in a three-dimensional frame of reference that consists of the reference X, Y in which the positions of the players and the referees can be determined, completed by a concurrent vertical axis Z whose origin corresponds for example to the ground. We can thus define a fixed geometrical reference with respect to the ground. The realization of a filmed sequence of the sporting action traditionally calls upon several video cameras which are generally established around the ground, in altitude and / or on the level of the ground, the cameras being able to be fixed or mobile and being in addition with focal variable.

This known arrangement allows the director to choose from several images available simultaneously that which is broadcast instantly, in particular according to the course of the game. The method according to the invention proposes to use a network of video cameras which are of a particular type but which are the cameras which allow the conventional capture of the real video images used for the conventional production of the filmed sequence, ie In other words, it does not in any way call upon a particular and additional network of cameras for the real-time acquisition of the instantaneous position of the players on the ground and of the balloon.

The cameras used are particular within the meaning of the invention because each of them comprises a system generating data representative of its position in the space of the ground, that is to say with respect to the previously mentioned reference point, movements of the camera, and its development, ie ^to e the focal length of the optical system. Thanks to the use of such cameras called "motion control", and after an initialization step of each camera in the coordinate system, it is possible to use each image of each camera to analyze the content in order to acquire in real time, from all the images supplied simultaneously by the network cameras, the instantaneous position x, y of each actor and of the balloon x, y, z. As In addition, each camera can be equipped with telemetry means.

Each camera can be of analog type, the images then being digitized, or be digital cameras.

The notion of simultaneity for the purpose of image analysis naturally requires that the network cameras be synchronized, that is to say that each of them generates a clock signal. We are thus able to know at each instant the position in the image of a camera of each actor and / or of the balloon and to associate with this position a triplet, or a doublet of coordinates x, y, z .

Thanks to the arrangement of networked and coordinated cameras, a computer unit forming part of the installation for implementing the method according to the invention makes it possible to cyclically produce a matrix of mathematical digital data which are representative of the position of each actor and the ball. This method of capturing spatial coordinates is particularly advantageous because it uses the same cameras as used for the realization of real images and it does not require ^'to implant transmitters radiofréq influence on the actors and on the ball, wh ich will transmitters are not admitted in the regulations on the equipment of the players and which impose moreover to have on each ground a network of radio frequency receivers.

Due to the twenty-four frames per second cycle used in video, a new matrix can be produced twenty-four times per second. The means for analyzing the video images of the network cameras are of known type (s) such as, for example, those sold by the company HYMATOM.

According to the most common technique, the images are digitized and then analyzed pixel by pixel, an analysis which, in addition to determining the coordinates sought, is then used during the phase of insertion of the virtual elements.

According to another method, and if the optical parameters of design and development of the cameras used are perfectly known, a photogrammetric analysis of each image is carried out, without going through a digitization step, that is to say say by implementing the principles of stereophotography. In this case, it is not necessary to have a large number of crossed cameras, one or two cameras being able to make it possible to reconstruct the three-dimensional coordinates of all the characteristic points of each video image, in particular by using the video signal. basic for retransmission called "broadcast" signal. To facilitate the acquisition of the position of the mobile formed by the ball, it can for example be called upon by two crossed cameras assigned to the pursuit of the ball and its immediate environment, this capture being manual or automated. The image analysis means for determining the coordinates of the actors may for example include a real-time image analysis calculator for each player, all the computers being managed by a centralization module which manages in particular the passage of a player from one camera to another, which manages conflicts such as occlusions and collisions, automatically or with the help of a personal assistant. Processing by real-time calculation requires that the computer unit be synchronized with the network of cameras themselves synchronized.

From said data matrices it is possible to construct at least one virtual element of information or comment using a computer unit for producing visual elements, which is also synchronized.

This virtual and visual comment element, in particular in three dimensions, is intended to be inserted in the real video images filmed so as to allow the broadcasting in real time or in deferred of a filmed sequence comprising the comment element inserted.

The insertion of the virtual element for the production of "augmented" video images intended to be broadcast is carried out by known computer means comprising at least one computer operating a real-time insertion program.

We will now give examples of the virtual elements intended to be inserted in order to constitute a visual commentary on the sporting action.

On the basis of spatio-temporal data previously determined, it is for example possible to calculate the trajectories of the ball and the players, to construct a virtual representation of these trajectories and to insert them into the images as shown diagrammatically in FIG. 1 on r which is shown the overall trajectory of the ball B on the ground and in the air, from one goal to another.

The sequence of images including the insertion of the ball's trajectory is broadcast slightly delayed. It can also be supplemented with three-dimensional virtual representations such as "mu rs" M which represent, for example, the level at which the first attacker or the last defender, these virtual elements being linked to the corresponding player, that is to say that their insertion in real time is "dynamic" in that at each instant of the sequence commented by the virtual elements, the latter are recalculated and inserted.

In Fig ure 2, there is shown the insertion of a volume representation of a virtual "cone" of the shooting space available to a player at the time of his strike in direction of the goal. This representation could be supplemented with calculated data such as the distance of the player from the goal, the distance and the running time of the player before hitting, etc.

Figures 3,4 and 5 show other angles of shots embellished with a virtual shooting cone with for example the addition of the trajectory of the ball.

The representation illustrated in fig ure 6 shows the use of a virtual wall for the illustration and the commentary of the rule of the offside, the wall M being "attached" to the last defender and moving with the latter. All of the data obtained from the real-time capture and analysis of the filmed images allows the insertion of many other elements of comments.

For example, you can insert the name of each player next to their image (see Figure 7). We can at any time associate with each player statistical data relating to their activity during the match (distances traveled, average or maximum speed, number of ball strokes, time in possession of the ball, etc.). This type of statistical analysis can be performed, and its results inserted globally, for each team. In the case of general images of the field taken from above, substantially vertically from the ground, it is possible to insert the tactical theoretical diagram of each team (player placement and movement modes), or to insert the actual playing diagram resulting from a prior statistical analysis of the data stored during an immediately preceding game period (game action, half-time, etc.).

Other statistics such as the number and precision of the shots on goal can also be the subject of virtual visual comments by insertion in images representing the goal of densities of "impacts" in and around the goal, agmented of the chronology of the shots on goal, agmented of the chronology of the shots on goal.

We can also materialize virtually, in two or three dimensions, the space in which the ball has most often circulated.

The method according to the invention also allows the realization of purely virtual sequences of the type of those produced in video games or the insertion of a virtual element allowing for example to simulate a hypothetical movement of a player compared to a real action. filmed and thus invent an alternative scenario of reality.

Such a scenario also allows the explanation of complex game rules such as offside. The creation of virtual game sequences also makes it possible to "enter" the virtual camera on the ground and to rebroadcast a real prior action from another point of view, for example from the point of view of a player or the ball or of the referee.

Claims

REVENDI CATIONS 1. Method for producing a filmed sequence of a sporting activity comprising an area of evolution on which evolves at least one physical person, in particular a player (J) or a competitor, and a moving object, including a ball (B), characterized in that it consists of: - film in real time a sequence of sports activity using a network of cameras, each of which is equipped with a system generating data representative of its position in space relative to a fixed reference point (X , Y, Z) related to the terrain, its movements, its optical focus and a clock signal; - acquire in real time the instantaneous position of each person (x, y) moving on the ground and of the mobile object (x, y, z) by analysis, by means of a computer unit, of the video images filmed by said network of cameras; - cyclically produce a matrix of digital data representative of the position of each person and of the mobile object in said frame and store the successive matrices in a computer processing unit; - construct at least one virtual visual element (M, C) of information and / or commentary by means of a computer unit for producing visual elements as a function of said data; - synchronize the network of cameras and computer units; and - fabricate a video image from a filmed video image and said virtual element constructed by inserting it into the filmed video image so as to allow the broadcast, in real time or delayed, of a filmed sequence of the 'activity sport comprising said virtual visual element of information and / or commentary. 2. Production method according to resell ication 1, characterized in that the stage of construction of the virtual element consists in: - perform an analysis and processing of said data to calculate information representative of the evolution over time of a person or of the moving object (speed-acceleration-trajectory-mobility-etc.); and - construct a virtual representation of this information intended to be inserted. 3. Production method according to claim 1, characterized in that the step of constructing the virtual element consists in: - carrying out an analysis and a processing of said data to calculate the relative positions of the people and / or of the object mobile in relation to the playing field; and - construct a "fixed" virtual visual object associated with said relative positions (shooting cone - position of a wall - viewing angle) and intended to be inserted. 4. Prod uction method according to claim 1, characterized in that the step of construction of the virtual element consists in: - carry out an analysis and a processing of said data to determine the evolution over time of at least one person and / or of the mobile object; and - associate a virtual element of known structure with the said person or with the mobile element to construct a virtual object to be inserted mobile during the course of the sequence. 5. Production method according to claim 1, characterized in that the step of constructing the virtual element consists in: - perform an analysis and processing of said data to calculate statistical information relating to the progress of the sporting activity and / or the evolution and movement of people and the mobile object; and - ^• build a virtual representation of this information intended to be inserted. 6. Production method according to any one of the preceding claims, characterized in that the step of acq uisition in real time of the instant position of each person evolving in the field consists in acq uter the position of these people in a coordinate system with at least two dimensions (X, Y) corresponding to the dimensions of the terrain of evolution. 7. Production method according to one of the preceding claims, characterized in that the step of real-time acquisition of the instantaneous position of the mobile object consists in acquiring the position of this object in a frame with three dimensions (X, Y, Z), two of which correspond to the dimensions of the terrain of evolution. 8. Production method according to the preceding claim, characterized in that the step of real-time acquisition of the instantaneous position of the mobile object consists in filming the mobile object by means of two cameras belonging to said network of cameras. 9. Production method according to any one of the preceding sells, characterized in that it comprises an additional step consisting in manufacturing a sequence of virtual video images from virtual elements constructed so as to allow the diffusion in differed from this sequence of virtual video images in association with the broadcasting of said filmed sequence. 10. I nstallation for the implementation of the method according to one of the preceding claims, characterized in that it comprises: - a network of cameras, each of which is fitted with a system generating data representative of its position in space relative to a fixed reference point linked to the terrain, its movements, its optical focusing and a signal clock; - a computer unit for real-time acq uisition of the instant position of each person moving on the ground and the object; - a computer unit for cyclically producing a matrix of digital data representative of the position of each person and of the mobile object in said reference and memorizing the matrices; - a computer unit for building at least one virtual visual element of information and / or comment as a function of said data; - means for synchronizing the network of cameras and the computer units; and - Means for manufacturing a video image from a filmed video image and said constructed virtual element.