TWI411292B - Method to encode real scene for virtual reality - Google Patents

Abstract

A method to encode real scene for the application of virtual reality is disclosed. It features synchronization between real scene playing and interactive objects, wherein the information contained in the real scene includes frames, the associated mileage, inclination of the ground plane or water plane, and the measured values of a variety of sensors; the interactive objects consist of user's action or the action of the machine operated by the user. The techniques implemented in the method contain splitting and merging of the road tracks, which intend to provide interconnected network for many users to join the competition or connection on line.

Description

Real-world virtualization coding method

The invention is a method for encoding a real scene, and realizing the synchronization between the live scene broadcast and the interactor. The real scene information includes the odometer corresponding to the frame, the ground tilt, and the values of the respective sensors, and the interactor includes the user's motion. Or the action of the appliance operated by the user.

The traditional virtual reality is modeled and designed by computer simulation software, which can be used to model the actual landscape in three dimensions, but the time it takes There are not only many, manpower and expenses, but their fidelity is not as good as the actual scene. Therefore, there is a technique that replaces the modeling of the landscape with the actual photograph of the landscape, but such landscape photos cannot produce a sense of continuity, such as the streetscape provided by GOOGLE MAP. Therefore, the use of the actual filming of the scenery becomes a real scene, and it has the characteristics that the virtual reality can interact with the viewer or the user, and it is indeed a technology and product worth developing.

One of the objects of the present invention is to make a real scene using a film that actually captures a landscape, and to have a feature similar to that of a virtual reality that can interact with a viewer or a user.

Another object of the present invention is to provide other information of the shooting location, such as latitude and longitude coordinates, altitude, odor or air composition (such as sea breeze taste, fishy smell, etc.), wind speed, temperature, humidity, etc., when photographing the real scene. Because more sensors are added to the camera by providing more sensors, such as gas sensors, thermometers and hygrometers, anemometers, global positioning systems (GPS), altimeters, etc., a more complete record can be achieved.

Another object of the present invention is to provide a method for encoding a real scene to achieve real-time playback and synchronization with an interactive person. The information of the real scene includes an odometer corresponding to a frame, ground or water surface inclination, and various sensing. The value of the device, the interaction of the user's action or the device operated by the user.

Another object of the present invention is to provide a traceable use of the real scene when the live video clip is encoded and interacted with the activist. Calendar, convenient to count popular routes and recommended routes or popular routes, or a variety of leaderboards, to provide reference for the latecomers to exercise. In addition, the physical condition and physical progress of the athlete can also be recorded. For each section, the difficulty level of the individual can be recorded separately, such as the average speed, the completion time, the heartbeat, and the like.

Another object of the present invention is to provide header and trailer encoding of live movie clips, each clip can be connected in the order of head and tail encoding, and can handle the connection of multiple loops, allowing the operator to select any path of multiple loops, thus Establish a true road network to allow users to go deep into the streets.

In order to achieve the above object, the method for encoding the real scene of the present invention is as shown in FIG. 1. Step 1 and film capture, and obtain real scene and corresponding geographic and environmental sensing information; Step 2: Movie fragment encoding, The clipped video clips are set as segmentation points according to the number of consecutive frames or the forks; in step 3, the frame coding, the video frames and the corresponding geographic and environmental sensing information are sequentially changed into array data files; 4, the film clip uses the resume record.

Step one, filming

When shooting a wide field of view, you can use a panoramic shot or at least a camera, or a stereo camera to help virtualize the post-production, while Also use the odometer and Global Positioning System (GPS) to provide the shooting location, etc., in addition to provide other information about the location, such as odor or air components (such as the smell of sea breeze, fishy smell, etc.), wind speed, temperature, humidity, etc. Therefore, the present invention provides more sensors, such as a gas sensor, a thermometer and hygrometer, a microphone, an anemometer, an altimeter, an illuminometer, etc., which are added to the camera to achieve a more complete record.

The progress of this step needs to pay attention to the sampling frequency of each sensor. For example, the GPS samples every 1 second, and the latitude and longitude information is mainly obtained. The possible sampling frequency of the frame is 30-1000 sheets per second. The odometer is 20 times per second, and the accelerometer or inclinometer is 30 times per second. Other sensors have different sampling frequencies. Basically, the sampling frequency needs to cover the bandwidth of the sensing signal. In step 3, the array data can be completed by interpolation according to the coding of the cell.

Step 2: Movie fragment encoding

Sculpt the captured film, remove the blurry or paused film segment, and set the segmentation point according to the number of consecutive frames or the intersection; if the segment is based on the number of consecutive frames, the movie segment The length is basically fixed, for example 1-10 Mbyte. The data format of the video clip is shown in Figure 2. It includes identification code (ID), clip length, movie format, name, starting point latitude and longitude coordinates, ending point latitude and longitude coordinates, identification code (ID) of the first connected video clip, and Second, the identification code (ID) of the video clip is connected. What I want to emphasize here is every film. The identification code (ID) of the segment is unique, and the encoding method can be performed by using a radio frequency identification (RFID) encoding method. With a unique identification code, you can effectively manage a large number of movie clips, no matter the tracking history of each video clip and the resume used, it is more accurate and faster.

The purpose of the real-life film segmentation is to allow users to penetrate the streets and create a road network that is close to real and two-way, providing multiplayer competition or joint participation. Since the real road network often has multiple loops, the real-time film is segmented into a fixed-length segment by using the segmentation technique, and the multi-cluster ring-view movie is self-contained, and the segment is encoded head-to-tail, that is, the first interface ID. (the identification code of the forward-connected video clip), the second interface ID (the identification code of the video clip that is connected backwards), and the clips are connected in the order of the head-to-tail encoding to handle the multiple loops connecting the branches, and then use the composite technology. In this way, the operator can make any path that selects multiple routes and establish a real road network.

Video transmissions can be downloaded via the Internet, or through a subscription or channel to provide a full-area video disc, or a partitioned video disc for users or consumers. After the movie is loaded into the memory of the computer, basically, a stack is provided by the computer to store a plurality of movie clips centered on the player's location. The stacking is sorted according to the distance from the place where the sportsman is located, wherein the film of the former interface is the first priority, the latter connection point is the second priority, and the others are prioritized in order.

The movie clips of the front and rear joints are placed on the stack. If you encounter a road, the connection point will be a scene segment of the fork. Interactive video playback or electricity According to the playback, the connection point is mainly the GPS latitude and longitude information before and after the movie clip.

In addition, another purpose of the live movie segmentation is to control the playback content of the player via a network line. If the film clip is cut into a short segment, for example, one minute, if there is no exclusive hardware decoding, the video cannot be played continuously, and the code can be encrypted in the file name, so that the average person is not easy to crack.

Step 3 3. Real-view frame coding

The format of the real image frame coding is shown in Figure 3. The frame number of each frame is that all the frames of each film segment are arranged in the order of shooting or arranged in the order of editing. Each frame is given the mileage measured by the odometer when shooting. The latitude and longitude coordinates obtained by GPS, the ground inclination measured by the inclinometer, the sound actually recorded by the microphone, or the music or sound, the wind speed and the wind direction meter, or the wind speed and direction The odor and concentration recognized or post-produced by the gas sensor, the actual measured or post-production illumination of the illuminometer, the altitude measured by the altimeter, and the temperature and humidity measured by the thermometer and hygrometer. It is necessary to pay attention to the content of the video frame here, which may be a combined image taken by at least one camera, or a panoramic image taken by a panoramic camera, or a stereoscopic image taken by a stereo camera.

For continuous frame data, it can be directly copied on a DVD disc or other storage medium such as an SD card, and other paired data is unlocked and transmitted by the network control server, so that even if No copy of the film The law is used separately, that is, the film can be copied and distributed separately, and the matching of other materials can be separated by the network or certified distribution for regulated and paid services.

The above-mentioned video data does not need to be sent out, but depends on whether the model purchased by the user has relevant equipment. When connected to the network, the server can be authenticated to judge the Internet. The equipment is equipped with items: for example, the three-axis motion platform can simulate the fluctuation and tilt of the ground; the load can be adjusted due to the ups and downs; the speaker can play the sound or soundtrack; the fan Wind speed is available; synthetic gas provides odour; adjustable light provides light and dark; sprayer provides humidity or air conditioning to provide ambient temperature and humidity.

Step 4: Use the resume record for the video clip

For the situation and record of the actual use of the present invention by the athlete, the following services can be provided: after the real road section is coded, when interacting with the activist, the traceable use history can be provided for the real scene, which is convenient for statistically popular routes and recommended routes. Or a popular route, or a variety of leaderboards, to provide a reference for the latecomers to exercise. In addition, the physical condition and physical progress of the athlete can also be recorded. For each section, the difficulty level of the individual can be recorded separately, such as the average speed, the completion time, the heartbeat, and the like. If more than one person is involved, the user ID (USER ID) can also be recorded. The privacy issue can be reduced by using the USER ID. Figure 4 shows the use of a history data format for tracking video clips using a unique encoding technique such as Radio Frequency Identification (RFID).

Since the film clip can be used with an elliptical machine, a stepper, a treadmill, a bicycle, a horse riding machine, a racing car, a rowing machine, etc., the machine ID must be provided to the central control server to judge and compare the classification separately. . The contents of the Machine ID are shown in Figure 5.

Embodiment 1 Interactive exercise

For walking, running, riding a bicycle, riding, racing, etc., the present invention can be implemented with an elliptical machine, a treadmill, a bicycle machine, a horse riding machine, a racing machine, etc., using an interface card to interact with the film, the present invention The hardware composition of the virtual reality implementation is shown in FIG. 6 , which includes the fitness equipment 11 (including speed, steering, and load); the situation simulation device 12 such as a horn, an air conditioner, an adjustable illumination, an odor generator, an electric fan, and the like; The control interface card 13 can record heartbeat, speed, steering, etc., can regulate load changes, decode play, display, provide button or touch input, and connect to a computer 15 (PC or Netbook, or Notebook or Smartbook or MID) using a USB interface. Or use a wireless interface such as Bluetooth or Wi-Fi to connect to the computer. The multi-axis motion platform 14 can provide attitude changes of 1-6 axes. The more common ones are yaw, front and rear pitch (pitch). Roll three-axis rotary motion or add X, Y, Z three-axis linear motion; DVD 16 or SD card and other data storage media to store encoded real-life movies or solid-state flash drives to store encoded real-life movies Projector Projector 17 may be provided with a large-screen projection real ratio The matching movie, of course, the projector can also be replaced by a large TV screen; the PC or Netbook, or Notebook is responsible for providing information exchange and managing interactive data files for the multiplayer competition.

Taking a bicycle as an example, the module of Figure 6 can be connected through the Internet to provide a multi-person ride experience, a multi-player experience, or an increase in various competition possibilities, such as the Tour de France, the ring race, and the town. Contest, ring county competition, etc. With the technology of the present invention, virtual competitions are easy to hold, not afraid of wind and rain, day and night can be held, safe, no traffic control, no accidents, drivers will not collide with each other, and fairness will not be cheating.

Example 2 Tourism introduction or tour

The digital content produced by the present invention is played on a tour bus in cooperation with a television or a projector, and the purpose thereof is to tunnel the street view or the scenery time, and to trace the past and the future. Because the festivals of different countries have different scenes, or there are beautiful scenery in spring, summer, autumn and winter, the tourist season may not catch up. For example, when traveling on a weekday, you can watch a scene of a street in the New Year, or play a Christmas street scene on a TV or projector. Or during the day, play a special scene with bright lights at night. Or play the scene of cherry blossoms full of streets, or snowy streetscapes, or maple red streetscapes. If you play the city before the city was rebuilt, the past street scene will let visitors know and remember the history of the tourist destination. If you play the city after the reconstruction, the future of the street scene, you can promote the tourist city and the country's vision for tourists.

This embodiment requires, in addition to the video encoding of the film that has been previously taken by the present invention. In addition, the transport vehicle carrying passengers should be equipped with a precise positioning system. The positioning system can be combined with Global Positioning System (GPS) and Geographic Information System (GIS), but its precision should be less than 0.5 meters, or added to the vehicle. An odometer is attached to the wheel and can be achieved by electromagnetic or photoelectric or Hall sensing.

Since the original coded film already has a precise positioning index, it is possible to simultaneously play the same scene as the location outside the window, but it is a different scene. Since the large-scale transportation vehicles are mostly divided into the left side and the right side, the coded film provided can be shot at least 270 degrees when shooting, and the corresponding coded movie can be compared regardless of whether the tourist sits in front, left or right.

Because the human eye has a visual persistence, the film needs at least 30 frames in 1 second, and the content of the film that people see will be smooth; therefore, the machine must be fixed every 1/30 second, and the mileage must be transmitted to the corresponding frame. To play the movie.

Embodiment 3 "Video clip inserted in advertisement"

Treating an ad clip as a movie clip also has a front-to-back connection point, so when the user arrives at the ad store or near the scenic spot, the central control server can read the disc or the ad clip into the memory, in advance On the screen and on the map, the store or scenic spots are urged to click on the user. After that, the guide arrow will appear on the screen, allowing the user to follow the arrow indicator to visit or consume. These advertisement segments are basically downloaded online. Arrows are also available online, so they are always up to date.

Real-time virtualized video player can be used to price advertisements. You can insert horse racing subtitles at any time, remind athletes to visit and purchase, and calculate the time for sportsmen to watch advertisements as the basis for collecting fees.

Further, for special attractions (culture, specialty sales, etc.) of advertising, video conferencing can be used to achieve the function of online shopping, that is, the user selects a network camera (IP Camera) and a user's own network at a special attraction. Road cameras, through SKYPE and other video conferencing software images and dialogues to interact with distant stores, such as viewing goods and inquiries, users can also step on the pedals, even WII pressure pedals, forward and left and right rotation, when If you step on the ground, you will travel in the real scene of the store; if you stop in place, it means to stay somewhere in the real scene of the store; if you turn around, the real scene will change direction. Once you want to shop, you can stop and watch the products and inquiry of distant stores through the video and dialogue of video conferencing software such as SKYPE. In this way, users can go shopping at home at any time through the system.

Embodiment 4 Diversified multimedia geography

National Geographic magazine is shooting local customs and does not provide other information about the location, such as world coordinates, smell or air components (such as the smell of sea breeze, fishy smell, etc.), wind speed, temperature, humidity, etc., so it is not complete. Record. If we provide more sensors, such as gas sensors, thermometers and hygrometers, anemometers, GPS, altimeters, etc., add to the camera, you can achieve a more complete record. If you can shoot multiple times in different seasons or weather, this is inevitable. It can make the record more comprehensive, and at the same time make the interactive tour more realistic and people can experience it more deeply.

In addition, the present invention records the multi-path landscape, which allows the user to have fun of re-exploration. For example, for most people, the place to go abroad, because of the limited time, is often a horse to see flowers, if you want to go back after going home, The aftertaste or more in-depth understanding can be accomplished by the techniques of the present invention, while rejuvenating while moving, or by taking some of the strokes that are too late to finish, which can be accomplished using the techniques of the present invention. In this way, the world will narrow its distance and people will be closer.

Example 5 Previewing the new environment

For a new city, you can simulate it before you go, which can reduce the strangeness. In addition, if you want to move to a new city, you can use the technology of the present invention to achieve a faster familiarity with the new city.

1‧‧‧Step 1 - Filming

2‧‧‧Step 2 - Film Fragment Coding

3‧‧‧Step 3 - Frame Coding

4‧‧‧Step 4 - Recording history of film clips

11‧‧‧Fitness equipment (including speed, steering gear, load)

12‧‧‧ Situational Simulator

13‧‧‧Control interface card

14‧‧‧Multi-axis motion platform

15‧‧‧ computer

16‧‧‧DVD storage of encoded live video

17‧‧‧Projector (Projector)

18‧‧‧Central Control Server Computer

Figure 1 Real virtual video format

Figure 2 Real-life virtual video clip data format generated using a unique encoding technique such as Radio Frequency Identification (RFID)

Figure 3: Format data format for real-world virtualization

Figure 4. Data format for tracking the use of video clips using unique encoding techniques such as Radio Frequency Identification (RFID)

Figure 5 Real virtual frame data format

Figure 6 The hardware composition of the real virtual implementation of the present invention

1‧‧‧Step 1 - Filming

2‧‧‧Step 2 - Film Fragment Coding

3‧‧‧Step 3 - Frame Coding

4‧‧‧Step 4 - Recording history of film clips

Claims (16)

A method for encoding real-life multimedia to achieve real-time multimedia playback and interactive action of the interacter, the steps including step one, film shooting, real-life video and corresponding geographic/environmental sensing information, including a frame, each 30-1000 seconds, corresponding to the latitude and longitude coordinates, mileage, ground or water surface inclination, and the value of each environmental sensor; Step 2, the video clip code, the real-life movie is cut into video clips according to the continuous frame The number or the intersection setting segmentation point; Step 3, frame coding, each of the coded film segments is given the mileage measured by the odometer and the corresponding geographic/environmental sensing information when interpolating or tween shot respectively. In order, the array data file with the positioning index is sequentially changed; in step 4, the real-time multimedia playback and the interaction of the interaction are synchronized, and the mileage generated by the action of the interactor is obtained in each fixed sampling period, and the array data file of the third step is obtained. Find the frame corresponding to the mileage and play it. The method of claim 1, further comprising the step of: using a history record of the video clip, the history record comprising a user identification code (USER ID), a status and a record of the user using the real scene, a popular route and a recommended route or The statistics of popular routes, or various rankings, provide reference for the latecomers to exercise; in addition, they can also record the physical condition and physical progress of the athletes. For each section, the difficulty level of the individual can be recorded separately, including the average speed. Completion time, heartbeat, etc. The method of claim 1, wherein the odometer has a precision of less than 0.5 meters, and the wheel attached to the live-viewing vehicle can be achieved by electromagnetic or photoelectric or Hall sensing. The method of claim 1, wherein the environmental sensor comprises at least one of a gas sensor, a temperature and humidity meter, an anemometer, an illuminometer, an altimeter, and the like. The method of claim 1, wherein the interactor comprises an appliance operated by a user or a user. The method of claim 5, wherein the user-operated appliance refers to at least one of an elliptical machine, a treadmill, a bicycle machine, a horse riding machine, a racing machine, a rowing machine, a ski machine, a glider, and the like. The method of claim 1, wherein the synchronization of the live video playback and the interaction of the interactive action is performed by using a control interface card to allow the user to operate the device to interact with the video playback system, and the original encoded video is already available. Accurate mileage positioning index, so you can simultaneously play the same scene as the location of the interactive action, and the control interface card can record the user's heartbeat, speed, steering, etc., can adjust the load change, decode the play, display, provide the button or Touch input. The method of claim 7, wherein each of the video playback systems is further connected to the central control server computer to provide information exchange and control interactive data files for the multiplayer competition, and live video of the respective users. Synchronization of play and individual user actions, still using a control interface at each user The card allows the user to operate the device to interact with the video playback system. The original coded video has an accurate mileage positioning index, and the same scenes corresponding to the locations where the respective user actions generate the mileage are synchronized. The method of claim 1, wherein the film segment coding uses a segmentation and synthesis technique to encode the live video clips in a head-to-tail format, and the segments are connected in a head-to-tail coding sequence to handle the connection of the multiple loops. The person chooses any path that selects multiple routes and establishes a real road network. A method for encoding real-life multimedia, so that the real-life multimedia becomes a tourism introduction or navigation multimedia geographic information, and the television or projector plays the real-life scene on the transportation vehicle carrying the passenger, and the video coding that has been taken in advance is required. The transport vehicle carrying passengers should be equipped with a precision positioning system. The positioning system can be combined with Global Positioning System (GPS) and Geographic Information System (GIS). The precision should be less than 0.5 meters, or an odometer can be installed on the vehicle. The odometer is attached to the wheel and can be achieved by electromagnetic or photoelectric or Hall sensing. Since the original coded film has accurate positioning index, it can simultaneously play the same scene as the location outside the window. But it is a different scene. The method of claim 10, wherein the encoded content comprises: a filming, obtaining a real-life film and corresponding geographic and environmental sensing information; and second, the film segment encoding, the sliced film segment is based on the number of consecutive frames. Or the intersection is used as a segmentation point; third, the frame coding, the film frames and the corresponding geographic and environmental sensing information are sequentially changed into an array data file. The method of claim 10, wherein the passenger is mostly divided into left and right ride vehicles, and the provided coded film is photographed at least 270 degrees when photographed, regardless of whether the passenger sits in front, left, or right. The encoded video can be viewed against the video. A virtual interactive system with real-life multimedia, comprising a coded real-life multimedia, including a frame, 30-1000 frames per second, corresponding latitude and longitude coordinates, mileage, ground or water surface inclination, and various environmental senses The value of the detector, the number of miles corresponding to each frame of the real scene should be within 0.5 meters; the hardware device interacting with the real scene multimedia includes fitness equipment, which transmits the user's sports conditions such as heartbeat, speed, steering, etc. The control interface card is received, and the load control signal provided by the control interface card is received; the situation simulation device implements the temperature, humidity, odor, wind speed, illumination, sound and the like in the encoded real-life multimedia; the computer has a network function. Connect with the control interface card to play the encoded live video; control the interface card to record the user's heartbeat, speed, steering, etc., can adjust the load change, decode the play, display, provide the button or touch input, and use the serial Or USB interface or wireless interface such as Bluetooth or Wi-Fi interface to the computer; display device, display live video . Such as the system of patent scope 13, wherein the fitness equipment is at least an elliptical machine, running One of the fitness equipment such as step machine, bicycle machine, horse riding machine, racing machine, rowing machine skiing gliding wing. For example, the system of Patent Application No. 13 further includes a multi-axis motion platform for riding exercise equipment to provide a posture change of 1-6 axes, at least yaw, front and rear pitch (pitch) and roll (roll) three axes. One of the linear motions of the rotary motion and the X, Y, and Z axes. For example, the system of patent scope 13 is further added to the central control server computer, which is responsible for providing information exchange and managing interactive data files for the multiplayer connection competition.