KR20140009913A - Method and appratus for processing virtual world - Google Patents

Abstract

A virtual world processing apparatus and a method thereof are disclosed. According to embodiments, an interaction between a real world and a virtual world can be realized by delivering sensed information on the photographed image of the real world to the virtual world by using the characteristics of an image sensor which is information about the characteristics of the image sensor. [Reference numerals] (321) Reception unit; (322) Processing unit; (323) Transmission unit

Description

METHOD AND APPRATUS FOR PROCESSING VIRTUAL WORLD}

The following embodiments relate to an apparatus and a method for processing a virtual world, and more particularly, to an apparatus and a method for applying sensing information measured by an image sensor to a virtual world.

Recently, interest in immersive games has increased. At the "E3 2009" Press Conference, Microsoft combined its sensor console, Xbox360, with a separate sensor device consisting of a Depth / Color camera and microphone array, providing users' full-body motion capturing, facial recognition, and speech recognition technologies. Introduced "Project Natal", which allows users to interact with virtual worlds without a controller. In addition, Sony applied position / direction sensing technology that combined color camera, marker, and ultrasonic sensor to Play Station3, its game console, so that the user can interact with the virtual world by inputting the motion trajectory of the controller. Wand "was released.

The interaction between the real world and the virtual world has two directions. The first is to reflect the data information obtained from the sensor of the real world to the virtual world, and the second is to reflect the data information obtained from the virtual world to the real world through an actuator.

In the present specification, a new method for an apparatus and method for applying information sensed from a real world using an environmental sensor to a virtual world is presented.

According to one or more exemplary embodiments, an apparatus for processing a virtual world may include: a receiver configured to receive sensing information of a captured image and sensor characteristics of characteristics of the image sensor from an image sensor; A processor configured to generate control information for controlling an object of a virtual world based on the sensing information and the sensor characteristic; And a transmission unit transmitting the control information to the virtual world.

According to another aspect of the present invention, there is provided a method of processing a virtual world, the method comprising: receiving sensor information on a characteristic of the image sensor and sensing information about a captured image from an image sensor; Generating control information for controlling an object of a virtual world based on the sensing information and the sensor characteristic; And transmitting the control information to the virtual world.

1 is a diagram illustrating a virtual world processing system that controls information exchange between a real world and a virtual world, according to an exemplary embodiment.
2 is a diagram for describing an augmented reality system, according to an exemplary embodiment.
3 is a diagram illustrating a configuration of a virtual world processing apparatus according to an exemplary embodiment.
4 is a flowchart illustrating a virtual world processing method according to an exemplary embodiment.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. Like reference symbols in the drawings denote like elements.

1 is a diagram illustrating a virtual world processing system that controls information exchange between a real world and a virtual world, according to an exemplary embodiment.

Referring to FIG. 1, a virtual world processing system according to an embodiment of the present invention may include a real world 110, a virtual world processing apparatus, and a virtual world 140.

The real world 110 may represent a sensor that senses information about the real world 110 or a sensory device that implements information about the virtual world 140 in the real world 110.

In addition, the virtual world 140 may represent a sensory media playback device that plays content including sensory effect information that may be implemented in the virtual world 140 itself or in the real world 110 implemented by a program.

According to an embodiment, a sensor may sense and transmit information on a user's motion, state, intention, shape, etc. of the real world 110 to the virtual world processing device.

According to an embodiment, the sensor may transmit a sensor capability 101, a sensor adaptation preference 102, and a sensed information 103 to the virtual world processing device.

The sensor characteristic 101 is information about the characteristic of the sensor. Sensor adaptation preference 102 is information indicative of the degree to which a user of a sensor prefers to the characteristics of the sensor. The sensing information 103 is information obtained by the sensor detecting the real world 110.

The virtual world processing apparatus according to an embodiment may include an adaptation adaptation real world to virtual world (120), a virtual world information (VWI) 104, and an adaptation adaptation real world to virtual world / Virtual World to Real World) 130.

The adaptive RV 120 is information that can apply the sensing information 103 detected by the sensor to the virtual world 140 based on the sensor characteristic 101 and the sensor adaptation preference 102 to the virtual world 140. I can convert it. According to an embodiment, the adaptive RV 120 may be implemented as a real world to virtual world engine (RV engine).

According to an embodiment, the adaptive RV 120 may convert the virtual world information (VWI) 104 using the converted sense information 103.

The VWI 104 is information about a virtual object of the virtual world 140.

The adaptive RV / VR 130 encodes the transformed VWI 104 to generate a Virtual World Effect Metadata (VWEM) 107 that is metadata about the effect applied to the virtual world 140. Can be. According to an embodiment, adaptive RV / VR 130 generates VWEM 107 based on Virtual World Capabilities (VWC) 105 and Virtual World Preferences (VWP) 106. can do.

The VWC 105 is information about the characteristics of the virtual world 140. In addition, the VWP 106 is information indicating the degree of preference of the user for the characteristics of the virtual world 140.

In addition, adaptive RV / VR 130 may send VWEM 107 to virtual world 140. At this time, the virtual world 140 is applied to the VWEM 107, so that an effect corresponding to the sensing information 103 may be implemented in the virtual world 140.

According to one side of the present invention, the effect event occurring in the virtual world 140 may be driven by the sensor device which is an actuator in the real world 110.

The virtual world 140 may generate sensory effect metadata (SEM) 111 by encoding sensory effect information, which is information about an effect event occurring in the virtual world 140. According to an embodiment, the virtual world 140 may include a sensory media playback device for playing content including sensory effect information.

The adaptive RV / VR 130 may generate sensory information 112 based on the SEM 111. Sensory information 112 is information about an effect event implemented in the sensory device of the real world 110.

The adaptive VR 150 may generate information about a sensory device command (SDCmd) 115 that controls the operation of the sensory device of the real world 110. According to an embodiment, adaptive VR 150 may use SDCmd () based on information on Sensory Device Capabilities (SDCap) 113 and information on User Sensory Preference (USP) 114. Information may be generated.

The SDCap 113 is information on characteristics of the sensory device. In addition, the USP 114 is information indicating the degree of preference of the user for the effect implemented in the sensory device.

2 is a diagram for describing an augmented reality system, according to an exemplary embodiment.

Referring to FIG. 2, the augmented reality system according to an exemplary embodiment may acquire an image representing the real world by using the media storage device 210 or the real-time media acquisition device 220. Also, the augmented reality system may acquire sensor information representing the real world using various sensors 230.

Here, the AR camera according to an embodiment may include a real-time media acquisition device 220 and various sensors 230. The augmented reality camera may acquire image or sensor information representing the real world for mixing between the real world information and the virtual object.

The AR container 240 is a device having not only real-world information but also information on a mixing method between the real world and a virtual object. For example, the augmented reality container 240 may include information about which virtual object to mix with which of the real world information at any point in time.

The augmented reality container 240 may request virtual object information from the AR content 250 based on information about a mixing method between the real world and the virtual object. Here, the augmented reality content 250 is a device that includes virtual object information.

The augmented reality content 250 may return virtual object information corresponding to a request of the augmented reality container 240. In this case, the virtual object information may be expressed based on at least one of three-dimensional graphics, audio, video, or text representing the virtual object. In addition, the virtual object information may also include an interaction between a plurality of virtual objects.

The visualization unit 260 may simultaneously visualize real-world information included in the augmented reality container 240 and virtual object information included in the augmented reality content 250. In this case, the interaction unit 270 may provide an interface through which the user can interact with the virtual object through the visualized information. In addition, the interaction unit 270 may update the virtual object by the interaction between the user and the virtual object, or may update the mixing method between the real world and the virtual object.

Hereinafter, a configuration of a virtual world processing apparatus according to an embodiment will be described in detail with reference to FIG. 3.

3 is a diagram illustrating a configuration of a virtual world processing apparatus according to an exemplary embodiment.

Referring to FIG. 3, the virtual world processing apparatus 320 according to an embodiment includes a receiver 321, a processor 322, and a transmitter 323.

The receiver 321 may receive the sensor information about the characteristic of the image sensor 311 and the sensing information for the captured image 315 from the image sensor 311. Here, the image sensor 311 may capture a still image, and may capture a video. For example, the image sensor 311 may include at least one of a photographing sensor and a video sensor.

The processor 322 generates control information for controlling an object of the virtual world based on the sensing information and the sensor characteristic. For example, the processor 332 may generate control information when a value related to a specific element included in the sensing information is within an allowable range defined by the sensor characteristic.

The transmitter 323 transmits control information to the virtual world.

At this time, the operation of the virtual world may be controlled based on the received control information.

For example, assume that the image sensor 311 is an augmented reality camera. The image sensor 311 may acquire the captured image 315 by photographing the real world 310. The image sensor 311 may extract the plurality of feature points 316 included in the captured image 315 by analyzing the captured image 315.

Here, the feature point may be mainly extracted from the boundary surfaces included in the captured image 315, and may be expressed in three-dimensional coordinates.

In some cases, the image sensor 311 may first extract feature points 316 related to the boundary of the closest object or the boundary of the largest object among the boundary surfaces included in the captured image 315.

The image sensor 311 may transmit the sensing information including the extracted plurality of feature points to the virtual world processing apparatus 320.

The virtual world processing apparatus 320 may extract a feature point from the sensing information transmitted from the image sensor 311, and generate control information including the extracted feature point.

Therefore, the virtual world processing apparatus 320 according to an embodiment may generate a control signal for the virtual scene 330 corresponding to the real world 310 using only a small amount of information (for example, a feature point). Can be.

In this case, the virtual world may control the virtual object based on the plurality of feature points included in the control information.

More specifically, the virtual world may represent the virtual scene 330 corresponding to the real world 310 based on the plurality of feature points. In this case, the virtual scene 330 may be represented in three-dimensional space. The virtual world may represent a plane for the virtual scene 330 based on the feature points.

In addition, the virtual world may simultaneously display the virtual scene 330 and the virtual objects 331 corresponding to the real world 310.

According to an embodiment, the sensing information and sensor characteristics received from the image sensor 311 may correspond to the SI 103 and the SC 101 of FIG. 1, respectively.

For example, the sensing information received from the image sensor 311 may be defined as shown in Table 1.

Sensed Information (SI, 103)

-Camera sensor type

-AR Camera type

Here, the AR camera type may basically include a camera sensor type. The camera sensor type may include an element of a resource element, a camera location element, and a camera orientation element, and an attribute of a focal length attribute, an aperture attribute, a shutter speed attribute, and a filter attribute.

At this time, the resource element includes a link to the image captured by the image sensor, the camera location element includes information related to the position of the image sensor measured using the Global Position System (GPS) sensor, and the camera orientation element May include information related to the attitude of the image sensor.

The focal length attribute includes information related to the focal length of the image sensor, the aperture attribute includes information related to the aperture of the image sensor, the shutter speed attribute includes information related to the shutter speed of the image sensor, and the filter attribute includes the image It may include information related to the filter signal processing of the sensor. Here, the filter type may include a UV filter, a polarizing light filter, a neutral density filter, a diffusion filter, a star filter, and the like.

In addition, the augmented reality camera type may further include a feature element and a camera position element.

In this case, the feature element may include a feature point related to the boundary surface in the captured image, and the camera position element may include information related to the position of the image sensor measured using a position sensor that is distinguished from the GPS sensor.

As described above, the feature point is a point mainly generated at boundary surfaces in the image photographed by the image sensor, and may be used to represent a virtual object in an augmented reality environment. More specifically, a feature element including at least one feature point may be used as an element representing a face by a scene descriptor. More details related to the operation of the scene description will be described later.

The camera position element may be utilized to measure the position of the image sensor in a room or a tunnel where it is difficult to measure the position using the GPS sensor.

On the other hand, the sensor characteristics received from the image sensor 311 may be defined as shown in Table 2.

Sensor Capability (SC, 101)

-Camera sensor capability type

AR Camera capability type

Here, the AR camera capability type may basically include a camera sensor capability type. The camera sensor characteristic type may consist of a Supported Resolution List element, a focal length range element, an aperture range element and a shutter speed range element.

In this case, the supported resolution list element includes a list of resolutions supported by the image sensor, the focal length range element includes a range of focal lengths supported by the image sensor, and the aperture range element is an aperture range supported by the image sensor The shutter speed range element may include a range of shutter speeds supported by the image sensor.

In addition, the augmented reality camera characteristic type may further include a maximum feature point element and a camera position range element.

At this time, the maximum feature point element may include the maximum number of feature points that may be detected by the image sensor, and the camera position range element may include a range of positions that may be measured by the position sensor.

Table 3 shows an XML syntax (eXtensible Markup Language Syntax) for a camera sensor type according to an embodiment.

<! - ##################################################### ## ->
<!-Camera Sensor Type->
<! - ##################################################### ## ->
<complexType name = "CameraSensorType">
<complexContent>
<extension base = "iidl: SensedInfoBaseType">
<sequence>
<element name = "Resource" type = "anyURI"/>
<element name = "CameraOrientation" type = "siv: OrientationSensorType" minOccurs = "0"/>
<element name = "CameraLocation" type = "siv: GlobalPositionSensorType" minOccurs = "0"/>
</ sequence>
<attribute name = "focalLength" type = "float" use = "optional"/>
<attribute name = "aperture" type = "float" use = "optional"/>
<attribute name = "shutterSpeed" type = "float" use = "optional"/>
<attribute name = "filter" type = "mpeg7: termReferenceType" use = "optional"/>
</ extension>
</ complexContent>
</ complexType>

Table 4 shows semantics for a camera sensor type according to one embodiment.

Name Definition CameraSensorType Tool for describing sensed information with respect to a camera sensor. Resource Describes the element that contains a link to image or video files. CameraLocation Describes the location of a camera using the structure defined by GlobalPositionSensorType. CameraOrientation Describes the orientation of a camera using the structure defined by OrientationSensorType. focalLength Describes the distance between the lens and the image sensor when the subject is in focus, in terms of millimeters (mm). aperture Describes the diameter of the lens opening. It is expressed as F-stop, e.g. F2.8. It may also be expressed as f-number notation such as f / 2.8. shutterSpeed Describes the time that the shutter remains open when taking a photograph in terms of seconds (sec). filter Describes kinds of camera filters as a reference to a classification scheme term that shall be using the mpeg7: termReferenceType defined in 7.6 of ISO / IEC 15938-5: 2003. The CS that may be used for this purpose is the CameraFilterTypeCS defined in A.x.x.

Table 5 shows XML syntax for a camera sensor characteristic type according to one embodiment.

<! - ##################################################### ## ->
<!-Camera Sensor capability type->
<! - ##################################################### ## ->
<complexType name = "CameraSensorCapabilityType">
<complexContent>
<extension base = "cidl: SensorCapabilityBaseType">
<sequence>
<element name = "SupportedResolutions" type = "scdv: ResolutionListType" minOccurs = "0"/>
<element name = "FocalLengthRange" type = "scdv: ValueRangeType" minOccurs = "0"/>
<element name = "ApertureRange" type = "scdv: ValueRangeType" minOccurs = "0"/>
<element name = "ShutterSpeedRange" type = "scdv: ValueRangeType" minOccurs = "0"/>
</ sequence>
</ extension>
</ complexContent>
</ complexType>

<complexType name = "ResolutionListType">
<sequence>
<element name = "Resolution" type = "scdv: ResolutionType" maxOccurs = "unbounded"/>
</ sequence>
</ complexType>

<complexType name = "ResolutionType">
<sequence>
<element name = "Width" type = "nonNegativeInteger"/>
<element name = "Height" type = "nonNegativeInteger"/>
</ sequence>
</ complexType>

<complexType name = "ValueRangeType">
<sequence>
<element name = "MaxValue" type = "float"/>
<element name = "MinValue" type = "float"/>
</ sequence>
</ complexType>

Table 6 shows the semantics for the camera sensor characteristic type according to one embodiment.

Name Definition CameraSensorCapabilityType Tool for describing a camera sensor capability. SupportedResolutions Describes a list of resolution that the camera can support. ResolutionListType Describes a type of the resolution list which is composed of ResolutionType element. ResolutionType Describes a type of resolution which is composed of Width element and Height element. Width Describes a width of resolution that the camera can perceive. Height Describes a height of resolution that the camera can perceive Focallengrange Describes the range of the focal length that the camera sensor can perceive in terms of ValueRangeType. Its default unit is millimeters (mm).
NOTE The minValue and the maxValue in the SensorCapabilityBaseType are not used for this sensor. ValueRangeType Defines the range of the value that the sensor can perceive. MaxValue Describes the maximum value that the sensor can perceive. MinValue Describes the minimum value that the sensor can perceive. ApertureRange Describes the range of the aperture that the camera sensor can perceive in terms of valueRangeType.
NOTE The minValue and the maxValue in the SensorCapabilityBaseType are not used for this sensor. ShutterSpeedRange Describes the range of the shutter speed that the camera sensor can perceive in terms of valueRangeType. Its default unit is seconds (sec).
NOTE The minValue and the maxValue in the SensorCapabilityBaseType are not used for this sensor.

Table 7 shows XML syntax for an augmented reality camera type according to one embodiment.

<! - ##################################################### ## ->
<!-AR Camera Type->
<! - ##################################################### ## ->
<complexType name = "ARCameraType">
<complexContent>
<extension base = "siv: CameraSensorType">
<sequence>
<element name = "Feature" type = "siv: FeaturePointType" minOccurs = "0" maxOccurs = "unbounded"/>
<element name = "CameraPosition" type = "siv: PositionSensorType" minOccurs = "0"/>
</ sequence>
</ extension>
</ complexContent>
</ complexType>

<complexType name = "FeaturePointType">
<sequence>
<element name = "Position" type = "mpegvct: Float3DVectorType"/>
</ sequence>
<attribute name = "featureID" type = "ID" use = "optional"/>
</ complexType>

Table 8 shows semantics for the augmented reality camera type according to one embodiment.

Name Definition ARCameraType Tool for describing sensed information with respect to an AR camera. Feature Describes the feature detected by a camera using the structure defined by FeaturePointType. FeaturePointType Tool for describing Feature commands for each feature point. Position Describes the 3D position of each of the feature points. featureID To be used to identify each feature. CameraPosition Describes the location of a camera using the structure defined by PositionSensorType.

Table 9 shows XML syntax for an augmented reality camera characteristic type according to one embodiment.

<! - ##################################################### ## ->
<!-AR Camera capability type->
<! - ##################################################### ## ->
<complexType name = "ARCameraCapabilityType">
<complexContent>
<extension base = "siv: CameraSensorCapabilityType">
<sequence>
<element name = "MaxFeaturePoint" type = "nonNegativeInteger" minOccurs = "0"/>
<element name = "CameraPositionRange" type = "scdv: RangeType" minOccurs = "0"/>
</ sequence>
</ extension>
</ complexContent>
</ complexType>

Table 10 shows semantics for the augmented reality camera characteristic type according to one embodiment.

Name Definition ARCameraCapabilityType Tool for describing an AR camera capability. MaxFeaturePoint Describes the maximum number of feature points that the camera can detect. CameraPositionRange Describes the range that the position sensor can perceive in terms of RangeType in its global coordinate system.
NOTE The minValue and the maxValue in the SensorCapabilityBaseType are not used for this sensor.

Table 11 shows XML syntax for a scene descriptor type according to an embodiment.

<!-############################################## #############->
<!-Scene Descriptor Type->
<!-############################################## #############->
<complexType name = "SceneDescriptorType">
<sequence>
<element name = "image" type = "anyURI"/>
</ sequence>
<complexType name = "plan">
<sequence>
<element name = "ID" type = "int32"/>
<element name = "X" type = "float"/>
<element name = "Y" type = "float"/>
<element name = "Z" type = "float"/>
<element name = "Scalar" type = "float"/>
</ sequence>
</ complexType>
<complexType name = "feature">
<sequence>
<element name = "ID" type = "int32"/>
<element name = "X" type = "float"/>
<element name = "Y" type = "float"/>
<element name = "N" type = "float"/>
</ sequence>
<complexType>
</ complexType>

Here, the image element included in the scene descriptor type may include a plurality of pixels. The plurality of pixels may depict an ID of a plan or an ID of a feature.

Where the plan is X _plan , Y _plan , Z _plan And Scalar, and referring to Equation 1, the scene descriptor may represent a face using a face equation including X _plan , Y _plan , and Z _plan .

In addition, the feature is a type corresponding to a feature element included in the sensing information, and the feature may include an X _feature , a Y _feature, and a Z _feature . In this case, the feature may represent three-dimensional points (X _feature , Y _feature , Z _feature ), and the scene descriptor may represent a face using three-dimensional points located at (X _feature , Y _feature , Z _feature ). .

4 is a flowchart illustrating a virtual world processing method according to an exemplary embodiment.

Referring to FIG. 4, the virtual world processing method according to an exemplary embodiment receives sensing information on a captured image and sensor characteristics on characteristics of an image sensor from an image sensor (410).

The virtual world processing method generates control information for controlling an object of the virtual world based on the sensing information and the sensor characteristic (420).

The virtual world processing method transmits control information to the virtual world (430).

At this time, the operation of the virtual world may be controlled based on the received control information. Since the details described with reference to FIGS. 1 through 3 may be applied to each step illustrated in FIG. 4, a detailed description thereof will be omitted.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

A receiver configured to receive sensing information of a captured image and sensor characteristics of the characteristics of the image sensor from an image sensor;
A processor configured to generate control information for controlling an object of a virtual world based on the sensing information and the sensor characteristic; And
Transmission unit for transmitting the control information to the virtual world
Virtual world processing device comprising a. The method of claim 1,
The image sensor
And at least one of a photographing sensor and a video photographing sensor. The method of claim 1,
The detection information is
A resource element comprising a link to an image captured by the image sensor;
A camera location element containing information related to the position of the image sensor measured using a Global Position System (GPS) sensor; And
Camera orientation element containing information related to the pose of the image sensor
Virtual world processing device comprising a. The method of claim 1,
The detection information is
A focal length attribute including information related to a focal length of the image sensor;
An aperture property including information related to an aperture of the image sensor;
A shutter speed attribute including information related to a shutter speed of the image sensor; And
Filter attributes including information related to filter signal processing of the image sensor
Virtual world processing device comprising a. The method of claim 3,
The detection information is
A feature element comprising a feature point associated with an interface in the captured image; And
A camera position element comprising information relating to the position of the image sensor measured using a position sensor distinct from the GPS sensor
The virtual world processing device further comprising. The method of claim 1,
The sensor characteristics
A Supported Resolution List element containing a list of resolutions supported by the image sensor;
A focal length range element including a range of focal lengths supported by the image sensor;
An aperture range element including an aperture range supported by the image sensor; And
Shutter speed range element including a range of shutter speeds supported by the image sensor
Virtual world processing device comprising a. The method according to claim 6,
The sensor characteristics
A maximum feature point element comprising a maximum number of feature points that can be detected by the image sensor; And
Camera position range element containing the range of positions that can be measured by the position sensor
The virtual world processing device further comprising. The method of claim 1,
The processor extracts at least one feature point included in the captured image from the sensing information,
The transmission unit transmits the at least one feature point to the virtual world.
And the virtual world represents at least one plane included in the virtual world based on the at least one feature point. Receiving sensor information on a characteristic of the image sensor and sensing information about a captured image from an image sensor;
Generating control information for controlling an object of a virtual world based on the sensing information and the sensor characteristic; And
Transmitting the control information to the virtual world
Virtual world processing method comprising a. A computer-readable recording medium having recorded thereon a program for executing the method of claim 9.

Images