CN115116415A - Virtual musical instrument playing method, virtual musical instrument playing device and electronic equipment - Google Patents

Abstract

The application discloses a virtual musical instrument playing method, a virtual musical instrument playing device and electronic equipment, and belongs to the technical field of terminals. The virtual musical instrument playing method is applied to an electronic device including a real-feeling pixel sensor, and includes: acquiring motion information of a main control object relative to a controlled object in a target interface through a real sensing pixel sensor; playing the target audio based on the motion information; wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different sound parts of the same musical instrument; the target audio includes audio corresponding to at least one musical tone of at least one musical instrument.

Description

Virtual musical instrument performance method, virtual musical instrument performance device, and electronic device

technical field

The present application belongs to the technical field of terminals, and in particular relates to a method for playing a virtual musical instrument, a device for playing a virtual musical instrument, and an electronic device.

Background technique

Music is a part of the human soul, and musical instruments are the carrier of music. However, for some large-sized musical instruments, such as pianos, drum sets, etc., users cannot carry them with them, so that users cannot play musical instruments anytime and anywhere, causing great trouble to many music lovers.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to provide a method for playing a virtual musical instrument, a device for playing a virtual musical instrument, and an electronic device, which can realize the virtual musical instrument function of the electronic device, so that the user can play the musical instrument anytime and anywhere.

In a first aspect, an embodiment of the present application provides a method for playing a virtual musical instrument, which is applied to an electronic device including a RealSense pixel sensor. Motion information; based on the motion information, the target audio is played; wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument; the target audio includes at least one musical tone of at least one musical instrument corresponding to 's audio.

In a second aspect, an embodiment of the present application provides a virtual musical instrument performance device, which is applied to an electronic device including a real-sensing pixel sensor. The device includes: an acquisition unit configured to obtain, through the real-sensing pixel sensor, the relative relative value of a master control object in a target interface. the motion information of the controlled object; the playback unit is used to play the target audio based on the motion information; wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument; target The audio includes audio corresponding to at least one tone of at least one musical instrument.

In a third aspect, an embodiment of the present application provides an electronic device, the electronic device includes a processor and a memory, the memory stores a program or an instruction that can be executed on the processor, and the program or instruction is executed by the processor to implement the first aspect The steps of the virtual instrument playing method.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, where a program or an instruction is stored thereon, and when the program or instruction is executed by a processor, the steps of the virtual musical instrument performance method according to the first aspect are implemented.

In a fifth aspect, an embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or an instruction to implement the steps of the virtual musical instrument performance method of the first aspect .

In a sixth aspect, an embodiment of the present application provides a computer program product, where the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the virtual musical instrument performance method of the first aspect.

In the virtual musical instrument performance method provided by the embodiment of the present application, for the electronic device including the real-sensing pixel sensor, the motion information of the master object relative to the controlled object in the target interface of the electronic device is obtained through the real-sensing pixel sensor. Wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument. On this basis, the target audio is played according to the motion information of the master object relative to the controlled object, and the target audio includes audio corresponding to at least one musical tone of at least one musical instrument. In this way, the human-computer interaction mode of the electronic device is added, and the virtual musical instrument can be played through the motion information of the master object in the target interface of the electronic device relative to the controlled object, thereby realizing the virtual musical instrument function of the electronic device, thereby Allows users to play musical instruments anytime, anywhere.

Description of drawings

1 is a schematic flowchart of a virtual musical instrument performance method provided by an embodiment of the present application;

Fig. 2 is the structural representation of the drum kit provided by the embodiment of this application;

3 is a schematic diagram of marking a controlled object provided by an embodiment of the present application;

4 is a circuit diagram of an RGB pixel provided by an embodiment of the present application;

FIG. 5 is a circuit diagram of a real-sensing pixel provided by an embodiment of the present application;

6 is a schematic diagram of a comparison between RGB pixels and RealSense pixels provided by an embodiment of the present application;

7 is one of the schematic diagrams of the pixel layout of the RealSense pixel sensor provided by the embodiment of the present application;

FIG. 8 is the second schematic diagram of the pixel layout of the RealSense pixel sensor provided by the embodiment of the present application;

FIG. 9 is a circuit diagram of a real-sensing pixel sensor provided by an embodiment of the present application;

10 is a structural block diagram of a virtual musical instrument performance device provided by an embodiment of the application;

11 is a structural block diagram of an electronic device provided by an embodiment of the present application;

FIG. 12 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art fall within the protection scope of this application.

The terms "first", "second" and the like in the description and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in sequences other than those illustrated or described herein, and distinguish between "first", "second", etc. The objects are usually of one type, and the number of objects is not limited. For example, the first object may be one or more than one. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the associated objects are in an "or" relationship.

The embodiment of the first aspect of the present application proposes a virtual musical instrument performance method. The execution subject of the technical solution of the virtual musical instrument performance method provided by the embodiment of the present application may be a virtual musical instrument performance device, which may be determined according to actual use requirements. The embodiment is not limited. In order to describe the virtual musical instrument performance method provided by the embodiments of the present application more clearly, in the following method embodiments, the execution subject of the virtual musical instrument performance method is exemplarily described as a virtual musical instrument performance apparatus.

The virtual musical instrument performance method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios thereof.

As shown in FIG. 1 , an embodiment of the present application provides a method for playing a virtual musical instrument, which is applied to an electronic device including a RealSense pixel sensor. The method may include the following steps 102 and 104:

Step 102 : obtain motion information of the master object in the target interface relative to the controlled object through the real-sensing pixel sensor.

Wherein, the above-mentioned electronic device may be a smart device with a camera, such as a smartphone and a tablet computer, and the above-mentioned electronic device may also be a smart electronic device with an AR (Augmented Reality, augmented reality) function, such as a smartphone, a tablet that supports the AR function Computers, AR glasses and other smart devices. The specific form of the above electronic device can be set by the user according to the actual situation, which is not specifically limited here.

Further, the above-mentioned target interface is an imaging interface of a real scene in an electronic device. Specifically, when the electronic device is a smart device with a photographing device, the target interface is a photographing preview interface of the electronic device. In the case where the electronic device is a smart device with AR function, the target interface is a virtual display interface of the electronic device, that is, an AR imaging interface. The specific form of the above target interface can be limited according to the specific situation of the electronic device used, and is not specifically limited here.

Further, the above-mentioned master object is a moving object, and the above-mentioned controlled object is a stationary object. During the performance of the virtual musical instrument, the above-mentioned master object is the object that controls the virtual musical instrument to sound, and the above-mentioned controlled object is the object associated with the virtual musical instrument. voice.

Specifically, the above-mentioned main control object may include objects that the user can move, such as chopsticks, metal bars, etc., and the above-mentioned main control object may also include the user's hand. The above-mentioned controlled objects include objects other than physical musical instruments, such as wooden blocks, planks, discs, etc., and the above-mentioned controlled objects may also include interface areas in the target interface, such as multiple planes such as desktop or ground in the target interface. Divide the area.

The number of the above-mentioned controlled objects may be multiple, and the multiple controlled objects correspond to multiple interface areas or multiple objects in the above-mentioned target interface one-to-one. Specifically, in the case where the electronic device is a smart device provided with a photographing device, the plurality of controlled objects correspond one-to-one with the plurality of objects or the plurality of interface areas photographed by the electronic device. In the case where the electronic device is a smart device with an AR function, the multiple controlled objects correspond to multiple objects or multiple interface areas in the AR imaging interface of the electronic device one-to-one.

In the actual application process, the user can mark the above-mentioned master object and the controlled object to establish the relationship between the controlled object and the target musical instrument, and to establish the control relationship between the master object and the controlled object, Also set the master object as the object that controls the sound of the target instrument.

Specifically, in the virtual musical instrument performance method proposed in the embodiments of the present application, the user can shoot the target scene through an electronic device with a shooting function, or perform AR imaging on the target scene through an electronic device with an AR function. On this basis, multiple objects or multiple interface areas in the photographing interface or AR imaging interface of the electronic device are identified and confirmed to determine multiple controlled objects. On this basis, the user can perform custom annotations on the identified multiple objects or multiple interface areas to establish a one-to-one correspondence between multiple objects or multiple interface areas in the actual scene and multiple musical instruments , or establish a one-to-one correspondence between multiple objects or multiple interface areas in the actual scene and multiple voices in an instrument.

Among them, the above-mentioned target musical instruments can be musical instruments with their own percussion objects such as drums, chimes, row drums, dulcimers, etc., and the above-mentioned target musical instruments can also be pianos, electronic organs, erhu, guzheng, harmonica, etc. that do not have percussion objects, but need to be pressed , plucked, played, etc. For the specific type of the above target musical instrument, the user can select it according to the actual situation, which is not specifically limited here.

It can be understood that different musical instruments have different musical tones, and the same musical instrument also has multiple sounding parts, and different sounding parts correspond to different musical tones. When playing an instrument, a complete piece of music can only be obtained through the cooperation of different tones. Therefore, when custom labeling the above-identified multiple objects or multiple interface areas, that is, when custom labeling multiple controlled objects in the target interface, specifically, the above-mentioned multiple controlled objects can be automatically labeled. Different musical instruments can be defined and marked, and the above-mentioned multiple controlled object groups can also be defined and marked as different parts of the same musical instrument, such as the 88 keys of a piano, each key of an electronic organ, and each bronze bell of a chime. In this way, by virtually playing multiple objects or multiple interface areas in the actual scene, a complete piece of music can be obtained, thereby realizing the virtual musical instrument function of the electronic device, increasing the human-computer interaction mode of the electronic device, and Allows users to play musical instruments anytime, anywhere.

Further, for musical instruments that have their own percussion objects, such as drums, chimes, row drums, and dulcimers, the movable objects of the user in the actual scene or the user's hands can be custom marked as the percussion objects in the above-mentioned musical instruments, such as Drumsticks, hammers for hitting chimes, bamboo for dulcimer, etc., in order to increase the authenticity and interactivity of users' virtual playing.

Illustratively, as shown in FIG. 2, an actual drum kit may include eight sounding parts, one kick drum 202, one snare drum 204, three toms 206 to 210, and three cymbals 212 to 216, while the drum kit also includes Two drumsticks. When the user simulates a drum kit to perform virtual playing through a smart device with a camera function, he or she can place multiple objects in the shooting scene of the electronic device according to the actual structure of the drum kit, such as objects 302 to 320 in Fig. 3 . Objects 302 to 320 are custom labeled. Specifically, as shown in FIG. 3 , the object 302 is defined as the bass drum 202 in the drum kit, the object 304 is defined as the snare drum 204 in the drum kit, and the objects 306, 308 and 310 are respectively defined as the three drum kits in the drum kit For toms 206, 208, and 210, objects 312, 314, and 316 are defined as three cymbals 212, 214, and 216 in the drum kit, respectively, and objects 318 and 320 are defined as two drumsticks in the drum kit, respectively.

Exemplarily, when a user simulates a piano through an AR-enabled smart device and performs virtual playing, the user can divide the desktop in the AR display scene of the electronic device into 88 areas according to the actual structure of the piano, and perform the above-mentioned 88 areas. Customize the annotations to establish a one-to-one correspondence between the 88 areas in the AR display scene and the 88 keys of the actual piano to build a virtual piano. Specifically, the above-mentioned 88 areas are respectively denoted as area 1 to area 88, and the 88 keys of the piano are respectively denoted as keys 1 to 88. By custom labeling the 88 areas, the area 1 to area 88 in the AR display scene are established One-to-one correspondence with piano keys 1 to 88. At the same time, a control relationship between the user's hands and the above-mentioned areas 1 to 88 is established, that is, the user's hands are set as objects for controlling the virtual piano to sound.

Further, the above-mentioned controlled object is a stationary object, and the above-mentioned main control object is a moving object. On this basis, the above motion information is the motion information of the master object in the target interface relative to each controlled object, and the motion information may specifically include the real-time position information of the master object and the duration of the master object at each position. And the real-time movement speed of the master object. The above-mentioned real-time motion speed is specifically the real-time motion speed of the master object between every two positions, and the real-time motion speed is used to represent the tap speed of the master object to each controlled object. Further, the above-mentioned real-time location information includes the location coordinates of at least one location of the master object in the target interface.

In addition, in an actual application process, when virtual playing is performed by the electronic device, the plurality of controlled objects can also be marked as a plurality of parts of a plurality of musical instruments. Specifically, when performing virtual performance through the above-mentioned electronic device, the user can mark multiple objects or multiple interface areas in the target interface of the electronic device, that is, the shooting preview interface or the AR imaging interface, so as to establish the actual scene. The association between multiple objects or multiple interface areas and different parts of multiple instruments.

For example, the user can divide the shooting preview interface or AR imaging interface of the electronic device into two areas, left and right, and then create multiple objects in the actual scene by custom annotations for multiple objects or multiple interface areas in the left area. The relationship between each object or multiple interface areas and the first instrument, and through the custom annotation of multiple objects or multiple interface areas in the right area, establish multiple objects or multiple objects in this part of the actual scene. The relationship between the first interface area and the second musical instrument can realize virtual playing of multiple parts of multiple musical instruments at the same time, which further increases the human-computer interaction mode of electronic equipment.

Step 104: Based on the motion information, play the target audio.

Wherein, the above-mentioned target audio includes audio corresponding to at least one musical tone of at least one musical instrument.

Further, the above-mentioned motion information is the motion information of the master object in the target interface of the electronic device relative to a plurality of controlled objects, and the motion information includes the real-time position information of the master object in the target interface, the master object in each The dwell time of the position and the real-time movement speed of the master object. The above-mentioned real-time motion speed is specifically the real-time motion speed of the master object between every two positions, and the real-time motion speed is used to represent the tap speed of the master object to each controlled object. The above-mentioned real-time position information includes the position coordinates of at least one position of the master object in the target interface.

In the actual application process, the above motion information can be used to represent the control state of the main control object in the target interface to multiple controlled objects, that is, the above motion information is used to express the target object corresponding to the main control object in the actual scene , the touch situation of other multiple objects or multiple interface areas corresponding to multiple controlled objects. The playing mode of the actual musical instrument is represented by the motion information of the master object relative to the multiple controlled objects, so as to achieve the purpose of virtual playing and obtain the target audio.

Specifically, when performing virtual performance through the above-mentioned electronic device, multiple objects or multiple interface areas in the shooting preview interface or virtual display interface of the electronic device are marked, thereby establishing multiple controlled objects and multiple musical instruments or A one-to-one correspondence between multiple parts of the same instrument. At the same time, the master control object in the target interface is marked to establish a performance control relationship between the master control object and the controlled object. On this basis, the motion information of the master object relative to the controlled object in the target interface is obtained through the real-sensing pixel sensor of the electronic device to determine the target object corresponding to the master object in the actual scene. Corresponding to the touch situation of other objects or multiple interface areas, when it is determined that the master object touches a controlled object, the instrument audio corresponding to the controlled object is played, so as to realize the virtual Instrument performance function to get the target audio.

Exemplarily, when a user simulates a drum kit through a smart device with a camera function to perform virtual playing, he or she can place multiple objects in the shooting scene of the electronic device according to the actual structure of the drum kit, such as objects 302 to 320 in FIG. 3 . , and then perform custom annotation on the above objects 302 to 320 . Specifically, as shown in FIG. 2 and FIG. 3 , the object 302 is defined as the bass drum 202 in the drum kit, the object 304 is defined as the snare drum 204 in the drum kit, and the objects 306, 308 and 310 are respectively defined as the drum kit in the drum kit. The three toms 206, 208 and 210 of the set, the objects 312, 314 and 316 are respectively defined as the three cymbals 212, 214 and 216 in the drum set, and the objects 318 and 320 are respectively defined as the two drums of the drum set hammer.

On this basis, the motion information of the object 318 and the object 320 relative to the objects 302 to 316 is obtained in real time through the real-time pixels, that is, the fixed position coordinates of the objects 302 to 316 are obtained, and the real-time position coordinates and movement speeds of the objects 318 and 320 are obtained. and how long objects 318 and 320 stay at each location. Further, according to the fixed position coordinates of the objects 302 to 316 and the real-time position coordinates of the objects 318 and 320, the pixel distances between the objects 318 and 320 and the objects 302 to 316, respectively, are calculated. When the pixel distance between the object 318 or the object 320 and any one of the objects 302 to 316 is smaller than the preset threshold, it is determined that the object 318 or the object 320 taps the object once, and simultaneously obtains the object 318 or the object 320 to tap the object , that is, the acquisition duration for which the pixel distance between the object 318 or the object 320 and the object is smaller than the preset threshold. On this basis, the electronic device plays the audio of the drum sound part corresponding to the object according to the specific number of times the object 318 or the object 320 hits the object and the duration and speed of each hitting the object.

For example, within a certain period of time, the electronic device detects that the object 318 approaches the object 302 three times continuously at a speed of 0.8 m/s, 0.4 m/s, and 0.4 m/s, respectively, and three times the pixels between the object 318 and the object 302 The durations when the distance is less than the preset threshold are 2 seconds, 2 seconds, and 1 second, respectively. At this point, the electronic device plays the audio of the drumstick hitting the kick drum in the drum kit three times in a row. Among them, the volume of the bass drum audio played for the first time is 80 decibels, which corresponds to the above 0.8 m/s, and the duration is 2 seconds; the volume of the bass drum audio played for the second time is 40 decibels, the decibel number Corresponds to the above 0.4 m/s, and the duration is 2 seconds; the volume of the third drum audio is played at 40 decibels, which corresponds to the above 0.4 m/s, and the duration is 1 second.

Exemplarily, when a user simulates a piano through an AR-enabled smart device and performs virtual playing, the user can divide the desktop in the AR display scene of the electronic device into 88 areas according to the actual structure of the piano, and perform the above-mentioned 88 areas. Customize the annotations to establish a one-to-one correspondence between the 88 areas in the AR display scene and the 88 keys of the actual piano to build a virtual piano. Specifically, the above-mentioned 88 areas are respectively denoted as area 1 to area 88, and the 88 keys of the piano are respectively denoted as keys 1 to 88. By custom labeling the 88 areas, the area 1 to area 88 in the AR display scene are established One-to-one correspondence with piano keys 1 to 88. At the same time, a control relationship between the user's hands and the above-mentioned areas 1 to 88 is established, that is, the user's hands are set as objects for controlling the virtual piano to sound.

On this basis, the motion information of the user's hands relative to area 1 to area 88 is obtained in real time through the real-sense pixels, that is, the center coordinates of the area from area 1 to area 88 are obtained, and the real-time position coordinates and movement speed of the head of each finger of the user's hands are obtained. And the length of stay of the heads of the fingers of the user's hands at each position. Further, according to the area center coordinates of area 1 to area 88 and the real-time position coordinates of the head of each finger of the user's hands, the pixel distance between the head of each finger of the user's hands and the area center of area 1 to area 88 is calculated. When the pixel distance between the head of a finger of the user and the area center of any one of the areas 1 to 88 is smaller than the preset threshold, it is determined that the head of the finger taps the area once, and the head of the finger is obtained at the same time. The duration of hitting the area, that is, the duration in which the pixel distance between the head of the user's finger and the center of the area is less than a preset threshold is obtained. On this basis, the electronic device plays the key audio corresponding to the region according to the specific number of times the user's finger head taps the region and the duration and speed of each tap on the region.

For example, the electronic device detects that the user's right index finger taps the area 56 at a speed of 0.4 m/s, and at the same time, the electronic device detects that the user's left middle finger taps the area 18 at a speed of 0.8 m/s. The duration of the user's right index finger hitting the area 56 is 2 seconds, and the user's left middle finger hitting the area 18 is 1 second. At this time, the electronic device will start to play the audio of the keys 56 corresponding to the area 56 and the keys 18 corresponding to the area 18 at the same time. Among them, the audio playback volume of the key 56 is 40 decibels, and the decibel number corresponds to the above-mentioned 0.4 m/s, and the duration is 2 seconds; the audio playback volume of the key 18 is 80 decibels, and the decibel number is the same as the above-mentioned 0.8 m/s. corresponds to seconds and has a duration of 1 second.

With the above-mentioned virtual musical instrument performance method provided by the embodiment of the present application, for an electronic device including a RealSense pixel sensor, the motion information of the master object relative to the controlled object in the target interface of the electronic device is obtained through the RealSense pixel sensor. Wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument. On this basis, the target audio is played according to the motion information of the master object relative to the controlled object, and the target audio includes audio corresponding to at least one musical tone of at least one musical instrument. In this way, the human-computer interaction mode of the electronic device is added, and the virtual musical instrument can be played through the motion information of the master object in the target interface of the electronic device relative to the controlled object, thereby realizing the virtual musical instrument function of the electronic device, making Users can play instruments anytime, anywhere.

In the embodiment of the present application, the main control object is a moving object, and the main control object includes a movable object of the user or the user's hand; the controlled object includes an object other than a physical musical instrument, or an interface area in a target interface; motion The information includes the real-time position information of the main control object, the staying time of the main control object at each position, and the real-time motion speed of the main control object, wherein the real-time position information includes the position coordinates of at least one position; the target interface includes the shooting preview of the electronic device interface or virtual display interface.

Specifically, when virtual playing is performed through an electronic device, multiple objects or multiple interface areas other than the physical musical instrument in the shooting preview interface or virtual display interface of the electronic device are marked, that is, multiple objects or multiple interface areas in the target interface are marked. Each controlled object is marked, thereby establishing a one-to-one correspondence between multiple controlled objects and multiple musical instruments or multiple parts of the same musical instrument. At the same time, mark other movable objects or the user's hand in the target interface other than the controlled object, that is, mark the main control object in the target interface, so as to establish the main control object and the controlled object. The performance control relationship between them.

On this basis, the real-time position coordinates of the master object in the target interface, the staying time of the master object at each position, and the real-time movement speed of the master object between each two positions are obtained through the real-sensing pixel sensor of the electronic device. In order to determine the touch situation of the movable object or the user's hand corresponding to the master object in the actual scene to other objects or interface areas corresponding to the controlled objects. When it is determined that the master object touches a certain controlled object, the musical instrument audio corresponding to the controlled object is played, so as to realize the virtual musical instrument performance function of the electronic device and obtain the target audio.

In the above-mentioned embodiments provided in the present application, through the motion information of the master object in the target interface of the electronic device relative to the controlled object, the actual musical instrument can be simulated to perform, and the human-computer interaction mode of the electronic device is added, and the electronic device is realized. The virtual instrument function allows users to play instruments anytime, anywhere.

In this embodiment of the present application, the foregoing step 104 may specifically include the following steps 104a and 104b:

Step 104a: According to the motion information, determine the control parameters for the master object to control the controlled object.

The above motion information is the motion information of the master object in the target interface of the electronic device relative to a plurality of controlled objects, and the motion information includes the real-time position information of the master object in the target interface, the position of the master object at each position the duration of stay and the real-time movement speed of the master object.

The above-mentioned real-time motion speed is specifically the real-time motion speed of the master object between every two positions, and the real-time motion speed is used to represent the tap speed of the master object to each controlled object. The above-mentioned real-time position information includes the position coordinates of at least one position of the master object in the target interface.

Further, the above-mentioned control parameters include the triggered controlled object, the control duration, the control times and the control speed.

In the actual application process, the above motion information can be used to represent the control state of the master object in the target interface relative to multiple controlled objects. Through the motion information of the master object relative to the controlled objects, it is determined that the master object controls the controlled objects. Control parameters of the control object.

Specifically, the controlled object is a stationary object, and the main control object is a moving object. A controlled object that is specifically triggered is determined from the multiple controlled objects through the real-time position coordinates of the main control object in the target interface, that is, the object or the user's hand corresponding to the main control object is determined. A certain object or interface area of the controlled object, and the number of times of control is determined according to the number of times the master controlled object touches the controlled object. Further, according to the length of stay of the master object at the position corresponding to the controlled object, determine the control duration of the master object to the controlled object, and according to the real-time movement speed of the master object at the position corresponding to the controlled object , to determine the control speed of the master object to the controlled object, that is, to determine the tap speed of the master object to the controlled object.

Step 104b: Play the target audio according to the control parameters.

Wherein, the above-mentioned control parameters represent the control state of the main control object on multiple controlled objects, that is, the above-mentioned control parameters are used to describe the target objects corresponding to the main controlled objects in the actual scene, and the control parameters corresponding to the multiple controlled objects are used to describe the target objects in the actual scene Touch operations for other multiple objects or multiple interface areas.

Further, the above-mentioned control parameters include the triggered controlled object, the control duration, the number of times of control, and the control speed, and the above-mentioned target audio includes audio corresponding to at least one musical tone of at least one musical instrument.

Specifically, after determining a controlled object that is specifically triggered from a plurality of controlled objects according to the motion information of the master object relative to the controlled object, and then according to the number of times the master control object controls the controlled object, the control The duration and control speed are used to play the musical instrument audio corresponding to the controlled object, so as to realize the virtual musical instrument performance function of the electronic device. On this basis, through the continuous operation of multiple controlled objects by the user controlling the master control object, the performance of the complete music piece can be realized.

Specifically, when playing the musical instrument audio corresponding to the controlled object according to the number of times, the control duration, and the control speed of the controlled object by the master control object, the duration of the audio playback can be adjusted according to the above-mentioned control duration, and the duration of the playback audio can be adjusted according to the above-mentioned control duration. The number of times to adjust the number of times to play the audio, and you can also adjust the volume of the played audio according to the above control speed, so as to obtain the target audio that meets the user's requirements.

In the above-mentioned embodiment provided by the present application, when the target audio is played based on the motion information of the master object relative to the controlled object, it is specifically determined according to the motion information of the master object relative to the controlled object to determine the movement of the master object to control the controlled object. control parameters, and then play the target audio corresponding to the controlled object according to the control parameters. In this way, according to the specific control parameters of the master object to the controlled object, the target audio corresponding to the controlled object is played, so as to realize the virtual musical instrument function of the electronic device, so that the user can play the musical instrument anytime and anywhere.

In the embodiment of the present application, the above-mentioned control parameters include the controlled object, the control duration, the number of times of control, and the control speed. On this basis, the above-mentioned step 104b may specifically include the following step 104b1:

Step 104b1: Play audio corresponding to at least one musical tone of the musical instrument associated with the controlled object according to the control duration, the number of times of control, and the control speed.

Specifically, after determining a specifically triggered controlled object from a plurality of controlled objects according to the motion information of the master object relative to the controlled object, the controlled object is controlled according to the number of times and the control duration of the controlled object. And control the speed, play audio corresponding to at least one musical tone of the musical instrument associated with the controlled object, so as to realize the virtual musical instrument performance function of the electronic device. On this basis, through the continuous operation of multiple controlled objects by the user controlling the master control object, the performance of the complete music piece can be realized.

Specifically, in the case where the above-mentioned multiple controlled objects are associated with multiple musical instruments, after determining a controlled object controlled by the master control object, according to the number of times the master control object controls the controlled object, the control duration and the Controls the speed and plays the instrument audio associated with this plant.

Further, in the case where the above-mentioned multiple controlled objects are associated with multiple parts of the same musical instrument, after determining a certain controlled object controlled by the master controlled object, the controlled object is controlled according to the number of times the controlled object is controlled by the master control object. , control the duration, and control the speed, and play the audio of the instrument part associated with the controlled object.

Wherein, when the audio corresponding to the controlled object is played according to the control times, control duration and control speed of the controlled object by the master control object, the duration of the audio playback can be adjusted according to the above-mentioned control duration, and can be adjusted according to the above-mentioned control times. The number of times the audio is played, and the volume of the played audio can also be adjusted according to the above-mentioned control speed, so as to obtain the target audio that meets the user's requirements.

In the above-mentioned embodiment provided by the present application, the above-mentioned control parameters include the controlled object, the control time, the number of times of control and the control speed. Audio corresponding to at least one tone of the musical instrument associated with the controlled object is played. In this way, according to the specific control parameters of the master object to the controlled object, the instrument audio corresponding to the controlled object is played according to the corresponding playback parameters, and the virtual musical instrument can be flexibly and accurately controlled to sound, so as to obtain an accurate target that meets the user's requirements. Audio, realizes the virtual musical instrument function of electronic devices, so that users can play musical instruments anytime and anywhere.

In this embodiment of the present application, when the target interface includes at least two controlled objects, the above step 104a may specifically include the following steps 104a1 and 104a2:

Step 104a1: Acquire the real-time pixel distance between the master object and each controlled object in the target interface according to the motion information.

Wherein, the above-mentioned motion information is the motion information of the master object in the target interface of the electronic device relative to the plurality of controlled objects, the motion information includes real-time position information of the master object in the target interface, and the real-time position information includes the master object The position coordinates of at least one position of the object in the target interface.

Further, the above-mentioned multiple controlled objects are static objects in the target interface, and the above-mentioned master control objects are moving objects in the target interface. That is to say, when the target interface is fixed, the pixel position coordinates of the controlled object in the target interface are constant, and the pixel position coordinates of the master object in the target interface change in real time. On this basis, the real-time pixel distance between the master object and each controlled object can be determined according to the real-time pixel position coordinates of the master object and the fixed pixel position coordinates of each controlled object, so that the subsequent real-time pixel distance can be determined according to the determined real-time pixel position. The distance determines the controlled object specifically controlled by the master object.

Exemplarily, if the fixed pixel position coordinates of the first controlled object in the target interface are (x ₁ , y ₁ ), the real-time pixel position coordinates of the master object at a certain moment are (x ₂ , y ₂ ). At this time, the real-time pixel distance between the master object and the first controlled object is

Step 104a2: Determine a controlled object whose real-time pixel distance is less than a preset threshold as a controlled object that can play audio.

Among them, the above real-time pixel distance is the pixel distance between the main control object and each controlled object during the movement process, and the real-time pixel distance can represent the proximity state of the main control object relative to each controlled object during the movement process, that is In the actual scene, the movable object corresponding to the master object or the user's hand touches other objects or interface areas corresponding to the controlled objects.

Specifically, according to the real-time pixel position coordinates of the master object in the target interface and the fixed pixel position coordinates of each controlled object, after the real-time pixel distance between the master object and each controlled object is determined, the The real-time pixel distance between the master object and each object is compared with a preset threshold. On this basis, when the real-time pixel distance between a controlled object and the main control object is less than the preset threshold, the controlled object is considered to be the current control object of the main control object, that is, the actual scene is considered to be the same as the control object. The movable object or the user's hand corresponding to the master object successfully touches the object or area corresponding to the controlled object. At this time, the electronic device will play the musical instrument audio corresponding to the controlled object.

The specific value of the above-mentioned preset threshold value can be set by the user according to the actual situation such as the screen size of the device, the distance between actual objects, etc., which is not specifically limited here.

In the above-mentioned embodiment provided by this application, in the case that the target interface includes at least two controlled objects, the main control object and each controlled object in the target interface are obtained according to the motion information of the main control object relative to each controlled object. The real-time pixel distance is determined, and the controlled object whose real-time pixel distance is less than the preset threshold is determined as the controlled object that can play audio. In this way, the controlled object that can play audio is determined according to the real-time pixel distance between the main control object and each controlled object, which ensures the accuracy and real-time performance of the controlled object determination, thereby ensuring the accuracy and fluency of the target audio playback. , which improves the reliability of the virtual performance function of electronic devices.

In this embodiment of the present application, the foregoing step 104b1 may specifically include the following steps 104b11 to 104b14:

Step 104b11: Adjust the playback duration of the target audio according to the control duration.

Wherein, the above motion information includes the staying time of the master object at each position in the target interface. The duration of the object's control.

Further, after the control duration of the controlled object by the master object is determined, the playback duration of the target audio to be played corresponding to the controlled object is adjusted according to the control duration.

Step 104b12: Adjust the playback frequency of the target audio according to the control times.

The above motion information includes real-time position information of the master object in the target interface, and the real-time position information includes the position coordinates of at least one position of the master object in the target interface. On this basis, a specific controlled object is determined from the multiple controlled objects through the real-time position coordinates of the main control object in the target interface, that is, the object or the user's hand corresponding to the main control object is determined. The specific touch represents a certain object or interface area of the controlled object. On this basis, the number of times of control is determined according to the number of times the master object touches the controlled object.

Further, after determining the number of times the master control object controls the controlled object, the playing frequency of the target audio to be played corresponding to the controlled object is adjusted according to the number of times of control.

Step 104b13: Adjust the playback volume of the target audio according to the control speed.

Wherein, the above motion information includes the real-time motion speed of the master object in the target interface. The control speed, that is, to determine the tap speed of the master object to the controlled object.

Further, after determining the control speed of the master object to the controlled object, the playback volume of the target audio to be played corresponding to the controlled object is adjusted according to the control speed.

Step 104b14: Play audio corresponding to at least one musical tone of the musical instrument associated with the controlled object according to the playing duration, the playing frequency and the playing volume.

Specifically, after adjusting the playback duration, playback frequency, and playback volume of the target audio according to the control duration, number of times, and control speed of the controlled object by the master control object, the playback duration, playback frequency, and playback volume are adjusted according to the adjusted playback duration, playback frequency, and playback volume. Audio corresponding to at least one musical tone of the musical instrument associated with the controlled object, thereby realizing the virtual musical instrument performance function of the electronic device. On this basis, through the continuous operation of multiple controlled objects by the user controlling the master control object, the performance of the complete music piece can be realized.

Wherein, in the case where the above-mentioned multiple controlled objects are associated with multiple musical instruments, after determining a controlled object controlled by the master control object, according to the adjusted playback duration, playback frequency and playback volume, the playback and The instrument audio associated with the control object.

Further, in the case where the above-mentioned multiple controlled objects are associated with multiple voice parts of the same musical instrument, after determining a certain controlled object controlled by the master control object, according to the adjusted playback duration, playback frequency and playback volume , which plays the audio of the instrument part associated with the controlled object.

Exemplarily, when a user simulates a drum kit through a smart device with a camera function to perform virtual playing, he or she can place multiple objects in the shooting scene of the electronic device according to the actual structure of the drum kit, such as objects 302 to 320 in FIG. 3 . , and then perform custom annotation on the above objects 302 to 320 . Specifically, as shown in FIG. 2 and FIG. 3 , the object 302 is defined as the bass drum 202 in the drum kit, the object 304 is defined as the snare drum 204 in the drum kit, and the objects 306, 308 and 310 are respectively defined as the drum kit in the drum kit. The three toms 206, 208 and 210 of the set, the objects 312, 314 and 316 are respectively defined as the three cymbals 212, 214 and 216 in the drum set, and the objects 318 and 320 are respectively defined as the two drums of the drum set hammer.

On this basis, if within a certain period of time, the electronic device detects that the object 318 approaches the object 302 three times continuously at a speed of 0.8 m/s, 0.4 m/s, and 0.4 m/s, respectively, and three times the object 318 and the object 302 The durations when the pixel distance between them is less than the preset threshold are 2 seconds, 2 seconds, and 1 second, respectively. At this point, the electronic device plays the audio of the drum stick hitting the kick drum in the drum kit three times in a row. Among them, the volume of the bass drum audio played for the first time is 80 decibels, which corresponds to the above 0.8 m/s, and the duration is 2 seconds; the volume of the bass drum audio played for the second time is 40 decibels, the decibel number Corresponding to the above 0.4 m/s, and the duration is 2 seconds, the volume of the third drum audio is played at 40 decibels, which corresponds to the above 0.4 m/s, and the duration is 1 second.

Exemplarily, when a user simulates a piano through an AR-enabled smart device and performs virtual playing, the user can divide the desktop in the AR display scene of the electronic device into 88 areas according to the actual structure of the piano, and perform the above-mentioned 88 areas. Customize the annotations to establish a one-to-one correspondence between the 88 areas in the AR display scene and the 88 keys of the actual piano to build a virtual piano. Specifically, the above-mentioned 88 areas are respectively denoted as area 1 to area 88, and the 88 keys of the piano are respectively denoted as keys 1 to 88. By custom labeling the 88 areas, the area 1 to area 88 in the AR display scene are established One-to-one correspondence with piano keys 1 to 88. At the same time, a control relationship between the user's hands and the above-mentioned areas 1 to 88 is established, that is, the user's hands are set as objects for controlling the virtual piano to sound.

On this basis, if the electronic device detects that the user's right index finger taps the area 56 at a speed of 0.4 m/s, at the same time, the electronic device detects that the user's left middle finger taps the area 18 at a speed of 0.8 m/s. The duration of the user's right index finger hitting the area 56 is 2 seconds, and the user's left middle finger hitting the area 18 is 1 second. At this time, the electronic device will start to play the audio of the keys 56 corresponding to the area 56 and the keys 18 corresponding to the area 18 at the same time. Among them, the audio playback volume of the key 56 is 40 decibels, and the decibel number corresponds to the above-mentioned 0.4 m/s, and the duration is 2 seconds; the audio playback volume of the key 18 is 80 decibels, and the decibel number is the same as the above-mentioned 0.8 m/s. corresponds to seconds and has a duration of 1 second.

In the above-mentioned embodiment provided by the present application, after determining the triggered controlled object and the control duration, control times and control speed of the controlled object by the main control object according to the motion information of the main control object relative to the controlled object, according to The determined control duration, control times and control speed adjust the playback duration, playback frequency and playback volume of the target audio respectively, and then play the instrument associated with the triggered controlled object according to the adjusted playback duration, playback frequency and playback volume. Audio corresponding to at least one tone. In this way, according to the specific control parameters of the master object to the controlled object, and play the musical instrument audio corresponding to the controlled object according to the corresponding playback parameters, the virtual musical instrument can be flexibly and accurately controlled to sound, so as to obtain the target audio that meets the user's requirements. The virtual instrument function of electronic equipment is realized.

In the embodiment of the present application, the above-mentioned real-sensing pixel sensor includes a plurality of real-sensing pixels. On this basis, the above-mentioned step 102 may specifically include the following steps 102a to 102c:

Step 102a: Acquire pixel change values of a plurality of real-time pixels in real time.

Wherein, a plurality of RGB (Red-Green-Blue, three primary colors) pixels are further arranged on the above-mentioned RealSense pixel sensor. In the actual application process, the above-mentioned real-sensing pixel sensor can be obtained by inserting a plurality of real-sensing pixels into an image sensor composed of RGB pixels according to a preset density, so that the image sensor has the ability to capture the outline of a moving object at a high speed.

Wherein, the above-mentioned preset density can take values such as 2.5%, 3%, 3.5%, etc. In actual application, the user can set the specific value of the above-mentioned preset density according to actual needs, which is not specifically limited here.

Specifically, as shown in (a) of FIG. 7 , when a plurality of photorealistic pixels 702 are inserted into an image sensor composed of RGB pixels 704 , the above-mentioned plurality of photorealistic pixels 702 can be embedded among the plurality of RGB pixels 704 to form a corresponding pixel array. Among them, each pixel array is composed of 32*32 pixel points.

Further, since the entire pixel array is laid on the middle part of the substrate, in order to facilitate the installation and fixation of the pixel array, a blank part may be reserved on the substrate. Specifically, as shown in (b) of FIG. 7 , the pixel array 708 and the substrate 706 are both rectangular structures, the width of the blank part reserved on the upper part of the substrate 706 is L ₁ , and the width of the blank parts on both sides is L ₂ , the width of the blank part reserved at the lower part is L ₃ .

In the actual application process, the above-mentioned L ₁ can be 72 μm, the L ₂ can be 88 μm, and the L ₃ can be 96 μm. For the above-mentioned specific values of L ₁ , L ₂ , and L ₃ , the user can choose according to the actual situation. settings, which are not specifically limited here.

Further, as shown in FIG. 8, when a plurality of real-sensing pixels 802 are inserted into an image sensor composed of RGB pixels 804, the above-mentioned multiple real-sensing pixels 802 can also be placed above the multiple RGB pixels 804, such as RGB pixels. Two rows of RealSense pixels 802 are inserted above 804 to form a corresponding pixel array.

On this basis, the circuit diagram of the above-mentioned image sensor can be shown in FIG. 9 , and two rows of real-sensing pixels C are inserted above the multiple rows of RGB pixels. Among them, the RGB pixels include R pixels, B pixels, Gr pixels and Gb pixels. The RGB pixels are used to photosensitive the actual scene, and the light signals collected by the RGB pixels are amplified by the first signal amplifier 904 to obtain the first voltage. The first analog-to-digital converter 906 converts the first voltage signal into a first digital pixel value, thereby outputting a high-definition color image. On the other hand, the real-sensing pixels use a fixed exposure and a higher frequency to collect ambient brightness, and amplify the light signal collected by the real-sensing pixels through the second signal amplifier 902 to obtain a second voltage signal, which is then passed through the second analog-to-digital conversion. The controller 908 converts the second voltage signal into a second digital pixel value, and outputs it to the image processor for calculation, so as to obtain motion state information in the field of view for reference in subsequent processing.

In the actual application process, the user can set the arrangement of the above-mentioned multiple RealSense pixels in the RGB pixels according to the actual situation, which is not specifically limited here.

Further, the internal circuit structure of the RGB pixel is shown in FIG. 4 . As shown in FIG. 4 , the internal circuit structure of the RGB pixel is a PPD (Pinned Photodiode Pixel, pinned photodiode pixel) structure. The PPD structure includes a photosensitive area of PD1, ie, a photosensitive diode, and 4 transistors, ie, a reset transistor RST, a floating switch TX1, a row selector SET, and a signal amplifier SF.

It can be understood that the PPD structure allows the introduction of a CDS (Correlated Double Sampling, correlated double sampling) circuit, so that the KTC noise introduced by the reset, and the 1/f noise and the offset noise introduced by the MOS transistor can be eliminated. The 1/f noise is also called flicker noise or low-frequency noise. The working process of the above-mentioned PPD structure may specifically include the following steps 202 to 210:

Step 202: Exposure. RST and TX1 are turned on at the same time to empty PD1, and then the two are disconnected to start exposure. The "electron-hole pairs" generated by light irradiation will be separated by the presence of the PD1 electric field, so that electrons move to the N region, and holes move to the Area P.

Step 204: Reset. At the end of the exposure, RST is activated, resetting FD1 to high.

Step 206: read out the reset level. After the reset is completed, read out the reset level of FD1, which includes the offset noise of the MOS transistor, the 1/f noise and the KTC noise introduced by the reset, and store the read-out reset level signal in the first capacitor.

Step 208: Charge transfer. Activation of TX1 completely transfers charge from photoreceptor PD1 to FD1.

Signal level readout: read out the voltage signal of FD1 to the second capacitor, the voltage signal includes the signal generated by photoelectric conversion and the offset noise, 1/f noise, and KTC noise introduced by reset.

Step 210: Signal output. Subtract the signals stored in the two capacitors. If the CDS circuit is used, the main noise in the pixel can be eliminated. After the noise is eliminated, the signal is then analog amplified by the signal amplifier SF, and then sampled by analog-to-digital conversion, which can be digitized. signal output.

Further, the internal circuit structure of the RealSense pixel is shown in FIG. 5 . As shown in FIG. 5 , the internal circuit of the RealSense pixel includes a first current amplification unit 502 , an analog-to-digital conversion unit 503 , a logic determination unit 504 , a control unit 505 , a switch unit 506 and a second current amplification unit 507 .

The output end of the first current amplifying unit 502 is connected to the input end of the analog-to-digital conversion unit 503, the output end of the analog-to-digital conversion unit 503 is connected to the input end of the logic judgment unit 504, and the output end of the logic judgment unit 504 is connected to the control unit 505 input connection. Further, the switch unit 506 is disposed between the first current amplifying unit 502 , the second current amplifying unit 507 and the control unit 505 .

Specifically, in the working process of the real-sensing pixel circuit, referring to the circuit diagram shown in FIG. 5 , the light collected by the photodiode PD1 is amplified by the first current amplifying unit 502 and then enters the analog-to-digital conversion unit 503 . The amplified electrical signal is converted into a digital signal, and the digital signal is continuously sent to the logic judging unit 504. When the logic judging unit 504 judges that the variation of the digital signal exceeds a preset threshold range, that is, when the logic judging unit 504 judges When the variation of the digital signal is larger than the preset maximum threshold VH or smaller than the minimum threshold VL, the control unit 505 generates a corresponding control signal according to the logic judgment result of the logic judgment unit 504 , and transmits the control signal to the switch unit 506 , to control the switch unit 506 to open the switch, and the second terminal of the switch unit 506 outputs an analog signal.

Wherein, VH and VL are respectively determined according to the digital signal value of the real-sensing pixel at the last clock frequency and the preset threshold, VH is the digital signal value of the real-sensing pixel at the last clock frequency plus the preset threshold, and VL is the last clock frequency. The digital signal value of the photorealistic pixel is subtracted by a preset threshold.

Further, the above-mentioned switch unit 506 can be a multiplexing switch, to select one or more signals from the analog or digital input signals obtained by all the RealSense pixels and forward them to the signal processing system inside the RealSense pixel sensor. , so as to be read by the signal processing system.

In addition, the second current amplifying unit 507 and the switch unit 506 form another circuit structure, and this circuit structure is connected with the first current amplifying unit 502 , the analog-to-digital conversion unit 503 , the logic judgment unit 504 , the control unit 505 and the switch unit 506 The formed circuit structures cooperate with each other to realize the phase focus function.

It can be seen that the difference between RealSense pixels and RGB pixels is that RGB pixels need to integrate light information in a period of time and then read them out one by one in sequence, and the length of the above period is related to the frame rate. The real-sensing pixels can independently perceive the change of the brightness of the external environment in real time with the pixel clock frequency, and convert the change of the brightness of the environment into the change of the pixel value. If the change of the pixel value of a certain RealSense pixel exceeds the preset threshold, the system will be reported to the system for reading, and a data packet with coordinate information, brightness information, and time information will be output. Therefore, compared with RGB pixels, RealSense pixels can more clearly capture the motion information of objects in motion. As shown in FIG. 6 , serial number 602 is the image seen by the human eye, serial number 604 is the image recorded by RGB pixels, and serial number 606 It is an image recorded by RealSense pixels.

Step 102b: In the case where the pixel change value of the target real-sensing pixel is outside the preset threshold range among the plurality of real-sensing pixels, output the target information.

Wherein, the above target information includes: position information, ambient brightness information, and time information of the target real-sensing pixel. As shown in FIG. 5, compared with the circuit diagram of the RGB pixel, the circuit diagram of the real-sensing pixel adds a logic judgment unit 504, and the logic judgment unit 504 is used to judge whether the difference between the pixel value at the current moment and the pixel value at the previous moment is greater than the predetermined value. Set a threshold, and output the pixel value at the current moment when the difference is greater than the preset threshold.

Specifically, the real-sensing pixels are used to sense changes in the brightness of the external environment in real time, and convert the changes of the brightness of the environment into changes in current. As shown in FIG. 5 , the light collected by the photodiode PD1 is amplified by the first current amplifying unit 502 and then enters the analog-to-digital conversion unit 503. The analog-to-digital conversion unit 503 converts the amplified electrical signal into a digital signal, and the digital signal is continued. It is sent to the logic judgment unit 504 . When the logic judging unit 504 judges that the digital signal exceeds the preset threshold range, that is, when the logic judging unit 504 judges that the digital signal is greater than the preset maximum threshold VH or smaller than the minimum threshold VL, the amplified signal is amplified by the analog signal output module. Digital-to-analog conversion is performed on the voltage signal of the pixel to obtain the readout result detected by the real-sensing pixel, and the read-out result includes the position information, ambient brightness information, and time information of the real-sensing pixel.

Wherein, VH and VL are respectively determined according to the digital signal value of the real-sensing pixel at the last clock frequency and the preset threshold, VH is the digital signal value of the real-sensing pixel at the last clock frequency plus the preset threshold, and VL is the last clock frequency. The digital signal value of the photorealistic pixel is subtracted by a preset threshold.

Step 102c: Determine motion information according to the target information.

Wherein, the acquisition frequency of motion information is related to the acquisition frequency of pixel change values of a plurality of real-sensing pixels.

Further, the above-mentioned target information includes position information, ambient brightness information, and time information of the real-sensing pixels. After outputting the above-mentioned position information, environmental brightness information, and time information, the above-mentioned multiple pieces of information can be forwarded to the signal processing system inside the RealSense pixel sensor, so as to determine, according to the above-mentioned position information, environmental brightness information, and time information, that the main control object is relative to the many motion information of a controlled object.

It can be understood that the real-time performance of the real-sensing pixels is high, so the position information displayed by the images obtained by the real-sensing pixels is more accurate, so that the specific position of the moving object can be quickly and accurately found in the image, and accurate positioning can be achieved. At the same time, the subtle position changes of the moving objects can be recorded through at least one frame of preview image. Therefore, the real-sensing pixels in the electronic device can accurately capture the position changes of the moving objects and ensure the accuracy of the determination of the motion information of the moving objects.

Exemplarily, when the motion information of the above-mentioned master control object relative to the controlled object is obtained through the real-sensing pixel sensor, as shown in FIG. 5 , the ambient light brightness is collected through the PD1 corresponding to each real-sensing pixel in the real-sensing pixel sensor, The brightness is converted into a voltage signal after being amplified by the first current amplifying unit 502, and then the voltage signal is input to the analog-to-digital conversion unit 503, and the amplified voltage signal is converted into a digital signal by the analog-to-digital conversion unit 503, and the digital signal continues It is sent to the logic judgment unit 504 . At this time, if the preset VH of a certain real-sensing pixel in the real-sensing pixel sensor is 12 and VL is 8, and the digital signal obtained by converting the brightness of the ambient light collected by the real-sensing pixel is 14, the logic determination unit 504 will determine The digital signal exceeds the preset threshold range of the real-sensing pixel, that is, it is considered that the detection object corresponding to the real-sensing pixel has acted. At this time, the logic judgment unit 504 will send the logic judgment result to the control unit 505, and the control unit 505 will then generate a corresponding control signal according to the logic judgment result of the logic judgment unit 504, and transmit the control signal to the switch unit 506 to The control switch unit 506 turns on the switch to output the position information, ambient brightness information, and time information of the photorealistic pixel.

In the above-mentioned embodiment provided by this application, the above-mentioned real-sensing pixel sensor is provided with a plurality of real-sensing pixels, and when determining the motion information of the master object relative to the multiple controlled objects, the pixel change values of the multiple real-sensing pixels are acquired in real time, and then the pixel change values of the multiple real-sensing pixels are obtained in real time, and then When the pixel change value of the target real-sensing pixel in the multiple real-sensing pixels is greater than or equal to the preset threshold, output the position information, ambient brightness information and time information of the target real-sensing pixel, and determine the relative position of the main control object relative to the multiple real-sensing pixels according to the output information. The motion information of the controlled object. In this way, due to the high real-time performance of real-sensing pixels, imaging through real-sensing pixels in electronic devices can quickly and accurately determine the specific position of the main control object in the target interface, and achieve precise positioning, so that the main control object can be accurately captured. The position change ensures the accuracy of determining the motion information of the master object relative to the controlled object, thereby ensuring the accuracy of audio playback.

In this embodiment of the present application, before the above step 102, the above virtual performance method may further include the following steps 100 and 101:

Step 100: in the case of receiving the first input from the user to the target object or the target interface area of the target interface, set the target object or the target interface area of the target interface as the controlled object, and establish a relationship between the controlled object and the target musical instrument. relationship between.

The above-mentioned first input is a user's touch input to the target interface of the electronic device, and the touch input is used to mark multiple objects or multiple interface areas in the target interface, so as to determine the corresponding objects or interface areas. is the controlled object, and establishes an association relationship between the controlled object and the target musical instrument. That is, the above-mentioned first input is used to mark the controlled object.

Further, the above-mentioned target interface is an imaging interface of a real scene in an electronic device. Specifically, when the electronic device is a smart device with a photographing device, the target interface is a photographing preview interface of the electronic device. In the case where the electronic device is a smart device with AR function, the target interface is a virtual display interface of the electronic device, that is, an AR imaging interface.

Specifically, in the virtual musical instrument performance method proposed in the embodiments of the present application, the user can shoot the target scene through an electronic device with a shooting function, or perform AR imaging on the target scene through an electronic device with an AR function. On this basis, identify and confirm multiple objects or multiple interface areas in the photographing interface of the electronic device, or identify and confirm multiple objects or multiple interface areas in the AR imaging interface of the electronic device. On this basis, the user can customize the identified multiple objects or multiple interface areas through touch input on the target interface, so as to determine the corresponding object or interface area as the controlled object, and establish The association between the controlled object and the target instrument.

Among them, the above-mentioned target musical instruments can be musical instruments with their own percussion objects such as drums, chimes, row drums, dulcimers, etc., and the above-mentioned target musical instruments can also be pianos, electronic organs, erhu, guzheng, harmonica, etc. that do not have percussion objects, but need to be pressed , plucked, played, etc. For the specific type of the above target musical instrument, the user can select it according to the actual situation, which is not specifically limited here.

It can be understood that different musical instruments have different musical tones, and the same musical instrument also has multiple sounding parts, and different sounding parts correspond to different musical tones. When playing an instrument, a complete piece of music can only be obtained through the cooperation of different tones. Therefore, when the above-mentioned identified multiple objects or multiple interface areas are custom-marked, the above-mentioned multiple objects or multiple interface areas can be custom-marked as different musical instruments, and the above-mentioned multiple objects can also be customized. Or multiple interface areas can be custom-labeled as different voices of the above target instruments. In this way, a complete piece of music can be obtained by virtually playing multiple objects or multiple interface areas in the actual scene, thereby realizing the virtual musical instrument function of the electronic device and increasing the human-computer interaction mode of the electronic device.

Illustratively, as shown in FIG. 2 , an actual drum kit may include eight sounding parts, one kick drum 202 , one snare drum 204 , three toms 206 to 210 , and three cymbals 212 to 216 . When a user performs virtual playing through a smart device with a camera function, he or she can place multiple objects in the shooting scene of the electronic device according to the actual number and specific arrangement of the sounding parts of the drum kit, such as objects 302 to 316 in FIG. 3 . On this basis, the above objects 302 to 316 are custom marked. Specifically, as shown in FIG. 2 and FIG. 3 , the object 302 is defined as the bass drum 202 in the drum set, and the object 304 is defined as the army drum in the drum set. Drum 204, which defines objects 306, 308, and 310 as the three toms 206, 208, and 210 in the drum kit, and objects 312, 314, and 316 as the three cymbals, 212, 214, and 216 in the drum kit, respectively .

Step 101: In the case of receiving the second input from the user to the master control object, set the master control object as an object for controlling the target musical instrument to sound.

Wherein, the above-mentioned second input is a user's touch input to the master control object in the target interface of the electronic device, and the touch input is used to set the master control object as an object for controlling the target musical instrument to sound. That is, the above-mentioned second input is used to mark the master object.

Further, the above-mentioned main control object is a moving object, and the main control object includes a user movable object or a user's hand. In the actual application process, through the user's touch input to the master object in the target interface, the master object is set as the object used to control the target musical instrument to sound, so as to establish the relationship between the master object and the controlled object. control relationship.

Among them, for drums, chimes, row drums, dulcimers and other musical instruments that have their own percussion objects, the objects that the user can move in the actual scene or the user's hands can be customized as percussion objects in the above instruments, such as drums. mallet, hammer for hitting chimes, bamboo for dulcimer, etc., so as to increase the authenticity and interactivity of users when they play virtual playing.

Exemplarily, as shown in FIG. 3 , when the user simulates a drum set to perform virtual playing, objects 318 and 320 may also be placed in the shooting scene of the electronic device, and the objects 318 and 320 are respectively defined as the parts of the drum set. Two drumsticks.

In the above-mentioned embodiments provided in this application, the user can set multiple objects or multiple interface areas in the target interface as controlled objects through touch input on the target interface, and establish an association between the controlled object and the target musical instrument relationship, and set the master object in the target interface, such as the user's hand or other movable objects, as the object used to control the sound of the target instrument. In this way, a complete piece of music can be obtained by virtually playing multiple objects or multiple interface areas in the actual scene, thereby realizing the virtual musical instrument function of the electronic device and increasing the human-computer interaction mode of the electronic device.

To sum up, the virtual musical instrument performance method proposed in the embodiment of the first aspect of the present application realizes the accurate extraction of the speed and displacement information of the moving object through the ability of real-time pixel sensor to grasp high-precision dynamic objects in real time. On this basis, combined with AR applications, the virtual instrument performance function of electronic devices is realized, thereby providing users with more functions and a smoother human-computer interaction method, and improving the overall performance of electronic devices.

In the virtual musical instrument performance method provided by the embodiments of the first aspect of the present application, the execution body may be a virtual musical instrument performance apparatus. In the embodiment of the present application, the virtual musical instrument performance device provided by the embodiment of the second aspect of the present application is described by taking the virtual musical instrument playing device performing the above-mentioned virtual musical instrument playing method as an example.

As shown in FIG. 10 , an embodiment of the present application provides an apparatus 1000 for playing a virtual musical instrument, which is applied to an electronic device including a real-sensing pixel sensor. The apparatus may include the following acquisition unit 1002 and playback unit 1004 .

The acquiring unit 1002 is used to acquire the motion information of the master object in the target interface relative to the controlled object through the real-sensing pixel sensor;

Playing unit 1004, for playing target audio based on motion information;

Wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument; the target audio includes audio corresponding to at least one musical tone of at least one musical instrument.

With the above-mentioned virtual musical instrument performance device 1000 provided by the embodiment of the present application, for an electronic device including a real-sensing pixel sensor, the acquiring unit 1002 acquires the motion information of the master object relative to the controlled object in the target interface of the electronic device through the real-sensing pixel sensor. Wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument. On this basis, the playing unit 1004 plays target audio according to the motion information of the master object relative to the controlled object, where the target audio includes audio corresponding to at least one musical tone of at least one musical instrument. In this way, the human-computer interaction mode of the electronic device is added, and the virtual musical instrument can be played through the motion information of the master object in the target interface of the electronic device relative to the controlled object, thereby realizing the virtual musical instrument function of the electronic device, making Users can play instruments anytime, anywhere.

In the embodiment of the present application, the main control object is a moving object, and the main control object includes a movable object of the user or the user's hand; the controlled object includes an object other than a physical musical instrument, or an interface area in a target interface; motion The information includes the real-time position information of the main control object, the staying time of the main control object at each position, and the real-time motion speed of the main control object, wherein the real-time position information includes the position coordinates of at least one position; the target interface includes the shooting preview of the electronic device interface or virtual display interface.

In the above-mentioned embodiments provided in the present application, through the motion information of the master object in the target interface of the electronic device relative to the controlled object, the actual musical instrument can be simulated to perform, and the human-computer interaction mode of the electronic device is added, and the electronic device is realized. The virtual instrument function allows users to play instruments anytime, anywhere.

In the embodiment of the present application, the virtual musical instrument performance device 1000 further includes: a processing unit 1006, configured to determine the control parameters of the master control object to control the controlled object according to the motion information; the playback unit 1004 is specifically configured to: according to the control parameters, play the target audio.

In the above-mentioned embodiment provided by this application, the virtual musical instrument performance device 1000 further includes a processing unit 1006, when the target audio is played based on the motion information of the master object relative to the controlled object, the processing unit 1006 performs The motion information of the control object is determined, and the control parameter of the master control object to control the controlled object is determined, and the playing unit 1004 further plays the target audio corresponding to the controlled object according to the control parameter. In this way, according to the specific control parameters of the master object to the controlled object, the target audio corresponding to the controlled object is played, thereby realizing the virtual musical instrument function of the electronic device, so that the user can play the musical instrument anytime and anywhere.

In the embodiment of the present application, the control parameters include the controlled object, the control duration, the number of times of control, and the control speed, and the playback unit 1004 is specifically configured to: play at least the musical instrument associated with the controlled object according to the control duration, the number of times of control, and the control speed. The audio corresponding to a musical tone.

In the above-mentioned embodiment provided by the present application, the above-mentioned control parameters include the controlled object, the control duration, the control times and the control speed. The speed is controlled, and the audio corresponding to at least one musical tone of the musical instrument associated with the controlled object is played. In this way, according to the specific control parameters of the master object to the controlled object, the instrument audio corresponding to the controlled object is played according to the corresponding playback parameters, and the virtual musical instrument can be flexibly and accurately controlled to sound, so as to obtain an accurate target that meets the user's requirements. Audio, realizes the virtual musical instrument function of electronic devices, so that users can play musical instruments anytime and anywhere.

In the embodiment of the present application, when the target interface includes at least two controlled objects, the processing unit 1006 is specifically configured to: obtain the real-time pixel distance between the main control object and each controlled object in the target interface according to the motion information; A controlled object whose real-time pixel distance is less than a preset threshold is determined as a controlled object that can play audio.

In the above-mentioned embodiment provided by this application, in the case that the target interface includes at least two controlled objects, the processing unit 1006 obtains the main control object and each controlled object in the target interface according to the motion information of the main control object relative to each controlled object. The real-time pixel distance of the controlled object, and then the controlled object whose real-time pixel distance is less than the preset threshold is determined as the controlled object that can play audio. In this way, the controlled object that can play audio is determined according to the real-time pixel distance between the main control object and each controlled object, which ensures the accuracy and real-time performance of the controlled object determination, thereby ensuring the accuracy and fluency of the target audio playback. , which improves the reliability of the virtual performance function of electronic devices.

In the embodiment of the present application, the playback unit 1004 is specifically used to: adjust the playback duration of the target audio according to the control duration; adjust the playback frequency of the target audio according to the number of controls; adjust the playback volume of the target audio according to the control speed; , playing frequency and playing volume, and play the audio corresponding to at least one musical tone of the musical instrument associated with the controlled object.

In the above-mentioned embodiment provided in this application, the processing unit 1006 determines the triggered controlled object and the control duration, control times and control speed of the controlled object by the main control object according to the motion information of the controlled object relative to the controlled object. Afterwards, the playback unit 1004 adjusts the playback duration, playback frequency and playback volume of the target audio respectively according to the determined control duration, control times and control speed, and then according to the adjusted playback duration, playback frequency and playback volume, playback and the triggered Audio corresponding to at least one tone of the musical instrument associated with the control object. In this way, according to the specific control parameters of the master object to the controlled object, and play the musical instrument audio corresponding to the controlled object according to the corresponding playback parameters, the virtual musical instrument can be flexibly and accurately controlled to sound, so as to obtain the target audio that meets the user's requirements. The virtual instrument function of the electronic device is realized, so that the user can play the instrument anytime and anywhere.

In the embodiment of the present application, the real-sensing pixel sensor includes a plurality of real-sensing pixels; the acquiring unit 1002 is specifically configured to: acquire the pixel change values of the multiple real-sensing pixels in real time; the processing unit 1006 is specifically used to: target real-sensing pixels among the multiple real-sensing pixels When the pixel change value of the pixel is outside the preset threshold range, output target information, where the target information includes at least one of the following: position information, ambient brightness information, and time information of the target real-sensing pixel; according to the target information, determine the motion information ; wherein, the acquisition frequency of motion information is related to the acquisition frequency of pixel change values of a plurality of real-sensing pixels.

In the above-mentioned embodiment provided by the present application, the above-mentioned real-sensing pixel sensor is provided with a plurality of real-sensing pixels, the acquisition unit 1002 acquires the pixel change values of the multiple real-sensing pixels in real time, and the processing unit 1006 obtains the pixel change of the target real-sensing pixel among the multiple real-sensing pixels. When the value is greater than or equal to the preset threshold, the position information, ambient brightness information and time information of the target real-sensing pixel are output, and the motion information of the master object relative to the multiple controlled objects is determined according to the output information. In this way, due to the high real-time performance of real-sensing pixels, imaging through real-sensing pixels in electronic devices can quickly and accurately determine the specific position of the main control object in the target interface, and achieve precise positioning, so that the main control object can be accurately captured. The position change ensures the accuracy of determining the motion information of the master object relative to the controlled object, thereby ensuring the accuracy of audio playback.

In this embodiment of the present application, the processing unit 1006 is further configured to: set the target object or the target interface area of the target interface to be controlled in the case of receiving the first input from the user to the target object or the target interface area of the target interface object, and establish an association relationship between the controlled object and the target musical instrument, wherein the first input is used to mark the controlled object; in the case of receiving the second input of the user to the master object, the master object is set as The object used to control the sounding of the target instrument, wherein the second input is used to mark the master object.

In the above-mentioned embodiments provided by this application, the processing unit 1006 sets multiple objects or multiple interface areas in the target interface as controlled objects according to the user's touch input on the target interface, and establishes a relationship between the controlled object and the target musical instrument. , and set the master object in the target interface, such as the user's hand or other movable objects, as the object used to control the sound of the target instrument. In this way, a complete piece of music can be obtained by virtually playing multiple objects or multiple interface areas in the actual scene, thereby realizing the virtual musical instrument function of the electronic device and increasing the human-computer interaction mode of the electronic device.

The virtual musical instrument performance apparatus 1000 in this embodiment of the present application may be an electronic device, or may be a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices other than the terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a notebook computer, a PDA, a vehicle electronic device, a Mobile Internet Device (MID), an augmented reality (AR)/virtual reality (VR) ) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS) ), personal computer (personal computer, PC), television (television, TV), teller machine or self-service machine, etc., which are not specifically limited in the embodiments of the present application.

The virtual musical instrument performance apparatus 1000 in this embodiment of the present application may be an apparatus having an operating system. The operating system may be an Android (Android) operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The virtual musical instrument performance device 1000 provided by the embodiment of the second aspect of the present application can implement each process implemented by the method embodiment in FIG. 1 , and to avoid repetition, details are not described here.

Optionally, as shown in FIG. 11 , an embodiment of the present application further provides an electronic device 1100, including a processor 1102 and a memory 1104, and the memory 1104 stores a program or instruction that can run on the processor 1102, the program or When the instruction is executed by the processor 1102, each step of the above-mentioned embodiment of the virtual musical instrument performance method of the first aspect can be realized, and the same technical effect can be achieved. To avoid repetition, details are not repeated here.

It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

FIG. 12 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 1200 includes but is not limited to: a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, a processor 1210 and other components .

Those skilled in the art can understand that the electronic device 1200 may also include a power supply (such as a battery) for supplying power to various components, and the power supply may be logically connected to the processor 1210 through a power management system, so as to manage charging, discharging, and power management through the power management system. consumption management and other functions. The structure of the electronic device shown in FIG. 12 does not constitute a limitation on the electronic device. The electronic device may include more or less components than the one shown, or combine some components, or arrange different components, which will not be repeated here. .

The electronic device 1200 in this embodiment of the present application can be used to implement each step of the above-mentioned first aspect of the virtual musical instrument performance method embodiment.

The processor 1210 is configured to acquire motion information of the master object relative to the controlled object in the target interface through the real-sensing pixel sensor.

The audio output unit 1203 is used for playing the target audio based on the motion information.

Wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument; the target audio includes audio corresponding to at least one musical tone of at least one musical instrument.

In the embodiment of the present application, for the electronic device 1200 including the RealSense pixel sensor, the processor 1210 thereon obtains the motion information of the master object relative to the controlled object in the target interface of the electronic device through the RealSense pixel sensor. Wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument. On this basis, the audio output unit 1203 plays target audio according to the motion information of the master object relative to the controlled object, where the target audio includes audio corresponding to at least one tone of at least one musical instrument. In this way, the human-computer interaction mode of the electronic device is added, and the virtual musical instrument can be played through the motion information of the master object in the target interface of the electronic device relative to the controlled object, thereby realizing the virtual musical instrument function of the electronic device, making Users can play instruments anytime, anywhere.

Optionally, the master object is a moving object, and the master object includes a movable object of the user or the user's hand; the controlled object includes an object other than the physical musical instrument, or an interface area in the target interface; the motion information includes the main object. The real-time position information of the control object, the stay time of the main control object at each position, and the real-time movement speed of the main control object, wherein the real-time position information includes the position coordinates of at least one position; the target interface includes the shooting preview interface of the electronic device or the virtual UI.

In the above-mentioned embodiments provided in the present application, through the motion information of the master object in the target interface of the electronic device relative to the controlled object, the actual musical instrument can be simulated to perform, and the human-computer interaction mode of the electronic device is added, and the electronic device is realized. The virtual instrument function allows users to play instruments anytime, anywhere.

Optionally, the processor 1210 is further configured to: determine, according to the motion information, a control parameter for the master object to control the controlled object; the audio output unit 1203 is specifically configured to: play the target audio according to the control parameter.

In the above-mentioned embodiments provided in this application, when the target audio is played based on the motion information of the master object relative to the controlled object, the processor 1210 determines that the master object controls the controlled object according to the motion information of the master object relative to the controlled object. The control parameter of the object, and the audio output unit 1203 then plays the target audio corresponding to the controlled object according to the control parameter. In this way, according to the specific control parameters of the master object to the controlled object, the target audio corresponding to the controlled object is played, thereby realizing the virtual musical instrument function of the electronic device, so that the user can play the musical instrument anytime and anywhere.

Optionally, the control parameters include the controlled object, the control duration, the number of times of control, and the control speed, and the audio output unit 1203 is specifically used to: play at least one musical tone of the musical instrument associated with the controlled object according to the control duration, the number of times of control, and the control speed. corresponding audio.

In the above-mentioned embodiment provided by this application, the above-mentioned control parameters include the controlled object, the control duration, the control times and the control speed. and control speed, and play audio corresponding to at least one tone of the musical instrument associated with the controlled object. In this way, according to the specific control parameters of the master object to the controlled object, and play the musical instrument audio corresponding to the controlled object according to the corresponding playback parameters, the virtual musical instrument can be flexibly and accurately controlled to sound, so as to obtain the target audio that meets the user's requirements. The virtual instrument function of the electronic device is realized, so that the user can play the instrument anytime and anywhere.

Optionally, when the target interface includes at least two controlled objects, the processor 1210 is specifically configured to: obtain the real-time pixel distance between the master object and each controlled object in the target interface according to the motion information; The controlled object whose distance is less than the preset threshold is determined as the controlled object that can play audio.

In the above-mentioned embodiment provided by the present application, in the case that the target interface includes at least two controlled objects, the processor 1210 obtains the main control object and each controlled object in the target interface according to the motion information of the main control object relative to each controlled object. The real-time pixel distance of the controlled object, and then the controlled object whose real-time pixel distance is less than the preset threshold is determined as the controlled object that can play audio. In this way, the controlled object that can play audio is determined according to the real-time pixel distance between the main control object and each controlled object, which ensures the accuracy and real-time performance of the controlled object determination, thereby ensuring the accuracy and fluency of the target audio playback. , which improves the reliability of the virtual performance function of electronic devices.

Optionally, the audio output unit 1203 is specifically used to: adjust the playback duration of the target audio according to the control duration; adjust the playback frequency of the target audio according to the number of controls; adjust the playback volume of the target audio according to the control speed; Frequency and playback volume, play audio corresponding to at least one musical tone of the instrument associated with the controlled object.

In the above-mentioned embodiment provided in this application, the processor 1210 determines the triggered controlled object and the control duration, control times and control speed of the controlled object by the main control object according to the motion information of the controlled object relative to the controlled object. Afterwards, the audio output unit 1203 adjusts the playback duration, playback frequency and playback volume of the target audio respectively according to the determined control duration, control times and control speed, and then according to the adjusted playback duration, playback frequency and playback volume, playback and triggered Audio corresponding to at least one tone of the musical instrument associated with the controlled object. In this way, according to the specific control parameters of the master object to the controlled object, and play the musical instrument audio corresponding to the controlled object according to the corresponding playback parameters, the virtual musical instrument can be flexibly and accurately controlled to sound, so as to obtain the target audio that meets the user's requirements. The virtual instrument function of the electronic device is realized, so that the user can play the instrument anytime and anywhere.

Optionally, the real-sensing pixel sensor includes a plurality of real-sensing pixels; the processor 1210 is specifically configured to: obtain the pixel change values of the multiple real-sensing pixels in real time; in the multiple real-sensing pixels, the pixel change value of the target real-sensing pixel is located within the preset threshold range. In other cases, the target information is output, wherein the target information includes at least one of the following: position information, environmental brightness information, and time information of the target real-sensing pixel; according to the target information, determine the motion information; wherein, the acquisition frequency of the motion information is related to the frequency of the motion information. The acquisition frequency of the pixel change value of each real-sensing pixel is related.

In the above-mentioned embodiment provided in this application, the above-mentioned real-sensing pixel sensor is provided with a plurality of real-sensing pixels, the processor 1210 obtains the pixel change values of the multiple real-sensing pixels in real time, and the pixel change value of the target real-sensing pixel in the multiple real-sensing pixels is greater than When it is equal to the preset threshold, the position information, ambient brightness information and time information of the target real-sensing pixel are output, and the motion information of the master object relative to the multiple controlled objects is determined according to the output information. In this way, due to the high real-time performance of real-sensing pixels, imaging through real-sensing pixels in electronic devices can quickly and accurately determine the specific position of the main control object in the target interface, and achieve precise positioning, so that the main control object can be accurately captured. The position change ensures the accuracy of determining the motion information of the master object relative to the controlled object, thereby ensuring the accuracy of audio playback.

Optionally, the processor 1210 is further configured to: set the target object or the target interface area of the target interface as the controlled object in the case of receiving the first input from the user to the target object or the target interface area of the target interface, and Establish an association relationship between the controlled object and the target musical instrument, wherein the first input is used to mark the controlled object; in the case of receiving the second input from the user to the master control object, the master control object is set to be used for controlling The object on which the target instrument sounds, wherein the second input is used to label the master object.

In the above-mentioned embodiments provided in this application, the processor 1210 sets multiple objects or multiple interface regions in the target interface as controlled objects according to the user's touch input on the target interface, and establishes a relationship between the controlled object and the target musical instrument. , and set the master object in the target interface, such as the user's hand or other movable objects, as the object used to control the sound of the target instrument. In this way, a complete piece of music can be obtained by virtually playing multiple objects or multiple interface areas in the actual scene, thereby realizing the virtual musical instrument function of the electronic device and increasing the human-computer interaction mode of the electronic device.

It should be understood that, in this embodiment of the present application, the input unit 1204 may include a graphics processor (Graphics Processing Unit, GPU) 12041 and a microphone 12042. camera) to process the image data of still pictures or videos. The display unit 1206 may include a display panel 12061, which may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 1207 includes at least one of a touch panel 12071 and other input devices 12072 . The touch panel 12071 is also called a touch screen. The touch panel 12071 may include two parts, a touch detection device and a touch controller. Other input devices 12072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which are not described herein again.

The memory 1209 may be used to store software programs as well as various data. The memory 1209 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, image playback function, etc.), etc. Additionally, memory 1209 may include volatile memory or non-volatile memory, or memory 1209 may include both volatile and non-volatile memory. Wherein, the non-volatile memory may be Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (Erasable PROM, EPROM), Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. Volatile memory can be random access memory (Random Access Memory, RAM), static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous random access memory) DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synch link DRAM) , SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 1209 in this embodiment of the present application includes, but is not limited to, these and any other suitable types of memory.

The processor 1210 may include one or more processing units; optionally, the processor 1210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations involving an operating system, a user interface, and an application program, etc. Modem processors mainly deal with wireless communication signals, such as baseband processors. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 1210.

Embodiments of the present application further provide a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, each process of the above-mentioned first aspect of the virtual musical instrument performance method embodiment is implemented, and can To achieve the same technical effect, in order to avoid repetition, details are not repeated here.

The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disk, and the like.

An embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement each process of the first aspect of the virtual musical instrument performance method embodiment, and The same technical effect can be achieved, and in order to avoid repetition, details are not repeated here.

It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, a system-on-chip, a system-on-a-chip, or a system-on-a-chip, or the like.

The embodiments of the present application provide a computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the various processes of the above-mentioned first aspect virtual musical instrument performance method embodiment, and can achieve The same technical effect, in order to avoid repetition, will not be repeated here.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in the reverse order depending on the functions involved. To perform functions, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to some examples may be combined in other examples.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present application can be embodied in the form of computer software products that are essentially or contribute to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk , CD-ROM), including several instructions to enable a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the inspiration of this application, without departing from the scope of protection of the purpose of this application and the claims, many forms can be made, which all fall within the protection of this application.

Claims (10)

1. a virtual musical instrument performance method, is characterized in that, is applied to the electronic equipment comprising real-sense pixel sensor, and described virtual musical instrument performance method comprises: Obtain the motion information of the master object in the target interface relative to the controlled object through the real-sensing pixel sensor; Based on the motion information, play the target audio; Wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument; the target audio includes audio corresponding to at least one musical tone of at least one musical instrument. 2. virtual musical instrument performance method according to claim 1, is characterized in that, described master control object is motion object, and described master control object comprises user's movable object or user's hand; The controlled object includes objects other than physical musical instruments, or an interface area in the target interface; The motion information includes the real-time position information of the main control object, the stay time of the main control object at each position, and the real-time movement speed of the main control object, wherein the real-time position information includes at least one position. Position coordinates; The target interface includes a shooting preview interface or a virtual display interface of the electronic device. 3. virtual musical instrument performance method according to claim 1, is characterized in that, described based on described motion information, playing target audio frequency, comprises: According to the motion information, determine the control parameters of the master object to control the controlled object; According to the control parameters, the target audio is played. 4. virtual musical instrument performance method according to claim 3, is characterized in that, described control parameter comprises controlled object, control duration, control number of times and control speed; Playing the target audio according to the control parameters includes: Play audio corresponding to at least one musical tone of the musical instrument associated with the controlled object according to the control duration, the number of times of control and the control speed. 5 . The virtual musical instrument performance method according to claim 3 , wherein, in the case that the target interface includes at least two controlled objects, it is determined according to the motion information that the controlled objects are controlled by the master object. 6 . Describe the control parameters of the controlled object, including: According to the motion information, obtain the real-time pixel distance between the master object and each controlled object in the target interface; A controlled object whose real-time pixel distance is less than a preset threshold is determined as a controlled object that can play audio. 6. virtual musical instrument performance method according to claim 4, is characterized in that, according to described control duration, described control frequency and described control speed, play at least one musical tone of the musical instrument associated with described controlled object Corresponding audio, including: Adjust the playback duration of the target audio according to the control duration; Adjust the playback frequency of the target audio according to the control times; Adjust the playback volume of the target audio according to the control speed; According to the playback duration, the playback frequency and the playback volume, audio corresponding to at least one musical tone of the musical instrument associated with the controlled object is played. 7. The virtual musical instrument performance method according to any one of claims 1 to 6, wherein the real-sensing pixel sensor comprises a plurality of real-sensing pixels; The acquisition of the motion information of the master object in the target interface relative to the controlled object through the real-sensing pixel sensor includes: acquiring the pixel change values of the plurality of RealSense pixels in real time; In the case where the pixel change value of the target photorealistic pixel in the plurality of photorealistic pixels is outside the preset threshold range, output target information, wherein the target information includes at least one of the following: position information of the target photorealistic pixel , ambient brightness information, time information; According to the target information, determine the motion information; Wherein, the acquisition frequency of the motion information is related to the acquisition frequency of pixel change values of the plurality of real-sensing pixels. 8. The virtual musical instrument performance method according to any one of claims 1 to 6, characterized in that, before obtaining the motion information of the master object in the target interface relative to the controlled object through the real-sensing pixel sensor , the virtual musical instrument performance method further includes: In the case of receiving the first input from the user to the target object or the target interface area of the target interface, the target object or the target interface area of the target interface is set as the controlled object, and the controlled object and the target musical instrument are established. The association relationship between, wherein, the first input is used to mark the controlled object; In the case of receiving a second input from the user to the master object, the master object is set as an object for controlling the target musical instrument to sound, wherein the second input is used to mark the master object. 9. A virtual musical instrument performance device, characterized in that, applied to an electronic device comprising a real-sensing pixel sensor, and the virtual musical instrument performance device comprises: an acquisition unit, used for acquiring the motion information of the master object in the target interface relative to the controlled object through the real-sensing pixel sensor; a playing unit for playing target audio based on the motion information; Wherein, different controlled objects correspond to different musical instruments, or different controlled objects correspond to different parts of the same musical instrument; the target audio includes audio corresponding to at least one musical tone of at least one musical instrument. 10. An electronic device, characterized in that it comprises a processor and a memory, wherein the memory stores a program or an instruction that can be executed on the processor, and when the program or instruction is executed by the processor, the implementation as claimed in the claims Steps of the virtual musical instrument performance method described in any one of 1 to 8.

Images