CN103309447B - The virtual data being carrier with mankind's both hands obtains and transmission method - Google Patents

Abstract

The present invention provides a virtual data acquisition and transmission method using human hands as the carrier. The steps are as follows: the first step, the user makes a gesture in front of the terminal device, and the terminal device recognizes the user's gesture; the second step, according to the first The result of gesture recognition in the first step, the terminal device sends the corresponding data to the cloud; the third step, the cloud verifies the user's identity, and queries and updates the data information in the user's hand, so as to complete the corresponding operation of grabbing data from the device to the hand, and the The corresponding operation of putting data into the device from the hand, the corresponding operation of clearing the data in the hand, or the operation of transferring data between hands. In the fourth step, the cloud sends the corresponding operation result to the terminal device, and the terminal device displays the operation result. The present invention can realize that human beings use their own body and hands as the carrier of virtual data acquisition and transmission, and complete the acquisition and sharing of digital information between electronic devices or even between people by virtue of intuitive actions.

Description

Virtual Data Acquisition and Transmission Method Using Human Hands as Carriers

technical field

The invention relates to the technical field of computer data interaction, in particular to a virtual data acquisition and transmission method using human hands as carriers.

Background technique

At present, digital information is read and transmitted through physical media in computers and network systems. If users want to obtain and transmit these digital information, they can only use certain file system functions to achieve their goals. Traditional data access or transmission methods are based on computer primitive operations, including computer local "read/write" primitives and network layer "send/receive" primitives.

There are also motion recognition methods in the prior art, such as the patented invention of Chinese Patent Publication No. 200610058529, which provides a data acquisition device, data acquisition method and program. Solve the problem that since the material data is acquired in the specified acquisition order, when the acquisition order is not continuous, it is necessary to mount or redesignate the same recording medium multiple times. The data acquisition device that acquires material data according to the acquisition queue is composed of the following parts: (a) a file search unit that searches for a scheduled document within the range of the recording medium specified as the acquisition source of the material data regardless of the order of acquisition specified by the acquisition queue. Acquired material files; (b) the search result display unit, when there are material files to be acquired, it takes all corresponding material files as priority acquisition objects and explicitly displays them on the operation screen.

Another example is the patented invention of Chinese Patent Publication No. 201110278157, which provides a wireless network communication system and a data transmission method thereof. The method mainly includes: connecting REC (Radio Equipment Control, wireless equipment control), RE (Radio Equipment, wireless equipment) and switching network to each other through optical fibers, one REC manages at least one of the REs, and one of the REs belongs to one of the REC management, communication between the REC and the RE, and between two RECs through the switching network. The embodiment of the present invention solves the problems of too many layers, numerous interfaces and low resource utilization in the existing system.

The defect of the current virtual data acquisition and transmission method is that the method based on computer primitives allows machines rather than humans to be the leader of data dissemination, and instructional operations are not intuitive and too complicated for humans, and require a certain amount of work. Learn, not master by nature.

Contents of the invention

In view of the defects in the prior art, the purpose of the present invention is to provide a human-oriented virtual data acquisition and transmission method, which realizes that human beings use their own body and hands as the carrier of virtual data acquisition and transmission, and rely on intuitive actions To complete the acquisition and sharing of digital information between electronic devices, and even between people.

In order to achieve the above-mentioned purpose, the virtual data acquisition and transmission method using human hands as the carrier according to the present invention, the user completes grabbing data from the device in the hand through corresponding gestures, carries the data in the body, and puts the data into (copy or migrate) to another device, or pass (copy) data from one person to another.

Method of the present invention comprises the steps:

In the first step, the user makes a gesture in front of the terminal device, and the terminal device recognizes the user's gesture;

The user makes gestures on the touch screen, wherein there are four kinds of user gestures including "grab", "put", "give" and "throw", and distinguish the left hand from the right hand;

"Grab": When the user makes a "grab" action on the screen, the hand that makes the action gets the file displayed on the screen;

"Put": When the user performs the "Put" action on the screen, if his hand is not "empty", the file in his hand will be put into the corresponding device through the screen;

"Give": Two hands belonging to two users are required to cooperate to complete the action, that is, the file is transferred from one hand to the other;

"Throw": When the user performs the "throw" action, the file in the hand of the action will be cleared;

"File": a collection of information stored on a computer with a computer hard disk as a carrier;

The recognition of user gestures refers to the use of a touch screen and a human body motion capture camera to identify the four gesture primitives of "grab", "put", "give" and "throw". All gestures can be detected by the human body motion capture camera. Captured, while "grab" and "drop" are captured by a body motion capture camera or touch screen.

In the second step, according to the result of gesture recognition in the first step, the terminal device sends the corresponding data to the cloud.

The terminal device will send the action identified by the action module and the current user identity information to the cloud, and if it detects that the action is "give", it will also send the involved files and file information to the cloud;

This part can be transmitted using the standard http protocol. For the convenience of various application developers in the future, a web service can also be established to use the soap protocol for transmission.

In the third step, the cloud verifies the user's identity, and queries and updates the data information in the user's hand, so as to complete the corresponding operation of grabbing data from the device to the hand, putting the data from the hand into the device, and clearing the data in the hand. Corresponding operation, or the operation of data transfer between hands.

There is a user name stored in the cloud, and the cloud will record the corresponding operations and files under the corresponding user's counterpart based on the data received in the second step.

If the operation is "grab", the received file will be saved under the corresponding user's corresponding hand, so as to complete the corresponding operation of grabbing data from the device to the hand;

If the operation is "Put", the file of the corresponding subordinate of the corresponding user is transmitted to the terminal device, so as to complete the corresponding operation of putting the data from the hand into the device;

If the operation is "give", copy the files under the corresponding hands of the "giving" party to the corresponding hands of the "given" party;

If the operation is "throw", the file of the corresponding user of the corresponding user will be emptied.

The purpose of the above data storage and operation is to ensure that the data acquisition and transmission completed by the cloud can be consistent with the data capture and delivery expected by users

In the fourth step, the cloud sends the corresponding operation result to the terminal device according to the operation in the third step, and the terminal device displays the operation result.

The cloud sends information such as whether the operation is successful or not back to the terminal device, and the terminal device displays the result in text or multimedia form to give feedback to the user.

As mentioned above, the cloud refers to a cloud server, but it can also be replaced by an ordinary server.

The cloud verifies the identity of the user, using technologies such as face recognition, fingerprint authentication, or human body current authentication. Preferably, a method based on face recognition is used to realize user identity authentication. The user's face information is captured by the camera installed on the terminal device, and the user's face information and corresponding user name are stored in the cloud, and the recognition of the user's face is completed by the cloud. The reason for adopting the method based on face recognition to realize the user identity authentication method is to allow the user to complete the user identity authentication without carrying any external devices and doing redundant operations. Using the cloud to identify the user's face can solve the large amount of calculation of face recognition.

Preferably, the files under the user's corresponding hands are defined as follows:

A user has two hands, and the left and right hands can carry different data respectively. A maximum of M files can be held in one hand;

"Empty": the user's hand is "empty", which means that the number of files in the user's hand is 0;

"Full": the number of files in the user's hand reaches M;

"Timeout": A file in the user's hand has been operated for more than T from the last operation to the current time, it is considered "timeout", and the "timeout" file will disappear from the user's hand;

"Not timed out": A file in the user's hand from the latest operation to the current time length is less than T, it is considered "not timed out", the "not timed out" file will remain in the hand, can be done as "give" and " put" operation;

The above M and T can be specifically set according to user needs.

Preferably, the specific implementation method of the face recognition is as follows: a face detection module is set at the client end, a face recognition module is set at the cloud server end, the face detection module shoots user images through a camera installed on the client end, and utilizes human The face detection algorithm searches for face pictures in the video pictures, and sends the detected face pictures in the video pictures to the face recognition module on the server side. The face recognition module records the user's face data. After receiving the face picture, the face recognition module will calculate and analyze the picture, and the recognized result is the user name.

The present invention realizes operator identity and gesture recognition through three human-computer interaction technologies of touch control, face recognition and human motion recognition, solves data buffering, transmission and name space problems through cloud storage, and uses high-performance cloud computing The background solves the image analysis, the concurrency pressure generated by the joint operation of thousands of users, and the large-scale and long-delay learning process required for face recognition.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention is designed to realize the acquisition and sharing of digital information between electronic devices, and even among people, by human beings through their bodies, especially their hands, with the most natural intuition. Shield the transmission and sharing of data information for additional operations (such as copying of storage media, network transmission), making people a virtual carrier for data transmission, and realizing a natural (closer to human habits) data transmission and sharing mode.

Users "grab" and "drop" on the screen with their real hands to "transfer (copy or migrate)" data from one electronic device to another. The user's left and right hands can grab and put different files, and even "give" the data in the hands to another user. The present invention enables "data access", the most frequent operation between computers and users, to realize virtual data transmission that users can grab and release with both hands by technologies such as human-computer interaction and cloud computing.

Description of drawings

Other characteristics, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

Figure 1 is the action diagram definition in the embodiment of the present invention.

detailed description

The present invention will be described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

This embodiment provides a virtual data acquisition and transfer method using humans as the carrier. Users can use corresponding gestures to complete grabbing data from the device in their hands, carrying the data on the body, and putting (copying or migrating) the data to another A device that transfers (replicates) data from one hand to another.

In the first step, the user makes a gesture in front of the terminal device, and the terminal device recognizes the user's gesture;

The user makes a gesture on the touch screen, and there are four user gestures including "catch", "put", "give" and "throw", and distinguish between the left hand and the right hand;

"Grab": When the user makes a "grab" action on the screen, the hand that makes the action gets the file displayed on the screen;

"Put": When the user performs the "Put" action on the screen, if his hand is not "empty", the file in his hand will be put into the corresponding device through the screen;

"Give": Two hands belonging to two users are required to cooperate to complete the action, that is, the file is transferred from one hand to the other;

"Throw": When the user performs the "throw" action, the file in the hand of the action will be cleared;

"File": a collection of information stored on a computer with a computer hard disk as a carrier;

The present invention uses a touch screen and a human motion capture camera to identify the four gesture primitives of "grab", "put", "give" and "throw". All gestures can be recognized by the body motion capture camera, while "grab" and "drop" can also be recognized by the touch screen. Action diagrams are defined in Figure 1. In Figure 1, a-d are gestures based on the real-time depth-sensing camera; e-f are gestures based on the touch screen.

On the touch screen, the "grab" action is that at least three fingers including the thumb of the same hand touch the screen and slide from outside to inside. The "grabbed" files are the corresponding files located within the range of the geometric centers of three (or four, five) finger touch points on the screen. For the "grab" operation of a non-touch screen (such as a projection), the finger movement is exaggerated as a joint movement of the arm and hand, and recognition requires the assistance of a motion capture camera. The action of "putting" is just the opposite of "grasping". It is a movement of sliding from inside to outside with at least four fingers including the thumb. The action of "give" is to hold with both hands. "Throwing" means raising your hand to your ear and shaking it twice.

In the second step, according to the result of gesture recognition in the first step, the terminal device sends the corresponding data to the cloud.

The terminal device will send the action identified by the action module and the current user identity information to the cloud, and if it detects that the action is "give", it will also send the involved files and file information to the cloud;

The standard http protocol can be used for transmission. For the convenience of various application developers in the future, a web service can also be established to use the soap protocol for transmission.

In the third step, the cloud verifies the user's identity, and a method based on face recognition can be used to realize user identity authentication. And query and update the data information in the user's hand, so as to complete the corresponding operation of grabbing data from the device to the hand, the corresponding operation of putting the data from the hand into the device, the corresponding operation of clearing the data in the hand, or the hand-to-hand The operation of data transfer between.

1. There is a user name stored in the cloud, and the cloud will record the corresponding operations and files under the corresponding user's counterpart based on the data received in the second step.

If the operation is "grab", the received file will be saved under the corresponding user's corresponding hand, so as to complete the corresponding operation of grabbing data from the device to the hand;

If the operation is "Put", the file of the corresponding subordinate of the corresponding user is transmitted to the terminal device, so as to complete the corresponding operation of putting the data from the hand into the device;

If the operation is "give", copy the files under the corresponding hands of the "giving" party to the corresponding hands of the "given" party;

If the operation is "throw", the file of the corresponding user of the corresponding user will be emptied.

The purpose of the above data storage and operation is to ensure that the data acquisition and transmission completed by the cloud can be consistent with the data capture and delivery expected by the user.

In this embodiment, the face recognition can be realized by the following modules:

Face detection module: The face detection module is located on the client side. It shoots user images through the camera installed on the client side, uses the face detection algorithm to search for face pictures in the video pictures, and the face detection module will detect the images in the video pictures. The face picture sent to the face recognition module on the server side;

Face recognition module: The face recognition module is located on the server side, and the face recognition module records the user's face data. After receiving the face picture sent by the client, the face recognition module will calculate and analyze the picture, and The recognized result (namely the user name) is sent back to the client; the current user name is the result obtained by the analysis of the server-side face recognition module. When the client receives the user name returned by the server, the current user in the client is will be changed to the username;

Action recognition module: The action recognition module is located on the client side, and it consists of two parts: a gesture recognition module based on a touch screen and a gesture recognition module based on a real-time depth-sensing camera. According to the different usage scenarios, you can choose to capture the user's hand movements or whole body movements through the real-time depth-sensing camera or multi-touch screen device, and then recognize the user's gestures. Once the action recognition module detects that a defined action has occurred, the client will send the action recognized by the action module and the identity of the current user recognized by the face recognition module to the server, and corresponding to the relevant action, The involved files and file information will also be transmitted to the server, and the data cache module at the server will process the received data.

Data cache module: The data cache module is located on the server side, receives the data sent by the client, performs corresponding calculations on the data, stores relevant information, and returns the result to the client. The motion recognition module of the client will send the recognized motion to the server. The files and file information involved will also be transmitted to the server if required by the relevant action. According to different actions, the server will perform different processing, and finally return the processing result.

2. Supplementary instructions for the implementation of gestures corresponding to cloud operations

In this embodiment, a file set S{F1, F2, F3, . The files F in S are arranged in descending order according to the length of time they are put in, that is, they are put in first and then put in later.

(1) When the user takes the "grab" action and grabs the file F:

If the number of files in S is less than M, add file F at the end of S, and the pointer P is assigned the size of S, that is, P points to the file at the end of set S;

If the number of files in S is equal to M, delete the first file F1 in S, add file F at the end of S, and assign the pointer P to the size of S, that is, P points to the file at the end of the set S;

(2) When the user makes a "release" action on a certain device:

If the pointer P is not equal to 0, the backup of the file Fp pointed to by the pointer P in the user's hand is sent to the device, and the pointer P is reduced by one (that is, it points to the previous file);

If the pointer P is equal to 0 and the number of files in the hand is not 0, the pointer P is assigned the size of S, that is, it points to the end file of S, and the backup of the file Fp pointed to by the pointer P in the user's hand is sent to the device;

If the number of files in hand is 0, the "put" action operation fails;

(3) When the user makes a "throw" action:

Then clear all the files in the hand of the action;

(4) When the user makes a "give" action

If one hand A participating in the "give" action is not "empty" and the other hand B is "empty", then copy all the files in the file collection SA of the hand A to SB, and SB refers to the hand B Document collection; otherwise the "give" action fails.

In this way, when the user completes a "grab" action, his corresponding hand obtains a backup of the file just grabbed from the screen. With the file in hand, the user can "put" the backup of the file to a certain machine through the screen. The "put" operation will not make the user lose the file in his hand, so the user can "put" the file in his hand any number of times without "timeout". The "grab" and "put" of the file in hand are similar to the push and pop of the stack structure.

"To" means to pass data from one user's hand to that user's or another user's hand. "Give" will only happen if both hands involved in the action have one and only one hand is not "empty", while the other hand is "empty". When the "give" happens, the hand without the file will get a backup of all the files in the hand with the file.

There are only two ways to make the hand empty, that is, to make the hand state "empty". The first way is to use the "throw" action. When the "throw" action is made, the corresponding hand will be emptied. The second way is to wait for a file in one hand to "time out". Whenever a certain file in one hand participates in the operation of "putting", "taking" and "giving", the latest operation time of this file is updated to the current time.

In the fourth step, the cloud sends the corresponding operation result to the terminal device according to the operation in the third step, and the terminal device displays the operation result.

The cloud sends information such as whether the operation is successful or not back to the terminal device, and the terminal device displays the result in text or multimedia form to give feedback to the user.

The invention makes users feel that they have data in their hands, but the real data transmission path actually goes around to the cloud. For example, when a user makes a "grab" action, the file he wants to grab will be sent to the cloud together with the corresponding user information. The cloud will store these data under the changed user name, and send it to the corresponding client machine when the user performs the "put" action, or copy it to the corresponding other client machine when the user performs the "give" action. under a username, or delete it on "throw" or "timeout". Specifically, face recognition is used to verify user identity, so as to ensure that the cloud sends data to the correct machine.

The present invention realizes operator identity and gesture recognition through three human-computer interaction technologies of touch control, face recognition and human motion recognition, solves data buffering, transmission and name space problems through cloud storage, and uses high-performance cloud computing The background solves the image analysis, the concurrency pressure generated by the joint operation of thousands of users, and the large-scale and long-delay learning process required for face recognition.

The present invention is designed to realize the acquisition and sharing of digital information between electronic devices, and even among people, by human beings through their bodies, especially their hands, with the most natural intuition. Shield the transmission and sharing of data information for additional operations (such as copying of storage media, network transmission), making people a virtual carrier for data transmission, and realizing a natural (closer to human habits) data transmission and sharing mode.

Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims (7)

1. the virtual data being carrier with mankind's both hands obtains and a transmission method, it is characterized in that, comprises the steps:

The first step, user makes gesture before terminal device, and terminal device identifies user's gesture;

Described user makes gesture on the touchscreen, wherein user's gesture have four kinds comprise " grabbing ", " putting ", " to " and " throwing away ", and distinguish left hand and the right hand;

" grab ": when user has done " a grabbing " action facing to screen, that hand that he does action just obtains the file that screen shows;

" put ": when user has done " putting " action facing to screen, if the hand that he does action is not " sky ", the file in his hand has just put into corresponding equipment by screen;

" give ": need two hands belonging to two users to coordinate execution, namely file is sent in another hand from a hand;

" throw away ": when user has done " throwing away " action, the hand file that he does action will be cleared;

" file ": take hard disc of computer as carrier storage information aggregate on computers;

Described user's gesture to be identified, refer to that using touch-screen and human action to catch camera identifies " grabbing ", " put ", " give " and " throwing away " these four gesture primitive, all gestures can be caught camera by human action and capture, and " grabbing " and " putting " catches camera by human action or touch-screen catches;

Second step, according to the result of first step gesture identification, corresponding data are sent to high in the clouds by terminal device;

The action that action module can recognize by terminal device and current user identities information are sent to high in the clouds, if monitor action be " to ", also the information of involved file and file also can be sent to high in the clouds;

3rd step, high in the clouds identifying user identity, and inquire about and upgrade data message in user's hand, with this complete to capture from equipment data in one's hands in respective operations, data are put into from hand the respective operations of equipment, the respective operations that data in hand are emptied, or the operation that between hand and hand, data are transmitted;

High in the clouds stores user name, the data that high in the clouds can receive according to second step, by corresponding operating and file record under the corresponding hand of respective user, that is:

If be operating as " grabbing ", under the corresponding hand file received being stored in respective user, with this complete to capture from equipment data in one's hands in respective operations;

If be operating as " putting ", by the document backup of the correspondence subordinate of respective user to terminal device, complete with this respective operations data being put into from hand equipment;

If be operating as " to ", by " giving " one side the corresponding hand of user under file copy under the corresponding hand of user of " by " side;

If be operating as " throwing away ", the file of the correspondence subordinate of respective user is emptied;

Above data store and operation guarantees that the data acquisition that high in the clouds completes can be consistent with transmission with the data grabber that user expects with transmission;

4th step, high in the clouds, according to the operation of the 3rd step, sends corresponding operating result to terminal device, terminal device display operating result.

2. the virtual data being carrier with mankind's both hands according to claim 1 obtains and transmission method, it is characterized in that described high in the clouds identifying user identity adopts the method based on recognition of face to realize authenticating user identification; User's facial information is caught by the camera being installed on terminal device, and high in the clouds stores user's face information and respective user name, and the identification of user's face is completed by high in the clouds.

3. the virtual data being carrier with mankind's both hands according to claim 2 obtains and transmission method, it is characterized in that, described recognition of face concrete methods of realizing is: arrange face detection module in client, server end arranges face recognition module beyond the clouds, face detection module is by assembling camera shooting user image on the client, utilize Face datection algorithm in image picture, search for face's picture, and the face picture obtained will be detected in image picture be sent to the face recognition module of server end, face recognition module records the face data of user, after receiving the face picture that client sends, this face picture of face recognition module meeting computational analysis, the result identified and user name.

4. the virtual data being carrier with mankind's both hands according to claim 3 obtains and transmission method, and it is characterized in that, described recognition of face is by realizing with lower module:

Face detection module: face detection module is positioned at client, by assembling camera shooting user image on the client, utilize Face datection algorithm in image picture, search for face's picture, face detection module can detect the face picture obtained and be sent to the face recognition module of server end in image picture;

Face recognition module: face recognition module is positioned at server end, face recognition module records the face data of user, after receiving the face picture that client sends, face recognition module can this face picture of computational analysis, and the result identified and user name are sent back to client; Active user be called server end face recognition module analyze acquired results, when client receive server end return come user name after, the active user in client will be modified to this user name;

Action recognition module: action recognition module is positioned at client, it is by the gesture recognition module based on touch screen and form based on gesture recognition module two parts of real time field depth induction camera, according to the difference of use scenes, select the hand motion or the double that are caught user by real time field depth induction camera or multiple point touching screen equipment, and then identify user's gesture, once motion identification module detects the action that there occurs certain and define, the identity of the action that action module can recognize by client and the active user that face recognition module recognizes is sent to server end, simultaneously corresponding with relevant action, involved file and the information of file also can be sent to server end, by the data cache module of server end, the data received are processed,

Data cache module: data cache module is positioned at server end, receives the data that client is sent, calculates accordingly, relevant information stored, and result is returned to client to data; The action recognized can be sent to server end by the action recognition module of client, if relevant action if required, involved file and the information of file also can be sent to server end, according to different actions, server end can carry out different process, finally returns the result of process.

5. be carrier with the mankind's both hands virtual data obtains and transmission method according to any one of claim 1-4, it is characterized in that, the file under the corresponding hand of user in described 3rd step, is defined as follows:

A user has two hands, and left hand and the right hand can carry different data respectively, can take M file at most in a hand;

" empty ": certain hand of user is " empty " quantity of documents meant in the hand of user is 0;

" full ": the quantity of documents in user's hand reaches M;

" time-out ": certain file in user's hand is operated into current time span more than T from the last time, then think " time-out ", the file of " time-out " can disappear from hand;

" time-out ": certain file in user's hand is operated into current time span from the last time and is less than T, then think " time-out ", the file of " time-out " can be stayed in hand, can be done " to " and the operation of " putting ";

Above M, T specifically need to arrange according to user.

6. be carrier with the mankind's both hands virtual data obtains and transmission method according to any one of claim 1-4, it is characterized in that, by successful operation, whether information sends it back terminal device in described high in the clouds, and terminal device shows this successful operation whether result with feedback user with word or multimedia form.

7. the virtual data being carrier with mankind's both hands according to claim 6 obtains and transmission method, and it is characterized in that, described high in the clouds, refers to Cloud Server, or common server.