CN106803997B - A system and method for detecting client playback status in audio and video live broadcast - Google Patents

Abstract

The present invention relates to a system and method for detecting the playback state of a client in live audio and video broadcasting, wherein the system includes: a collection end, a server and a client, the collection end is connected to the server, and the server is connected to the client; Add the status detection information mark to the audio and video data; the server detects the status detection information mark in the audio and video data, and then decodes the image at this time; the client can obtain the audio and video data from the server and play it; the client is playing For audio and video data, if the mark of the state detection information is detected, the mark of the state detection information is recorded in real time, and a screenshot of the playback area is taken at the same time; if the decoded image corresponding to the mark of the same state detection information is inconsistent with the sent screenshot data Or if the server does not receive the screenshot data sent by the client within the limited time, it is determined that the playback status of the client is abnormal. The invention can accurately detect the current state of the live broadcast client.

Description

A system and method for detecting client playback status in audio and video live broadcast

technical field

The invention belongs to the technical field of network live broadcasting, and in particular relates to a system and method for detecting the playback state of a client in audio and video live broadcasting.

Background technique

Live broadcast, as a way of real-time display, is sought after by more and more people, especially young people. Live broadcasts are divided into text and picture live broadcasts and video live broadcasts. Traditional TV stations mostly focus on video live broadcasts, such as "News Network" for news and "Spring Festival Gala" for performing arts. In the Internet age, live broadcasts with graphics and texts are the main content, such as live broadcasts of sports events and news. In the mobile Internet era, text, pictures, and videos can all be broadcast live. Through the Internet, participants can be the person who initiates the live broadcast, or the person who watches or listens to the live broadcast. Whether the live broadcast status is normal is crucial for both the person who initiates the live broadcast and the person who watches or listens to the live broadcast.

The current audio and video live broadcast status detection only judges whether the live broadcast is successful by whether the data arrives and the signaling is cleared. However, for those who initiate the live broadcast, they do not know the actual operation of the audio and video live broadcast, so this live broadcast status detection technology is not very accurate.

Contents of the invention

In order to solve the above-mentioned technical problem of inaccurate detection of the live broadcast status of the client, the embodiment of the present invention proposes a system and method for detecting the playback status of the client during audio and video live broadcast.

A system for detecting the playback state of a client in an audio and video live broadcast, comprising: a collection terminal, a server and a client, wherein the collection terminal is connected to the server, and the server is connected to the client;

The acquisition end adds the status detection information mark to the collected audio and video data, and uploads the audio and video data with the status detection information mark to the server;

The server detects the received audio and video data, and if the flag of the state detection information is detected, the image at this time is decoded;

The client obtains the audio and video data from the server and plays them; when the client is playing the audio and video data, if it detects the mark of the state detection information, it will record the mark of the state detection information in real time, and take a screenshot of the playback area at the same time, and save the screenshot data and The marked data of the status detection information is sent to the server in real time;

The server compares whether the decoded image corresponding to the tag of the same state detection information is consistent with the sent screenshot data, and if it is inconsistent or the server does not receive the screenshot data and the tag data of the state detection information sent by the client within a limited time, then judge The playback status of the client is abnormal.

Further, when the client sends the screenshot data and the tag data of the state detection information to the server in real time, it also sends the client's identity information together.

Further, the marked frame added with state detection information is a key frame.

Further, the mark of the state detection information is a time stamp and/or a barrage.

Further, when it is determined that the playback status of the client is abnormal, the server feeds back the abnormal playback status of the client to the user who initiates the audio and video live broadcast, or feeds back the identity information of the client whose playback status is abnormal to the server administrator.

A method for detecting the playback state of a client in an audio and video live broadcast, the method comprising:

Acquisition step: the acquisition terminal collects audio and video data;

Marking step: the acquisition terminal regularly adds a mark of status detection information to the collected audio and video data;

Uploading step: after the marking step, the acquisition end uploads the marked audio and video data with status detection information to the server;

The first detection step: the server detects the received audio and video data, if the server detects the mark of the state detection information, the image at this time is decoded;

The second detection step: the client obtains the audio and video data from the server. When the client is playing the audio and video data, if it detects the mark of the state detection information, it will record the mark of the state detection information in real time, and take a screenshot of the playback area at the same time. The screenshot data and the marked data of the status detection information are sent to the server in real time;

Judgment step: the server receives the screenshot data sent by the client and the tag data of the state detection information, and compares whether the decoded image corresponding to the tag of the same state detection information is consistent with the sent screenshot data, if not consistent or within a limited time, the server If the screenshot data sent by the client and the tag data of the state detection information are not received, it is determined that the playback state of the client is abnormal.

Further, in the second detection step, when the client sends the screenshot data and the tag data of the state detection information to the server in real time, it also sends the client's identity information together.

Further, the marked frame added with state detection information is a key frame.

Further, the mark of the state detection information is a time stamp and/or a barrage.

Further, in the judging step, when it is determined that the playback status of the client is abnormal, the server feeds back the abnormal playback status of the client to the user who initiated the audio and video live broadcast, or feeds back the identity information of the client whose playback status is abnormal to the server administrator.

Beneficial effects of the embodiment of the present invention: the system and method for detecting the playback status of the client in the audio and video live broadcast proposed by the embodiment of the present invention, compared with the technical solutions in the prior art, such as whether the data arrives, whether the interface is called successfully, etc., can More accurate and timely detection of the current status of the client during the live broadcast.

Description of drawings

Fig. 1 is the structural block diagram of the system that detects client playback state in the audio-video live broadcast that the embodiment of the present invention proposes;

Fig. 2 is the flow chart of the method for detecting the playback status of the client in the audio and video live broadcast proposed by an embodiment of the present invention;

Fig. 3 is a flow chart of a method for detecting a playback state of a client in an audio and video live broadcast according to another embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings. However, those skilled in the art know that the present invention is not limited to the drawings and the following embodiments.

The system for detecting the playback state of the client in the audio and video live broadcast proposed by the embodiment of the present invention, as shown in Figure 1, the system includes: a collection terminal 11, a server 12 and a client 13, wherein the collection terminal 11 is connected to the server 12, and the server 12 is connected with client 13.

Acquisition terminal 11 collects audio-video data, and regularly adds time stamp in the key frame of the audio-video data collected, thereby makes audio-video data carry state detection information; Acquisition terminal 11 uploads the audio-video data that has added time stamp to server 12. The key frame refers to a frame that can be coded independently without other frame images as a reference, and can also be called an independent frame, which generally refers to an I frame. There can be one or more key frames.

Client 13 downloads audio and video data from server 12 or forwards audio and video data to client 13 by server 12, and client 13 plays the audio and video data from server 12; Client 13 is when playing audio and video data, if detect A time stamp is added in a certain frame of audio and video data, then record the time stamp data, and simultaneously take screenshots of the playback area, and send the screenshot data and the time stamp data together with the identity information of the client 13 to the server 12 in real time, the described The identification information may be an ip address or a MAC address.

The server 12 detects the audio and video data uploaded by the acquisition end. If it is detected that a time stamp has been added to a certain frame of data, the image corresponding to the frame of data with the time stamp added will be decoded, and the decoded image will be converted into a decoded image. RGB data, which store the time stamp data, the RGB data of the decoded image and the identity information of the client in association.

The server 12 waits to receive the screenshot data and timestamp data sent by the client and the identity information of the client 13. For each timestamp data stored in the server, if the server 12 receives the time stamp sent by the client 13 within a limited time Stamp the screenshot data corresponding to the data, then carry out RGB conversion to the sent screenshot data, and convert it into the RGB data of the sent screenshot, and the server 12 compares the RGB data of the screenshot sent by the client 13 with the RGB of the saved decoded image through a graphic comparison algorithm Whether the data is consistent; if inconsistent or the server 12 does not receive the screenshot data corresponding to the timestamp data sent by the client 13 within a limited time, then it is determined that the client 13 plays an abnormal state of audio and video, and performs operations such as real-time alarm notifications, For example, notify the person who initiated the live broadcast that the audio and video status of the client is abnormal, or feed back the identity information of the client whose playback status is abnormal to the server administrator; if consistent, then determine that the audio and video playback status of the client 13 is normal.

The system can detect the playing status of multiple clients at the same time.

Furthermore, the limited time can be set by the user (for example, the person who initiates the live broadcast), or can be set automatically by the server. For each timestamp, the limited time may be the same or different.

In the above-mentioned embodiment, the time stamp is used as the mark of the state detection information. Those skilled in the art know that other time mark methods can also be used, or other types of mark methods can be used, such as adding a barrage to a certain frame, and adding the barrage It can be used as a mark of state detection information; or the timestamp and bullet chatting can be used together as a mark of state detection information. At this time, the time stamp and bullet chatting can be set in the same frame or in different frames.

In the method for detecting the playback state of the client in the audio and video live broadcast proposed by the embodiment of the present invention, in this embodiment, the time stamp is used as the mark of the state detection information. The method proposed in this embodiment is shown in Figure 2, and the method includes:

Step S21, collection step: the collection terminal collects audio and video data;

Step S22, marking step: the acquisition end regularly adds time stamps to the key frames of the collected audio and video data, so that the audio and video data carry state detection information; the key frames refer to the fact that no other frame images are required for reference , the frame that can be encoded independently can also be called an independent frame, which generally refers to an I frame. There can be one or more key frames.

Step S23, uploading step: the acquisition end uploads the audio and video data with the time stamp added to the server;

Step S24, the first detection step: the server detects the audio and video data uploaded by the acquisition end, and when the server detects that a time stamp has been added to a certain frame of data, it will decode the image corresponding to the frame of data with the time stamp added, and decode The converted image is converted to the RGB data of the decoded image, and the timestamp data and the RGB data of the decoded image are saved in association;

Step S25, the second detection step: the client downloads audio and video data from the server or the server forwards audio and video data to the client, and the client plays the audio and video data from the server; if the client detects certain audio and video data when playing audio and video data If a time stamp is added to the frame data, record the time stamp data, take a screenshot of the playback area at the same time, and send the screenshot data and time stamp data together with the client's identification information to the server in real time; the identification information can be ip address or MAC address;

Step S26, judging step: the server waits to receive the screenshot data and timestamp data sent by the client, as well as the identification information of the client. For the screenshot data corresponding to the timestamp data, RGB conversion is performed on the sent screenshot data to convert it into the RGB data of the sent screenshot; the server compares the RGB data of the sent screenshot with the RGB data of the saved decoded image through a graphic comparison algorithm Whether it is consistent; if it is inconsistent or the server does not receive the screenshot data corresponding to the time stamp data sent by the client within a limited time, it will be determined that the audio and video playback status of the client is abnormal, and real-time alarm notifications and other operations will be performed, such as notifications to initiate live broadcasts The person who has the audio and video playback status of the client is abnormal, or feeds back the identity information of the client whose playback status is abnormal to the server administrator; if they are consistent, it is determined that the audio and video playback status of the client is normal.

Another embodiment of the present invention proposes a method for detecting the playback state of a client in a live audio and video broadcast. In this embodiment, bullet chatting is used as a mark for detecting status information, and bullet chatting is added to certain frames of audio and video data. The method proposed in this embodiment is shown in Figure 3, including:

Step S31, collection step: the collection terminal collects audio and video data;

Step S32, marking step: the acquisition end regularly adds barrage to some frames of the collected audio and video data, so that the audio and video data carry state detection information;

Step S33, uploading step: the acquisition end uploads the audio and video data added to the barrage to the server;

Step S34, the first detection step: the server detects the audio and video data uploaded by the acquisition terminal, and when the server detects that a barrage has been added to a certain frame of data, it will decode the image corresponding to the frame of data that has added the barrage, and the decoded The converted image is converted to the RGB data of the decoded image, and the RGB data of the barrage and the decoded image are saved in association;

Step S35, the second detection step: the client downloads audio and video data from the server or the server forwards audio and video data to the client, and the client plays the audio and video data from the server; if the client detects certain audio and video data when playing audio and video data If the frame data is added to the barrage, then record the barrage data, and simultaneously take screenshots of the playing area, and send the screenshot data and the barrage data together with the identification information of the client to the server in real time; the identity information can be an ip address or MAC address;

Step S36, judging step: the server waits to receive screenshot data and barrage data sent by the client, as well as the identity information of the client. For the screenshot data corresponding to the barrage data, RGB conversion is performed on the sent screenshot data to convert it into the RGB data of the sent screenshot; the server compares the RGB data of the sent screenshot with the RGB data of the stored decoded image through a graphic comparison algorithm Whether it is consistent; if it is inconsistent or the server does not receive the screenshot data corresponding to the barrage data sent by the client within a limited time, it will be determined that the status of the video played by the client is abnormal, and real-time alarm notifications and other operations will be performed, such as notifying the person who initiated the live broadcast There is an abnormality in the audio and video playback status of the client, or the identity information of the client whose playback status is abnormal is fed back to the server administrator; if they are consistent, it is determined that the video playback status of the client is normal.

Those skilled in the art will understand that the logic and/or steps shown in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, which can be specifically implemented in on any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus, or device and execute them), or Used in conjunction with these instruction execution systems, devices or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device.

More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. The program is processed electronically and stored in computer memory.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The embodiments of the present invention have been described above. However, the present invention is not limited to the above-mentioned embodiments. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A system for detecting the playback state of a client in an audio and video live broadcast, characterized in that it comprises: a collection end, a server and a client, wherein the collection end is connected to the server, and the server is connected to the client; The acquisition terminal regularly adds the mark of the state detection information in the audio and video data collected, and uploads the audio and video data added with the state detection information mark to the server; wherein, the time mark is used as the mark of the state detection information; The server detects the received audio and video data, and if it detects the flag of the state detection information, it decodes the image at this time, converts the decoded image into the RGB data of the decoded image, and stores the flag and decoding of the state detection information in association The RGB data of the image and the identity information of the client; The client obtains the audio and video data from the server and plays them; when the client is playing the audio and video data, if it detects the mark of the state detection information, it will record the mark of the state detection information in real time, and take a screenshot of the playback area at the same time, and save the screenshot data and The marked data of the state detection information is sent to the server in real time together with the identity information of the client; The server performs RGB conversion on the screenshot data sent by the client received within a limited time, converts it into the RGB data of the sent screenshot, and compares the RGB data of the decoded image corresponding to the flag of the same state detection information with the RGB data of the sent screenshot Whether it is consistent, if it is not consistent or the server does not receive the screenshot data sent by the client and the tag data of the status detection information within the limited time, it is determined that the playback status of the client is abnormal; The system can detect the playing state of multiple clients at the same time. 2. The system according to claim 1, wherein the identity information is an ip address or a MAC address. 3 . The system according to claim 2 , wherein the marked frame added with the state detection information is a key frame. 4 . 4. The system according to claim 1, wherein the state detection information is marked with a time stamp and/or a barrage. 5. The system according to any one of claims 1 to 4, characterized in that, when it is determined that the playback status of the client is abnormal, the server feeds back the abnormal playback status of the client to the user who initiated the audio and video live broadcast, or reports to the server management The identification information of the client whose playback status is abnormal is reported by the staff. 6. A method for detecting the playback state of a client in an audio and video live broadcast, characterized in that the method comprises: Acquisition step: the acquisition terminal collects audio and video data; Marking step: the acquisition terminal regularly adds the mark of the state detection information in the audio and video data collected; wherein, the time mark is used as the mark of the state detection information; Uploading step: after the marking step, the acquisition end uploads the marked audio and video data with status detection information to the server; The first detection step: the server detects the received audio and video data, if the server detects the flag of the state detection information, it decodes the image at this time, converts the decoded image into the RGB data of the decoded image, and saves the state in association Marking of detection information and RGB data of decoded image and identity information of client; The second detection step: the client obtains the audio and video data from the server. When the client is playing the audio and video data, if it detects the mark of the state detection information, it will record the mark of the state detection information in real time, and take a screenshot of the playback area at the same time. The screenshot data and the marked data of the state detection information are sent to the server in real time together with the identity information of the client; Judgment step: the server receives the screenshot data sent by the client and the tag data of the state detection information, and performs RGB conversion on the screenshot data sent by the client received within a limited time, and converts it into the RGB data of the sent screenshot, and compares the corresponding Whether the RGB data of the decoded image marked with the same state detection information is consistent with the RGB data of the sent screenshot, if they are inconsistent or the server does not receive the screenshot data sent by the client and the marked data of the state detection information within a limited time, then determine whether the client The playback status of the terminal is abnormal; The method can detect the playing state of multiple clients at the same time. 7. The method according to claim 6, wherein the identity information is an ip address or a MAC address. 8 . The method according to claim 7 , wherein the marked frame added with the state detection information is a key frame. 9. The method according to claim 6, wherein the state detection information is marked with a time stamp and/or a barrage. 10. The method according to any one of claims 6 to 9, characterized in that, in the judging step, when it is determined that the playback status of the client is abnormal, the server feeds back the abnormal playback status of the client to the user who initiated the audio and video live broadcast , or feedback the identity information of the client whose playback status is abnormal to the server administrator.