WO2025019831A2

WO2025019831A2 - Systems and methods for passive detection of unauthorized screen recording of video content on a device

Info

Publication number: WO2025019831A2
Application number: PCT/US2024/038882
Authority: WO
Inventors: Johan Lindquist; Klaus Schenk
Original assignee: Verimatrix GmbH; Verimatrix Inc
Current assignee: Verimatrix GmbH; Verimatrix Inc
Priority date: 2023-07-20
Filing date: 2024-07-19
Publication date: 2025-01-23
Anticipated expiration: 2026-01-20
Also published as: WO2025019831A3

Abstract

Systems and methods for passive detection of unauthorized screen recording of video content on a device are disclosed. In one embodiment, a method for detecting un-authorized screen recording of video playback includes playing back media content on a display by a playback device, capturing sensor information over time during playback of the media content, where sensor information includes at least one characteristic of the playback device and at least one characteristic of the playback being performed, sending the sensor information to a recording detection server, and determining, by the recording detection server from the sensor information, whether an external device captured the visual appearance of the media content as it was being played back on the display.

Description

SYSTEMS AND METHODS FOR PASSIVE DETECTION OF UNAUTHORIZED

SCREEN RECORDING OF VIDEO CONTENT ON A DEVICE

FIELD OF THE INVENTION

[0001] The present invention relates to detection of screen recording to create an unauthorized copy of multimedia video content.

BACKGROUND OF THE INVENTION

[0002] Digital representation, storage, distribution, and duplication of digital media have become very popular because they are inexpensive, easy to use and maintain the quality of the media. These advantages however have enabled widespread, illegal distribution and use of copyrighted material, such as unauthorized distribution of digital images and videos over the Internet.

[0003] Many different approaches have been presented to secure digital media against unauthorized use. For example, digital encryption technology is effective to enable secure communication and delivery. However, once encrypted content is decrypted and presented in a form that is visible or audible to humans, that content can be re-rec- orded and an unsecured copy can be obtained. One common method of unauthorized copying is capturing the content displayed on the screen of a playback device while video is being played. The capture can be by a video camera pointed at the screen or by a capture device attached to the screen.

BRIEF DESCRIPTION OF THE DRAWINGS

[0004] In the drawings, like reference characters generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.

[0005] Fig. 1 illustrates watermark sequences in video frames, according to one embodiment of the invention.

[0006] Figs. 2A, 2B, and 2C conceptually illustrate a content distribution server, a playback device, and a recording detection server, respectively, in accordance with an embodiment of the invention. [0007] Fig. 3 is a flow diagram illustrating a process for detecting screen recording using sensor information of a playback device in accordance with embodiments of the invention.

DETAILED DESCRIPTION

[0008] Turning now to the drawings, systems and methods for detecting unauthorized screen recording in accordance with embodiments of the invention are illustrated are described. Many embodiments of the invention consider at least two scenarios for unauthorized screen recording: 1 ) aiming a camera at the screen of a playback device to record the video while content is being played back and 2) capturing video playing on a playback device by an HDMI cable connected to the playback device or a digitizing overlay on the screen.

[0009] Detection can be facilitated by collecting information from sensors on the playback device, for example by an on-device agent. The information can be combined into a heuristic that can then be used to determine a confidence level that screen recording is occurring. The heuristic can be evaluated based on the likelihood of conditions defined by the sensor information and as associated with a particular scenario. In some embodiments, the sensor information is processed on the playback device to generate a confidence level. In other embodiments, the sensor information or the heuristic is sent from the playback device to a recording detection server that processes the information to generate a confidence level.

System Architecture

[0010] A video distribution system in accordance with an embodiment of the invention is illustrated in FIG. 1 . Systems for detecting unauthorized screen recording can include playback of video content that is streamed from a server and/or stored on the playback device. The system 100 includes a number of different playback devices 102 connected to a content distribution server 104 over the Internet 106. The playback devices 102 can include playback applications configured to receive and decode encoded video from the content distribution server 104. The video distribution system can also include a recording detection server 107, which can be configured to receive sensor information from playback devices 102 over the Internet 106 and determine whether screen recording is occurring from the sensor information. Although video transmitted to playback devices via the internet is mentioned above, video can be transmitted to playback devices in any manner as appropriate to specific applications in accordance with many embodiments of the invention, including over a local access network (LAN).

[0011] A content distribution server in accordance with many embodiments of the invention can load a content distribution application as machine readable instructions from memory or other storage. A content distribution server in accordance with an embodiment of the invention is illustrated in FIG. 2A. The content distribution server 202 includes a processor 204, non-volatile memory 208 that includes a content distribution application 210, and a network interface 212. In the illustrated embodiment, the non-volatile memory 208 is a machine-readable media that can be utilized to store the machine-readable instructions that can configure the processor 204 to execute the instructions, such as according to processes discussed further below. The non-volatile memory 208 contains a content distribution application 210, which can be utilized to configure the processor 204 to provide video content to one or more playback devices using network interface 212.

[0012] Playback devices in accordance with many embodiments of the invention can load a playback application as machine readable instructions from memory. A playback device in accordance with an embodiment of the invention is illustrated in FIG. 2B. The playback device 252 includes a processor 254, one or more sensors 256, non-volatile memory 258 that includes a playback application 260, a network interface 262, and a screen 264. In the illustrated embodiment, the non-volatile memory 258 is a machine- readable media that can be utilized to store the machine-readable instructions that configure the processor 254 to execute the instructions, such as according to processes discussed further below. Here, the non-volatile memory 258 contains the instructions of a playback application 260, which can be utilized to configure the processor 254 to decode video. In many embodiments, a playback application can be loaded from any kind of memory or storage device including volatile memory in accordance with many embodiments of the invention. The non-volatile memory 258 can also include a detection agent 261 that can collect information from one or more sensors 256. The sensor information can be processed on the playback device or sent to a recording detection server to determine whether screen recording is taking place with video being played back on the device. Playback devices can include, but are not limited to, mobile devices (smartphones), tablets, set-top boxes connect to displays, computing devices, etc.

[0013] Recording detection servers in accordance with many embodiments of the invention can generate watermarks for application to video content. Recording detection servers can receive sensor information from playback devices and determine whether screen recording is occurring with video being played back, such as by processes that will be discussed further below. A recording detection server in accordance with an embodiment of the invention is illustrated in FIG. 2C. The recording detection server 272 includes a processor 274, non-volatile memory 278 that includes a content distribution application 280, and a network interface 282. In the illustrated embodiment, the nonvolatile memory 278 is a machine-readable media that is utilized to store the machine- readable instructions that configure the processor 274 to execute the instructions. Here, the non-volatile memory 278 contains the instructions of a detection application 280, which can be utilized to execute processes such as those discussed further below. In many embodiments, a detection application can be loaded from any kind of memory or storage device including volatile memory in accordance with many embodiments of the invention.

[0014] Although a video distribution system is described above with respect to a specific content distribution server and playback devices, any of a variety of encoding, transmitting or decoding systems can be utilized in the encoding, decoding and transmission of video as appropriate to specific applications in accordance with many embodiments of the invention.

Sensor Information

[0015] As discussed above, a playback device may include one or more sensors that can provide information useful to detecting screen recording. Any of a variety of sensors may be utilized as appropriate to detecting any particular scenario. Sensors can include network status (cellular, wifi, ethernet, etc ), HDMI interface, ambient light, location or motion (GPS, accelerometer, compass, etc.). [0016] Additional sensor information can include characteristics of the device or playback conditions, such as playback behavior, metadata of the content, volume levels, screen brightness, battery level, etc.

[0017] Some sensors may be polled or the information taken at certain configurable time intervals, while others can be event driven (e.g., received a seek command). Information can be collected by an on-device agent. For example, a sensor collection agent can be injected (e.g., using an SDK) into the playback application that facilitates playback on the device. The sensor collection agent can include capabilities such as, but not limited to, debugger detection, hooking detection, virtual machine/emulator detection, etc. Sensor information from by different playback devices may be provided (e.g., using a sensor collection agent) to a recording detection server. The recording detection server may collect sensor information from playback devices to determine patterns of normal or abnormal (suspicious) behavior during playback.

[0018] In several embodiments, the agent can receive updates from a recording detection server or other server to reconfigure thresholds, intervals, and/or the manner in which information is collected. Some such updates can be based on sensor information collected by other playback devices as will be discussed further below.

[0019] While specific sensor types are discussed above, one skilled in the art will recognize that any of a variety of sensors may be utilized in accordance with embodiments of the invention.

Sensor Conditions

[0020] Information reflecting the state or status of a sensor on a playback device can be referred to as sensor condition. The time window over which the sensor condition is recorded can be tied to the time length of the content being played back. Conditions that can be indicative of screen recording can include at least the following. In accordance with certain embodiments of the invention, some conditions may be more or less relevant to different scenarios that are intended to be detected. Accordingly, different rulesets for detection may include or not include any of the conditions listed here and/or may weigh conditions differently. Sensors

[0021] A GPS sensor can be used to provide location information. The location information may indicate that the device has not moved and has stayed in the same location, which can be indicative of unauthorized recording conditions.

[0022] Motion sensors can include gyroscopes and accelerometers. Similarly, if a motion sensor shows no abrupt movements it can be indicative of recording. Motion information from a motion sensor may be normalized. Motion may be detected as “peaks” in the data.

[0023] An ambient light sensor may sense a light intensity over time. When an ambient light sensor indicates that lighting conditions are constant and reflective of a controlled environment, that is not completely dark and not very bright, it can be indicative of recording. Certain intensities of ambient light (e.g., in lumens) may be better for capturing video of a screen and minimizing reflections (e.g., of a camera). If the measured intensity falls within a certain range of an optimal intensity, it may indicate screen recording.

Device Characteristics

[0024] Certain properties or characteristics of a playback device (e.g., a mobile device) can be indicative of recording. Similar to sensors as discussed above, maintaining constant conditions may show a controlled environment for optimal and uninterrupted recording. These characteristics may be obtained, for example, from the device operating system, applications on the device, and/or application programming interfaces (APIs).

[0025] Characteristics that may indicate recording can include, but are not limited to, power status where the device is receiving constant power, e.g., connected to external power; power status where battery level is constantly high; screen brightness has maintained a constant level; and volume of the device being unchanged through the content.

[0026] The audio output of the device being set to a connected Bluetooth device or another external audio device may indicate recording. [0027] An onboard thermometer or temperature sensor may provide temperature information. A constant temperature or a temperature that seems to be maintained within a small range can indicate indoor use and/or controlled conditions.

Playback Behavior and Content Metadata

[0028] Playback behavior can also indicate screen recording conditions. Information regarding the manner in which the content is played back can be obtained, for example, from the device operating system and/or a playback application on the device.

[0029] When the piece of content is played at regular speed from start to finish can be indicative of recording. Further indicators can be when pauses, seeks, or trick play (e.g., fast forward or rewind) are not used or occur minimally, and when the introduction portion of the content is skipped.

[0030] A video is played from start to finish at regular speed, played at the highest bitrate available, with no pausing or trick play (e.g., fast forward or reverse) or other type of user interaction. Longer pieces of content may tend more often to be recorded, therefore the length of a video may indicate recording.

[0031] Multiple videos are played in the same manner over a period of time.

[0032] All episodes within a series or season are played. Furthermore, when introductions are not skipped and/or end credits are not skipped to proceed to the next episode.

Confidence Level

[0033] Sensor information can be used to determine a confidence level that indicates the likelihood of suspected screen recording. In some embodiments, the sensor information is structured for machine learning. Sensor information collected during times of normal playback behavior, e.g., when recording is not suspected, can be used as baseline data to train a machine learning model. Suspicion of screen recording can be determined as outliers in the data set over time. In some embodiments of the invention, sets of sensor information can be collected where a set corresponds to a playback session (e.g., start of playback to end of playback). The data may be formatted such that the set falls within a row each piece of sensor information is a field in the row. [0034] Machine learning models may be used in a variety of ways. In several embodiments of the invention, a large data set may be used on an untrained model. In other embodiments, a large data set of assumed non-malicious behavior may be used as baseline training data to train a model and/or synthetic training data of malicious behavior can be created by emulating screen recording.

[0035] In other embodiments of the invention, thresholds can be set for a combination of values of sensor information. For example, a set of sensors on the playback device have conditions matching those described further above would be indicative screen recording. For a scenario of capture by HDMI cable or an overlay on the screen, information would be focused more on display hardware, e.g., what type of screen, whether there are known flaws in the DRM or Android display stack, is it a common device used by pirates, etc. In additional embodiments of the invention, thresholds may be determined by comparing playback behavior of the same piece of content across different playback devices and/or by different users.

[0036] A number of playback sessions may be collected for a single user. The sessions can be accumulated until a threshold of malicious behavior is met.

[0037] An example heuristic for detection can include collecting data sets for playback sessions, normalizing and processing the data to create statistics representing normal playback behavior. Characteristics can include, but are not limited to, number of seeks used, number of pauses used, number of times introductions are skipped, number of times entire episodes are played without stopping, etc. Values for these characteristics can be assigned a percentage of all playbacks that exhibit that value of the characteristic. A playback session can then be evaluated for where they fall on the distributions of the characteristics given the values present in that playback session to determine if it deviates from normal behavior.

Processes for Detecting Screen Recording

[0038] A process for detecting screen recording of a playback device in accordance with an embodiment of the invention is shown in FIG. 3. The process 300 can include receiving (302), by a playback device, video content for playback or playing back stored content already on the playback device. The content may be streamed from a streaming service using mechanisms such as adaptive bitrate streaming (ABR).

[0039] Sensors on the playback device provide (304) information during playback. As discussed further above, sensor information can include information collected by sensors, device characteristics, and/or playback behavior.

[0040] In some embodiments, the sensor information can be sent (306) to a detection server for processing. The detection server can improve its detection capabilities by comparing behavior on other playback devices, e.g., crowdsourcing sensor information from large numbers of playback devices. For example, baseline normal behavior (e.g., authorized playback) can be developed and used to tune thresholds. In other embodiments, the processing can be performed on the playback device. Several embodiments of the invention may utilize machine learning models to learn baseline normal behavior and detecting suspicious behavior (e.g., screen recording) as deviating from the baseline normal behavior.

[0041] The sensor information can be combined into a heuristic as discussed further above. A determination can be made (308) whether screen recording was performed on the playback device and a confidence level calculated using a heuristic, thresholds, and/or a machine learning model.

[0042] Although a specific process is discussed above with respect to Fig. 3, one skilled in the art will recognize that any of a variety of processes may be utilized in accordance with embodiments of the invention.

[0043] While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as an example of one embodiment thereof. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

WHAT IS CLAIMED IS:

1. A method for detecting unauthorized screen recording of video playback, the method comprising: playing back media content on a display by a playback device; capturing sensor information over time during playback of the media content, where sensor information includes at least one characteristic of the playback device and at least one characteristic of the playback being performed; sending the sensor information to a recording detection server; and determining, by the recording detection server from the sensor information, whether an external device captured the visual appearance of the media content as it was being played back on the display.

2. The method of claim 1 , further comprising sending from the recording detection server to the playback device a configuration file that instructs the playback device to change an aspect of capturing sensor information.

3. The method of claim 1 , wherein the capturing sensor information is performed by a sensor collection agent injected by an SDK into a playback application.

4. The method of claim 1 , wherein sensor information includes a GPS location.

5. The method of claim 1 , wherein sensor information includes information from an accelerometer.

6. The method of claim 1 , wherein sensor information includes information from an ambient light sensor.

7. The method of claim 1 , wherein sensor information includes battery level.

8. The method of claim 1 , wherein sensor information includes screen brightness level.

9. The method of claim 1 , wherein sensor information includes volume level.

10. The method of claim 1 , wherein sensor information includes temperature.

11. The method of claim 1 , wherein sensor information includes playback behavior including number of pauses and trick plays.

12. The method of claim 1 , wherein sensor information includes length of the media content and the amount of the total length played.

13. The method of claim 1 , wherein sensor information includes whether an introduction of the media content was skipped.

14. The method of claim 1 , wherein sensor information includes whether end credits of the media content was skipped.

15. The method of claim 1 , wherein capturing sensor information includes storing individual pieces of sensor information within fields in a row and the row represents a single playback session.

16. The method of claim 1 , wherein determining whether an external device captured the visual appearance of the media content includes inputting at least some of the sensor information into a machine learning model and obtaining an output that includes a confidence level that screen recording occurred.

17. The method of claim 1 , wherein determining whether an external device captured the visual appearance of the media content comparing at least some of the sensor information to thresholds.

18. A system for detecting unauthorized screen recording of video playback comprising: a playback device; and a recording detection server; where the playback device comprises: a processor; and memory comprising a playback application and a detection agent; where the recording detection server comprises: a processor; and memory comprising a detection application; where the detection agent when executing directs the playback device processor to: capture sensor information over time during playback of a media content, where sensor information includes at least one characteristic of the playback device and at least one characteristic of the playback being performed; and sending the sensor information to a recording detection server; where the detection agent when executing directs the recording detection server processor to: determine from the sensor information whether an external device captured the visual appearance of the media content as it was being played back on the display.

19. The system of claim 18, where the detection agent further directs the recording detection server processor to send to the playback device a configuration file that instructs the playback device to change an aspect of capturing sensor information.

20. The system of claim 18, wherein the capturing sensor information is performed by a sensor collection agent injected by an SDK into a playback application.

21. The system of claim 18, wherein sensor information includes a GPS location.

22. The system of claim 18, wherein sensor information includes information from an accelerometer.

23. The system of claim 18, wherein sensor information includes information from an ambient light sensor.

24. The system of claim 18, wherein sensor information includes battery level.

25. The system of claim 18, wherein sensor information includes screen brightness level.

26. The system of claim 18, wherein sensor information includes volume level.

27. The system of claim 18, wherein sensor information includes temperature.

28. The method of claim 1 , wherein sensor information includes playback behavior including number of pauses and trick plays.

29. The method of claim 1 , wherein sensor information includes length of the media content and the amount of the total length played.

30. The method of claim 1 , wherein sensor information includes whether an introduction of the media content was skipped.

31. The method of claim 1 , wherein sensor information includes whether end credits of the media content was skipped.

32. The method of claim 1 , wherein capturing sensor information includes storing individual pieces of sensor information within fields in a row and the row represents a single playback session.

33. The method of claim 1 , wherein determining whether an external device captured the visual appearance of the media content includes inputting at least some of the sensor information into a machine learning model and obtaining an output that includes a confidence level that screen recording occurred.