WO2021153507A1 - Imaging device, control method for imaging device, control program, information processing device, control method for information processing device, and control program - Google Patents

Abstract

An imaging device provided with: a communication unit for performing communication by a first communication scheme and/or by a second communication scheme capable of higher-speed or larger-capacity communication than the first communication scheme; a function control unit for switching the communication schemes in the communication unit; and a transmission control unit for controlling the transmission of original data on the basis of a transmission method that corresponds to the communication scheme in the communication unit.

Description

Image pickup device, control method of image pickup device, control program, information processing device, control method and control program of information processing device

This technology relates to an image pickup device, a control method of the image pickup device, a control program, an information processing device, a control method of the information processing device, and a control program.

The communication method used for the Internet etc. is defined by the name "G (generation)", and new communication methods are proposed and spread every few years. Currently, 4G (4th generation mobile communication system) is widely used, and 5G (5th generation mobile communication system), which is a new generation communication method, is becoming widespread (Patent Document 1). Compared to 4G, which is the previous generation, 5G is characterized by high-speed, large-capacity communication, low delay, and multiple simultaneous continuations. In addition, 6G, which is the next communication method after 5G, has already been proposed.

Japanese Unexamined Patent Publication No. 2019-004277

However, it is common to start commercial services in some countries and regions before the new communication method becomes widespread. Therefore, when the range where the new communication method can be used and the range where it cannot be used are mixed, when the bandwidth is disturbed and the new communication method cannot be used stably, or when the data in the perfect state cannot be transmitted. There is. Therefore, the user may have to use the new communication method and the conventional communication method together.

This technology has been made in view of these points, and is capable of transmitting data in a perfect state even when a plurality of communication methods are used in combination, an image pickup device, a control method of the image pickup device, a control program, and the like. It is an object of the present invention to provide an information processing apparatus, a control method and a control program of the information processing apparatus.

In order to solve the above-mentioned problems, the first technique is a communication unit that communicates with a first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method. This is an imaging device including a function control unit for switching a communication method in the communication unit and a transmission control unit for controlling transmission of original data based on a transmission method according to the communication method in the communication unit.

Further, the second technique switches the communication method in the communication unit that communicates by the first communication method and / or the second communication method capable of higher speed or larger capacity communication than the first communication method, and the communication unit. This is a control method of an imaging device that controls the transmission of original data based on the transmission method according to the communication method in.

Further, the third technique switches the communication method in the communication unit that communicates by the first communication method and / or the second communication method capable of higher speed or larger capacity communication than the first communication method, and the communication unit. This is a control program that causes a computer to execute a control method of an image pickup device that controls transmission of original data based on a transmission method according to a communication method in.

Further, the fourth technique is from a communication unit that communicates with the image pickup device by a second communication method capable of higher speed or larger capacity communication than the first communication method and / or the first communication method, and the image pickup device. Information including a first data that constitutes the original data transmitted by the first communication method and a reception control unit that receives the second data that constitutes the original data transmitted by the second communication method. It is a processing device.

Further, in the fifth technique, transmission is performed by the first communication method from an image pickup apparatus that communicates by a second communication method capable of higher speed or larger capacity communication than the first communication method and / or the first communication method. This is an information processing method for receiving the first data constituting the original data and the second data constituting the original data transmitted by the second communication method.

Further, the sixth technique is to transmit by the first communication method from the image pickup apparatus that communicates by the first communication method and / or the second communication method capable of higher speed or larger capacity communication than the first communication method. It is an information processing program that causes a computer to execute an information processing method for receiving the first data constituting the original data and the second data constituting the original data transmitted by the second communication method.

It is a block diagram which shows the structure of an information processing system 10. It is a block diagram which shows the structure of the image pickup apparatus 100. It is a block diagram which shows the structure of the terminal apparatus 200. It is a block diagram which shows the structure of the information processing apparatus 300. It is explanatory drawing of the detection method of shortage in an information processing apparatus. It is explanatory drawing of the image processing apparatus 400. It is explanatory drawing of another example of a video processing apparatus 400. It is a flowchart which shows the process in the image pickup apparatus 100. It is explanatory drawing of area information generation. It is a sequence diagram explaining the transmission / reception of data between an image pickup apparatus and an information processing apparatus. It is a block diagram which shows an example of the structure of MOJO. It is explanatory drawing of the 1st aspect of coverage in MOJO. It is explanatory drawing of the 2nd aspect of coverage in MOJO.

Hereinafter, embodiments of the present technology will be described with reference to the drawings. The explanation will be given in the following order.
<1. Embodiment>
[1-1. Configuration of information processing system 10]
[1-2. Configuration of image pickup device 100]
[1-3. Configuration of terminal device 200]
[1-4. Configuration of information processing device 300]
[1-5. Processing in the image pickup apparatus 100]
[1-6. Processing in the image pickup apparatus 100 and the information processing apparatus 300]
<2. Application example of this technology>
[2-1. MOJO configuration]
[2-2. First aspect of coverage in MOJO]
[2-3. Second aspect of coverage in MOJO]
<3. Modification example>

<1. Embodiment>
[1-1. Configuration of information processing system 10]
First, the configuration of the information processing system 10 will be described with reference to FIG. The information processing system 10 includes an image pickup device 100, a terminal device 200, and an information processing device 300. The image pickup device 100 and the terminal device 200 are used by the photographer. The information processing device 300 is used by a person who receives data (video data, image data, audio data, etc.) generated by the photographer in shooting with the image pickup device 100. The person who receives the data is a director as a shooting instructor who requests the photographer to interview, and in that case, the information processing device 300 is used in a broadcasting station or the like where the shooting instructor works.

The image pickup device 100 and the information processing device 300, and the terminal device 200 and the information processing device 300 are connected to each other via a network such as the Internet. Further, the image pickup device 100 and the terminal device 200 are connected and associated with each other by, for example, short-range wireless communication.

The imaging device 100 takes a picture and transmits the data generated by the picture to the information processing device 300 using a communication method such as 5G or 4G.

The terminal device 200 also has a camera function, and the photographer can transmit the data generated by the shooting to the information processing device 300 using a communication method such as 5G or 4G. Further, the terminal device 200 can also receive the data generated by the image pickup device 100 taking a picture and transmit the data to the information processing device 300. Further, the terminal device 200 can also be used to receive the shooting instruction information from the information processing device 300 and present the shooting instruction information to the photographer.

The information processing device 300 receives the data transmitted from the image pickup device 100 and the terminal device 200. The information processing device 300 can also transmit shooting instruction information to the terminal device 200.

This information processing system 10 is used, for example, in a broadcasting system that broadcasts using data generated by shooting with an imaging device 100.

[1-2. Configuration of image pickup device 100]
Next, the configuration of the image pickup apparatus 100 will be described with reference to the block diagram of FIG. The image pickup device 100 includes a control unit 101, an optical image pickup system 102, a lens drive driver 103, an image sensor 104, an image signal processing unit 105, a code code unit 106, a storage unit 107, a first communication unit 108, a second communication unit 109, and an input. A configuration including a unit 110, a display unit 111, a microphone 112, a position information acquisition unit 113, an attitude information acquisition unit 114, a radio wave intensity sensor 115, an area information acquisition unit 116, a determination unit 117, a function control unit 118, and a transmission control unit 119. Has been done.

The control unit 101 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The CPU controls the entire image pickup apparatus 100 and each part by issuing commands by executing various processes according to the program stored in the ROM. Further, the control unit 101 stores the image pickup device 100 identification information for identifying the image pickup device 100, which is required when connecting to the terminal device 200 and the information processing device 300, and the control unit 101 stores the terminal device 200 and information processing. Controls the connection with the device 300.

The optical image pickup system 102 includes an image pickup lens for concentrating light from a subject on an image pickup element 104, a drive mechanism for moving the image pickup lens to perform focusing and zooming, a shutter mechanism, an iris mechanism, and the like. There is. These are driven based on the control signals from the control unit 101 and the lens drive driver 103. The optical image of the subject obtained through the optical image pickup system 102 is imaged on the image pickup device 104.

The lens drive driver 103 is composed of, for example, a microcomputer or the like, and by moving the image pickup lens by a predetermined amount along the optical axis direction under the control of the control unit 101, autofocus is performed so as to focus on the target subject. Further, according to the control from the control unit 101, the operation of the drive mechanism, the shutter mechanism, the iris mechanism, etc. of the optical imaging system 102 is controlled. As a result, the exposure time (shutter speed) and the aperture value (F value) are adjusted.

The image sensor 104 photoelectrically converts the incident light from the subject obtained through the image sensor into an electric charge amount and outputs an image pickup signal. Then, the image sensor 104 outputs the pixel signal to the image signal processing unit 105. As the image pickup device 104, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like is used.

The image signal processing unit 105 is a sample hold for maintaining a good S / N (Signal / Noise) ratio by CDS (Correlated Double Sampling) processing with respect to the image pickup signal output from the image sensor 104, and AGC (Auto Gain Control). ) Processing, A / D (Analog / Digital) conversion, etc. are performed to create an image signal. Further, the image signal processing unit 105 performs a recording process on the image signal for recording, and a display process on the image signal for display.

The codec unit 106 performs coding processing for recording or communication, for example, on the image signal subjected to the predetermined processing.

The storage unit 107 is a large-capacity storage medium such as a hard disk or a flash memory. The video data and image data processed by the image signal processing unit 105 and the codec unit 106 are stored in a compressed state or an uncompressed state based on a predetermined standard. In addition, EXIF (ExchangeableImageFileFormat) including additional information such as information on the saved data, imaging position information indicating the imaging position, and imaging time information indicating the imaging date and time is also stored in association with the data.

The first communication unit 108 is a communication module for transmitting and receiving data and various information to and from the information processing device 300 by a communication method other than 5G. Communication methods other than 5G include 4G (4th generation mobile communication system) / LTE (Long Term Evolution). Here, the communication method other than 5G will be described as 4G. Some 5G communication modules also have 4G functions, in which case one communication unit may be used. A communication method including 4G, which is slower than 5G and has a small communication capacity, corresponds to the first communication method in the claims.

The second communication unit 109 is a communication module for transmitting and receiving data and various information to and from the information processing device 300 using 5G. 5G corresponds to the second communication method in the claims.

Some 5G communication modules also have 4G functions. In that case, one communication unit may be used, but when 5G and 4G are used at the same time, there are two communication units for 5G and 4G. You will need one.

The imaging device 100 may include a communication unit that can be connected to the Internet and other devices such as a wireless LAN (Local Area Network), WAN (Wide Area Network), and WiFi (Wireless Fidelity). In addition, communication between the image pickup device 100 and the terminal device 200 includes short-range wireless communication such as NFC (Near Field Communication) and ZigBee (registered trademark), WiFi tethering, USB (Universal Serial Bus) tethering, and Bluetooth (registered trademark) tethering. It may be a tethering connection such as.

The input unit 110 is for the photographer to give various instructions to the image pickup apparatus 100. When an input is made to the input unit 110 by the photographer, a control signal corresponding to the input is generated and supplied to the control unit 101. Then, the control unit 101 performs various processes corresponding to the control signal. The input unit includes a shutter button for shutter input, a physical button for various operations, a touch panel, a touch screen integrally configured with a display as a display unit 111, and the like.

The display unit 111 stores a through image, which is an image signal for display that has been processed for display by the image signal processing unit 105 and the codec unit 106, and has been subjected to image processing for recording by the image signal processing unit 105. It is a display device such as a display that displays images / videos, GUI (Graphical User Interface), etc. stored in unit 107.

The microphone 112 is for recording audio in video recording.

The position information acquisition unit 113 is provided with GPS (Global Positioning System) and SLAM (Simultaneous Localization and Mapping) functions, and acquires the current position of the terminal device 200 as coordinate information, for example.

The attitude information acquisition unit 114 detects the angular velocity of an IMU (Inertial Measurement Unit) module or the like. The IMU module is an inertial measurement unit, and detects the posture, orientation, etc. of the image pickup device 100 by obtaining a three-dimensional angular velocity and acceleration by an acceleration sensor, an angular velocity sensor, a gyro sensor, or the like in two or three axial directions.

The radio wave intensity sensor 115 is a sensor for measuring the radio wave intensity of the 5G millimeter wave band in the communication by the first communication unit 108. The radio field strength information detected by the radio wave strength sensor 115 is supplied to the determination unit 117.

The area information acquisition unit 116 acquires information indicating a cover area that is a usable range of 5G around the terminal device 200. The area information can be acquired by receiving the area information indicating the 5G usable range around the terminal device 200 provided by the telecommunications carrier, based on the position information of the terminal device 200 acquired by the position information acquisition unit 113.

The area information acquisition unit 116 can also acquire area information by creating a map showing the 5G usable range around the image pickup apparatus 100 by itself. The creation of area information will be described later.

The determination unit 117 determines whether the imaging device 100 can currently use 5G based on the radio wave intensity information from the radio wave intensity sensor 115 and the area information from the area information acquisition unit 116. Whether or not 5G can be used can be determined based on whether or not the radio field intensity in the 5G frequency band is equal to or higher than a predetermined value. Further, it is possible to determine whether 5G is available by confirming whether or not the imaging device 100 is a dolphin in the area where 5G can be used by referring to the area information. The determination result is supplied to the function control unit 118.

The function control unit 118 switches the communication method used by the image pickup apparatus 100 depending on whether 5G is available or not, depending on the determination result of the determination unit 117. If 5G is unavailable, turn on the first communication unit 108 to communicate with 4G, and if 5G is available, turn on the second communication unit 109 to communicate with 5G. Control.

In addition, "5G available" means a state in which stable communication can be performed when the radio wave intensity is above a certain level, a state in which the image pickup apparatus 100 exists within the 5G usable range, or both states. do. Further, "5G cannot be used" means a state in which stable communication cannot be performed due to a radio wave intensity of a certain level or less, a state in which the image pickup device 100 exists outside the 5G usable range, or both states. It shall be.

The transmission control unit 119 transfers the image pickup device 100 to the information processing device 300 depending on whether the current communication method of the image pickup device 100 is 4G by the first communication section 108 or 5G by the second communication section 109. It controls the transmission of data. In the present embodiment, data is controlled to be transmitted to the information processing apparatus 300 in a streaming format in both 4G and 5G.

5G is capable of higher speed, larger capacity, and lower latency communication than 4G, and can transmit data in a streaming format of tens of megabits (bits per second). Therefore, data transfer equivalent to the download format can be performed only by saving the data in the information processing device 300 on the data receiving side.

Streaming is a method that enables the use of data while receiving the data. For example, it can be played back while receiving video data and audio data. There are advantages such as the waiting time until the receiving side can use the data can be significantly shortened, and the storage capacity of the receiving side is not squeezed. FPU (Field Pickup Unit) and SNG (Satellite News Gathering) are used. For example, taking advantage of the low delay, it is possible to have a conversation between the interview site and the studio in broadcasting. Unlike the download method, the data is not saved on the receiving side, so it is necessary to receive the data by streaming each time the data is used. There is a drawback that a large-scale transmission device is required, and there is a drawback that the video quality on the receiving side is not stable when transmitting by a relatively inexpensive IP.

Downloading is a method that enables the use of data after the data has been received. The data cannot be used until the download is completed, but since the downloaded data is saved on the receiving side, the data can be used at any time without receiving the data again after downloading once. In addition, it takes longer to use the data than streaming, but once the download is completed, the receiving side can acquire the data in a perfect state without deterioration. It also has the advantage that the equipment is cheap because it can be used with a general personal computer. There is a drawback that it is difficult to predict the timing of data reception completion on the receiving side.

There are three modes of transmission control by the transmission control unit 119. Hereinafter, three modes of transmission control will be described.

In the first transmission control mode, the original data to be transmitted is intentionally divided into the first data and the second data in order to reduce the data size of the data transmitted from the image pickup apparatus 100 to the information processing apparatus 300. To send.

Since the first data is transmitted by the first communication unit 108 in 4G and the second data is transmitted by the second communication unit 109 in 5G, the first data is larger than the second data. The data size should be small.

In this first transmission control, the communication speed and communication capacity of 4G are limited as compared with 5G. Therefore, when communicating with 4G, the first data, which is a part of the original data, is streamed. Is transmitted to the information processing apparatus 300. Then, when 5G becomes available and 5G communication is established, the remaining data constituting the original data is transmitted to the information processing apparatus 300 in a streaming format as the second data. As a result, data can be efficiently transmitted according to the communication method. Further, it is possible to prevent the data transmission from being delayed when trying to transmit data having a large data size even though the communication is performed by 4G.

When the original data is video data, there is a method of thinning out the pixels of the video data as a data division method. By thinning out the pixels of the video data, the resolution of the video data is lowered. Specifically, when communicating in 4G, the first data is transmitted by thinning out the pixels to reduce the data size as compared with the case of transmitting in 5G, and the pixels thinned out by the first data after the establishment of 5G. The second data including the above is transmitted by 5G. In this case, the second data may be video data consisting of only the thinned pixels or video data including the thinned pixels. By synthesizing the first data and the second data in the information processing device 300 on the receiving side, the original video data having the original resolution can be restored.

Also, as a method of dividing video data, there is a method of thinning out frames of video data. Specifically, when communicating in 4G, the first data is transmitted by thinning out frames to reduce the data size as compared with the case of transmitting in 5G, and the frame thinned out by the first data after 5G is established. The second data including the above is transmitted by 5G. In this case, the second data may be video data consisting only of thinned frames, or video data including thinned frames. By synthesizing the first data and the second data in the information processing device 300 on the receiving side, the original video data at the original frame rate can be restored. There is also a method such as reducing the I frame at the time of inter-frame compression.

Further, as a method of dividing the video data, there is also a method of using a lower frequency coefficient when frequency analysis such as discrete cosine transform is performed on the information representing the image of one frame constituting the video data. In the case of the discrete cosine transform, the imaging device 100 on the transmitting side records the coefficient of each frequency of the discrete cosine transform that can be transmitted together with the time information. When there is a transmission request from the information processing device 300 on the receiving side, the image pickup apparatus 100 retransmits the coefficient of the remaining frequency that could not be transmitted to the information processing apparatus 300. The information processing device 300 combines the coefficient of the remaining frequency sent from the image pickup device 100 with the coefficient of the received frequency already recorded by the information processing device 300, so that the image pickup device 100 originally wanted to send the highest frequency coefficient. It is possible to restore the frame image of the image quality of. By doing this all the time and frames, the high quality video data that the image pickup apparatus 100 wanted to send can be completely restored.

In this way, when 5G is not available and the data that can be transmitted by communicating with 4G is limited, the original data data is divided and first, the video data is equivalent to the amount of data that can roughly represent the video. By transmitting in 4G, it is possible to realize streaming transmission with the best effort in 4G.

Also, as a method of dividing video data, there is a process of separating audio and video. Specifically, when communicating in 4G, only the audio data is transmitted as the first data, and after the establishment of 5G, all the data constituting the video data other than the audio data is transmitted in 5G as the second data. ..

Furthermore, as a method of dividing video data, there are division of high frequency components (edge components), division based on motion vectors in frames, and the like. Since the high frequency component is an edge component that shows the outline of the subject or the like in the image, it is considered that the subject in the image can be recognized on the receiving side even if only the high frequency component is transmitted first in 4G. Then, by transmitting all the remaining frequency components after the establishment of 5G, the information processing apparatus 300 can restore the original video data.

When the audio data is composed of a plurality of channels, the audio data of a specific channel is divided into each channel, the audio data of a specific channel is transmitted in 4G as the first data, and the audio data of the remaining channels is used as the second data after the establishment of 5G. It is also possible to send as.

Note that the first data whose data size has been reduced as described above may be transmitted first, and the entire original data including the first data may be transmitted as the second data.

The above-mentioned division methods may be used in combination. For example, thinning out pixels and thinning out frames, thinning out frames and separating audio data, and so on.

The second mode of transmission control is transmission control when the communication method is switched from 5G to 4G when the band is insufficient due to the disturbance of the band or when the image pickup apparatus 100 is out of the 5G usable area. The untransmitted data that has not been transmitted due to the loss of data caused by the switching of the communication method from 5G to 4G is transmitted as the second data.

Therefore, the first data is the transmitted data transmitted in 4G, and the second data is the untransmitted data of the shortage that could not be transmitted due to the switching from 5G to 4G. By transmitting the untransmitted data generated by the loss when the communication by 5G is established, the data transmission can be completed within a short time after the end of shooting.

In both the first division mode and the second division mode, the second data is the original data including the first data, or all the data constituting the original data other than the first data. You can say that.

Further, in the second transmission control, data may be transmitted in response to a data transmission request from the information processing device 300 as an external device. When the transmission control unit 119 receives a transmission request for a missing portion of data from the information processing device 300, the transmission control unit 119 controls to transmit the data corresponding to the transmission request to the information processing device 300. The details of the second transmission mode will be described later together with the description in the information processing apparatus 300.

The third mode of transmission control is that if the communication is switched to 4G while transmitting the original data in 5G, the transmission of the original data is temporarily stopped, and when the communication by 5G is established again, the original data is transmitted. Is to restart. Alternatively, the data may not be transmitted during the communication by 4G, and the original data may be transmitted only when the communication by 5G is established. As a result, the original data can be transmitted without degrading the quality.

When all the transmission of the first data and the second data is completed, the data transfer equivalent to the download format can be performed only by saving the data in the information processing device 300 which is the data receiving side.

Which of the above-mentioned transmission control modes to be used may be set to one of the modes by default, may be decided by the photographer, and either 4G or 5G is frequently used. The image pickup apparatus 100 may be determined according to the information such as the default.

The image pickup device 100 is configured as described above. The image pickup device 100 includes devices specialized in camera functions such as digital cameras, single-lens reflex cameras, camcoders, professional cameras, and professional photography devices, as well as smartphones, tablet terminals, wearable devices, portable game machines, etc. having camera functions. It may be. Further, the function control unit 118 and the transmission control unit 119 may be realized by executing a program.

[1-3. Configuration of terminal device 200]
Next, the configuration of the terminal device 200 will be described with reference to FIG. The terminal device 200 includes a control unit 201, a storage unit 202, a first communication unit 203, a second communication unit 204, an input unit 205, a display unit 206, a microphone 207, a camera 208, a position information acquisition unit 209, and an attitude information acquisition unit 210. It is configured to include a radio wave strength sensor 211, an area information acquisition unit 212, a determination unit 213, a function control unit 214, and a transmission control unit 215.

The control unit 201 is composed of a CPU, RAM, ROM, and the like. The CPU controls the entire terminal device 200 and each part by issuing commands by executing various processes according to a program stored in the ROM. Further, the control unit 201 stores the terminal device 200 identification information that identifies the terminal device 200, which is required when connecting to the image pickup device 100 and the information processing device 300, and stores the first communication unit 203 or the second communication unit. The control unit 201 controls the connection between the image pickup device 100 and the information processing device 300 via the 204.

The storage unit 202 is a large-capacity storage medium such as a hard disk or a flash memory. The storage unit 202 stores data generated by shooting with the camera 208, various applications used in the terminal device 200, and the like.

The first communication unit 203 is a communication module for transmitting and receiving data and various information to and from the information processing device 300 by 4G, which is a communication method other than 5G. Some 5G communication modules also have a 4G function, and in that case, only one communication unit may be used.

The second communication unit 204 is a communication module for transmitting and receiving data and various information to and from the information processing device 300 using 5G.

Some 5G communication modules also have 4G functions. In that case, one communication unit may be used, but when 5G and 4G are used at the same time, there are two communication units for 5G and 4G. You will need one.

The terminal device 200 may include a communication unit that can be connected to the Internet such as wireless LAN, WAN, WiFi, and other devices. Further, the communication between the image pickup device 100 and the terminal device 200 may be short-range wireless communication such as NFC or ZigBee, or tethering connection such as WiFi tethering, USB tethering, or Bluetooth (registered trademark) tethering.

The input unit 205 is for the photographer to input information and messages to the terminal device 200, and to give various instructions. When an input is made to the input unit 205 by the photographer, a control signal corresponding to the input is generated and supplied to the control unit 201. Then, the control unit 201 performs various processes corresponding to the control signal. In addition to the physical buttons, the input unit 205 includes a touch panel, a touch screen integrally configured with the display as the display unit 206, and the like.

The display unit 206 is a display device such as a display that displays images, images, GUIs, and the like.

The microphone 207 is used as a voice input device for recording voice during video recording, voice input operation, voice call, video call, and the like.

The camera 208 is composed of a lens, an image pickup element, an image signal processing unit, a codec unit, and the like, and captures images and images to generate image data, image data, and the like.

The position information acquisition unit 209, the attitude information acquisition unit 210, the radio wave intensity sensor 211, the area information acquisition unit 212, the determination unit 213, the function control unit 214, and the transmission control unit 215 are the same as those provided in the image pickup apparatus 100. Is omitted.

The terminal device 200 is configured as described above. Specific examples of the terminal device 200 include cameras, smartphones, tablet terminals, personal computers, wearable devices, portable game machines, and the like.

[1-4. Configuration of information processing device 300]
Next, the configuration of the information processing apparatus 300 will be described with reference to FIG. The information processing device 300 includes a control unit 301, a first communication unit 302, a second communication unit 303, a storage unit 304, a reception control unit 305, a transmission request unit 306, and a data processing unit 307.

The control unit 301 is composed of a CPU, a RAM, a ROM, and the like, and the CPU controls the entire information processing device 300 and each unit by executing various processes according to a program stored in the ROM and issuing a command. ..

The first communication unit 302 is a communication module for transmitting and receiving data and various information to and from the image pickup device 100 and the terminal device 200 by 4G, which is a communication method other than 5G.

The second communication unit 303 is a communication module for transmitting and receiving data and various information to and from the image pickup device 100 and the terminal device 200 using 5G.

Some 5G communication modules also have 4G functions. In that case, one communication unit may be used, but when 5G and 4G are used at the same time, there are two communication units for 5G and 4G. You will need one.

The storage unit 304 stores and manages received data, identification information of the image pickup device 100 and the terminal device 200, and the like.

The reception control unit 305 controls to receive the first data transmitted by the image pickup apparatus 100 by 4G, the second data transmitted by 5G, or the original data.

The transmission requesting unit 306 detects whether or not the data transmitted and received from the imaging device 100 is insufficient, and if there is a shortage, requests the imaging device 100 to transmit the insufficient data. The transmission request is transmitted using either 4G by the first communication unit 302 or 5G by the second communication unit 303, using the communication method available at that time.

For example, as shown in FIG. 5, the transmission requesting unit 306 divides the received data into data by a predetermined unit time, and compares the data size, parity, etc. for each divided unit time between the transmitting side and the receiving side. It is possible to detect whether there is a shortage (deficiency due to the communication method being switched from 5G to 4G). When it is detected that the parity and data size are different from those of the transmitting side on the receiving side, a flag is set as a transmission request part in the range where there is a difference, and the transmission request is transmitted to the image pickup apparatus together with the flag information. The information processing device 300 on the receiving side needs to receive data size information and parity information from the imaging device 100 on the transmitting side. There is also a method of setting a predetermined threshold value for the data size and determining that the data is insufficient in the unit time when the data size is equal to or less than the predetermined value.

Upon receiving the transmission request, the image pickup apparatus 100 transmits the data of the portion indicated by the flag information to the information processing apparatus 300 by 5G after the establishment of the 5G communication by the second communication unit.

When the image pickup device 100 intentionally divides the original data into the first data and the second data and transmits the original data as in the first aspect of the transmission control of the image pickup device 100 described above, the transmission request is made. Not needed. This is because the image pickup apparatus 100 transmits the second data after the transmission of the first data without making a transmission request. However, this description does not exclude the request for transmission from the information processing apparatus 300. For example, when the second data is not transmitted for a predetermined period, the information processing device 300 may make a transmission request to the image pickup device 100.

The data processing unit 307 performs a process of restoring the original data from the first data and the second data received by the reception control unit 305.

Data restoration includes synthesis and replacement. The synthesis is to restore the complete original data in the information processing apparatus 300 by synthesizing the first data and the second data.

For example, in the case of the first data in which the pixels are thinned out from the original data and the second data consisting of the pixels thinned out in the first data, the pixels of the first data are complemented by the second data. The original data can be restored.

Further, in the case of the second data consisting of the first data obtained by thinning out the frames from the original data and the frames thinned out by the first data, the pixels of the first data are complemented by the second data. The original data can be restored.

When the first data is audio data and the second data is video-only data,
Original data can be restored by performing known video / audio synthesis processing in synchronization with a common time axis.

The replacement means that when the second data received later includes the first data received earlier, the complete video data is restored in the information processing apparatus 300 by replacing the first data with the second data. It is a thing.

The information processing device 300 is configured as described above. The information processing device 300 is composed of a program, and the program may be installed in a server or the like in advance, or may be distributed as a download or a storage medium so that the user can install the program on the server or the like by himself / herself. Further, the information processing device 300 may be configured on the cloud. Further, the information processing device 300 is not only realized by a program, but may also be realized by combining a dedicated device, a circuit, or the like by hardware having the function.

As shown in FIG. 6, the information processing device 300 is connected to an external video processing device 400 that performs editing processing for broadcasting on the data. The video processing device 400 includes at least a control unit 401, a communication unit 402, and a switcher 403, and operates in, for example, a broadcasting station 500. Further, the broadcasting station 500 is provided with a studio 600.

At Studio 600, producers, directors, assistant directors, cameramen, etc. of broadcasting station 500 (hereinafter collectively referred to as staff) perform work for broadcasting. The studio 600 also includes at least a CCU 601 and a camera 602. The CCU 601 processes and outputs an image captured by the camera 602, or transmits data or the like. The camera 602 photographs casters and the like reporting in the studio 600.

The control unit 401 is composed of a CPU, RAM, ROM, and the like, and the CPU controls the entire image processing device 400 and each unit by executing various processes according to a program stored in the ROM and issuing commands. In addition, the control unit 401 performs video editing processing for broadcasting. The editing process includes, for example, video division and combination, cropping, trimming, editing, subtitling, CG compositing, and effect addition. However, the editing process for broadcasting is not limited to these, and any process performed for broadcasting data may be used.

The communication unit 402 is a communication module for transmitting and receiving various data, various information, and the like to and from the information processing device 300 and the like. Communication methods include wireless LAN, WAN, WiFi, 4G, 5G and the like.

Switcher 403 is a device that switches and mixes a plurality of input signals. When there are a plurality of image pickup devices 100, the data captured by the plurality of image pickup devices 100 and transmitted to the image processing device 400 via the information processing device 300 is collected in the switcher 403. Then, the switcher 403 selects which image pickup device 100 data is to be used for broadcasting by switching. Further, the switcher 403 transmits the return data to the information processing device 300 when the video processing device 400 performs an editing process for broadcasting on the data to generate the return data. The return data is transmitted to the terminal device 200 via the information processing device 300, for example.

The video processing device 400 may be composed of hardware or a program, and the program may be installed on a server or the like having the function of the information processing device 300, or may be installed on another server. It may be configured on the cloud.

As shown in FIG. 7A, the image processing device 400 may have a function as an information processing device 300, and as shown in FIG. 7B, the information processing device 300 has a function as a video processing device 400. May be good. Although the broadcasting station and the studio are omitted in FIG. 7, the information processing device 300 and the video processing device 400 may operate at the broadcasting station as in FIG.

[1-5. Processing in the image pickup apparatus 100]
Next, the processing in the image pickup apparatus 100 will be described with reference to the flowchart of FIG.

First, in step S101, information used for determining whether or not 5G is available in the determination unit 117, such as 5G radio wave intensity information acquired by the radio wave intensity sensor 115 and area information acquired by the area information acquisition unit 116, is acquired.

Next, in step S102, the determination unit 117 determines whether the image pickup apparatus 100 is currently available in 4G or 5G. The determination result is supplied to the function control unit 118.

If 5G is not available as a result of the determination, the process proceeds from step S103 to step S104, and the function control unit 118 turns on the first communication unit 108 and performs function control so as to perform communication with 4G (step). No of S103).

Next, in step S105, the transmission control unit 119 controls transmission so as to transmit the first data by streaming using 4G.

On the other hand, in step S103, when the image pickup apparatus 100 is currently able to use 5G, the process proceeds from step S103 to step S106, so that the function control unit 118 turns on the second communication unit 109 and communicates with 5G. Function control is performed (Yes in step S103).

Next, in step S107, it is confirmed whether the transmission control unit 119 has already started transmitting the first data in 4G. If the transmission of the first data has been started in 4G, the process proceeds to step S108 (Yes in step S107).

Next, in step S108, the transmission control unit 119 controls transmission so as to transmit the second data by streaming using 5G. In this case, the second data is all the data constituting the original data other than the data transmitted as the first data in 4G, or the original data including the first data.

On the other hand, if the transmission of the first data is not started in 4G in step S107, the process proceeds to step S109 (No in step S107). Then, in step S109, the transmission control unit 119 controls transmission so as to transmit the original data by streaming using 5G. In this step S109, if the data can be transmitted in 5G in a state where the transmission of the first data has not yet started in 4G, the original data is transmitted without dividing the data from the viewpoint of data transmission efficiency. This is preferable.

Then, in step S110, the transmission control unit 119 confirms whether or not the transmission of the first data and the second data, or the original data is completed. When the data transmission is completed, the process ends (Yes in step S110). If the data transmission is not completed, steps S101 to S110 are repeated until the data transmission is completed (No in step S110).

If 5G is available and the original data is being transmitted in 5G in step S109 and 5G becomes unavailable, the subsequent data transmission will be the first data in 4G in step S105. To send. Then, when 5G becomes available again, in step S108, the shortage data due to the transmission of the first data is transmitted as the second data, and the original data thereafter is transmitted.

Transmission control in the image pickup apparatus 100 is performed as described above.

Here, the generation of area information by the area information acquisition unit 116 will be described with reference to FIG.

In order to generate area information, the model number information of the module of the base station in which the image pickup device 100 communicates in the millimeter wave band of 5G, and the information indicating the direction of the base station with reference to the image pickup device 100 are required.

The model number information of the base station module can be obtained from, for example, when it is described in SSID (Service Set Identifier), BSSID (Basic Service Set Indentifier), or the like. If it is not described in SSID or BSSID, the information provided by the manufacturer of the base station module can be accessed and acquired based on SSID or the like. Once the model number information of the base station module is acquired, the maximum radio field strength of the base station module can be estimated by referring to the product information provided by the manufacturer.

The coverage range of the base station module that communicates in the millimeter wave band of 5G is about 100 m in radius per location, and the shape of the coverage range is often circular or fan-shaped. In addition, the radio field strength of the base station module is strongest in the vicinity of the base station module and weakens as the distance from the base station module increases (attenuates in proportion to the square of the distance). Furthermore, millimeter waves are also characterized by high straightness.

Area information indicating the 5G usable range can be generated by using the above-mentioned characteristics of millimeter waves. Since millimeter waves have high straightness, it can be estimated that the base station module exists in the direction in which the radio waves are strongest. By changing the direction of the image pickup device 100, the posture information acquisition unit 114 can detect the direction in which the radio wave is strongest as the direction and posture of the image pickup device 100. The direction can be obtained based on, for example, north, south, east, and west.

Further, the distance from the current position of the image pickup apparatus 100 to the base station can be estimated from the relationship between the above-mentioned distance and the radio wave intensity. The position of the base station can be estimated from this direction and position, and the coverage range of the base station can be estimated by assuming that the coverage range is about 100 m in radius based on the position of the base station. As a result, as shown in FIG. 8, area information indicating the 5G usable range can be generated.

It is also possible to create a heat map of the radio field intensity from the change in the radio wave intensity detected by the radio wave intensity sensor 115 when the photographer possessing the image pickup device 100 moves between two points. Area information indicating the 5G usable range can also be generated by using this heat map.

By using such a method, area information can be obtained even in a situation where area information cannot be obtained from a communication carrier such as a foreign country. Further, the area information generation by the image pickup apparatus 100 can be performed in the background during shooting, or can be performed even when shooting is not performed.

When the image pickup device 100 generates the area information, the area information can be transmitted to the terminal device 200 and shared. As a result, the display unit 111 of the image pickup apparatus 100 can be used as a viewfinder at the time of shooting, or the area information can be displayed on the terminal apparatus 200 to confirm the 5G usable range while using the data for confirmation.

Although the above-described embodiment has been described as processing in the image pickup apparatus 100, it is also possible in the terminal apparatus 200 to perform communication control and transmission control in accordance with switching between 4G and 5G. Further, since the terminal device 200 also has a function as a camera, that is, a function as an image pickup device, the terminal device 200 also corresponds to an image pickup device within the scope of claims.

Further, the image pickup device 100 and the terminal device 200 are paired and connected by short-range wireless communication, the data generated by the image pickup device 100 is transmitted to the terminal device 200, and the terminal device 200 processes the data in 5G. It may be transmitted to the device 300.

[1-6. Processing in the image pickup apparatus 100 and the information processing apparatus 300]
Here, a transmission request from the information processing device 300 and data transmission from the imaging device 100 in response to the transmission request will be described with reference to FIG. Here, it will be described assuming that the data is video data.

First, in step S201, the imaging device 100 on the transmitting side transmits the video data S (t), and the information processing device 300 receives the video data S (t). t represents time information and has a value in the range from the reproduction start position (t = 1) to the reproduction end position (t = end) of the video data S (t).

Next, in step S202, the information processing device 300 transmits the communication band information C (t) to the image pickup device 100, and the image pickup device 100 receives the communication band information C (t). The communication band information can be expressed by the following equation 1 using the video data S (t).

[Equation 1]
C (t) = S (t) / Δt

Next, in step S203, the imaging device 100 transmits the video data S (t + 1) scaled by the communication band information C (t) to the information processing device 300, and the information processing device receives the video data S (t + 1).

As shown in step S204, the image pickup apparatus 100 repeats steps S201 to S203, continues with (t + 2), (t + 3), ..., And transmits up to the video data S (end) to the information processing apparatus 300. .. The video data transmitted in the processing up to this point, which is not transmitted in response to the transmission request from the information processing apparatus 300, corresponds to the first data.

Next, in step S205, the transmission requesting unit 306 of the information processing apparatus 300 transmits the remaining video data S'(t) that could not be transmitted at each time to the imaging apparatus 100 so as to transmit from t = 1 to t = end. Make a request.

Then, in step S206, the image pickup apparatus 100 transmits the remaining video data S'(t) from t = 1 to t = end. The remaining video data S'(t) corresponds to the second data.

Then, when the transmission of the remaining video data S'(t) from the imaging device 100 to the information processing device 300 is completed, all the data constituting the original video data is transmitted to the information processing device 300.

As described above, the data transmission request from the information processing device 300 and the data transmission from the image pickup device 100 can be performed.

The transmission control unit 119 of the image pickup apparatus 100 may record the untransmitted data in the original data and start transmitting the untransmitted data as the second data after the 5G communication is established.

According to this technology, data that could not be transmitted due to the switching of the communication method from 4G to 5G can be reliably transmitted and the complete original data can be restored on the receiving side. Further, by intentionally dividing the data and transmitting the data having a small data size in 4G and transmitting the remaining data having a large data size in 5G, it is possible to efficiently transmit the data according to the communication method. can.

This technology can be used, for example, for breaking news in news programs and MOJO. In breaking news, immediacy is prioritized over quality of video, so if 5G is unavailable, first data is first transmitted in 4G, and breaking news is performed using the first data. However, it is desirable to use high-quality video for reporting after breaking news. Therefore, when 5G becomes available and 5G communication is established after the first data transmission, the second data is transmitted by 5G. Then, by using the first data and the second data, it is possible to report after the breaking news with high quality video.

In this way, in reporting, for example, only voice data is transmitted as the first data, and breaking news is performed only by voice. Then, it is also possible to report after the breaking news by using the video data as the second data and the audio data as the first data. Further, the method of first transmitting the voice data as the first data is also useful when the voice translation is required for reporting. If the video processing is prioritized, the video data other than the audio may be sent first, and the audio data may be sent later.

<2. Application example of this technology>
[2-1. MOJO configuration]
Next, a specific application example of this technology will be described. This technology can be applied to a new reporting system called MOJO (Mobile Journalism).

MOJO is journalism in which journalists, the general public, etc. use the terminal device 200 to collect data and report. With the spread of terminal devices 200 such as the Internet and smartphones, even ordinary citizens or freelance journalists who do not have a dedicated imaging device 100 or video editing device can shoot, edit, and provide news materials by using the applications in the terminal device 200. It will be possible to easily carry out the above, and MOJO will enable more immediate coverage than before.

First, with reference to FIG. 11, an example of the configuration in the case of conducting coverage / reporting in the MOJO format will be described. Coverage and coverage in MOJO format consists of, for example, journalists 1000, broadcasting stations 2000, and servers 3000. The broadcasting station 2000, the server 3000, and the imaging device 100 and the terminal device 200 used by the journalist 1000 are connected via a network such as the Internet. The image pickup device 100, the terminal device 200, and the information processing device 300 are the same as those described in the embodiment.

Journalist 1000 is a person who goes to the interview destination, which is the site of the incident, conducts interviews, and provides the contents of the interview (video data, image data, audio data, text, etc.) to the broadcasting station 2000. The journalist 1000 interviews using the imaging device 100 and / or the terminal device 200.

Broadcasting station 2000 is a business entity and / or equipment that gives an instruction for coverage to journalist 1000, edits the content of coverage provided by journalist 1000, and broadcasts it.

The server 3000 is an information processing device 300 according to the present technology, and is also responsible for transmitting and receiving information, data, instructions, and the like between the broadcasting station 2000 and the journalist 1000. The server 3000 may be managed and operated by the broadcasting station 2000 itself or an affiliated company or an affiliated organization of the broadcasting station 2000, or may be managed and operated by a company other than the broadcasting station 2000 and used by the broadcasting station 2000. good.

The server 3000 (information processing device 300) is configured to include a control unit 301, a first communication unit 302, a second communication unit 303, and a storage unit 304.

The control unit 301 is composed of a CPU, a RAM, a ROM, and the like, and the CPU controls the entire information processing device 300 and each unit by executing various processes according to a program stored in the ROM and issuing a command. ..

The first communication unit 302 is a communication module for transmitting and receiving data and various information to and from the image pickup device 100 and the terminal device 200 by 4G, which is a communication method other than 5G. The second communication unit 303 is a communication module for transmitting and receiving data and various information to and from the image pickup device 100 and the terminal device 200 using 5G.

The storage unit 304 stores and manages received video data, identification information and functional information of the image pickup device 100 and the terminal device 200. The information processing device 300 may transmit the video data stored in the storage unit 304 as an archive to the image pickup device 100 and the terminal device 200 as needed or requested.

Broadcasting station 2000 has at least a control room 2100 and a studio 2200. In the control room 2100 and studio 2200, producers, directors, assistant directors, photographers, etc. (hereinafter collectively referred to as staff) of broadcasting station 2000 perform work for program broadcasting.

The control room 2100 includes at least a control unit 2101, a switcher 2102, a monitor 2103, and a communication unit 2104. The control unit 2101 controls each part and the whole of the control room 2100. The control unit 2101 also transmits and receives video data and the like between the switcher 2102 and the monitor 2103, and also transmits and receives video data and the like between the control room 2100 and the CCU 2201 (Camera Control Unit) of the studio 2200.

The switcher 2102 is a device that switches or mixes a plurality of input signals, and is used by, for example, a staff member in charge of switching operations of images displayed on the monitor 2103. The monitor 2103 is a display device that displays various images such as an image before broadcasting, an image for broadcasting, and an image that has been processed for broadcasting.

Also, Studio 2200 is equipped with at least CCU2201 and Camera 2202. The CCU 2201 processes and outputs an image captured by the camera 2202, or transmits image data or the like. The camera 2202 photographs casters and the like reporting in the studio 2200.

As shown in FIG. 11, for example, coverage instructions and coverage information are transmitted from the broadcasting station 2000 to the journalist 1000 via the server 3000. The coverage information includes, for example, a date, a journalist's ID, a coverage ID, a coverage location, a title, setting information of the image pickup apparatus 100, and the like.

Then, when the journalist 1000 conducts an interview based on the interview instruction and the interview information, the journalist 1000 transmits the interview contents to the broadcasting station 2000 via the server 3000. The content of the interview includes, for example, video, still image, audio, text, etc. obtained by the interview.

At the broadcasting station 2000, the staff composes, records, edits, etc. the program based on the interview contents sent from the journalist 1000, and broadcasts the interview contents in the program.

[2-2. First aspect of coverage in MOJO]
Next, the first aspect of the interview in MOJO will be described with reference to FIG.

Journalists at MOJO include citizen journalists (citizen journalists), video journalists, and broadcast journalists.

Citizen journalists are ordinary citizens who provide the captured images to broadcasting station 2000 and post them on the Internet. The spread of terminal devices 200 such as smartphones has made it easier for ordinary citizens to take pictures of incidents. Since we will be interviewing when we happen to encounter the scene of the incident, we can report more immediately than video journalists and broadcast journalists.

Video journalists are staff members of broadcasting station 2000, freelance journalists, etc., and their profession is to provide broadcasting station 2000 with images taken with devices that are more portable than professional photography equipment such as smartphones and digital cameras. Is the one who is. Therefore, video journalists can report faster than broadcast journalists.

A broadcast journalist is a staff member of a broadcasting station 2000, a freelance journalist, etc., who is professionally engaged in providing a video shot using a professional shooting device to the broadcasting station 2000. Since professional photography equipment is heavier and larger than terminal devices 200 such as smartphones and imaging devices 100 such as general digital cameras, it arrives at the scene of the incident and starts coverage compared to citizen journalists and video journalists. It takes time. However, since it is shot with a professional shooting device, it is possible to provide detailed coverage with higher image quality than the images shot by citizen journalists and video journalists.

FIG. 12 is a first coverage mode of MOJO in which broadcasting is performed based on coverage of a video journalist who uses the terminal device 200 and a broadcast journalist who uses the image pickup device 100 as a professional photography device.

In this first interview mode, when the broadcasting station 2000 learns of the occurrence of the incident, it requests the interview from the video journalist and the broadcast journalist via the server 3000 or directly.

The video journalist who received the interview request shoots and edits as an interview using the terminal device 200. In this example, since the terminal device 200 is more portable and portable than the professional photography equipment, the video journalist arrives at the incident site earlier than the broadcast journalist who uses the professional photography equipment. You can do the interview. In addition, the broadcasting station 2000 can request a video journalist who is near the scene of the incident and has the terminal device 200 to interview the video journalist as it is. On the other hand, since it takes more time to prepare the professional photography equipment than the terminal device 200, it is considered that it may be later than the broadcast journalist to arrive at the scene of the incident and conduct an interview.

The video journalist using the terminal device 200 transmits the video data, which is the content of the interview, to the broadcasting station 2000 via the server 3000 when the shooting and editing at the incident occurrence site are completed. The broadcasting station 2000 that has received the video data reports the first report using the video taken by the terminal device 200. This enables more immediate coverage than when only conventional professional photography equipment is used. Such an interview method is suitable for reporting the content of an incident in real time, for example, when an incident occurs during a live broadcast such as a news program broadcast during the day. It should be noted that the video journalist who uses the terminal device 200 can continue shooting as it is at the scene where the incident occurs. Further, the case where the terminal device 200 transmits the edited video data to the broadcasting station 2000 as the content of the interview is shown. For example, the editing performed by the terminal device 200 may be a process for broadcasting, or a video journalist may use it. Only the text data such as the coverage memo may be transmitted to the broadcasting station 2000 as the coverage content together with the video, and the broadcasting station 2000 may edit the processing for broadcasting or the like.

A broadcast journalist using professional shooting equipment sends video data to the broadcasting station 2000 via the server 3000 when shooting and editing at the scene of the incident is completed. The broadcasting station 2000 that has received the video data uses the video data to make further reports after the first report. Since the professional shooting device has higher performance than the terminal device 200, it is possible to perform detailed reporting with higher image quality than the image shot by the terminal device 200. Such an interview method is suitable for, for example, to additionally interview an incident interviewed by a video journalist during the day and to broadcast more detailed contents in an evening or evening news program. However, the content may be the original coverage of a broadcast journalist different from the video journalist. Broadcast journalists can also continue shooting at the scene of the incident.

[2-3. Second aspect of coverage in MOJO]
FIG. 13 shows a second coverage mode of MOJO in which broadcasting is performed based on coverage of a citizen journalist who uses the terminal device 200 and a broadcast journalist who uses the image pickup device 100 as a professional photography device.

In this second coverage mode, there is an external server 4000 for providing services on the Internet, which is different from the server 3000 used by the broadcasting station 2000. Services on the Internet include various SNS (Social Network Service), video distribution sites, blog sites, and Internet TV.

Since the Citizen journalist is a general citizen who has no employment relationship with the broadcasting station 2000, in this second interview mode, the broadcasting station 2000 and the Citizen journalist will know the occurrence of the incident separately. Therefore, Citizen journalists voluntarily conduct interviews even if there is no interview request from the broadcasting station 2000. Citizen journalists can shoot and edit as an interview as soon as they discover an incident by chance or encounter an incident.

The Citizen journalist who encountered the incident uses the terminal device 200 to take pictures as an interview. In general, since the terminal device 200 is superior in portability, portability, and communication function as compared with professional photography equipment, it is considered that citizen journalists can post and provide coverage earlier than broadcast journalists. .. Citizen journalists may also attach text to the footage they shoot. In some cases, the video captured by a citizen journalist may be edited in the server of the broadcasting station 2000 or the broadcasting station 2000.

Citizen journalists using the terminal device 200 post the contents of the interview to SNS and video distribution sites on the Internet when shooting and editing are completed. This is the first report without going through the broadcasting station 2000. This enables more immediate coverage than the conventional coverage provided by the broadcasting station 2000.

Broadcasting station 2000 regularly searches SNS, video distribution sites, blog sites, etc., and when it finds out that a citizen journalist is posting an incident on the Internet, he requests the citizen journalist to provide information and posts the content. get.

The broadcasting station 2000 that acquired the posted content reports using the posted content. This enables more immediate coverage than the coverage and coverage of video and broadcast journalists.

In addition, the broadcasting station 2000 may request a citizen journalist to receive the supply of coverage contents such as video data in real time. This makes it possible to perform real-time coverage (live broadcasting) using images taken by Citizen journalists.

In addition, a broadcast journalist using a professional shooting device transmits video data to the broadcasting station 2000 via the server 3000 when shooting and editing are completed. The broadcasting station 2000 that has received the video data uses the video taken by the camera to make further reports after the first report. Since the professional shooting device has higher performance than the terminal device 200, it is possible to perform detailed reporting with higher image quality than the image shot by the terminal device 200. In addition, the broadcasting station 2000 can also perform real-time reporting (live broadcasting) using images transmitted from broadcast journalists.

The above is an example of MOJO's coverage. It should be noted that the modes of coverage in FIGS. 12 and 13 are merely examples.

With such a reporting method, live coverage or real-time video streaming may be used. In such live broadcasts and real-time video streaming, journalists may want to see what the broadcaster 2000 is currently broadcasting and how the footage they are shooting is being broadcast. .. This technique is useful in such cases.

Specifically, the journalist 1000 (citizen journalist, video journalist, and broadcast journalist) possesses an image pickup device 100 and a terminal device 200, and shoots and edits with the image pickup device 100 at the incident occurrence site. Then, the shooting data is transmitted to the broadcasting station 2000 via the server 3000. The broadcasting station 2000 broadcasts using the shooting data, and transmits the broadcast contents as return video data to the terminal device 200 of the journalist 1000. By displaying the return video data on the terminal device 200, the journalist 1000 can confirm the broadcast content on the terminal device 200 while taking a picture with the image pickup device 100. It is also possible to give instructions, consultations, meetings, etc. between the broadcasting station 2000 and the journalist 1000 by voice call.

<3. Modification example>
Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical idea of the present technology are possible.

In the embodiment, video data has been described as a specific example, but this technology can be applied to all kinds of data such as image data, audio data, text data, statistical data, measurement data, programs, and application data.

This technology can be applied not only to news reports such as MOJO, but also to all types of broadcasting such as live dramas and live variety shows.

In the embodiment, the first communication method is 4G and the second communication method is 5G, but the communication method is not limited to them. The second communication method may be any communication method as long as it can perform high-speed, large-capacity and / or low-delay communication as compared with the first communication method. Therefore, when a communication method having a higher speed, a larger capacity, and / or a lower delay than 5G, for example, 6G (6th generation mobile communication system) is put into practical use in the future, the first communication method will be 5G and the second communication. This technology can be applied by setting the method to 6G. The same applies to the communication method after 6G.

Further, the communication method is not limited to two, and this technology can be applied to a device capable of communicating with three or more communication methods.

The present technology can also have the following configurations.
(1)
A communication unit that communicates with a first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method.
A function control unit that switches the communication method in the communication unit, and
An imaging device including a transmission control unit that controls transmission of original data based on a transmission method according to a communication method in the communication unit.
(2)
When the communication unit communicates by the first communication method, the transmission control unit transmits the first data constituting the original data, and the communication unit communicates by the second communication method. In the case, the imaging apparatus according to (1), wherein the second data constituting the original data is controlled to be transmitted.
(3)
The imaging apparatus according to (2), wherein the first data and the second data are data obtained by dividing the original data in order to reduce the data size of the original data.
(4)
The imaging device according to (2) or (3), wherein the first data has a smaller data size than the second data.
(5)
The first data is data obtained by thinning out frames of video data generated by shooting, and the second divided data is data including frames thinned out in the first data (2). The imaging device according to any one of (4).
(6)
The first data is data obtained by thinning out the pixels of the video data generated by shooting, and the second data is data including the pixels thinned out in the first data (2). The imaging device according to any one of 5).
(7)
The imaging according to any one of (2) to (6), wherein the first data is audio data included in the video data generated by shooting, and the second data is data other than the audio data. Device.
(8)
The second data is described in any one of (2) to (7), which is the original data including the first data, or all the data constituting the original data other than the first data. Imaging device.
(9)
The second data includes untransmitted data that has not been transmitted because the communication by the communication unit is switched from the second communication method to the first communication method (2) to (8). The imaging apparatus according to any one.
(10)
The imaging device according to (9), wherein the transmission control unit transmits the second data in response to a request from an external device.
(11)
The transmission control unit does not transmit the original data when the communication unit communicates by the first communication method, and the original data when the communication unit communicates by the second communication method. The imaging apparatus according to any one of (1) to (10), which controls to transmit the image.
(12)
When the transmission control unit transmits the first data constituting the original data by the first communication method and then communicates by the second communication method, the transmission control unit transmits the original data as the second data. The imaging apparatus according to any one of (1) to (11).
(13)
The imaging device according to any one of (1) to (12), wherein the second communication method enables communication with a lower delay than the first communication method.
(14)
A communication unit that communicates with a first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method.
A function control unit that switches the communication method in the communication unit, and
A control method for an imaging device that controls transmission of original data based on a transmission method according to a communication method in the communication unit.
(15)
A communication unit that communicates with a first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method.
A function control unit that switches the communication method in the communication unit, and
A control program that causes a computer to execute a control method of an image pickup device that controls transmission of original data based on a transmission method according to a communication method in the communication unit.
(16)
A communication unit that communicates with the image pickup apparatus by the first communication method and / or the second communication method capable of higher speed or larger capacity communication than the first communication method.
Receives the first data constituting the original data transmitted by the first communication method from the image pickup apparatus and the second data constituting the original data transmitted by the second communication method. An information processing device including a reception control unit.
(17)
It is determined whether the first data transmitted from the imaging device is insufficient with respect to the original data, and if the first data is insufficient with respect to the original data, the imaging device is subjected to the above. The information processing apparatus according to (16), which includes a transmission requesting unit that requests transmission of insufficient data as second data.
(18)
When the first data is insufficient with respect to the original data, the second data is transmitted by the second communication method after the second communication method is established (16) or (17). ).
(19)
The information processing apparatus according to any one of (16) to (18), comprising a data processing unit that restores the original data based on the first data and the second data.
(20)
Original data transmitted by the first communication method from an image pickup device that communicates by the first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method. An information processing method for receiving the first data to be configured and the second data constituting the original data transmitted by the second communication method.
(21)
The original data transmitted by the first communication method from the image pickup device that communicates by the first communication method and / or the second communication method capable of higher speed or larger capacity communication than the first communication method. An information processing program that causes a computer to execute an information processing method for receiving the first data to be configured and the second data constituting the original data transmitted by the second communication method.

100 ... Imaging device 108 ... First communication unit 109 ... Second communication unit 118 ... Function control unit 119 ... Transmission control unit 300 ... Information processing device 305 ... Reception control unit 306 ... Transmission request unit 307 ... Data processing unit

Claims (21)

A communication unit that communicates with a first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method.
A function control unit that switches the communication method in the communication unit, and
An imaging device including a transmission control unit that controls transmission of original data based on a transmission method according to a communication method in the communication unit. When the communication unit communicates by the first communication method, the transmission control unit transmits the first data constituting the original data, and the communication unit communicates by the second communication method. In this case, the imaging device according to claim 1, wherein the second data constituting the original data is controlled to be transmitted. The imaging device according to claim 2, wherein the first data and the second data are data obtained by dividing the original data in order to reduce the data size of the original data. The imaging device according to claim 2, wherein the first data has a smaller data size than the second data. The first data is data obtained by thinning out frames of video data generated by shooting, and the second divided data is data including frames thinned out in the first data according to claim 2. The imaging device described. The first data is the data obtained by thinning out the pixels of the video data generated by shooting, and the second data is the data including the thinned out pixels in the first data according to claim 2. Imaging device. The imaging device according to claim 2, wherein the first data is audio data included in video data generated by shooting, and the second data is data other than the audio data. The imaging device according to claim 2, wherein the second data is the original data including the first data, or all data constituting the original data other than the first data. The imaging device according to claim 2, wherein the second data includes untransmitted data that has not been transmitted because the communication by the communication unit is switched from the second communication method to the first communication method. .. The imaging device according to claim 9, wherein the transmission control unit transmits the second data in response to a request from an external device. The transmission control unit does not transmit the original data when the communication unit communicates by the first communication method, and the original data when the communication unit communicates by the second communication method. The imaging device according to claim 1, wherein the image pickup device is controlled to transmit the data. When the transmission control unit transmits the first data constituting the original data by the first communication method and then communicates by the second communication method, the transmission control unit transmits the original data as the second data. The image pickup apparatus according to claim 1. The imaging device according to claim 1, wherein the second communication method enables communication with a lower delay than the first communication method. A communication unit that communicates with a first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method.
A function control unit that switches the communication method in the communication unit, and
A control method for an imaging device that controls transmission of original data based on a transmission method according to a communication method in the communication unit. A communication unit that communicates with a first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method.
A function control unit that switches the communication method in the communication unit, and
A control program that causes a computer to execute a control method of an image pickup device that controls transmission of original data based on a transmission method according to a communication method in the communication unit. A communication unit that communicates with the image pickup apparatus by the first communication method and / or the second communication method capable of higher speed or larger capacity communication than the first communication method.
Receives the first data constituting the original data transmitted by the first communication method from the image pickup apparatus and the second data constituting the original data transmitted by the second communication method. Reception control unit and
Information processing device equipped with. It is determined whether the first data transmitted from the imaging device is insufficient with respect to the original data, and if the first data is insufficient with respect to the original data, the imaging device is subjected to the above. The information processing apparatus according to claim 16, further comprising a transmission requesting unit that requests transmission of insufficient data as second data. The 17th aspect of claim 17, wherein when the first data is insufficient with respect to the original data, the second data is transmitted by the second communication method after the second communication method is established. Information processing device. The information processing apparatus according to claim 16, further comprising a data processing unit that restores the original data based on the first data and the second data. Original data transmitted by the first communication method from an image pickup device that communicates by the first communication method and / or a second communication method capable of higher speed or larger capacity communication than the first communication method. An information processing method for receiving the first data to be configured and the second data constituting the original data transmitted by the second communication method. The original data transmitted by the first communication method from the image pickup device that communicates by the first communication method and / or the second communication method capable of higher speed or larger capacity communication than the first communication method. An information processing program that causes a computer to execute an information processing method for receiving the first data to be configured and the second data constituting the original data transmitted by the second communication method.

Images