JP2018011120A - Imaging apparatus, control method for imaging apparatus, and program - Google Patents

Abstract

An object of the present invention is to maintain high image quality while avoiding pressure on a memory that stores encoded data obtained by encoding a captured image. A frequency converter (103) generates conversion coefficient data obtained by frequency-converting plane data. The quantization unit (104) quantizes the transform coefficient data to generate quantized data. The entropy encoding unit (105) encodes any of plane data, transform coefficient data, and quantized data. The control unit (110) encodes plane data when the used capacity of the memory (107) for storing the encoded data is less than the threshold value, and transform coefficient data for one frame when the memory used capacity exceeds the threshold value. After encoding, control is performed so that the quantized data is encoded. [Selection] Figure 1

Description

  The present invention relates to an imaging apparatus that encodes captured image data, a control method for the imaging apparatus, and a program.

  In recent years, with the spread of digital cameras and digital camcorders, it has become widespread to shoot still images and moving images by digital methods. Generally, these cameras compress and record still image or moving image data using an encoding technique typified by JPEG or H.264. An imaging apparatus capable of compressing and recording the raw RAW image data is also disclosed.

  In encoding techniques typified by JPEG and H.264, in order to reduce the amount of data to be recorded, the amount of data can be reduced by lossy encoding that quantizes relatively less important information. In general, lossless encoding without quantization may be performed. Patent Document 2 discloses an image compression apparatus having a configuration for switching between lossless encoding and lossy encoding.

JP 2014-179853 A International Publication No. 2012/160626

  Here, as an assumption example when switching between lossless encoding and lossy encoding is performed, there is a pressure avoidance of a memory that temporarily stores image data such as still images and moving images taken continuously. For example, if there is sufficient free memory space, lossless coding is performed without image quality degradation, and when the free memory space decreases, switching to lossy coding reduces the amount of generated code. , Avoid memory pressure. In particular, in lossless encoding, the amount of generated code may be very large depending on the image, and if the generated code amount exceeds the free space of the memory, the memory will fail. For this reason, when the free space of the memory is reduced, the loss due to the compression of the memory is avoided by switching from lossless encoding to lossy encoding to reduce the generated code amount.

  However, immediately after switching from lossless encoding to lossy encoding, even in lossy encoding, the amount of code generated by encoding may not be much different from that in lossless encoding. There is a risk that the avoidance of pressure will be insufficient. On the other hand, in order to suppress the amount of generated code due to lossy encoding, if quantization is performed to significantly reduce information of relatively low importance, the image quality will be greatly degraded, and encoding switching has been performed. Variations in image quality at locations will become noticeable.

  Accordingly, an object of the present invention is to reduce fluctuations in image quality when switching encoding while avoiding compression of a memory that stores encoded data obtained by encoding a captured image.

  According to the present invention, frequency conversion means for performing frequency conversion on a frame image to generate first coefficient data, and second coefficient data are generated by quantizing the first coefficient data based on a quantization parameter. A quantization unit; an encoding unit that encodes input data to generate encoded data; and a control unit that controls encoding. The control unit includes a plurality of frames generated from a plurality of frame images. When the encoded data is stored in the memory, if the capacity occupied by the encoded data in the memory is less than a predetermined threshold, the frame image data is encoded, and the capacity occupied by the encoded data in the memory is predetermined. When the threshold value is equal to or greater than the threshold, the first coefficient data for at least one frame is encoded, and then the second coefficient data is controlled to be encoded.

  According to the present invention, it is possible to reduce fluctuations in image quality when switching between encodings while avoiding pressure on a memory that stores encoded data obtained by encoding a captured image.

It is a figure which shows the example of schematic structure of the imaging device of 1st Embodiment. It is a figure used for description of plane conversion. It is a figure used for description of a subband at the time of carrying out DWT processing of plane data. It is a figure used for description of the reference place at the time of determining a quantization step value. It is a figure which shows the memory use capacity transition in an assumption example, and the input to an encoding part. It is a flowchart which shows operation | movement of a control part. It is a figure which shows the memory use capacity transition of 1st Embodiment, and the input to an encoding part. It is a figure which shows the schematic structural example of the imaging device of 2nd Embodiment. It is a figure which shows the memory use capacity transition of 2nd Embodiment, and the input to an encoding part.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.
<First Embodiment>
FIG. 1 is a diagram illustrating a schematic configuration of an imaging apparatus 100 according to the first embodiment.
The imaging apparatus 100 according to the present embodiment includes an imaging unit 101, a plane conversion unit 102, a frequency conversion unit 103, a quantization unit 104, an entropy encoding unit 105, a memory I / F unit 106, a memory 107, a recording processing unit 108, a recording A medium 109, a control unit 110, and a MUX 111 are included.

  The control unit 110 controls at least the plane conversion unit 102, the frequency conversion unit 103, the quantization unit 104, the entropy encoding unit 105, and the MUX 111.

  The imaging unit 101 includes a lens optical system, a diaphragm, a driving unit that controls and drives the lens optical system and the diaphragm, an imaging element, and the like. The lens optical system includes a focus optical system that enables focus adjustment, a zoom optical system that enables optical zoom, and the like. The drive unit includes a focus control and lens drive unit for the focus optical system, an aperture control and aperture drive unit for the aperture, and may further include a zoom control and lens drive unit for the zoom optical system. The imaging element is composed of a CCD image sensor or a CMOS sensor, and converts an optical image formed on the imaging surface by a lens optical system into an electrical signal. Then, the imaging unit 101 sends RAW (raw) data obtained by converting an electrical signal obtained by the imaging element into a digital signal to the plane conversion unit 102 as data of a frame image (hereinafter referred to as a frame) by imaging. .

  The plane conversion unit 102 converts the RAW data supplied from the imaging unit 101 into data for each color component. Hereinafter, converting RAW data into data for each color component is referred to as plane conversion. FIG. 2 is an explanatory diagram of plane conversion. In FIG. 2, RAW data 200 is data corresponding to a so-called Bayer array, and R in the figure represents red, G (G1, G2) represents green, and B represents blue. The plane conversion unit 102 converts the RAW data 200 into image data of four planes, that is, an R component plane 201R, a G1 component plane 201G1, a G2 component plane 201G2, and a B component plane 201B. In this embodiment, plane image data of each color component obtained by plane conversion of RAW data is sent to a frequency conversion unit 103 described later and a MUX 111 described later. Hereinafter, the image data of each plane of each color component obtained by plane conversion is referred to as plane data.

  The frequency conversion unit 103 performs frequency conversion processing on the plane data for each color component supplied from the plane conversion unit 102. As an example of the frequency conversion process, a case where a so-called discrete wavelet transform (hereinafter referred to as DWT) process is performed will be described with reference to FIG. In FIG. 3, it is assumed that the plane data 300 is any one of the four plane data described above. The frequency conversion unit 103 generates subband data 301 including conversion coefficient data for each frequency band by performing frequency decomposition processing for decomposing the plane data 300 into a plurality of frequency bands by DWT processing three times. In the present embodiment, the conversion coefficient data for each frequency band in the subband data 301 corresponds to the first coefficient data according to the present invention. Hereinafter, each frequency band in which the plane data 300 has been subjected to DWT processing is referred to as a subband. In subband data 301 of FIG. 3, subband “0” represents DC (direct current) component conversion coefficient data, and subbands “1” to “9” represent AC (alternating current) component conversion coefficient data. Represents. The subband data generated by the frequency conversion unit 103 is sent to a quantization unit 104 described later and a MUX 111 described later.

  The quantization unit 104 divides the transform coefficient data of each subband in the subband data by a quantization step value that is one of the quantization parameters to generate quantized data. That is, each transform coefficient data of each subband is compressed by being divided by the quantization step value. In the present embodiment, this quantized data corresponds to the second coefficient data according to the present invention. The quantized data obtained by the quantization processing by the quantization unit 104 is sent to the MUX 111 described later.

  The MUX 111 is a multiplexer that selects and outputs one data from a plurality of input data based on a control signal from the control unit 110. Data output from the MUX 111 is sent to the entropy encoding unit 105. In the case of the present embodiment, the MUX 111 includes the plane data output from the plane conversion unit 102, the subband data output from the frequency conversion unit 103, and the quantized data output from the quantization unit 104. , Three data are input. In the present embodiment, when the operation mode of the imaging apparatus 100 is the first operation mode, the control unit 110 controls the MUX 111 so that the plane data from the plane conversion unit 102 is output to the entropy encoding unit 105. To do. Further, when the operation mode of the imaging apparatus 100 is the second operation mode, the control unit 110 controls the MUX 111 so as to output the subband data from the frequency conversion unit 103 to the entropy encoding unit 105. When the operation mode of the imaging apparatus 100 is the third operation mode, the control unit 110 controls the MUX 111 so that the quantized data from the quantization unit 104 is output to the entropy encoding unit 105.

  The entropy encoding unit 105 performs entropy encoding on the data supplied from the MUX 111. Specifically, when the imaging apparatus 100 is in the first operation mode, for example, the entropy encoding unit 105 performs entropy encoding on the plane data supplied via the MUX 111. When the imaging apparatus 100 is in the second operation mode, the entropy encoding unit 105 performs entropy encoding on the subband data supplied via the MUX 111. Further, when the imaging apparatus 100 is in the third operation mode, the entropy encoding unit 105 performs entropy encoding on the quantized data supplied via the MUX 111. In addition, as a method of entropy coding, lossless coding such as Golomb coding, run-length coding, Huffman coding, arithmetic coding, etc. can be considered, but the method is not limited here and any A scheme may be used. The encoded data generated by the encoding in the entropy encoding unit 105 is sent to the memory I / F unit 106.

  Information indicating the code amount generated by the encoding by the entropy encoding unit 105 is fed back to the quantization unit 104. In the case of this embodiment, for example, when subband data is encoded in the second operation mode, for each subband by encoding transform coefficient data for each subband together with information on the generated code amount for one frame. The generated code amount information is sent to the quantization unit 104. In addition, when the quantized data is encoded for each subband in the third operation mode, the subband is generated by encoding the quantized data for each subband together with information on the generated code amount for one frame. Information on the amount of generated code for each is sent to the quantization unit 104. As described above, the entropy encoding unit 105 feeds back the generated code amount information for each subband in the second and third operation modes to the quantization unit 104 together with the generated code amount information for one frame.

  The quantization unit 104 refers to the generated code amount information fed back from the entropy encoding unit 105 and determines a quantization step value that is one of the quantization parameters. In the case of the present embodiment, the quantization unit 104 is fed back information on the generated code amount for one frame and the generated code amount for each subband. Therefore, the quantization unit 104 determines a fine quantization step value for each subband by referring not only to the generated code amount for one frame but also to the generated code amount for each subband.

  Hereinafter, the operation of the process in which the quantization unit 104 determines a fine quantization step value for each subband with reference to the generated code amount for each subband will be described with reference to FIG. In determining the quantization step value, the generated code amount for one frame is also referred to, but in the following description, the reference operation for the generated code amount for one frame is omitted.

  The horizontal axis in FIG. 4 indicates time, and the time point Ta indicates, for example, the shooting timing of the first frame among the time-series frames acquired by continuous shooting for continuously capturing still images, and the time point Tb indicates two frames. Assume that the shooting timing of the eye frame is shown. Note that the points in time Ta and Tb may be the shooting timings of the first frame and the second frame in each successive time-series frame by moving image shooting. Hereinafter, only a case where continuous shooting of still images is performed will be described as an example. The subband data 401 in FIG. 4 represents conversion coefficient data of each subband generated by frequency conversion processing from one plane data of a frame shot at the time Ta. Further, it is assumed that the subband data 402 represents conversion coefficient data of each subband generated by frequency conversion processing from the same plane data of the frame shot at the time Tb.

  Here, for example, when continuous shooting of still images is performed, it is considered that the images of the frames in the time series order have a high correlation in the time direction. Even when moving image shooting is performed, it is considered that the images of the frames in time-series order by the moving image shooting have a high correlation in the time direction. Therefore, when the quantization unit 104 quantizes the subband data 402 obtained at the time point Tb, the generated code amount for each subband of the subband data 401 obtained at the time point Ta one frame before is obtained. The quantization step value is determined for each subband. For example, when the transform coefficient data of the subband “5” in the subband data 402 is quantized, the generated code amount in the subband “5” in the subband data 401 is referred to and the subband data 402 is referred to. A quantization step value corresponding to “5” is determined. Thus, when determining the quantization step value of each subband, the quantization unit 104 uses the generated code amount in the same subband one frame before, which is considered to have a high correlation in the time direction, as a reference value. To do. Therefore, the quantization unit 104 includes a storage element that stores information on each generated code amount one frame before fed back from the entropy encoding unit 105.

  Returning to FIG. The memory I / F unit 106 arbitrates memory access requests from each unit of the imaging apparatus 100 and controls reading / writing with respect to the memory 107. The encoded data after the entropy encoding by the entropy encoding unit 105 described above is temporarily stored in the memory 107 via the memory I / F unit 106.

  The memory 107 is a storage area composed of, for example, a volatile memory for temporarily storing encoded data after entropy encoding and various data output from each unit of the imaging apparatus 100. The encoded data temporarily stored in the memory 107 is sent to the recording processing unit 108 via the memory I / F unit 106.

  The recording processing unit 108 generates recording data suitable for recording on the recording medium 109 from the encoded data read from the memory 107 and records the recording data on the recording medium 109. The recording medium 109 is a recording medium composed of, for example, a nonvolatile memory. Note that the imaging apparatus 100 according to the present embodiment can reproduce recorded data from the recording medium 109, but the description and illustration of the configuration and the like for the reproduction are omitted.

<Example of switching between lossless coding and lossy coding>
Here, when the encoded data of each frame by continuous shooting or the like is temporarily stored in the buffer memory and then recorded on the recording medium, lossless encoding and lossy encoding are performed according to the capacity occupied by the encoded data in the buffer memory. Let's assume an example of switching the conversion. This assumption example is an example for explaining a problem that may occur due to switching between lossless coding and lossy coding, and coding switching control described later according to the present embodiment shown in FIG. It is an example for demonstrating the difference.

  In this assumption example, it is assumed that in lossless encoding, entropy encoding is performed on plane data obtained by plane-converting captured RAW data. On the other hand, in lossy coding, it is assumed that entropy coding is performed on the quantized data after the quantization processing is performed on each subband data obtained by subjecting the plane data to frequency conversion processing. In this assumption example, switching from lossless encoding to lossy encoding is performed when the capacity occupied by the encoded data in the buffer memory (hereinafter referred to as “used capacity”) exceeds a predetermined threshold. Let's say.

  FIG. 5 illustrates the capacity of the buffer memory used in the case where the encoded data of each frame of still images taken continuously is temporarily stored in the buffer memory and then transferred to the recording medium and recorded in this example. It is a figure which shows the example of a transition. In the example of FIG. 5, the code amount per unit time of the encoded data that is entropy encoded and stored in the buffer memory is the transfer data amount per unit time that is read from the buffer memory and transferred to the recording medium. Indicates the case where the number is exceeded.

  The horizontal axis of the graph 500 in FIG. 5 indicates time, the vertical axis indicates the capacity of the buffer memory, and the broken line 502 indicates the transition of the used capacity of the buffer memory. Each scale on the horizontal axis of the graph 500 in FIG. 5 indicates the time for which encoded data of one frame by continuous shooting is stored in the buffer memory. Further, a time point T1 in FIG. 5 indicates a timing at which encoded data of, for example, the first frame obtained by continuous shooting is stored in the buffer memory. The same applies to the following, and time points T2 to T13... Indicate the timing at which the encoded data of the second to thirteenth frames are stored in the buffer memory. The time point T0 represents the timing of the start of photographing. From the time point T0 to T2, the memory usage indicated by the broken line 502 gradually increases, but the memory usage capacity decreases at the time point T2. This means that the capacity for the first encoded data has been made available in the buffer memory by completing the data transfer process from the buffer memory to the recording medium. Note that the data transfer from the buffer memory to the recording medium actually starts at time T1 when the first frame is stored in the buffer memory. In the example of FIG. 5, for the sake of easy understanding, an image in which the memory usage is freed by the first encoded data (data amount is subtracted) at the timing of completion of the data transfer at time T <b> 2. A broken line 502 is drawn.

  A time chart 501 in FIG. 5 represents the input timing of data input to the entropy encoding process. In the case of this assumption example, since the buffer memory has a sufficient capacity immediately after the start of continuous shooting, the plane data of each frame by continuous shooting is used as input data to the entropy encoding process, and lossless encoding is performed. Done. Therefore, the input data IN0 and IN1 in FIG. 5 are plane data, that is, data that has not been subjected to frequency conversion processing and quantization processing.

  On the other hand, a time point Tx in the graph 500 of FIG. 5 represents a timing at which the used capacity of the buffer memory indicated by the broken line 502 becomes equal to or greater than a predetermined threshold (Th or greater). As described above, when the used capacity of the buffer memory becomes equal to or greater than the predetermined threshold Th, the encoding method is switched to lossy encoding thereafter in order to reduce the amount of data stored in the buffer memory. In lossless encoding, the amount of generated code may increase depending on the image, and if the generated code amount exceeds the free space of the buffer memory, the encoded data cannot be stored, and the failure occurs. The encoding method is switched to lossy encoding. In the case of this assumption example, the amount of generated code is suppressed by performing lossy encoding such that entropy encoding is performed on the quantized data after the frequency conversion process and the quantization process. In the case of FIG. 5, after the time T12 after the time Tx when the used capacity of the buffer memory becomes equal to or greater than the threshold Th, the quantized data IN12, IN13,... After the frequency conversion process and the quantization process are performed. This is input data to the entropy encoding process.

  Here, in the quantization process, data is compressed by determining a quantization step value and dividing the transform coefficient data for each subband by the frequency conversion process by the quantization step value. The quantization step value is determined, for example, by using the correlation that the images of each frame that has been continuously shot have in the time direction. Specifically, referring to the generated code amount of an image encoded one frame before, the quantization step value is determined based on the generated code amount. Further, when referring to the generated code amount, as described with reference to FIG. 4 above, not only the generated code amount for one frame but also the sub-band generated by encoding the quantized data for each sub-band. The generated code amount is referred to. In this way, a finer quantization step value is set by changing the quantization step value for each subband in accordance with the generated code amount for each subband.

  However, in the case of the assumption example in FIG. 5, switching to lossy coding is performed after time T12, but lossless coding is performed before time T12, and frequency conversion processing and quantization processing are performed. Absent. For this reason, the quantization step value used when generating the quantized data of the input data IN12 is determined by referring only to the generated code amount for each plane data by the lossless encoding performed one frame before. Value. Therefore, the input data IN12 is data obtained by quantizing the transform coefficient data for each subband by the frequency transform process using the quantization step value determined with reference to the generated code amount for each plane data. That is, the input data IN12 is not a fine and appropriate quantization step value corresponding to the generated code amount for each subband, but a rough quantization step value corresponding to the generated code amount for each plane data. The conversion coefficient data is quantized data.

  When the quantized data such as the input data IN12 is entropy-encoded, for example, as shown by a broken line 503 in FIG. 5, a code amount that is not much different from the generated code amount when lossless encoding is performed is generated. May end up. Conversely, as indicated by the broken line 504 in FIG. 5, the amount of generated code is reduced, but image quality deterioration due to encoding may be increased. As in the example of the broken line 503, when a code amount that is not much different from that in the case of the lossless encoding occurs, the free space of the buffer memory is reduced without reducing the used capacity of the buffer memory even at the time T13, and the encoding is performed. Data may not be stored. On the other hand, if the amount of generated codes can be reduced as in the example of the broken line 504, the use capacity of the buffer memory can be reduced, and the possibility that the failure of the buffer memory can be avoided increases. However, in this case, since the image quality deterioration due to the encoding becomes large, the image quality fluctuation becomes conspicuous at the place where the encoding method is switched from the lossless encoding to the lossy encoding.

  Thus, in the case of the assumption example in FIG. 5, when switching between lossless coding and lossy coding, reference information for determining the quantization step value is insufficient, and the control of the code amount becomes rough, so that the memory capacity is reduced. There may be a problem that compression or deterioration of image quality and image quality fluctuation are conspicuous.

<Switching control between lossless coding and lossy coding in this embodiment>
Therefore, when switching between lossless encoding and lossy encoding in encoding of still images and moving image frames taken continuously, the imaging apparatus 100 according to the present embodiment reduces compression of the memory 107, image quality degradation, and image quality fluctuation. Switching control that can be avoided is realized.

FIG. 6 is a control flowchart when the imaging apparatus 100 of the present embodiment performs continuous shooting of still images, for example, and shows processing steps related to encoding of the shot frames.
Each process shown in the flowchart of FIG. 6 is not only realized by the hardware configuration of the imaging apparatus 100 shown in FIG. 1, but may be realized by executing the program according to the present embodiment by a CPU or the like. Good. In the following description, steps S201 to S214 of each process in the flowchart are abbreviated as S201 to S214. The process of the flowchart in FIG. 6 starts when continuous shooting of still images starts.

  When continuous shooting of still images is started, the control unit 110 first initializes the operation mode of the imaging apparatus 100. The initialized operation mode is referred to as an initial operation mode. In the present embodiment, the operation modes include an initial operation mode, the first operation mode, the second operation mode, and the third operation mode described above, and details of control in each operation mode will be described later. After setting the initial operation mode, the control unit 110 proceeds to the control process of S201.

  When continuous shooting is started in the imaging apparatus 100, RAW data of each frame by continuous shooting is output from the imaging unit 101, and the RAW data is supplied to the plane conversion unit 102. In step S <b> 201, the control unit 110 controls the plane conversion unit 102 to perform a RAW data plane conversion process. After S201, the control unit 110 proceeds to the control process of S202.

  In S202, the control unit 110 determines whether the operation mode of the imaging apparatus 100 is the second or third operation mode. If the control unit 110 determines in S202 that the operation mode is the second or third operation mode (YES), the control unit 110 proceeds to the control process of S207. On the other hand, when the control unit 110 determines in S202 that the operation mode is not the second or third operation mode (NO), the control unit 110 proceeds to the control process of S203. When the continuous shooting of the still image is started, the operation mode is set to the initial operation mode as described above. Therefore, the control unit 110 determines in S202 that the operation mode is not the second or third operation mode, The control process proceeds to S203.

  In step S203, the control unit 110 determines whether or not the used capacity of the memory 107 is equal to or greater than a predetermined threshold (Th or greater). The threshold Th is a threshold for memory usage, and is preset in the control unit 110 as a value smaller than the maximum capacity that can be stored in the memory 107. The control unit 110 knows the maximum capacity of the memory 107 and the speed of writing and reading, and by monitoring the amount of code generated by the encoding by the entropy encoding unit 105, it can determine which capacity of the memory 107 is used. We are aware of how it is moving. For this reason, the control unit 110 can determine whether or not the used capacity of the memory 107 is equal to or greater than the predetermined threshold Th. Although not shown in FIG. 1, the memory I / F unit 106 generates information on the memory usage capacity and notifies the control unit 110 of the information, and the control unit 110 uses the information on the memory usage capacity. It may be determined whether the used capacity of the memory 107 is equal to or greater than the threshold Th. If the controller 110 determines in S203 that the used capacity of the memory 107 is greater than or equal to the threshold Th (YES), the control unit 110 proceeds to S205, and if it determines that the used capacity is less than the threshold (less than Th) (NO). Control proceeds to S204. When continuous shooting of a still image is started, the capacity of the memory 107 has a margin and is less than the threshold value Th, so the control unit 110 advances the control process to S204.

  When the process proceeds to S204, the control unit 110 changes the initial operation mode when the continuous shooting of still images is started to the first operation mode, and then proceeds to the control process of S210. In S210, the control unit 110 controls the entropy encoding unit 105 to perform entropy encoding. At this time, the first operation mode is set. As described above, the first operation mode is an operation mode in which plane data is directly entropy-encoded. Therefore, in S210, the control unit 110 causes the plane data from the plane conversion unit 102 (that is, plane data not passing through the frequency conversion unit 103 and the quantization unit 104) to be directly input to the entropy coding unit 105. The MUX 111 is controlled. The encoded data generated by the entropy encoding unit 105 is stored in the memory 107 via the memory I / F unit 106. Therefore, the memory 107 at this time stores encoded data obtained by entropy encoding plain data that has not been subjected to frequency conversion processing and quantization processing, that is, encoded data by lossless encoding.

  As described above, after the entropy encoding in the first operation mode is performed in S210, the control unit 110 proceeds to the control process in S211. In S211, the control unit 110 determines whether or not the operation mode of the imaging device 100 is the first operation mode. If the control unit 110 determines in S211 that the operation mode is the first operation mode (YES), the control unit 110 proceeds to the control process in S213. If the control unit 110 determines that the operation mode is not the first operation mode (NO), the control unit 110 performs the control. Proceed to processing. Since the first operation mode is set as described above when continuous shooting of still images is started, the control unit 110 determines in S211 that the operation mode is the first operation mode, and S213. Proceed to the control process.

  In S213, the control unit 110 determines whether or not the above-described entropy encoding has been completed for RAW data of all frames by continuous shooting. When the control unit 110 determines in S213 that the encoding process for the RAW data of all the frames has not been completed (NO), the control unit 110 returns the process to S201, and the RAW data of the next frame is subjected to the above-described S201 and subsequent steps. The control process is performed.

  Returning to the processing of S201, the control unit 110 controls the plane conversion unit 102 to perform the plane conversion processing of the RAW data of the frame by continuous shooting as described above, and then proceeds to the control processing of S202. Since the operation mode at this point is set to the first operation mode as described above, the control unit 110 determines that the operation mode is not the second or third operation mode in S202, and advances the control process to S203. . In S203, the control unit 110 determines whether or not the used capacity of the memory 107 has become equal to or greater than the predetermined threshold Th. If it is determined that the used capacity is less than the predetermined threshold Th, the process proceeds to S204 and subsequent steps. .

  Then, the control unit 110 repeats the above-described series of processing for each frame by continuous shooting, and when it is determined in S213 that entropy encoding has been completed for RAW data of all frames by continuous shooting (YES). , The process proceeds to S214. In S214, the control unit 110 sets the operation mode of the imaging apparatus 100 to the initial operation mode, and then ends the continuous shooting process of the flowchart in FIG.

  On the other hand, the control unit 110 repeats the series of processes described above for each frame obtained by continuous shooting, and if it is determined in S203 that the used capacity of the memory 107 has become equal to or greater than the predetermined threshold Th, the control process proceeds to S205. .

  In S205, the control unit 110 sets the operation mode to the second operation mode, and then proceeds to the control process in S206. In S206, the control unit 110 controls the frequency conversion unit 103 to perform the frequency conversion process, and then proceeds to the control process of S210. In S206, an example is given in which subband decomposition (frequency decomposition) is performed three times as frequency conversion processing, but the number of subband decomposition may be two or more. In the present embodiment, as described above, the second operation mode is an operation mode in which the subband data after the frequency conversion processing of the plane data is entropy encoded. In the case of the second operation mode, the control unit 110 uses the entropy encoding unit 105 to convert the subband data (that is, the subband data not through the quantization unit 104) after the frequency conversion of the plane data by the frequency conversion unit 103. The MUX 111 is controlled so as to be input to. In S210, the encoded data generated by the entropy encoding unit 105 is stored in the memory 107 via the memory I / F unit 106. In the second operation mode, the memory 107 stores encoded data obtained by entropy-encoding subband data, that is, encoded data obtained by encoding transform coefficient data for each subband. In the case of the second operation mode, information on the generated code amount for each subband, which is generated when the transform coefficient data for each subband is encoded, is obtained.

  As described above, after the control process of S210 is performed in the second operation mode, the control unit 110 proceeds to the control process of S211. Since the second operation mode is set at this time, the control unit 110 determines that the operation mode is not the first operation mode in S211 and proceeds to the control process in S212.

  When the process proceeds to S <b> 212, the control unit 110 causes the storage element in the quantization unit 104 to store information on the generated code amount for each subband obtained by encoding in the second operation mode. After S212, the control unit 110 proceeds to the control process of S213. If the control unit 110 determines in S213 that the encoding process for the RAW data of all frames has not been completed, the control unit 110 returns the process to S201, and the control process after S201 described above for the RAW data of the next frame. I do.

  At this time, the control unit 110 performs the plane conversion process of the RAW data of the frame by continuous shooting in S201, and then proceeds to the control process of S202. Since the operation mode at this point is set to the second operation mode as described above, the control unit 110 determines that the operation mode is the second operation mode in S202, and advances the control process to S207.

  When the process proceeds to S207, the control unit 110 sets the operation mode to the third operation mode, and then proceeds to the control process of S208. In S208, the control unit 110 controls the frequency conversion unit 103 to perform the frequency conversion process, and then proceeds to the control process in S209. In S208, as in S206, an example is given in which subband decomposition (frequency decomposition) is performed three times as frequency conversion processing. However, the number of subband decomposition may be two or more. In S209, the control unit 110 controls the quantization unit 104 to perform the quantization process, and then proceeds to the control process in S210. At this time, the quantization unit 104 performs quantization processing with reference to the information of the generated code amount for each subband stored in the storage element in S212 when the second operation mode is set. That is, the quantization step value for each subband is determined with reference to the generated code amount of the same subband one frame before, and the quantization process is performed using the quantization step value.

  As described above, after the control process of S210 is performed in the third operation mode, the control unit 110 proceeds to the control process of S211. Since the third operation mode is set at this time, the control unit 110 determines that the operation mode is not the first operation mode in S211 and proceeds to the control process in S212.

  When the process proceeds to S212, the control unit 110 causes the storage element in the quantization unit 104 to store information on the generated code amount for each subband obtained by encoding in the third operation mode. After S212, the control unit 110 proceeds to the control process of S213. If the control unit 110 determines in S213 that the encoding process for the RAW data of all frames has not been completed, the control unit 110 returns the process to S201, and the control process after S201 described above for the RAW data of the next frame. I do.

  The control unit 110 proceeds to the control process of S202 after performing the plane conversion process of the RAW data of the frame by continuous shooting in S201. Since the operation mode at this point is set to the third operation mode as described above, the control unit 110 determines that the operation mode is the third operation mode in S202, and advances the control process to S207.

  When the process proceeds to S207, the control unit 110 has already set the operation mode to the third operation mode, and thus proceeds to the control process of the next S208 while maintaining the third operation mode. In S208, the control unit 110 controls the frequency conversion unit 103 to perform the frequency conversion process, and then proceeds to the control process in S209. In S209, the control unit 110 controls the quantization unit 104 to perform the quantization process, and then proceeds to the control process in S210. At this time, the quantization unit 104 refers to the information of the generated code amount for each subband stored in the storage element in S212 when the previous frame was encoded, and the quantization step value for each subband. And a quantization process for each subband using the quantization step value is performed.

  The control unit 110 controls each unit so that the subsequent frames are encoded in the third operation mode. Then, in the third operation mode, when entropy encoding is completed for RAW data of all frames by continuous shooting, the control unit 110 ends the processing after setting the operation mode of the imaging device 100 to the initial operation mode. To do.

  Although omitted in the flowchart of FIG. 6, when the use capacity of the memory 107 becomes less than the threshold Th during encoding in the third operation mode, the first operation mode is performed. Referring back to FIG. 6, the processing from S201 onward in the flowchart of FIG. 6 may be performed.

  7, the processing of the flowchart of FIG. 6 described above is performed in the imaging apparatus 100 of the present embodiment, and the encoded data of the frames continuously shot is temporarily stored in the memory 107 and then recorded on the recording medium 109. It is a figure which shows the example of a transition of the used capacity of the memory 107 at the time of doing. In the example of FIG. 7 as well, the code amount per unit time of the encoded data that is entropy-coded and stored in the memory 107 is the unit time that is transferred from the memory 107 to the recording medium 109, as in the example of FIG. It is assumed that the amount of data transferred per hit is exceeded.

  The horizontal axis of the graph 700 in FIG. 7 indicates time, the vertical axis indicates the capacity of the memory 107, and the broken line 702 indicates the transition of the used capacity of the memory 107. In addition, each scale on the horizontal axis of the graph 700 in FIG. 7 indicates a time during which one frame is shot by continuous shooting and the data of the frame is encoded and recorded in the memory 107. In addition, a time point T1 in FIG. 7 indicates a timing at which the encoded data of the first frame that is continuously shot is stored in the memory 107. Similarly, time points T2 to T29 indicate timings at which the encoded data of the second to 29th frames are stored in the memory 107. The time point T0 represents the timing of the start of photographing. From the time point T0 to the time point T2, the memory usage indicated by the broken line 702 gradually increases, but the memory usage capacity decreases at the time point T2. This indicates that the data 107 from the memory 107 to the recording medium 109 has been completed, so that the memory 107 has a capacity for the first encoded data. In the example of FIG. 7 as well, as in the example of FIG. 5 described above, the data transfer from the memory 107 to the recording medium 109 actually starts at time T 1 when the first frame is stored in the memory 107. Further, in the example of FIG. 7, for easy understanding, a broken line 702 is drawn with an image in which the memory usage is empty for the first encoded data at the timing of data transfer completion at time T2. Yes.

  A time chart 701 in FIG. 7 represents the timing of data input to the entropy encoding unit 105. In the case of the example in FIG. 7, since the capacity of the memory 107 is sufficient immediately after the start of continuous shooting, the plane data of each frame by continuous shooting is sent to the entropy encoding unit 105 in the first operation mode described above. It is input and entropy coding is performed. Therefore, the input data IN0 and IN1 in FIG. 7 are plane data from the plane conversion unit 102, and are data that has not been subjected to frequency conversion processing and quantization processing.

  On the other hand, a time point Tx in the graph 700 in FIG. 7 represents a timing at which the used capacity of the memory 107 indicated by the broken line 702 becomes equal to or higher than a predetermined threshold Th. In the present embodiment, when the used capacity of the memory 107 becomes equal to or greater than the predetermined threshold Th, the second operation mode is set between time T12 and T13 corresponding to at least one frame after time Tx. . Thereby, from time T12 to T13, the subband data after the frequency conversion processing by the frequency conversion unit 103 is used as input data IN12 to the entropy encoding unit 105. Accordingly, at this time, the entropy encoding unit 105 entropy-encodes the transform coefficient data for each subband, and the encoded data for each subband is stored in the memory 107 via the memory I / F unit 106. . Also, information on the generated code amount for each subband generated by encoding by the entropy encoding unit 105 is sent to the quantization unit 104 and stored in an internal storage element.

  In the present embodiment, the third operation mode is set at time T13 of the next frame after time T12. As a result, in the frame at time T13, the quantized data obtained by further quantizing the subband data after the frequency conversion processing by the frequency conversion unit 103 by the quantization unit 104 is input data IN13 to the entropy encoding unit 105. It is made. At this time, information on the generated code amount for each subband obtained by encoding for each subband performed between time T12 and T13 is stored in the storage element inside the quantization unit 104. Yes. Therefore, the quantization unit 104 determines the quantization step value for each subband with reference to the generated code amount information for each subband obtained one frame before for the frame at time T13. Can do. The entropy encoding unit 105 encodes the quantized data (IN13) after the quantization process is performed using the quantization step value for each subband in the quantization unit 104. The encoded data for each subband is stored in the memory 107 via the memory I / F unit 106. Further, for the frame at time T13, information on the generated code amount for each subband generated by encoding by the entropy encoding unit 105 is fed back to the quantization unit 104 and stored in an internal storage element. The generated code amount information may be held by the control unit 110. In this case, the control unit 110 may determine the quantization step value of the quantization unit 104 based on the generated code amount information. .

  Hereinafter, for each frame after time T13, similarly, the entropy encoding is performed after the quantization step value is determined and quantized based on the generated code amount for each subband one frame before. become. For example, if continuous shooting is completed at time T20, data transfer from the memory 107 to the recording medium 109 is performed until time T29 when all the encoded data of each frame remaining in the memory 107 is read and lost. Is called.

  As described above, the image capturing apparatus 100 according to the present embodiment can record a high-quality image by performing lossless encoding as much as possible. The imaging apparatus 100 according to the present embodiment avoids memory pressure by switching to lossy coding when the use capacity of the memory 107 exceeds the threshold Th and it is difficult to continue lossless coding. Image quality deterioration due to switching of the encoding system can be reduced. In addition, when switching from lossless encoding to lossy encoding, subband data encoding processing by frequency conversion processing is performed on an image of at least one frame, and the generated code for each subband at that time Quantity information is stored. In the next frame, the subband data is quantized by the frequency conversion process. When the quantization is performed, the quantization step is performed with reference to the generated code amount at the time of the previous frame encoding. The value is determined. As a result, in the quantization process after the coding method is switched, quantization using the optimum quantization step value referring to the generated code amount for each subband becomes possible, and the coding method is switched. Image quality deterioration can be reduced.

<Second Embodiment>
FIG. 8 is a diagram illustrating a schematic configuration of an imaging apparatus 800 according to the second embodiment.
The imaging apparatus 800 of this embodiment includes an imaging unit 801, a plane conversion unit 802, a frequency conversion unit 803, a quantization unit 804, an entropy encoding unit 805, a memory I / F unit 806, a memory 807, a recording processing unit 808, and a recording. A medium 809, a control unit 810, and a MUX 811. In the imaging apparatus 800 of FIG. 8, the description of the same configuration as the imaging apparatus 100 of FIG.

  In the image pickup apparatus 800 of the second embodiment shown in FIG. 8, the operations of the recording processing unit 808 and the control unit 810 are different from those of the recording processing unit 108 and the control unit 110 of the image pickup apparatus 100 of FIG. Details of operations of the recording processing unit 808 and the control unit 810 will be described later. Hereinafter, only the portions of the imaging apparatus 800 according to the second embodiment that are different from the imaging apparatus 100 according to the first embodiment will be described.

  For example, when a large number of accesses to the memory 807 are generated by various processing units not shown in FIG. 8 or external devices connected to the imaging device 100, the data reading speed from the memory 807 is high. Decreases. Here, for example, when the data reading speed of the memory 807 decreases while the encoded data of each frame is written to the memory 807 by continuous shooting or the like, the capacity of the memory 807 is compressed. become.

  FIG. 9A shows the memory 807 when the reading speed of the memory 807 decreases when the encoded data of each frame by continuous shooting or the like is temporarily stored in the memory 807 and then recorded on the recording medium 809. It is a figure which shows the example of a transition of the used capacity. In the example of FIG. 9A, the code amount per unit time of the encoded data encoded and stored in the memory 807 exceeds the transfer data amount per unit time transferred from the memory 807 to the recording medium 809. Suppose that In the transition example of FIG. 9A, the memory I / F unit 806 is set so that the writing priority is higher than the reading priority in writing and reading the encoded data to and from the memory 807. .

  The horizontal axis of the graph 900 in FIG. 9A represents time, the vertical axis represents the capacity of the memory 807, and the broken line 902 represents the transition of the used capacity of the memory 807. Each scale on the horizontal axis of the graph 900 in FIG. 9A indicates a time during which one frame is shot by continuous shooting and the data of the frame is encoded and stored in the memory 807. In FIG. 9A, time points T0, T1,..., T12,... Indicate the timing at which the encoded data of each frame after the imaging start timing is stored in the memory 807, as in the example of FIG. Yes. In the case of the transition example in FIG. 9A, since the reading speed of the memory 807 is lower than that in the above-described example in FIG. 7, the memory usage gradually increases between time points T0 and T4. In the case of the transition example in FIG. 9A, since the process of transferring the encoded data of the first frame from the memory 807 to the recording medium 809 is completed at time T4, the code of one frame is stored in the memory 807. This indicates that there is enough free space for data. In the example of FIG. 9A as well, the encoded data of the first frame is actually stored in the memory 807 in the data transfer from the memory 807 to the recording medium 809 as in the example of FIG. It starts from time T1.

  A time chart 901 in FIG. 9A represents the timing of data input to the entropy encoding unit 805 as in the example of FIG. Since the first operation mode is set in the same manner as described above during a period in which the memory 807 has a sufficient capacity after the start of continuous shooting, the input data IN0 and IN1 in FIG. This is plain data that has not been processed.

  A time point Tx in the graph 900 of FIG. 9A represents a timing when the used capacity of the memory 807 indicated by the broken line 902 becomes equal to or higher than a predetermined threshold value Th. Also in the second embodiment, as in the first embodiment, when the used capacity of the memory 807 is equal to or greater than the predetermined threshold Th, the second operation mode is set between the time points T8 and T9 after the time point Tx. Is set. As a result, between time points T8 and T9, the subband data after the frequency conversion processing by the frequency conversion unit 803 becomes the input data IN8 to the entropy encoding unit 805. At this time, the entropy encoding unit 805 encodes the transform coefficient data for each subband, and the encoded data for each subband is stored in the memory 807 via the memory I / F unit 806. Also, information on the generated code amount for each subband generated by encoding by the entropy encoding unit 805 is stored in a storage element inside the quantization unit 804.

  Also in the second embodiment, as in the first embodiment, the third operation mode is set at the time T9 of the next frame after the time T8. Thereby, in the frame at time T9, the quantized data obtained by further quantizing the subband data after the frequency conversion process is used as the input data IN9 to the entropy coding unit 805. At this time, the quantization unit 804 refers to the generated code amount information for each subband obtained in the second operation mode between the time points T8 and T9 for the frame at the time point T9. A quantization step value for each subband is determined. Then, the quantized data (the quantized data of IN9) after being quantized using the quantization step value for each subband in the quantizing unit 804 is encoded by the entropy encoding unit 805, and the sub data The encoded data for each band is stored in the memory 807. Thereafter, the third operation mode is continued.

  Here, in the case of the transition example in FIG. 9A, even when the encoding in the third operation mode is performed and the generated code amount is suppressed, the reading speed of the memory 807 is reduced. There is a possibility that the used capacity of the memory 807 reaches the maximum capacity. In this case, for example, by controlling the quantization step value of the quantization unit 804 to a value that greatly reduces the generated code amount, the memory 807 can be controlled so as not to reach the maximum capacity. Conceivable. However, if quantization is performed with a quantization step value that significantly reduces the amount of generated code, the image quality deteriorates rapidly.

  Therefore, in the second embodiment, by controlling the predetermined threshold Th for the memory usage capacity according to the data transfer rate of the memory 807, the usage capacity of the memory 807 is prevented from reaching the maximum capacity, It is possible to prevent sudden deterioration of image quality.

  In the imaging apparatus 800 of the second embodiment, the recording processing unit 808 obtains a data transfer rate when data is read from the memory 807 and recorded on the recording medium 809. For example, the recording processing unit 808 uses the difference between the encoded data read start time of the memory 807 by the memory I / F unit 806 and the encoded data read completion time of the memory 807 per unit time when reading data from the memory 807. Estimate the transfer amount. Further, for example, the recording processing unit 808 may estimate the transfer amount per unit time when data is read from the memory 807 based on the ratio between the data write amount and the read amount of the memory 807 by the memory I / F unit 806. . Such information on the transfer amount per unit time is sent to the control unit 810 as information indicating a data transfer rate when the data is read from the memory 807 and recorded on the recording medium 809. The control unit 810 changes the predetermined threshold Th based on the data transfer rate information received from the recording processing unit 808. Specifically, the threshold value Th for the memory usage capacity is changed so as to decrease as the data transfer rate becomes slower.

  Note that the threshold value can be changed according to the data transfer rate by using a method of referring to a correspondence table between a plurality of data transfer rates and a plurality of threshold values as an example. For example, the control unit 810 holds a correspondence table of a plurality of different data transfer rates and a plurality of threshold values in an internal storage element or the like, and supports a threshold corresponding to the data transfer rate supplied from the recording processing unit 808. The threshold is changed by reading from the table. Of course, the method of referring to the correspondence table is an example, and is not limited to this example. A threshold value corresponding to the data transfer rate may be obtained by calculation.

  FIG. 9B is a diagram showing an example in which the predetermined threshold Th described in FIG. 9A is changed to a low value like the threshold Th1 because the data transfer rate of the memory 807 is decreased. It is. Since the graph 920, the broken line 922, and the time chart 921 in FIG. 9B are the same as the graph 900, the broken line 902, and the time chart 901 in FIG. 9A described above, detailed description thereof will be given. Is omitted.

  In the case of the example in FIG. 9B, the threshold Th1 is changed to a value lower than the threshold Th in FIG. 9A. Therefore, the time Tx when the used capacity of the memory 807 exceeds the threshold Th1 is shown in FIG. It is before the example of a) on the time axis. In the example of FIG. 9B, the used capacity of the memory 807 becomes equal to or greater than the predetermined threshold Th1 between the time points T6 and T7. Therefore, switching from the first operation mode to the second operation mode is performed. Is performed at time T7. Therefore, between time points T7 and T8, the subband data after the frequency conversion process is used as input data IN7 to the entropy encoding unit 805.

  In the example of FIG. 9B, the switching from the second operation mode to the third operation mode is performed at time T8 of the next frame after time T7. Thereby, in the frame at time T8, frequency conversion processing and quantization processing are performed, and the quantized data becomes input data IN8 to the entropy coding unit 805. Thereafter, the third operation mode is continued.

  In the case of the example in FIG. 9B, since the threshold value Th1 is lower than the threshold value Th in FIG. 9A, the transition from the first operation mode to the second operation mode, and further the third The transition to the operation mode is performed when the used capacity of the memory 807 is small (the free capacity is large). That is, since the switching to the third operation mode is performed at an early stage when sufficient free space remains in the memory 807, even if the reading speed of the memory 807 is reduced, the used capacity of the memory 807 is the maximum. The possibility of reaching capacity is reduced. Therefore, in the third operation mode, the quantization step value of the quantization unit 804 does not have to be a value that greatly reduces the generated code amount, and as a result, the image quality deteriorates rapidly. Can be prevented.

  As described above, in the second embodiment, even when there are many accesses to the memory 807 and the data transfer speed of the memory 807 is slow, lossless encoding can be performed as much as possible to record a high-quality image. It is said. Similarly to the first embodiment, the imaging apparatus 800 according to the second embodiment avoids memory pressure by switching to lossy encoding when the used capacity of the memory 807 exceeds a threshold value. However, it is possible to reduce image quality deterioration due to switching of the encoding method. In addition, when switching the encoding method, as described above, subband data is encoded by frequency conversion processing on an image of at least one frame, and in the next frame, the previous frame is encoded. The quantization step value is determined with reference to the generated code amount. Thereby, image quality deterioration at the time of switching of an encoding system is reduced. Furthermore, the imaging apparatus 800 according to the second embodiment can change the threshold Th according to the data transfer speed. When the data transfer speed of the memory 807 is slow, the threshold Th for the memory usage capacity is set as follows. Change to a lower threshold Th1. Therefore, according to the second embodiment, the used capacity of the memory 807 can be prevented from reaching the maximum capacity, and the image quality is rapidly deteriorated after the operation mode is changed to the third operation mode. Can be prevented.

  In the second embodiment, the threshold value is changed according to the data transfer amount (transfer speed) per unit time of the memory 807. However, the control unit 810 performs, for example, the data transfer processing performance in the recording medium 809. The threshold value may be controlled based on (recording processing capability). In this example, the effect is exhibited not when the amount of access to the memory 807 is large but when the transfer speed of the recording processing unit 808 is maintained high. For example, the control unit 810 performs control so that the threshold is set high when the recording medium 809 has high-speed transfer processing capability, and is set low when the recording medium 809 has low-speed transfer processing capability. . Further, the control unit 810 may control the threshold based on both the data transfer speed of the memory 807 and the data transfer processing performance of the recording medium 809. For example, the control unit 810 may be configured to increase the threshold if both the data transfer speed of the memory 807 and the data transfer processing performance of the recording medium 809 are high, and to decrease the threshold if either of them is low. In addition, for example, the control unit 810 can set the threshold value if the memory 807 can be prevented from being compressed by the other even if either the data transfer speed of the memory 807 or the data transfer processing performance of the recording medium 809 is low. You may make it not control low.

  In the example of FIG. 8, the recording processing unit 808 is not controlled by the control unit 810, but the recording processing unit 808 may be controlled by the control unit 810. In this case, the control unit 810 manages the data transfer rate and the reading rate per unit time of the recording processing unit 808, and controls the threshold as described above. The information on the generated code amount may be held by the control unit 810, and the control unit 810 may determine the quantization step value of the quantization unit 804 based on the generated code amount.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  100,800 imaging device, 101,801 imaging unit, 102,802 plane conversion unit, 103,803 frequency conversion unit, 104,804 quantization unit, 105,805 entropy coding unit, 106,806 memory I / F unit, 107,807 Memory, 108,808 Recording processing unit, 109,809 Recording medium, 110,810 Control unit, 111,811 MUX

Claims (12)

Frequency conversion means for performing frequency conversion on the frame image to generate first coefficient data;
Quantization means for quantizing the first coefficient data based on a quantization parameter to generate second coefficient data;
Encoding means for encoding input data to generate encoded data;
Control means for controlling the encoding,
The control means includes
When storing a plurality of encoded data generated from a plurality of frame images in a memory, if the capacity occupied by the encoded data in the memory is less than a predetermined threshold, the data of the frame image is encoded. ,
When the capacity occupied by the encoded data in the memory exceeds a predetermined threshold value, the first coefficient data for at least one frame is encoded, and then the second coefficient data is encoded. An image pickup apparatus that is controlled as described above.   When the capacity occupied by the encoded data in the memory exceeds a predetermined threshold, the control means encodes the first coefficient data for at least one frame and then generates one frame before The imaging apparatus according to claim 1, wherein the second coefficient data that has been quantized is controlled to be encoded using a quantization parameter corresponding to a code amount. The frequency transform decomposes the frame image into a plurality of subbands to generate the first coefficient data for each of the plurality of subbands;
3. The quantization unit according to claim 2, wherein the quantization unit performs quantization for each of the plurality of subbands using a quantization parameter determined according to a generated code amount of a subband one frame before. Imaging device.   4. The control unit according to claim 1, wherein the control unit controls the predetermined threshold value based on a data transfer rate when the encoded data is read from the memory and transferred. The imaging device according to item.   5. The control unit according to claim 1, wherein the control unit controls the predetermined threshold according to a difference between a read start time and a read completion time of the encoded data of the frame for the memory. The imaging device described in 1.   4. The control unit according to claim 1, wherein the control unit controls the predetermined threshold according to a ratio between a write amount and a read amount of the encoded data per unit time with respect to the memory. The imaging device described in 1.   7. The control unit according to claim 1, wherein the control unit controls the predetermined threshold according to processing performance of a recording medium on which the encoded data read from the memory is transferred and recorded. The imaging apparatus of Claim 1. Plane conversion means for generating plane data obtained by converting RAW data of a frame image obtained by the imaging into a plane for each color component;
The frequency conversion means decomposes each plane data for each of the color components converted from the RAW data of the frame image into a plurality of subbands by the frequency conversion, and the first for each of the plurality of subbands. Generate coefficient data,
The quantization means quantizes the first coefficient data for each subband using a quantization parameter for each subband corresponding to a generated code amount by the encoding for each subband, and The imaging apparatus according to claim 1, wherein second coefficient data for each band is generated. A selection in which the frame image data, the first coefficient data, and the second coefficient data are input, and one of the input data is switched and selected and output to the encoding means Having means,
The control means includes
When the capacity occupied by the encoded data in the memory is less than a predetermined threshold, the selection means outputs the frame image data to the encoding means,
When the capacity occupied by the encoded data in the memory is equal to or greater than a predetermined threshold, after the selection means outputs the first coefficient data for at least one frame to the encoding means, 9. The image pickup apparatus according to claim 1, wherein the second coefficient data of one frame before is output to the encoding unit thereafter.   The imaging device according to any one of claims 1 to 9, wherein the frame image is each frame image of a still image captured by continuous shooting or each frame image of a moving image captured by a moving image. . A conversion step of performing frequency conversion on the frame image to generate first coefficient data;
A quantization step of quantizing the first coefficient data based on a quantization parameter to generate second coefficient data;
Encoding the input data to generate encoded data, and
In the encoding step,
When storing a plurality of encoded data generated from a plurality of frame images in a memory, if the capacity occupied by the encoded data in the memory is less than a predetermined threshold, the data of the frame image is encoded. ,
When the capacity occupied by the encoded data in the memory exceeds a predetermined threshold value, the first coefficient data for at least one frame is encoded, and then the second coefficient data is encoded. And a method of controlling the imaging apparatus.   The program for functioning the computer which an imaging device has as each means of the imaging device of any one of Claims 1 thru | or 10.

Images