[go: up one dir, main page]

WO2020111441A1 - Processeur et procédé de commande associé - Google Patents

Processeur et procédé de commande associé Download PDF

Info

Publication number
WO2020111441A1
WO2020111441A1 PCT/KR2019/009961 KR2019009961W WO2020111441A1 WO 2020111441 A1 WO2020111441 A1 WO 2020111441A1 KR 2019009961 W KR2019009961 W KR 2019009961W WO 2020111441 A1 WO2020111441 A1 WO 2020111441A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel data
data
binning
image sensor
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2019/009961
Other languages
English (en)
Korean (ko)
Inventor
류철규
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
LG Electronics Inc
Original Assignee
LG Electronics Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by LG Electronics Inc filed Critical LG Electronics Inc
Publication of WO2020111441A1 publication Critical patent/WO2020111441A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
    • H04N25/46—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by combining or binning pixels
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith

Definitions

  • the present invention relates to a processor and a control method thereof, and is connected to an image sensor and relates to a processor and a control method for improving the brightness of an original image using sensor binning.
  • the image sensor is a semiconductor that converts photons into electrons and displays them on a display or stores them in a storage device.
  • It is a light-receiving element that converts a light-receiving signal into an electrical signal, and converts the converted electrical signal into an analog signal to process the image signal. It is composed of ASIC part, and is classified into CCD, CMOS, and CIS (Contact Image Sensor).
  • CMOS image sensor In more detail, it is a CCD image sensor that moves electrons generated by light to the output unit using a gate pulse as it is, and converts electrons generated by light into voltages within each pixel and outputs them through various CMOS switches.
  • This is a CMOS image sensor.
  • the field of application of these image sensors is not only for household products such as digital cameras and mobile phones, but also for endoscopes used in hospitals and satellite telescopes orbiting the earth.
  • An embodiment of the present invention is to apply binning to original image data to generate binning data with improved brightness, fuse original image data and binning data, and generate a processor with improved brightness and a control method thereof.
  • the purpose is to provide.
  • the binning data and the pixel data are physically or temporally divided and received from the image sensor, and the processor generates fused pixel data by fusing the resolution information of the pixel data and the brightness information of the binning data. And it is an object to provide the control method.
  • the processor includes an interface unit 210 for receiving pixel data and binning data from an image sensor; And a control unit 220 that fuses the received pixel data and the received binning data to generate fused pixel data.
  • a control method of a processor includes receiving pixel data and binning data from an image sensor; And fusing the received pixel data and the received binning data to generate fused pixel data.
  • binning is applied to the original image data to generate binning data with improved brightness, and the original image data and binning data are fused to generate image data with enhanced brightness.
  • the file size of is the same, the brightness is improved, and the size of the image sensor can be kept the same, thereby improving user convenience.
  • binning data and pixel data are physically or temporally divided and received from an image sensor, and fusion pixel data is generated by fusing resolution information of pixel data and brightness information of binning data, , Since the size of the image sensor remains the same and the brightness of the image data is improved, user convenience can always be achieved.
  • FIG. 1 is a diagram illustrating a configuration diagram of an image sensor according to an embodiment of the present invention.
  • FIG. 2 is a flowchart illustrating a method of controlling an image sensor according to an embodiment of the present invention.
  • FIG. 3 is a diagram illustrating binning data generated by applying binning to pixel data and pixel data according to an embodiment of the present invention.
  • FIG. 4 is a diagram illustrating a system in which a control unit that fuses binning data and pixel data and another hardware configuration according to an embodiment of the present invention are combined.
  • FIG. 5 is a diagram illustrating fusion data generated by fusing pixel data and binning data according to an embodiment of the present invention.
  • FIG. 6 is a diagram illustrating a quad bayer to a normal bayer according to an embodiment of the present invention.
  • FIG. 7 is a diagram illustrating an embodiment of generating fused pixel data by fusing pixel data and binning data according to an embodiment of the present invention.
  • FIG. 8 is a diagram illustrating an embodiment in which a fusion chip generates binning data and a fusion chip generates fusion pixel data by fusing pixel data with binning data generated according to an embodiment of the present invention.
  • FIG. 9 is a diagram illustrating an embodiment of generating fusion pixel data by fusing pixel data with binning data generated by a fusion chip according to an embodiment of the present invention.
  • FIG. 10 is a view showing horizontal binning, vertical binning, and full binning according to an embodiment of the present invention.
  • FIG. 11 is a diagram illustrating generating binning data in an image sensor according to an embodiment of the present invention.
  • FIG. 12 is a diagram illustrating a configuration diagram of a processor according to an embodiment of the present invention.
  • FIG. 13 is a flowchart illustrating a method of controlling a processor according to an embodiment of the present invention.
  • FIG. 14 is a diagram illustrating a structural diagram of an image sensor system in which an image sensor and a processor are combined according to an embodiment of the present invention.
  • 15 is a diagram illustrating pixel data and binning data output from an image sensor according to an embodiment of the present invention.
  • 16 is a diagram illustrating fusion pixel data generated by fusing pixel data and binning data according to an embodiment of the present invention.
  • FIG. 17 is a diagram illustrating a quad bayer to a normal bayer according to an embodiment of the present invention.
  • FIG. 18 is a diagram illustrating that a processor receives pixel data and binning data output from an image sensor according to an embodiment of the present invention.
  • 19 is a diagram illustrating a fusion image generated by fusing an original image and a binning image according to an embodiment of the present invention.
  • 20 is a diagram illustrating generating fused pixel data inside a processor.
  • 21 is a diagram specifically showing generation of a fusion image inside a processor.
  • FIG. 1 is a diagram illustrating a configuration diagram of an image sensor according to an embodiment of the present invention.
  • the image sensor 100 includes a pixel unit 110, a row selection unit 120, an image synthesis unit 130, an analog-to-digital conversion unit 140, a control unit 150, and an amplifier unit 160. , Memory 170.
  • the pixel unit 110 converts the optical signal of the subject into an electrical signal, and outputs pixel data (original image) arranged in a matrix of a plurality of rows and columns.
  • the pixel unit 110 is arranged in a Bayer pattern consisting of a first line in which red pixels and green pixels are alternately arranged, and a second line in which blue pixels and green pixels are alternately arranged.
  • the row selector 120 receives a row address and generates a selection signal for selecting at least two rows from the plurality of rows.
  • the image synthesizing unit 130 generates binning data by applying binning to pixel data (original image data).
  • the analog-to-digital conversion unit 140 converts the pixel data selected by the row selection unit 120 and the generated binning data into a digital image signal and outputs the converted digital image signal.
  • the analog-to-digital conversion unit 140 physically separates pixel data and binning data and outputs them.
  • the analog-to-digital converter 140 outputs the pixel data and binning data in time.
  • the controller 150 fuses the output pixel data and the output binning data to generate fused pixel data.
  • the controller 150 may be a processor or an application processor.
  • the control unit 150 is designed as a fusion chip and may be mounted inside the processor.
  • the amplifier unit 160 amplifies pixel data.
  • the memory 170 stores pixel data.
  • the memory 170 is a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), card type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read (EPMROM) It may include a storage medium of at least one type of -only memory (PROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk.
  • PROM programmable read-only memory
  • FIG. 2 is a flowchart illustrating a method of controlling an image sensor according to an embodiment of the present invention.
  • the pixel unit 110 converts an optical signal of an object into an electrical signal, and outputs pixel data arranged in a matrix of a plurality of rows and columns (S210).
  • the pixel data corresponds to the original image data.
  • the row selector 120 receives a row address and generates a selection signal for selecting at least two rows from the plurality of rows (S220).
  • the image synthesis unit 130 applies binning to the pixel data to generate binning data (S230).
  • the analog-to-digital conversion unit 140 converts the pixel data selected by the row selection unit 120 and the generated binning data into a digital image signal and outputs it (S240).
  • the analog-to-digital conversion unit 140 physically separates pixel data and binning data and outputs them.
  • the analog-to-digital converter 140 outputs the pixel data and binning data in time.
  • the controller 150 fuses the output pixel data and the output binning data to generate fused pixel data (S250).
  • the controller 150 may be a processor or an application processor.
  • FIG. 3 is a diagram illustrating binning data generated by applying binning to pixel data and pixel data according to an embodiment of the present invention.
  • outputs of the image sensor 100 are pixel data 10 and binning data 20.
  • the image synthesis unit 130 generates binning data 20 by applying binning to the pixel data 10.
  • the pixel data 10 means original image data.
  • the exposure is the same as the pixel data 10
  • the brightness is the binning data 20 different from the pixel data.
  • the original image and the binning image are the same.
  • the binning image may be four times brighter than the original image.
  • the size of the original image may be 20 megapixels
  • the size of the binning image may be 5 megapixels.
  • the image synthesizing unit 130 may perform noise reduction of pixel data with a filter.
  • the original image is a dark image using 1/4 exposure compared to the binning image created. That is, the original image has more noise components compared to the binning image. This noise component is a factor that degrades the resolution when the controller 150 fuses.
  • the image synthesizing unit 130 performs noise reduction of pixel data with a filter.
  • the filter includes a Median Filter.
  • the median filter refers to a filter that obtains an intermediate value of a pixel density in an area around a certain pixel and processes it with a desired pixel density.
  • FIG. 4 is a diagram illustrating a system in which a control unit that fuses binning data and pixel data and another hardware configuration according to an embodiment of the present invention are combined.
  • the system includes an image sensor 100, a control unit 150, and a processor 200.
  • the image sensor 100 transmits the pixel data 10 and binning data 20 to the control unit 150.
  • the control unit 150 may be designed as a fusion chip.
  • the control unit 150 may be a processor or an application processor.
  • the control unit 150 is installed spaced apart from the image sensor 100 by a predetermined distance.
  • the control unit 150 is installed spaced apart from the image sensor 100 by a predetermined distance.
  • the image sensor 100, the control unit 150, and the processor 200 are connected to Miffy.
  • MIPI stands for MIPI, a mobile industry processor interface, and was created to standardize an interface for connecting a mobile processor 200 and peripheral devices.
  • Miffy's transmission speed is 9 Gps.
  • the ISP data transmission rate is 750 MHZ.
  • the output of the image sensor 100 becomes pixel data.
  • the pixel data is 16 megabytes, 30 frames are transmitted per second, and the coefficient is set to 1.2 in consideration of dummy data, and the transmission rate is 576 MHz. Therefore, if the transmission rate is 576 MHz, the processor 200 can process it.
  • the output of the image sensor 100 becomes pixel data and binning data.
  • the pixel data is 16 megabytes and the binning data is 4, 30 frames are transmitted per second, and the coefficient is set to 1.2 in consideration of dummy data, so that the transmission rate is 720 MHz. Therefore, if the transmission rate is 720 MHz, the processor 200 can process it.
  • the data amount of the pixel data and the data amount of the generated fused pixel data are the same. That is, binning data and pixel data generated by the image synthesis unit 130 are transmitted to the control unit 150.
  • the controller 150 fuses binning data and pixel data to generate fused pixel data, and the data amount of the fused pixel data is the same as the data amount of the pixel data.
  • FIG. 5 is a diagram illustrating generating fused pixel data by fusing pixel data and binning data according to an embodiment of the present invention.
  • FIG. 5(a) is a diagram showing fusion of pixel data and binning data to generate fusion pixel data
  • FIG. 5(b) is a diagram showing a mathematical algorithm when generating fusion pixel data.
  • the controller 150 fuses the pixel data 10 and binning data 20 to generate fused pixel data 30. Specifically, the controller 150 fuses the resolution information of the pixel data and the brightness information of the binning data to generate fused pixel data.
  • the pixel data is quad-bayer pixel data having the same attribute, and the same attribute means having the same color filter.
  • the color filter means any one of Gr, Red, and Blue.
  • adjacent pixel data of the first pixel data means second pixel data, third pixel data, and fourth pixel data.
  • the adjacent pixel data of the first pixel data Gr1 means the second pixel data Gr2, the third pixel data Gr3, and the fourth pixel data Gr4.
  • the binning data 20 includes Gr', R', B', and Gb'.
  • the image synthesizing unit 130 generates binning data Gr' by applying binning to Gr1, Gr2, Gr3, and Gr4.
  • the image synthesizing unit 130 generates binning data R'by applying binning to R1, R2, R3, and R4.
  • the image synthesis unit 130 generates binning data B'by applying binning to B1, B2, B3, and B4.
  • the image synthesizing unit 130 generates binning data Gb' by applying binning to Gr1, Gr2, Gr3, and Gr4.
  • the binning data 20 Gr', R', B', and Gb' mean brightness information.
  • applying binning means applying full binning, and the scope of the present invention is not limited to full binning, but includes both horizontal binning and vertical binning.
  • the controller 150 fuses the resolution information of the pixel data and the brightness information of the binning data to generate fused pixel data.
  • the resolution information of the first pixel data means a ratio obtained by dividing the first pixel data by an average value of the first pixel data, the second pixel data, the third pixel data, and the fourth pixel data.
  • the resolution information of Gr1 is Gr1/((Gr1 + Gr2 + Gr3 + Gr 4)/4).
  • the resolution information of Gr2 is Gr2/((Gr1 + Gr2 + Gr3 + Gr4)/4).
  • the resolution information of Gr2 is Gr2/((Gr1 + Gr2 + Gr3 + Gr4)/4).
  • the resolution information of Gr4 is Gr4/((Gr1 + Gr2 + Gr3 + Gr4)/4).
  • the resolution information of the first pixel data means a ratio between the first pixel data and the adjacent pixel data, and there is no unit.
  • the ratio between the pixel data and the adjacent pixel data is close to 1, but in the case of an edge, the ratio between the pixel data and the adjacent pixel data is greater than 1.
  • the controller 150 fuses the resolution information of the pixel data and the brightness information of the binning data to generate fused pixel data.
  • the meaning of fusion means that the resolution information and the brightness information are multiplied.
  • Fused pixel data is generated as a product of brightness information of binning data and resolution information of pixel data.
  • the first fused pixel data is Gr1
  • the first fused pixel data Gr1" Gr1' x Gr1/((Gr1 + Gr2 + Gr3 + Gr 4)/4).
  • the brightness of the fused pixel data is characterized by being at least twice brighter than the brightness of the first pixel data.
  • Fused pixel data is generated as a product of brightness information of binning data and resolution information of pixel data.
  • the brightness information of binning data becomes 4 times the brightness information of the original image.
  • the ratio of the original image it is possible to obtain an original image with 4 times the brightness.
  • the image quality is mainly affected by the amount of light collected in each pixel constituting the image sensor.
  • the chip size of the CMOS image sensor is getting smaller and the number of pixels is increasing, so the size of the pixel is getting smaller.
  • the back-illuminated sensor (BSI sensor) has moved the light-receiving unit (the portion that receives light) to the uppermost portion of the sensor to increase the light-receiving rate.
  • the size of the pixel continues to decrease, there is a problem that there is a limit.
  • binning is applied to the original image data to generate binning data with improved brightness, and original image data and binning data are fused to generate image data with improved brightness, thereby increasing the size of the image sensor Without it, the brightness of the image sensor can be improved.
  • the data amount of the pixel data and the data amount of the generated fused pixel data are the same. Therefore, according to the present invention, it is possible to improve the brightness of the image sensor while keeping the amount of data constant and not increasing the amount of data.
  • FIG. 6 is a diagram illustrating a quad bayer to a normal bayer according to an embodiment of the present invention.
  • control unit 150 converts the quad bayer 30 including the generated fusion pixel data into a normal bayer 40.
  • the control unit 150 converts the quad bayer 30 into a normal bayer 40, so that the processor 200 can recognize pixel data.
  • FIG. 7 is a diagram illustrating an embodiment of generating fused pixel data by fusing pixel data and binning data according to an embodiment of the present invention.
  • pixel data corresponds to the original image 710 and binning data corresponds to the binning image 720.
  • the fused pixel data corresponds to the fused image 730.
  • the controller 150 fuses the resolution information of the pixel data and the brightness information of the binning data to generate fused pixel data.
  • the meaning of fusion means that the resolution information and the brightness information are multiplied.
  • the brightness of the fused pixel data is characterized in that it is at least two times brighter than the brightness of the first pixel data.
  • the brightness information of the binning data becomes data of 4 times the brightness of the original image.
  • the ratio of the original image it is possible to obtain a fused image 730 with 4 times the brightness.
  • the present invention since the brightness of the generated fused image is improved, the calling is reduced, and the pixel size is not increased, the manufacturing cost of the image sensor is reduced.
  • FIG. 8 is a diagram illustrating an embodiment in which a fusion chip generates binning data and a fusion chip generates fusion pixel data by fusing pixel data with binning data generated according to an embodiment of the present invention.
  • the system includes an image sensor 100, a control unit 150, and a processor 200.
  • the image sensor 100 outputs pixel data 10.
  • the control unit 150 may be a fusion chip.
  • the control unit 150 generates binning data 20 based on the pixel data 10.
  • the controller 150 fuses the pixel data 10 and the binning data 20 to generate fused pixel data 30.
  • the controller 150 transmits the fused pixel data to the processor 200.
  • control unit 150 may be designed as a fusion chip and embedded in the processor 200.
  • control unit 150 may be a processor 200 or an application processor.
  • FIG. 9 is a diagram illustrating an embodiment of generating fusion pixel data by fusing pixel data with binning data generated by a fusion chip according to an embodiment of the present invention.
  • the system includes an image sensor 100, a control unit 150, and a processor 200.
  • the image sensor 100 outputs the pixel data 10 and binning data 20.
  • the control unit 150 may be a fusion chip.
  • the control unit 150 may be designed as a fusion chip and embedded in the processor 200.
  • the control unit 150 may be a processor 200 or an application processor.
  • the controller 150 fuses the pixel data 10 and binning data 20 to generate fused pixel data 30.
  • the controller 150 transmits the fused pixel data to the processor 200.
  • FIG. 10 is a view showing horizontal binning, vertical binning, and full binning according to an embodiment of the present invention.
  • FIG. 10(a) is a diagram showing horizontal binning
  • FIG. 10(b) is a diagram showing vertical binning
  • FIG. 10(c) is a diagram showing overall binning.
  • horizontal binning adds electric charges from adjacent pixels in a line and is applied as one pixel.
  • vertical binning adds charges from adjacent cells in a vertical direction in a plurality of lines and is applied as one pixel.
  • the whole binning is added in close and grouped charges in a two-dimensional plane, and is applied as one pixel.
  • binning adds the charges from adjacent pixels of the CCD sensor to increase the camera's sensitivity to light.
  • horizontal binning the charge from adjacent pixels in each line of the sensor is summed.
  • vertical binning the charges of adjacent pixels in the vertical direction in two rows are summed in the sensor.
  • Total binning is a combination of horizontal binning and vertical binning in which two adjacent pixels are summed in a two-dimensional plane.
  • the sensitivity of the image sensor may be increased by a factor of two.
  • the sensitivity of the image sensor can be increased up to 4 times.
  • the present invention has been mainly described for applying the whole binning to pixel data.
  • FIG. 11 is a diagram illustrating generating binning data in an image sensor according to an embodiment of the present invention.
  • the image sensor 100 includes a pixel unit 110, a row selection unit 120, an image synthesis unit 130, an analog-to-digital conversion unit 140, a control unit 150, and an amplifier unit 160. , Memory 170.
  • the pixel unit 110 converts an optical signal of an object into an electrical signal, and outputs pixel data arranged in a matrix of a plurality of rows and columns.
  • the row selector 120 receives a row address and generates a selection signal for selecting at least two rows from a plurality of rows.
  • the image synthesizing unit 130 generates binning data by applying binning to pixel data.
  • the image combining unit 1300 generates binning data by applying any one of horizontal binning, vertical binning, and full binning to pixel data.
  • the analog-to-digital conversion unit 140 converts the pixel data selected by the row selection unit 120 and the generated binning data into a digital image signal and outputs it.
  • the analog-to-digital conversion unit 140 physically separates pixel data and binning data and outputs them.
  • the analog-to-digital converter 140 outputs the pixel data and binning data in time.
  • the amplifier unit 160 amplifies the output of the pixel unit and transmits it to the analog-to-digital conversion unit 140.
  • the memory 170 stores pixel data.
  • the memory 170 includes a capacitor.
  • the memory 170 may be arranged for each column of the pixel unit 110.
  • the memory 170 is arranged for each column, there is an advantage of storing pixel data of a specific line included in the pixel unit 110.
  • the image synthesizing unit 130 stores the pixel data of the first line in the memory 170, and applies binning to the pixel data of the second line and the pixel data of the first line stored in the memory 170, and the binning data ( 1120). In addition, the image synthesizing unit 130 stores the pixel data of the third line in the memory 170 and applies binning to the pixel data of the fourth line and the pixel data of the third line stored in the memory 170, and binning Data 1120 is generated.
  • the image synthesis unit 130 outputs the first line, outputs the second line, outputs binning data from the first line and the second line, outputs a third line, and outputs a fourth line.
  • Binning data binning the third and fourth lines is output.
  • the image synthesizing unit 130 sequentially outputs a plurality of lines 1110.
  • the image synthesis unit 130 controls synchronization of the amplifier unit 160 and the analog-to-digital conversion unit 140.
  • FIG. 12 is a diagram illustrating a configuration diagram of a processor according to an embodiment of the present invention.
  • the processor 200 includes an interface unit 210, a control unit 220, a filter unit 230, and a converter 240.
  • the processor 200 includes an application processor.
  • the interface unit 210 receives pixel data and binning data from an image sensor.
  • the control unit 220 fuses the received pixel data and the received binning data to generate fused pixel data.
  • the filter unit 230 removes noise of pixel data and noise of binning data according to a control command from the control unit 220.
  • the converter 240 converts the quad bayer containing the generated fusion pixel data into a normal bayer.
  • the ISP Image single processor, 250 performs overall image processing to process and process raw data according to a control command from the control unit 220.
  • the scaler 260 converts the format of the input video signal to the resolution of the display panel according to a control command from the control unit 220.
  • the memory 270 stores fused pixel data according to a control command from the control unit 220.
  • FIG. 13 is a flowchart illustrating a method of controlling a processor according to an embodiment of the present invention. The present invention is performed by the control unit 220.
  • pixel data and binning data are received from an image sensor (S1310 ).
  • the received pixel data and the received binning data are fused to generate fused pixel data (S1320).
  • the controller 150 fuses the resolution information of the pixel data and the brightness information of the binning data to generate fused pixel data.
  • FIG. 14 is a diagram illustrating a structural diagram of an image sensor system in which an image sensor and a processor are combined according to an embodiment of the present invention.
  • the image sensor system 300 is configured by combining the image sensor 100 and the processor 200.
  • the image sensor 100 includes a pixel unit 110, a row selection unit 120, an image synthesis unit 130, an analog-to-digital conversion unit 140, an amplifier unit 160, and a memory 170.
  • the pixel unit 110 converts the optical signal of the subject into an electrical signal, and outputs pixel data (original image) arranged in a matrix of a plurality of rows and columns.
  • the row selector 120 receives a row address and generates a selection signal for selecting at least two rows from a plurality of rows.
  • the image synthesizing unit 130 generates binning data by applying binning to pixel data (original image data).
  • the analog-to-digital converter 140 converts the pixel data selected by the row selector 120 and the generated binning data into a digital image signal, and outputs the converted pixel data and the converted binning data.
  • the analog-to-digital conversion unit 140 physically separates pixel data and binning data and outputs them.
  • the analog-to-digital converter 140 outputs the pixel data and binning data in time.
  • the processor 200 includes an interface unit 210 and a control unit 220.
  • the interface unit 210 receives the pixel data 10 and binning data 20 from the image sensor 100.
  • the control unit 220 fuses the received pixel data 10 and the received binning data 20 to generate fused pixel data.
  • the image sensor 100 and the processor 200 may be integrated and designed in the same semiconductor chip package.
  • 15 is a diagram illustrating pixel data and binning data output from an image sensor according to an embodiment of the present invention.
  • the image sensor 100 outputs pixel data 10 and binning data 20.
  • the image sensor 100 outputs pixel data 10 and binning data 20 for each line.
  • the remaining lines are also divided into i) when only the pixel data 10 is output, and ii) when both the pixel data 10 and binning data 20 are output.
  • the pixel data 10 is transmitted in a packet unit, and includes a packet header (PH, 11), a packet footer (PF, 12), a frame start (FS, 13), and a frame end (FE, 14).
  • a packet is a data unit handled in data communication, and includes a packet header and a data portion.
  • the packet header 11 contains destination information of a packet.
  • the packet footer 12 contains information related to the pixel data of the packet.
  • the binning data 20 includes a packet header 21 and a packet footer 22.
  • the packet header 21 includes destination information of a packet.
  • the packet footer 220 includes information related to binning data.
  • the frame start (FS, 13) means that one frame starts
  • the frame end (FE, 14) means that one frame ends.
  • 16 is a diagram illustrating fusion pixel data generated by fusing pixel data and binning data according to an embodiment of the present invention.
  • FIG. 16(a) is a diagram showing fusion of pixel data and binning data to generate fused pixel data
  • FIG. 16(b) is a diagram showing a mathematical algorithm when generating fused pixel data.
  • the controller 150 fuses the pixel data 10 and binning data 20 to generate fused pixel data 30. Specifically, the controller 150 fuses the resolution information of the pixel data and the brightness information of the binning data to generate fused pixel data.
  • the pixel data is quad-bayer pixel data having the same attribute, and the same attribute means having the same color filter.
  • the color filter means any one of Gr, Red, and Blue.
  • adjacent pixel data of the first pixel data means second pixel data, third pixel data, and fourth pixel data.
  • the adjacent pixel data of the first pixel data Gr1 means the second pixel data Gr2, the third pixel data Gr3, and the fourth pixel data Gr4.
  • the binning data 20 includes Gr', R', B', and Gb'.
  • the image synthesizing unit 130 generates binning data Gr' by applying binning to Gr1, Gr2, Gr3, and Gr4.
  • the image synthesizing unit 130 generates binning data R'by applying binning to R1, R2, R3, and R4.
  • the image synthesis unit 130 generates binning data B'by applying binning to B1, B2, B3, and B4.
  • the image synthesizing unit 130 generates binning data Gb' by applying binning to Gr1, Gr2, Gr3, and Gr4.
  • Gr', R', B', and Gb' of the binning data 20 mean brightness information. Applying binning means applying full binning, and the scope of the present invention is not limited to full binning, and includes both horizontal binning and vertical binning.
  • the controller 150 fuses the resolution information of the pixel data and the brightness information of the binning data to generate fused pixel data.
  • the resolution information of the first pixel data refers to a ratio obtained by dividing the first pixel data by the average value of the first pixel data, the second pixel data, the third pixel data, and the fourth pixel data. .
  • the resolution information of Gr1 is Gr1/((Gr1 + Gr2 + Gr3 + Gr 4)/4).
  • the resolution information of Gr2 is Gr2/((Gr1 + Gr2 + Gr3 + Gr4)/4).
  • the resolution information of Gr2 is Gr2/((Gr1 + Gr2 + Gr3 + Gr4)/4).
  • the resolution information of Gr4 is Gr4/((Gr1 + Gr2 + Gr3 + Gr4)/4).
  • the resolution information of the first pixel data means a ratio between the first pixel data and the adjacent pixel data, and there is no unit.
  • the resolution of the first pixel data and the resolution of the fused pixel data are the same.
  • the ratio between the pixel data and the adjacent pixel data is close to 1, but in the case of an edge, the ratio between the pixel data and the adjacent pixel data is greater than 1.
  • the controller 150 fuses the resolution information of the pixel data and the brightness information of the binning data to generate fused pixel data.
  • the meaning of fusion means that the resolution information and the brightness information are multiplied.
  • Fused pixel data is generated as a product of brightness information of binning data and resolution information of pixel data.
  • the first fusion pixel data is Gr1
  • the first fusion pixel data Gr1" Gr1' x Gr1/((Gr1 + Gr2 + Gr3 + Gr 4)/4).
  • the second fused pixel data is Gr2
  • the second fused pixel data Gr2 Gr2'x Gr2/((Gr1 + Gr2 + Gr3 + Gr 4)/4).
  • the third fused pixel data is Gr3
  • the third fused pixel data Gr3" Gr3'x Gr3/((Gr1 + Gr2 + Gr3 + Gr 4)/4).
  • the fourth fusion pixel data is Gr4"
  • the fourth fusion pixel data Gr4" Gr4' x Gr4/((Gr1 + Gr2 + Gr3 + Gr 4)/4).
  • the brightness of the fused pixel data is characterized in that it is at least two times brighter than the brightness of the first pixel data.
  • the fused pixel data is generated as a product of brightness information of binning data and resolution information of pixel data.
  • the pixel data corresponds to the original image, that is, the brightness information of binning data becomes 4 times the brightness information of the original image.
  • the ratio of the original image it is possible to obtain an original image with 4 times the brightness.
  • the image quality is mainly affected by the amount of light collected in each pixel constituting the image sensor.
  • the chip size of the CMOS image sensor is getting smaller and the number of pixels is increasing, so the size of the pixel is getting smaller.
  • the back-illuminated sensor (BSI sensor) has moved the light-receiving unit (the portion that receives light) to the uppermost portion of the sensor to increase the light-receiving rate.
  • the size of the pixel continues to decrease, there is a problem that there is a limit.
  • binning is applied to the original image data to generate binning data with improved brightness, and original image data and binning data are fused to generate image data with improved brightness, thereby increasing the size of the image sensor Without it, the brightness of the image sensor can be improved.
  • the data amount of the pixel data and the generated fusion pixel data are the same. Therefore, according to the present invention, it is possible to improve the brightness of the image sensor while keeping the amount of data constant and not increasing the amount of data.
  • FIG. 17 is a diagram illustrating a quad bayer to a normal bayer according to an embodiment of the present invention.
  • a converter 230 is further included as a component, and the converter 230 includes a quad-bayer 30 including fused pixel data generated according to a control command from the controller 220 as a normal bayer ( 40).
  • the control unit 220 converts the quad bayer 30 into a normal bayer 40, so that the processor 200 can recognize pixel data.
  • 18 is a diagram illustrating that a processor receives pixel data and binning data output from an image sensor according to an embodiment of the present invention. 18 includes FIGS. 18(a) and 18(b).
  • 18(a) is a diagram illustrating that the processor 200 receives pixel data and binning data from an image sensor in one path.
  • 18(b) is a diagram illustrating that the processor 200 receives pixel data and binning data from an image sensor in two paths.
  • the processor 200 includes an application processor 200.
  • MIPI stands for MIPI, a mobile industry processor interface, and was created to standardize an interface for connecting a mobile processor 200 and peripheral devices.
  • the interface unit 210 of the processor 200 simultaneously receives binning data 20 and pixel data 10 from the image sensor 100.
  • the interface unit 210 temporally divides the binning data 20 and the pixel data 10 and receives them from the image sensor 100.
  • the interface unit 210 temporally divides the binning data 20 and the pixel data 10 and receives them as one line.
  • the interface unit 210 first receives pixel data of the first line, and receives pixel data and binning data 20 and pixel data 10 of the second line. . Accordingly, the interface unit 210 first receives the pixel data of the first line, and after a predetermined time has elapsed, then receives the pixel data 10 and the binning data 20 of the second line.
  • 18(b) is a diagram illustrating that the processor 200 receives pixel data and binning data from an image sensor in two paths.
  • the interface unit 210 of the processor 200 simultaneously receives binning data 20 and pixel data 10 from the image sensor 100.
  • the interface unit 210 physically separates the binning data 20 and the pixel data 10 and receives them from the image sensor 100.
  • the interface unit 210 physically separates the binning data 20 and the pixel data 10, and separately receives the pixel data and binning data through two paths.
  • the pixel data 10 is transmitted to the interface unit 210 through the first path 41 and binning data 20 ) Is transmitted to the interface unit 210 through the second path 42.
  • the processor 200 when the pixel data and the binning data are separately received through two paths, the processor 200 has an advantage that the pixel data and the binning data can be fused more efficiently.
  • 19 is a diagram illustrating a fusion image generated by fusing an original image and a binning image according to an embodiment of the present invention.
  • the image sensor 100 outputs pixel data 10 and binning data 20.
  • the pixel data 10 corresponds to the original image 1910
  • the binning data 20 corresponds to the binning image 720
  • the fused pixel data corresponds to the fused image 730.
  • the processor 200 fuses the resolution information of the pixel data 10 and the brightness information of the binning data 20 to generate fused pixel data.
  • the meaning of fusion means that the resolution information and the brightness information are multiplied.
  • the brightness of the fused pixel data is characterized by being more than twice the brightness of the pixel data.
  • the size of the original image 1910 is 20 MP (mega pixels), the brightness is 1 unit, the binning image 1920 is 5 MP (mega pixels), and the brightness is 4 units, the fusion The size of the image 1930 is 20 MP (mega pixels), and the brightness is 4 units.
  • the brightness information of the binning data becomes 4 times the brightness information of the original image.
  • the ratio of the original image it is possible to obtain a fused image 1930 with 4 times the brightness.
  • the present invention since the brightness of the generated fused image is improved, the calling is reduced, and the pixel size is not increased, the manufacturing cost of the image sensor is reduced.
  • 20 is a diagram illustrating generating fused pixel data inside a processor.
  • the processor 200 receives pixel data and binning data from two image sensors 100-1 and 100-2.
  • the interface unit 210 includes a first interface unit 212 and a second interface unit 214.
  • the first interface unit 212 receives the first pixel data and the first binning data from the first image sensor 100-1.
  • the second interface unit 214 receives the second pixel data and the second binning data from the second image sensor 100-2.
  • the resolution information of the first pixel data is different from the resolution information of the second pixel data.
  • the resolution information of the first pixel data may be 21 MP (mega pixels)
  • the resolution information of the second pixel data may be 16 MP (mega pixels).
  • the control unit 220 includes a first control unit 220-1 and a second control unit 220-2.
  • the first control unit 220-1 fuses the resolution information of the first pixel data and the brightness information of the first binning data to generate first fused pixel data.
  • the size of RAW data of the first fused pixel data may be 21 MP (megapixel).
  • the second controller 220-2 fuses the resolution information of the second pixel data and the brightness information of the second binning data to generate second fused pixel data.
  • the size of RAW data of the second fused pixel data may be 16 MP (megapixel).
  • the ISP 250 includes a first ISP 250-1 and a second ISP 250-2.
  • the first ISP 250-1 performs image processing to process and process raw data of the first fused pixel data.
  • the second ISP 250-2 performs image processing to process and process raw data of the second fused pixel data.
  • the scaler 260 includes a scaler YUV 1 261, a scaler YUV 2 262, a scaler YUV 3 263, and a scaler FD 264.
  • DMA Direct memory access
  • CPU central processing unit
  • AXI Advanced extensible interface
  • AXI Advanced extensible interface
  • a channel can be introduced to transmit data independently.
  • a channel is introduced so that each channel operates independently to continuously transmit data.
  • 21 is a diagram specifically showing generating a fusion image inside the processor.
  • the processor 200 receives the pixel data 10 and binning data 20 from the image sensor 100 in the interface unit 210.
  • the filter unit 230 reduces noise of the pixel data 10.
  • the filter unit 230 reduces noise in the binning data 20.
  • the filter unit 230 may reduce noise of the pixel data 10 according to a control command from the control unit 220.
  • noise of the binning data 20 can be reduced.
  • the original image is a dark image using 1/4 exposure compared to the binning image created. That is, the original image has more noise components compared to the binning image. This noise component is a factor that degrades the resolution when the controller 150 fuses.
  • the filter unit 230 performs noise reduction of the pixel data 10 according to a control command from the control unit 220.
  • the filter unit 230 includes a median filter.
  • the median filter 230 refers to a filter that obtains an intermediate value of a pixel density in an area around a certain pixel and processes it at a desired pixel density.
  • the controller 220 fuses the resolution information of the pixel data 10 and the brightness information of the binning data 20 to generate fused pixel data.
  • the controller 220 calculates a coefficient value of 2 x 2 pixel gain, fuses brightness information of binning data 20 to the calculated 2 x 2 pixel gain value, and generates fused pixel data.
  • the control unit 220 converts the fused pixel data
  • the fused pixel data becomes a Bayer image 40 with improved brightness.
  • Fused pixel data is generated as a product of brightness information of binning data and resolution information of pixel data.
  • binning data may be used instead of pixel data in a preview screen, thereby reducing power consumption.
  • the amount of data processed can be reduced to 1/4, thereby reducing power consumption. I can do it.
  • binning is applied to the original image data to generate binning data with improved brightness, and the original image data and binning data are fused to generate image data with enhanced brightness.
  • the file size of is the same, the brightness is improved, and the size of the image sensor can be kept the same, thereby improving user convenience.
  • binning data and pixel data are physically or temporally divided and received from an image sensor, and fusion pixel data is generated by fusing resolution information of pixel data and brightness information of binning data, , Since the size of the image sensor remains the same and the brightness of the image data is improved, user convenience can always be achieved.
  • the image display device and the operation method according to the present invention are not limited to the configuration and method of the above-described embodiments, and the above embodiments are all or part of each embodiment so that various modifications can be made. May be selectively combined.
  • the operation method of the video display device of the present invention can be implemented as a code that can be read by the processor on a recording medium that can be read by the processor provided in the video display device.
  • the processor-readable recording medium includes all kinds of recording devices in which data that can be read by the processor are stored. Examples of the recording medium that can be read by the processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and include those implemented in the form of carrier waves such as transmission through the Internet. .
  • the processor-readable recording medium may be distributed over a networked computer system so that the processor-readable code is stored and executed in a distributed manner.
  • the present invention is used in a related field of a processor that improves the brightness of an original image using sensor binning.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)

Abstract

La présente invention concerne un processeur et un procédé de commande associé, et concerne la réception de données de pixel et le compartimentage de données à partir d'un capteur d'image, et la convergence des données de pixel reçues et des données de compartimentage reçues de façon à générer des données de pixel convergentes.
PCT/KR2019/009961 2018-11-30 2019-08-08 Processeur et procédé de commande associé Ceased WO2020111441A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201862773187P 2018-11-30 2018-11-30
US62/773,187 2018-11-30
KR10-2018-0162029 2018-12-14
KR20180162029 2018-12-14

Publications (1)

Publication Number Publication Date
WO2020111441A1 true WO2020111441A1 (fr) 2020-06-04

Family

ID=70852802

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/KR2018/016000 Ceased WO2020111369A1 (fr) 2018-11-30 2018-12-17 Capteur d'image et procédé de commande associé
PCT/KR2019/009961 Ceased WO2020111441A1 (fr) 2018-11-30 2019-08-08 Processeur et procédé de commande associé

Family Applications Before (1)

Application Number Title Priority Date Filing Date
PCT/KR2018/016000 Ceased WO2020111369A1 (fr) 2018-11-30 2018-12-17 Capteur d'image et procédé de commande associé

Country Status (1)

Country Link
WO (2) WO2020111369A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220122219A1 (en) * 2020-10-21 2022-04-21 Samsung Electronics Co., Ltd. Device for improving image resolution in camera system having lens that permits distortion and operation method thereof
WO2023031943A2 (fr) 2021-09-06 2023-03-09 Biond Biologics Ltd. Agents de blocage du décollement de cd28
CN118509465A (zh) * 2024-07-22 2024-08-16 深圳市广明科技有限公司 一种刀闸位置双确认信号校验方法、设备及介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118175441B (zh) * 2024-04-24 2025-02-14 荣耀终端有限公司 图像传感器、图像处理方法、电子设备、存储介质及产品

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110234869A1 (en) * 2009-10-07 2011-09-29 Panasonic Corporation Solid-state image pickup device and image pickup apparatus
US20140071244A1 (en) * 2011-05-24 2014-03-13 Sony Corporation Solid-state image pickup device and camera system
US20150288950A1 (en) * 2013-08-16 2015-10-08 University Of New Brunswick Camera imaging systems and methods
KR20160031706A (ko) * 2014-09-15 2016-03-23 삼성전자주식회사 이미지의 잡음 특성 개선 방법 및 그 전자 장치
KR20160086135A (ko) * 2015-01-09 2016-07-19 삼성전자주식회사 이미지 센서와 이를 포함하는 이미지 처리 시스템

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101488074B1 (ko) * 2013-02-04 2015-01-29 주식회사 동부하이텍 이미지 센서와 이를 이용한 비닝 이미지 생성 방법
EP3059939A4 (fr) * 2013-12-05 2017-08-02 Olympus Corporation Dispositif d'imagerie et procédé de commande du dispositif d'imagerie
US9686485B2 (en) * 2014-05-30 2017-06-20 Apple Inc. Pixel binning in an image sensor
JP2016139988A (ja) * 2015-01-28 2016-08-04 株式会社東芝 固体撮像装置
CN108781278A (zh) * 2016-03-30 2018-11-09 Lg 电子株式会社 图像处理装置和移动终端

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110234869A1 (en) * 2009-10-07 2011-09-29 Panasonic Corporation Solid-state image pickup device and image pickup apparatus
US20140071244A1 (en) * 2011-05-24 2014-03-13 Sony Corporation Solid-state image pickup device and camera system
US20150288950A1 (en) * 2013-08-16 2015-10-08 University Of New Brunswick Camera imaging systems and methods
KR20160031706A (ko) * 2014-09-15 2016-03-23 삼성전자주식회사 이미지의 잡음 특성 개선 방법 및 그 전자 장치
KR20160086135A (ko) * 2015-01-09 2016-07-19 삼성전자주식회사 이미지 센서와 이를 포함하는 이미지 처리 시스템

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220122219A1 (en) * 2020-10-21 2022-04-21 Samsung Electronics Co., Ltd. Device for improving image resolution in camera system having lens that permits distortion and operation method thereof
US11758288B2 (en) * 2020-10-21 2023-09-12 Samsung Electronics Co., Ltd. Device for improving image resolution in camera system having lens that permits distortion and operation method thereof
WO2023031943A2 (fr) 2021-09-06 2023-03-09 Biond Biologics Ltd. Agents de blocage du décollement de cd28
CN118509465A (zh) * 2024-07-22 2024-08-16 深圳市广明科技有限公司 一种刀闸位置双确认信号校验方法、设备及介质

Also Published As

Publication number Publication date
WO2020111369A1 (fr) 2020-06-04

Similar Documents

Publication Publication Date Title
WO2020111441A1 (fr) Processeur et procédé de commande associé
WO2016013902A1 (fr) Appareil de prise de vues d'images et procede de prise de vues d'images
WO2013147488A1 (fr) Appareil et procédé de traitement d'image d'un dispositif d'appareil photographique
WO2017090837A1 (fr) Appareil de photographie numérique et son procédé de fonctionnement
WO2013157802A1 (fr) Appareil et procédé de traitement d'image d'un appareil photo
WO2015108341A1 (fr) Dispositif d'affichage, circuit de commande du dispositif d'affichage, dispositif électronique contenant le dispositif d'affichage et le circuit de commande, et système d'affichage
WO2020130496A1 (fr) Appareil d'affichage et procédé de commande associé
WO2017030262A1 (fr) Appareil photo, et procédé de commande associé
WO2019017698A1 (fr) Dispositif électronique, et procédé pour dispositif électronique comprimant des données d'image à plage dynamique élevée
WO2014079189A1 (fr) Procédé et dispositif de traitement d'image et dispositif de formation d'image
WO2017018614A1 (fr) Procédé d'imagerie d'objet mobile et dispositif d'imagerie
WO2020054949A1 (fr) Dispositif électronique et procédé de capture de vue
WO2019017641A1 (fr) Dispositif électronique, et procédé de compression d'image de dispositif électronique
WO2016024796A2 (fr) Appareil d'imagerie et son procédé d'imagerie
WO2016017906A1 (fr) Dispositif d'affichage, dispositif de correction d'affichage, système de correction d'affichage, et procédé de correction d'affichage
WO2017014404A1 (fr) Appareil de photographie numérique, et procédé de photographie numérique
WO2012002742A2 (fr) Module appareil de prises de vue et procédé pour entraîner ledit module
WO2022102972A1 (fr) Dispositif électronique comprenant un capteur d'image et son procédé de fonctionnement
WO2021210875A1 (fr) Dispositif électronique permettant de détecter un défaut dans une image sur la base d'une différence entre des sous-images acquises par un capteur à photodiodes multiples, et son procédé de fonctionnement
WO2019160191A1 (fr) Dispositif et procédé de traitement d'images
WO2019017586A1 (fr) Procédé de transmission de données d'image et de données associées à la commande de capture d'image, sur la base de la taille de données d'image et de la taille de données associées à la commande de capture d'image, et dispositif électronique le prenant en charge
WO2021085694A1 (fr) Capteur d'image, et caméra et dispositif électronique la comportant
WO2025135845A1 (fr) Procédé et appareil de reproduction de couleur d'image
WO2022045509A1 (fr) Appareil électronique et procédé de commande associé
WO2020045834A1 (fr) Appareil électronique et son procédé de commande

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19891009

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19891009

Country of ref document: EP

Kind code of ref document: A1