GB2162019A - Compensating for camera movement - Google Patents

Abstract

TV cameras can reproduce an image of a viewed scene accurately provided the camera is rigidly mounted or is mounted on a stabilised platform to keep it steady. It is possible to remove image perturbations, due to camera movement, from a solid state TV camera picture by using accelerometers to measure camera movement and then using signals indicative of the movement to modify the read-out pulses of the vertical shift registers 3 and the horizontal read-out register 4 in frame Transfer CCD ie advancing or retarding the pulses 20, 21, to compensate for movement in two orthogonal directions. Interline transfer or XY CCDs may be compensated for movement in a similar manner (fig. 4). The integration time of the CCD is preferably short (eg 1 ms) to avoid blurring due to movement. <IMAGE>

Description

SPECIFICATION Image sensing apparatus This invention relates to image sensing apparatus, for example television cameras, including solid state image sensors.

Television cameras can reproduce an image of a viewed scene accurately, provided the camera is rigidly mounted to keep it steady, or in the case of a camera mounted on board a moving vehicle or craft, the camera is mounted on a stabilised, gimballed platform of some kind. Any random movement of the camera, for example due to the motion of the vehicle or craft on which it is mounted can be effectively magnified in a narrow field-of-view by the high magnification ofthe lenses which may have to be used.

The object of the invention is to remove image perturbations from a solid state image thereby allowing steady pictures to be obtained when using a hand-held, news gathering television camera or a strap-down camera system ie a camera which is rigidly mounted on a vehicle or craft.

According to one aspect ofthe present invention, there is provided image sensing apparatus comprising a solid state image sensor which has a twodimensional array of photo-sites and is operable, during respective ones of a series of field-time periods, forforming successive patterns of electrical image signals representative of a radiation image received by the sensor, read-out control for causing the image signals of each said pattern to be read out from the sensor in series, and motion signal supply means for supplying motion signals representative of movement in space of said image sensor in the two dimensions of said array, said read-out control means being connected to receive said motion signals and being operable in response thereto to varythe read-outtimes of said signals and thereby to compensate for movement of the sensor in said two dimensions.

According to another aspect of the invention, there is provided image sensing apparatus comprising a solid state image sensor which has an array of photo-sensitive elements and is operable, during respective ones of a series of predetermined field-time periods, forforming successive patterns of electrical signals indicative of a radiation image received by the sensor, read-out means for applying read-out signals to said sensorto cause the signals of each said pattern to be read out from the sensor in series over a field-time period, control means for ensuring that each read-out pattern is formed as a result of radiation received by the sensor over a time interval which is relatively short compared with each said field-time period, motion sensing means for sensing the movement in space of said image sensor, and read-out modifying means which is operable in response to the motion sensing means to modify the operation of said read-out means so as to vary the instant within each field-time period at which the signal formed by any one-photosensitive element is read out, thereby compensating for the movement of the apparatus.

For a better understanding of the invention, reference will now be made, by way of example, to the accompanying drawing in which: Figure 1 shows a block diagram of a frame transfer charge-coupled solid state imager (FTCCD) as used in one embodiment of the invention, and Figure 2 shows a further embodiment of the invention where an XY-addressed device is employed, Figure 3 shows a block diagram of a circuit which may be used to effect image motion compensation for the figure 1 device, Figure 4 shows a block diagram of a circuitwhich may be used to effect image motion compensation for an interline transfer charge-coupled device (ITCCD) or for the device offigure 2, Figure 5 shows the displayed image from an image motion compensated FTCCD, and Figure 6 shows the displayed image from an image motion compensated ITCCD or XY-addressed device.

It is known to detect the movement of a television camera due to hand-shake (ifthecamera is hand-held) orto vibrations (if the camera is mounted on a vehicle or craft) using accelerometertechniques. The output signal obtained indicative of the camera movement can then be used to initiate the compensation required to obtain a steady image. Naturally, any steady movement which is required ie allowing panning to take place can be obtained by applying the output signal to an appropriate filtering system before the initiation of the compensating effect.

In solid state television cameras, solid state imagers of the types known as Frame Transfer Charge Coupled Devices (FTCCD) and Interline Transfer Charge Coupled Devices (ITCCD) are known. Each device has an image region where the image of a viewed scene is formed - a storage region where the image is held after being transferred from the image region, and a horizontal read-out register to which the information held in the storage region is transferred to give an output signal. The image and storage regions consist of a matrix of identical photo-sensitive picture elements called pixels, and the range of radiation wavelengths to which the solid state imager is sensitive depends on the photo-sensitive materials used in the pixels.In the FTCCD, the image and storage regions form a matrix having parallel column shift registers formed by the vertical columns in the matrix, and the pixels form the components in the shift registers. In the ITCCD, the image region columns are interlaced with storage region columns. In both devices, a photo-generated charge distribution is obtained on the image region which is representative of the image oftheviewed scene focussed onto the image region. By clocking the column shift registers, the image (represented by a charge distribution) is transferred to the storage region and then to the horizontal read-out register, where the output signal is read out and then can be amplified and filtered to form a conventional video signal. The image is focussed on to the image region and charge integrated for periods of typically 1/50 or 1/60 of a second.

The output signal indicative of camera movement obtained using accelerometertechniques (as mentioned previously) can be used to alterthe clocking of the vertical shift registers in the FTCCD and ITCCD imagersto produce a steady display image.

In figure 1, a FTCCD 1 is shown which comprises an image region 2, a storage region 3, and a horizontal read-out register 4. The output from the register 4 is connected to a display 6. The image region 2 and the storage region 3 are made up of parallel vertical column shift registers 7 which allowthe image formed in the image region 2 to be transferred to the storage region 3 by the clocking ofthe registers. The horizontal read-out register 4 accepts a stored row from the storage region 3 and as the vertical registers 7 are clocked, the register4 is also clocked to provide the output 5 which is then fed to the display 6. For example, points P and Q in the image region 2 representthe positions of stationary images of a viewed scene.By clocking the vertical shift registers 7, P and Q are transferred to the storage region 3 and are stored asP' and Q'. On further clocking P' and Q' are fed into the read-out register 4 and then to the display 6. P and Q in the image region 2 are then displayed as P" and Q" in the display 6.

Say that a particular point feature in a viewed scene is imaged at position A in the image region 2 during one field period. Camera movement causes the image of the same feature to be formed at respective positions B, C and D as shown during successive ones ofthe three following field periods. The movement of the camera is measured using accelerometertechniques and the signals obtained are used to initiate the compensationforthe image movement by altering the clocking ofthe vertical column shift registers 7 and the horizontal read-out register4 so that, during successive fields of the displayed picture, the image feature appears at only one point on the displayS.

During four consecutive field periods, the camera views the same scene and the feature of interest is imaged atfour points A, B, C and Din the image region 2 due to camera movement, each point corresponding to one field period le in the first field thefeature is imaged atA, in the second field the feature is imaged at B etc. In thefirstfield, the image pattern appearsatA while the camera is at rest or stationary and so no compensation is applied forthisfield lithe image pattern is clocked and displayed normaily as described previously for points P and Q. During the next field, the feature is imaged at B, showing thatthe camera has moved.The camera movement is measured using known accelerometer techniques and the output signals are used to advance the clocking of the image pattern to the output 5. Therefore, at the end of the second field period, the image pattern B is clocked into the storage region 3 using the same number of clock pulses as for the image pattern A (because A and B are in the same horizontal row in the image region 2) but a series of extra clock pulses are applied to the register4to advance the image pattern B" so that it coincides with the pattern A" on the display 6. During the third field, the feature is imaged at C due to camera movement. Extra clock pulses are applied to the vertical column shift registerwhich contains the image pattern C as it is transferred to the storge region 3.These extra pulses clock the pattern C' into the same horizontal row that A' and B' occupied on the previous two fields. As for B', extra pulses are also applied to the register4 so that the pattern C" coincides with pattern A" on the display 6. Similarly, during the forth field,thefeature is imaged at D, extra clock pulses are applied to the register 7 as for pattern C' to bring D' into the same horizontal row that A', B' and C' occupied, but the clock pulses applied to the read-out register4 are retarded so that pattern D" coincides with pattern A" on the display 6. Therefore, over four field periods the image position of the feature on the display 6 is displayed as a constant point after compensation for camera movement has taken place.

Solid state sensors of the generic type known as XY-addressed devices may also be used in a way which compensates for camera movement. Figure 2 shows an XY-addressed sensor 10. The sensor 10 comprises a matrix 11 of pixels which outputs signals to a horizontal scanning register 12 and a vertical scanning register 13.Registers 12 and 13 are able to interrogate a particular pixel as shown, scanning the device in a conventional television raster format The scanning registers 12 and 13 are designed so that a single clock pulse which is introduced at one end of the register, travels through it to the other end, driven by an external clock-pulse source. Movementcom- pensation in such a device is achieved by advancing or retarding the introduction of the scanning pulse into the registers using the signals obtained from the techniques previously mentioned for measuring camera movement.

In both the described embodiments, the time period for which the solid state imaging device is exposed to a viewed scene iethe exposure, has to be reduced to preventsmearing or blurring of the image. The exposure, normally 20 ms, is reduced to a typical value of 1 ms. The reduced exposure can be achieved by using a mechanical shutterega rotating aperture, or a liquid crystal shutter. The exposure may also be controlled by electronic means eg reverse clocking of the kind disclosed on our patent No. 2,083,968.For example, an image may be allowed to build up in the integration region of the sensor offigure 1 during the first, major part of the field-time period, this image then being 'dumped' by applying a reverse-direction clocking pulse sequence to the sensor so that the image charge pattern is transferred to the 'wrong' end of the image region and there flows away into the sensorsubstrate. Forward clocking may be used for this purpose in sensors processing an efficient antiblooming structure. Then, a further image is allowed to build up in the integration region over the remaining part of the field-time period, say 1 ms, and only this image is transferred to the storage region and read out.

The read-out rate from the device is maintained at a standard rate, typically 50 or 60 Hz.

Figure3illustratesan FTCCD 1 aswell asthe components required for image motion compensation shown generally at 20 (for vertical) and 21 (for horizontal). The device 1 operates in two modes, the integration mode where the image is formed in the image region 2, and the transfer mode where the image istransferredfrom the image region 2to the storage region 3.

In the integration mode, an electronic switch 22 routes signals from fixed potential 23 to the electrodes 1, IZ12 and Id3 of of the image region 2, whilst switch 24 feedssignalsfrom a line readout clock 25 to the electrodes PI'1, $'2 and $' ofthe storage region 3. The clock 25 operates during each line blanking period ie at the end of an output row and before another row is read out, to move a single row of stored charge from the storage region 3 a the horizontal readout register 4.

In the transfer mode, switches 22 and 24 direct clock pulses from the vertical clock 26 via gate 27 to both the image and storage regions 2 and 3. The electrode sets 1ZI,, 2, 3 and '1, '2 and m'3 operate in unison to swiftlytransferthe charge out ofthe image region 2 into the storage region 3. A line counter28 controls the gate 27 to ensure that the correct number of clock pulses is applied to the device 1, the number of pulses being equal to the numberofpixels inthevertical columns and being stored as a preset count value 29.

Compensation forvertical movement ofthe device 1 is achieved by altering the value 29 using the vertical motion image compensator 20, which comprises displacement sensing means 30, a filter 31, an A/D converter 32, and a binary adder/substractor 33. The sensing means 30 provides a signal 34 which is indicative of the vertical displacement which signal is then filtered by the filter 31 to remove effects due to constant panning motion, and is then digitised by the converter 32. The filtered and digitised signal 34' is then added to or subtracted from the count value 29 according to whether the vertical movement is in a downward or upward direction to provide the vertical compensation.

Similarly,the horizontal motion image compensator 21 comprises displacement sensing means 35, filter 36, A/D converter 37 and counter 38, the filter 36 removing the effects of panning and the converter 37 digitising the signal. However, as each line of charge is transferred to the readout register 4, it has to be shuffled to the right or left to achieve compensation. A horizontal clock 39 provides signals via AND gate 40 and reverse readout switch 41 to the clock lines Ib"i, $"2 and 0"3 clocks to readout the register 4, signals being provided when there is no output from the line blanking unit 42.The clock lines can be operated in both directions ieto the left or right, by reversing the clock pulses on any two of the three clock lines, the signal indicating the direction of operation being obtained from the counter 38. The counter 38 enables the register4to be read out by allowing signals from the clock 39 to pass through the gate 40 and the readout switch 41 when its output is 'HIGH'. A pixel counter43 having a maximum number of pixels in a row, say 40, counts the signals being fed to the switch 41 and switches off the AND gate 40 when the maximum value is reached.

The horizontal motion compensation is effected on each video line during the horizontal line blanking period which immediately proceeds the readout of that line, and if the line blanking period is not long enoughtoachievefull compensation,the horizontal clock 39 may be altered to have its frequency increased.

Image motion compensation for an interline transfer device (an ITCCD) or an XY-addressed device may be affected in a similar manner (like components being indicated by the same numbers as previously used) as shown in figure 4. The ITCCD 50 comprises columns 51 of alternate storage registers 52 and integrating columns 53, and an output register 54. The image is formed in the integrating columns 53 and is then transferred to the storage registers 52 before being read out from the output register 54.In this case the digitised signal 34' indicative of vertical motion of the camera is used to advance or retard the commencement of the vertical readout in 55 ie to advance or retard the feed ofthe video lines from the storage registers 52 into the output register 54, by adjusting the field timebase 56 and the line timebase or vertical clock 57. This alters the arrival time of the video field to the display device (not shown) and because the display is running to a constanttimebase, the effect is a vertical displacement of the image on the display, the degree to which the delay or advance for displacement being such that the camera movement between fields is corrected exactly.

A similar arrangement is used to control the commencement of the readout from the output register 54 to effect horizontal motion compensation ie the vertical clock 57 and the horizontal clock 58 are altered in 59. Ifthe compensation requires an advance of readout from register 54 which is greater than the line blanking period, the previous readout may be terminated prematurely, and the loading of lines into the output register 54 must be brought forward to aliowforthis.

Alternatively, an offset may be introduced so that the mean position for a steady camera produces a fixed amount of delay in the commencement ofthe readout for both the vertical and horizontal registers.

Displacement of the image from the mean position is then compensated for by increasing or decreasing the mean delay.

An XY-addressed device as shown in figure 2, may also be used with the circuit of figure 4. The device 10 has built-in shift registers 12,13 which are used to address the image area 11 and to readout the signal. A single pulse is introduced into each ofthe registers 12, 13 and passes through reading outthe lines or pixels consecutively. The apparatus shown in figure 4 is used to delay or advance the introduction ofthe pulses into the registers 12,13 at points marked H and V.

Figures 5 a nd 6 illustrate the picture areas which may be lost due to stabilisation of the camera using the techniques described above. For a FTCCD (figure 5) a frame shown shaded around the image is lost due to compensation for a shaking camera as described with reference to figure 3. A similar sized area is lost duetocompensation usingthefigure4technique (shown in figure 6) but part ofthis loss is due to an offset bias applied to the device, shown crosshatched, and is lost regardless ofwhetherthe camera is shaking or not.

Claims (13)

1. Image sensing apparatus comprising a solid state image sensor which has a two-dimensional array of photo-sites and is operable, during respective ones of a series of field-time periods, forforming successive patterns of electrical image signals representative of a radiation image received bythe sensor, read-out control for causing the image signals of each said pattern to be read outfrom the sensor in series, and motion signal supply means for supplying motion signals representative of movement in space of said image sensor in the two dimensions of said array, said read-out control means being connected to receive said motion signals and being operable in response theretotovarythe read-outtimes of said signals and thereby to compensate for movement of the sensor in said two dimensions.

2. Image sensing apparatus comprising a solid state image sensorwhich has an array of photosensitive elements and is operable, during respective ones of a series of predetermined field-time periods, forforming successive patterns of electrical signals indicative of a radiation image received by the sensor, read-out means for applying read-out signals to said sensorto cause the signals of each said pattern to be read out from the sensor in series over a field-time period, control means for ensuring that each read-out pattern is formed as a result of radiation received by the sensor over a time interval which is relatively short compared with each said field-time period, motion sensing means for sensing the movement in space of said image sensor, and read-out modifying means which is operable in response to the motion sensing means to modify the operation of said read-out means so asto varythe instantwithin each field-time period atwhich the signal formed by any one photo-sensitive element is read out, thereby compensating for the movementofthe apparatus.

3. Image sensing apparatus according to claim 1 or2, wherein said solid state image sensor is a frame transfer charge coupled device.

4. Image sensing apparatus according to claim 1 or2, wherein said solid state image sensor is an interlinetransferchargecoupled device.

5. Image sensing apparatus according to claim 1 or2, wherein said solid state image sensor is an XY-addressed device.

6. Image sensing apparatus according to any one of claims 1 to 5, wherein said read-out means is a clock pulse generatorwhich applies clock pulses to said image sensor.

7. Image sensing apparatus according to any preceding claim, wherein said control means includes an exposure control means.

8. Image sensing apparatus according to claim 7, wherein said exposure control means is a mechanical shutter.

9. Image sensing apparatus according to claim 7, wherein said exposure control means is a liquid crystal shutter.

10. Image sensing apparatus according to claim 7, wherein said exposure control means uses reverseclocking techniques.

11. Image sensing apparatus according to any preceding claim, wherein said motion sensing means comprises accelerometer means operable for measuring movement in a plurality of directions and for producing signals indicative of said movement.

12. Image sensing apparatus according to claim 11, wherein said read-out modifying means receives said signals and advances or retards said read-out means to obtain said compensation.

13. Imagesensing apparatus substantially as he- reinbefore described with reference to figures 1 to 4 of the accompanying drawings.