WO2013114988A1 - Information display device, information display system, information display method and program - Google Patents

Description

Information display device, information display system, information display method and program

The present invention relates to a technique for translating character information copied in an image.

In recent years, high-performance mobile phone terminals such as smartphones have been used, and as a result, various software implemented on mobile phone terminals have been developed.

The above software translates the characters described in the target from a captured image obtained by capturing the target with characters such as a signboard, poster, menu, etc. There is software that displays captured images with translated characters. Such software is useful when an operator of a mobile phone terminal travels abroad.

As a method for translating character information, a method for analyzing words and morphemes in the character information and a method using translated image data including translated character information in which the character information is translated in advance are known.

The method of analyzing words and morphemes has a problem that character information is translated into a meaning different from the original meaning or translated into an unnatural expression.

On the other hand, in the method using translated image data, collation image data including the same character information as the character information in the captured image data is retrieved from the database, and the translation image data previously associated with the retrieved image data indicates the translation. An image is displayed. For this reason, character information is not translated into a meaning different from the original meaning or translated into an unnatural expression.

However, since the captured image and the translated image are only displayed separately, there is a problem that it is difficult to understand the correspondence between the character information in the translated image and the translated character information in the photographed image.

On the other hand, Patent Document 1 aims to translate and display place names described on a map, and has two maps, an original map and a translated map in which place names described on the original map are translated. An information display device to be displayed on each screen is disclosed. In this information display device, it is possible to easily compare the character information in the captured image with the translated character information in the translated image, so that the correspondence between the character information in the translated image and the character information in the captured image is It can be easy to understand.

JP 2003-330588 A

Since place names are generally short and the length does not change much depending on the language, when the place name is translated as character information, the length of the character information does not change much before and after translation. For this reason, in the information display device disclosed in Patent Document 1, the range in which the place name is displayed does not change much before and after translation.

However, the length of the names of dishes listed on the menus and the texts written on the signs installed on famous places and historic sites may vary greatly depending on the language. For this reason, when the information display device disclosed in Patent Document 1 is applied to the translation of a dish name or a guide sentence, the range in which the character information is displayed before and after the translation changes greatly, and the character information before and after the translation is changed. The correspondence may be difficult to understand.

An object of the present invention is to provide an information display device, an information display system, and an information display method capable of easily displaying character information even if the lengths of character information before translation and character information after translation change significantly. And to provide a program.

An information display device according to the present invention includes a display unit, an imaging unit that performs imaging and outputs captured image data, collation image data indicating a collation image including character information, and translated character information obtained by translating the character information. A translation image data indicating a translation image including a reference point in a collation character range where the character information exists on the collation image and a reference point in a translation character range where the translation character information exists on the translation image A storage unit for storing character position information indicating a correspondence relationship; and comparing the captured image data with the verification image data to determine whether or not the character information is included in the captured image indicated by the captured image data. When the character information is included in the captured image and the collation unit to be determined, the reference points of the collation character range and the translated character range are associated with each other based on the character position information, and The focus image and the translated image and a control unit for displaying on the display unit.

An information display system according to the present invention is an information display system having a collation device and an information display device, wherein the collation device is obtained by translating collation image data indicating a collation image including character information and the character information. Translated image data indicating a translated image including translated character information, a reference point in the collated character range where the character information exists on the collated image, and a translated character range within which the translated character information exists on the translated image A storage unit for storing character position information indicating a correspondence relationship with a reference point, a first communication unit for receiving captured image data from the information display device, and comparing the captured image data with the verification image data. A collation unit that determines whether or not the character information is included in the captured image indicated by the captured image data; and the collation image data when the character information is included in the captured image. A first control unit that acquires the translated image data and the character position information from the storage unit and transmits the retrieved image data and the character position information as search information to the information display device. The information display device includes a display unit and an imaging unit. An imaging unit that outputs the captured image data, a second communication unit that transmits the captured image data to the verification device, and that receives the search information from the verification device, and the second communication A second control unit that displays the collation image and the translation image on the display unit in association with the reference points of the collation character range and the translation character range based on the search information received by the unit; Have

The information display method according to the present invention includes collation image data indicating a collation image including character information, translation image data indicating a translation image including translation character information obtained by translating the character information, and the character information on the collation image. Character position information indicating a correspondence relationship between a reference point in the collated character range in which the character string exists and a reference point in the translated character range in which the translated character information exists on the translated image is stored, and imaging is performed. Outputting image data, comparing the captured image data with the collation image data, determining whether or not the character information is included in the captured image indicated by the captured image data, and the character in the captured image When information is included, the collation image and the translation image are displayed in correspondence with the reference points of the collation character range and the translation character range based on the character position information.

The program according to the present invention includes collation image data indicating a collation image including character information, translation image data indicating a translation image including translation character information obtained by translating the character information, and the character information on the collation image. A computer connected to a storage unit for storing character position information indicating a correspondence relationship between a reference point in the collated character range to be performed and a reference point in the translated character range where the translated character information exists on the translated image; The procedure for performing imaging and outputting the captured image data and collating the captured image data with the verification image data to determine whether or not the character information is included in the captured image indicated by the captured image data When the character information is included in the captured image and the procedure, the reference points of the collation character range and the translated character range are associated with each other based on the character position information. To execute a procedure for displaying the collation image and the translated image.

According to the present invention, even if the length of the character information before translation and the character information after translation changes significantly, it becomes possible to display the character information in an easy-to-see manner.

It is a figure which shows the structure of the information display apparatus of the 1st Embodiment of this invention. It is a figure which shows an example of the user interface screen in 1 screen display mode. It is a figure which shows an example of the user interface screen in 2 screen display mode. It is a flowchart for demonstrating an example of operation | movement of the information display apparatus of the 1st Embodiment of this invention. It is a figure which shows the structure of the information display apparatus of the 2nd Embodiment of this invention. It is a figure which shows an example of the user interface screen in 2 screen opposing display mode. It is a flowchart for demonstrating an example of operation | movement of the information display apparatus of the 2nd Embodiment of this invention. It is a figure which shows the structure of the information display system of the 3rd Embodiment of this invention. It is a sequence diagram for demonstrating an example of operation | movement of the information display system of the 3rd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, components having the same function may be denoted by the same reference numerals and description thereof may be omitted.

(First embodiment)
FIG. 1 is a diagram illustrating the configuration of the information display apparatus according to the present embodiment. The information display device 1 shown in FIG. 1 includes an input button unit 11, a rear camera unit 12, a display unit 13, a control signal input unit 14, a storage unit 15, an image collation unit 16, and a control unit 17. Have. The information display device 1 is, for example, a smartphone, a tablet terminal, or an information processing device having the same size as those.

When the input button unit 11 is pressed by an operator who uses the information display device 1, the input button unit 11 outputs a pressing signal indicating that the input button unit 11 has been pressed.

The rear camera unit 12 is an imaging unit that performs imaging and outputs captured image data. In the present embodiment, the rear camera unit 12 captures an imaging target such as a menu or a signboard, and outputs captured image data indicating a captured image obtained by capturing the imaging target. It is assumed that character information is included in the shooting target.

The display unit 13 displays various images. In the present embodiment, the display unit 13 is set to one display mode of one screen display mode for displaying one image and two screen display mode for displaying two images. The display unit 13 displays a translation result image including translated character information obtained by translating the character information copied in the photographed image in the one-screen display mode, and the translation result image and the photographed image in the two-screen display mode. And a collation result image including the character information copied to. Detailed descriptions of the translation result image and the collation result image will be described later.

The display unit 13 is provided on a predetermined surface (hereinafter referred to as the front surface) of the information display device 1, and the rear camera unit 12 is provided on the back surface that is the surface opposite to the front surface of the information display device 1. Suppose that

Various information is input to the control signal input unit 14 from the operator. In the present embodiment, it is assumed that the control signal input unit 14 is a touch panel that detects contact or proximity by an input unit such as a finger or a stylus and outputs a detection signal indicating the contact or proximity position. The control signal input unit 14 is provided so as to overlap the display unit 13.

The storage unit 15 stores storage information D1 that is a database for translating character information included in captured image data into another language.

The storage information D1 has data strings C11 to C1n as records. Each data string C1i includes collation image data Xi, translated image data X1i, image position feature information Yi, and an image position conversion formula Zi. In the present embodiment, n is an integer of 2 or more, and i is an integer of 1 to n. There may be only one data string.

The collation image data Xi is image data that is collated with the captured image data output from the rear camera unit 12, and includes character information.

The translated image data X1i is image data including translated character information obtained by translating character information in the collation image data Xi into another language. The language of character information and translated character information is not particularly limited as long as they are different languages.

The image position feature information Yi and the image position conversion formula Zi include a collation character range in which character information in the collation image indicated by the collation image data Xi exists and a translation in which translation character information in the translation image indicated by the translation image data X1i exists. Character position information indicating a character range and correspondence is configured.

Specifically, the image position feature information Yi indicates the correspondence between the reference point in the collation character range and the reference point in the translated character range. The reference point is, for example, a writing point or end point of character information (translated character information) or a midpoint of the writing point and end point.

The image position conversion formula Zi is information indicating the ratio between the size of the collation character range in the collation image data Xi and the size of the translation character range in the translation image data X1i. For example, the image position conversion formula Zi indicates the ratio of the length from the writing point in the collation image to the end point of the character information and the length from the writing point in the translated image to the end point of the translated character information.

Note that the collation image data X1 to Xn constitute a collation image information group D11. The translation image data X11 to X1n constitute a translation image information group D12. The image position feature information Y1 to Yn constitutes an image position feature information group D13. The image position conversion formulas Z1 to Zn constitute an image position conversion formula group D14.

The image collation unit 16 collates the captured image data output from the rear camera unit 12 with each of the collation image data X1 to Xn in the storage information D1 stored in the storage unit 15, and thereby the characters in the captured image data. The collation image data having the same character information as the information is searched. Then, the image collation unit 16 outputs the retrieved collation image data as retrieval image data.

The control unit 17 controls the entire information display device 1 by performing transmission and reception of signals and image data with each unit of the information display device 1. Specifically, the control unit 17 includes a CPU 18 and a calculation unit 19 having the following functions.

The CPU 18 receives a press signal from the input button unit 11, receives a detection signal from the control signal input unit 14, and receives search image data from the image matching unit 16.

When the search image data is received, the CPU 18 confirms the display mode of the display unit 13.

When the display mode is the single screen display mode, the CPU 18 acquires the translation image data corresponding to the search image data from the storage unit 15 and displays the translation image indicated by the translation image data on the display unit 13 as a translation result image. When there are a plurality of search image data, the CPU 18 displays each of the translation images indicated by the plurality of translation image data corresponding to the plurality of search image data as a translation result image.

When the display mode is the two-screen display mode, the CPU 18 acquires translated image data and image position feature information corresponding to the search image data from the storage unit 15. Then, based on the image position feature information, the CPU 18 associates the reference position in the collation character range of the search image data with the reference position in the translation character range of the translation image data, so that the search image data and the translation image The collation image and the translation image indicated by each data are displayed on the display unit 13 as a collation result image and a translation result image.

More specifically, the CPU 18 sets a collation display area for displaying the collation result image and a translation display area for displaying the translation result image on the display unit 13, displays the collation image in the collation display area, and displays the translation display. Display translated images in the area.

At this time, the CPU 18 associates the display position of the reference point of the character information with the display position of the translation reference point, which is the reference point of the translated character information, based on the image position feature information, and displays the collation image and the translated image. indicate. More specifically, the CPU 18 matches the display position of the reference point in the collation display area with the display position of the translation reference point in the translation display area, and displays the collation result image and the translation result image. For example, when the display position of the reference point is set to the center of the collation display area, the CPU 18 sets the display position of the translation reference point to the center of the translation display area.

Further, the CPU 18 includes an image movement mode and a pen input mode as operation modes for performing an operation according to the detection signal, and operates in one operation mode. The image movement mode is an operation mode in which the collation result image and the translation result image are moved and scaled, and the pen input mode is an operation mode in which the operator writes an image.

When the display mode of the display unit 13 is the single screen display mode, the CPU 18 operates in the image moving mode. In this case, the CPU 18 moves and enlarges / reduces the translation result image according to the detection signal.

For example, when moving the translation result image, the operator performs a first movement operation to input a first movement instruction to move the translation result image to the control signal input unit 14. The first movement operation is, for example, an operation of bringing the input unit into contact with the translation display area and then moving the input unit to move the contact position between the input unit and the translation display area.

When the first movement operation is performed, the CPU 18 receives a detection signal corresponding to the first movement operation from the control signal input unit 14 as a first movement instruction. In this case, the CPU 18 moves the translation result image displayed on the display unit 13 in the movement distance and the movement direction according to the first movement instruction.

Also, when enlarging / reducing the translation result image, the operator performs a first enlargement / reduction operation for inputting a first enlargement / reduction instruction for enlarging / reducing the translation result image to the control signal input unit 14. The first enlargement / reduction operation is, for example, an operation of bringing the input unit into contact with a slider area set in advance in the control signal input unit 14.

When the first enlargement / reduction operation is performed, the CPU 18 receives a detection signal corresponding to the first enlargement / reduction operation from the control signal input unit 14 as a first enlargement / reduction instruction. In this case, the CPU 18 enlarges / reduces the translation result image displayed on the display unit 13 at an enlargement / reduction ratio corresponding to the first enlargement / reduction instruction.

Further, when the display mode is the two-screen display mode, the CPU 18 operates in the image moving mode or the pen input mode.

When operating in the image movement mode, the CPU 18 moves and scales the collation result image and the translation result image in accordance with the detection signal.

For example, when the operator performs a second movement operation for inputting a second movement instruction to move the movement target image, which is a collation result image or a translation result image, to the control signal input unit 14, the CPU 18 performs the second movement operation. The movement target image is moved according to the movement instruction, and based on the character position information corresponding to the movement target image, different from the movement target image so that the reference points of the collation character range and the translation character range correspond to each other. The collation image or the translation image is moved. Note that the second moving operation is, for example, an operation in which the input unit is brought into contact with the collation display region or the translation display region for displaying the target image, and then the input unit is moved to move the contact position of the input unit. is there.

For example, when a second movement operation using the target image as a translation result image is performed, the CPU 18 receives a detection signal corresponding to the second movement operation from the control signal input unit 14 as a second movement instruction. The CPU 18 moves the translation result image that is the movement target image in accordance with the second movement instruction, and also displays the display position in the translation display area of the translation reference point of the translated translation result image and the reference point of the matching result image The collation result image is moved so that the display position in the collation display area matches.

When the target image is a collation image, the CPU 18 performs the same process as the above process, and moves both the collation result image and the translation result image.

When the operator performs a second enlargement / reduction operation for inputting a second enlargement / reduction instruction to enlarge / reduce the enlargement / reduction target image, which is a collation result image or a translation result image, to the control signal input unit 14, the CPU 18 performs the second enlargement / reduction operation. In response to the expansion instruction, not only the enlargement / reduction target image but also the collation result image or the translation result image different from the enlargement / reduction target image are enlarged / reduced. The second enlargement / reduction operation is, for example, an operation of bringing the input unit into contact with the slider area in the control signal input unit 14. The slider area may be provided for each of the collation result image and the translation result image, or may be provided for one of the collation result image and the translation result image.

Specifically, when a second enlargement / reduction operation using the enlargement / reduction target image as a translation result image is performed, the CPU 18 outputs a detection signal corresponding to the second enlargement / reduction operation from the control signal input unit 14 to the second enlargement / reduction instruction. As received. In this case, the CPU 18 determines the enlargement / reduction ratio according to the second enlargement / reduction instruction, enlarges / reduces the translation result image displayed on the display unit 13 with the enlargement / reduction ratio, and uses the calculation unit 19 to enlarge / reduce the image. A collation character range in the collation result image different from the image is specified. Then, the CPU 18 enlarges / reduces the collation result image based on the collation character range so that the collation character range corresponds to the translation character range. For example, the CPU 18 expands the collation result image so that the expanded collation character range and the translated character range have the same size. At this time, the CPU 18 may enlarge or reduce each of the collation result image and the translation result image around the reference point and the translation reference point, or may enlarge or reduce around the point designated by the user. In addition, the process of the calculating part 19 is mentioned later.

Further, when operating in the pen input mode, the CPU 18 performs drawing on the collation result image and the translation result image according to the detection signal.

For example, when the operator performs a drawing operation for drawing on the drawing target image, which is a matching result image or a translation result image, on the control signal input unit 14, the CPU 18 checks not only the drawing target image but also a matching different from the target image. Drawing is also performed on the result image or the translation result image. The drawing operation is, for example, an operation in which the input unit is brought into contact with the collation display region or the translation display region for displaying the drawing target image, and then the input unit is moved to move the contact position of the input unit.

More specifically, when a drawing operation using the drawing target image as a translation result image is performed, the CPU 18 receives a detection signal corresponding to the drawing operation from the control signal input unit 14 as a drawing instruction. A drawing instruction is received.

The CPU 18 draws a figure corresponding to the drawing instruction in the translated character range in the translation result image that is the drawing target image. Then, the CPU 18 uses the calculation unit 19 to identify a collation character range in the collation result image that is different from the drawing target image, and draws the graphic in the identified collation character range.

The CPU 18 sets the operation mode and the operation mode based on the detection signal that is set on the control signal input unit 14 and indicates the position in the region for switching the operation mode of the CPU 18 or the display mode of the display unit 13. The display mode can be switched.

The calculation unit 19 is used when the CPU 18 determines the collation character range or the translation character range. When determining the collation character range, the CPU 18 transmits translation coordinate information indicating the size of the translation character range to the calculation unit 19, and when determining the translation character range, the collation coordinates indicating the size of the collation character range. Information is transmitted to the calculation unit 19.

The calculation unit 19 acquires the image position conversion formula from the storage unit 15 when receiving the translated coordinate information from the CPU 18, for example. The computing unit 19 calculates collation coordinate information indicating the size of the collation character range associated with the character position information based on the acquired image position conversion formula, and transmits the collation coordinate information to the CPU 18. The CPU 18 determines the collation character range based on the received collation coordinate information and the image position feature information acquired from the storage unit 15 via the CPU 18.

The calculation unit 19 can calculate the translation coordinate information and transmit it to the CPU 18 by the same process as described above even when the collation coordinate information is received from the CPU 18.

As described above, the CPU 18 uses the calculation unit 19 from the translation coordinate information indicating the size of the translated character range, and the size of the collation character range associated with the translated character range in the character position information. The CPU 18 and the calculation unit 19 function as the control unit 17 that determines the collation character range associated with the translation character range in the character position information from the translation character range.

Hereinafter, the user interface screen displayed on the display unit 13 will be described.

FIG. 2 is a diagram showing an example of a user interface screen displayed on the display unit 13 in the single screen display mode.

The user interface screen shown in FIG. 2 has a 1-screen / 2-screen switch display area 101, a pen input switch display area 102, a translation display area 103, and a slider area 105.

The 1-screen / 2-screen switch display area 101 is an area for switching the display mode of the display unit 13. When the CPU 18 receives a detection signal indicating a position in the 1-screen / 2-screen switch display area 101, the CPU 18 switches the display mode of the display unit 13.

The pen input switch display area 102 is an area for switching the operation mode of the CPU 18. When the display unit 13 is in the one-screen display mode, the CPU 18 operates in the screen movement mode. Therefore, even when the detection signal indicating the position in the pen input switch display area 102 is received, the operation mode is not switched.

The translation display area 103 is an area for displaying a translation result image. In FIG. 2, the translation result image corresponding to the captured image obtained by capturing the menu 2 is shown.

The slider area 105 is an area for enlarging / reducing the translation result image. When the CPU 18 receives the detection signal indicating the position in the slider area 105, the CPU 18 enlarges or reduces the translation result image according to the detection signal.

For example, when the contact position indicated by the detection signal is on the + side from the center in the slider area 105, the CPU 18 enlarges the translation result image, and the contact position indicated by the detection signal is on the − side from the center in the slider area 105. In the case, the translation result image is reduced. At this time, the CPU 18 determines the scaling factor in accordance with the distance from the center of the slider area 105 to the contact position.

FIG. 3 is a diagram illustrating an example of a user interface screen displayed on the display unit 13 in the two-screen display mode.

The user interface screen shown in FIG. 3 includes a one-screen / 2-screen switch display area 101, a pen input switch display area 102, a translation display area 103, a collation display area 104, a slider area 105, and a pen display area 106. And have. In FIG. 3, the same components as those in FIG. 2 are denoted by the same reference numerals, and the description thereof is omitted.

The collation display area 104 is an area for displaying a collation result image. FIG. 3 shows a matching result image corresponding to a captured image obtained by capturing the menu 2.

In the two-screen display mode, since the CPU 18 operates in the image movement mode or the pen input mode, when the detection signal indicating the position in the pen input switch display area 102 is received, the CPU 18 switches the operation mode of the CPU 18 itself. Therefore, every time the pen input switch display area 102 comes into contact with the input means, the operation mode of the CPU 18 is switched between the image movement mode and the pen input mode, so that the pen input switch display area 102 is equivalent to a toggle type switch. It will have a function.

In the pen input mode, the CPU 18 displays the pen display area 106 on the translation display area 103.

When the first detection signal indicates the position on the translation display area 103 after the pen display area 106 is displayed, the CPU 18 matches the tip of the pen display area 106 with the position.

When the collation character range is displayed at the position indicated by the detection signal, the CPU 18 confirms the translated character range corresponding to the collation character range based on the character position information. The CPU 18 draws the same graphic on the collation character range and the translated character range.

For example, when an underline is drawn in the translated character information in the translation display area 103, an underline is drawn in the character information in the collation display area 104.

At this time, the CPU 18 transmits translated coordinate information indicating the size of the translated character range equal to the length of the underline to the computing unit 19. The computing unit 19 calculates the size of the collation character range corresponding to the translated character range based on the character position information from the received translation coordinate information based on the image position conversion formula, and transmits it to the CPU 18. The CPU 18 draws an underline having a length equal to the size of the collation character range.

When the first detection signal indicates the position on the collation display area 104 after the pen display area 106 is displayed, the CPU 18 matches the position of the tip of the pen display area 106 to the pen display area 106. 106 is displayed.

In the image movement mode, when the CPU 18 receives a detection signal indicating the position in the slider area 105, the CPU 18 scales the collation result image and the translation result image according to the detection signal.

For example, when the operator 201 brings the input unit into contact with a position closer to + than the center in the slider area 105, the CPU 18 enlarges the translation result image and positions closer to − than the center in the slider area 105. When the contact is made, the translation result image is reduced. At this time, the CPU 18 determines the scaling factor according to the distance from the center in the slider area 105 to the contact position.

Note that when the display unit 13 is in the two-screen display mode, the CPU 18 can perform pen input to the translation display area 103 and can enlarge or reduce or move an image on the collation display area 104. It is possible to set the operation mode.

FIG. 4 is a flowchart for explaining an example of the operation of the information display apparatus 1 of the present embodiment.

First, the rear camera unit 12 captures an object and generates captured image data (step S301).

The rear camera unit 12 outputs the captured image data to the CPU 18. When accepting the captured image data, the CPU 18 outputs the captured image data to the image collating unit 16 (step S302).

When the image collation unit 16 accepts the captured image data, the image collation unit 16 collates the captured image data with each of the collation image data X1 to Xn in the storage information D1 stored in the storage unit 15, and character information included in the captured image data. Search for collation image data having the same character information. Then, the image matching unit 16 transmits the search result to the CPU 18 (step S303). At this time, if the collation image data can be retrieved, the image collation unit 16 transmits the retrieved collation image data as a search result. If the collation image data cannot be retrieved, the image collation unit 16 retrieves that there is no search image data. Send as a result.

When the CPU 18 receives the search result from the image collating unit 16, the CPU 18 confirms the search result and determines whether there is search image data in the collation image information group D11 (step S304).

When there is no search image data, the CPU 18 displays selection information for requesting selection of whether or not to perform imaging again, and that there is no collation image data having the same character information as the character information included in the captured image data. (Step S313).

Thereafter, when receiving a detection signal from the control signal input unit 14, the CPU 18 confirms the detection signal and determines whether or not to perform imaging again, thereby determining whether or not to end the process (step). S314). CPU18 complete | finishes a process, when not imaging again. On the other hand, when performing imaging again, the process of step S301 is performed.

If there is search image data in step S304, the CPU 18 sets the image movement mode in the CPU 18 itself, and sets the one-screen display mode in the display unit 13. Then, the CPU 18 extracts translation image data corresponding to the search image data from the storage information D1 stored in the storage unit 15, and displays the translation image indicated by the translation image data on the display unit 13 as a translation result image ( Step S305).

When the translation result image is displayed in step S305, the CPU 18 confirms the presence or absence of a detection signal (step S306).

If there is no detection signal, the CPU 18 confirms whether or not there is a pressing signal and determines whether or not the input button unit 11 has been pressed (step S307).

When the input button unit 11 is pressed, the CPU 18 ends the process.

On the other hand, if the input button unit 11 is not pressed, the CPU 18 returns to the process of S306.

If there is a detection signal in step S306, the CPU 18 determines whether or not the 1-screen / 2-screen switch display area 101 is touched based on the detection signal (step S308).

When the 1-screen / 2-screen switch display area 101 is touched in step S308, the CPU 18 switches the display mode of the display unit 13 and displays an image corresponding to the display mode after switching on the display unit 13 (step S309). .

For example, when the display unit 13 enters the single screen display mode, the CPU 18 displays the translation result image indicated by the translation image data extracted in step 305 on the display unit 13.

Further, when the display unit 13 enters the two-screen display mode, the CPU 18 confirms the image position feature information corresponding to the translation result image data in the storage information D1 stored in the storage unit 15. Based on the image position feature information, the CPU 18 associates the display position of the translation reference point in the translation result image data with the display position of the reference point in the matching result image data that is the search image data, and the translation result. The image data and the collation result image data are displayed on the display unit 13.

If the 1-screen / 2-screen switch display area 101 is not touched in step S308, the CPU 18 determines whether or not the operation mode is the pen input mode (step S310).

When the operation mode is the pen input mode, the CPU 18 superimposes and displays the pen display area 106 on the translation result image displayed on the display unit 13 (step S311).

The CPU 18 operates in the image movement mode when not operating in the pen input mode (S312).

CPU18 returns to the process of step S306, when the process of step S309, S311 and S312 is complete | finished.

Note that the storage information D1 may include a plurality of translated image information groups D12 having different translated character information languages. In this case, the CPU 18 selects one translated image information group D12 from the plurality of translated image information groups D12 based on the language used by the operator set in the information display device 1, and the selected translated image information. What is necessary is just to display the translation result image according to group D12. At this time, the CPU 18 may display an area for switching to a translation result image in another language on the display unit 13 so that the operator can select a language.

As described above, according to the present embodiment, the reference point of the collation character range in which the character information in the collation image data having the same character information as the captured image data exists, and the translated image in which the characters of the collation image data are translated A collation image indicated by the collation image data and a translation image indicated by the translation image data are displayed in correspondence with the translation reference point of the translation character range in which the translation character information in the data exists.

For this reason, even if the lengths of the character information included in the collation image and the translated character information included in the translated image change significantly, the character information and the translated character information can be displayed in an easy-to-read manner.

In the present embodiment, the collation image and the translation image are moved and scaled so that the reference point and the translation reference point correspond to each other. Therefore, even when the image is moved or scaled, the character information and the translated character information It becomes possible to make the correspondence easy to understand.

In the present embodiment, the information display device 1 further includes a control signal input unit 14, and the control unit 17 is a collation image or a translation image displayed on the display unit with respect to the control signal input unit 14. When an enlargement / reduction instruction for enlarging / reducing the enlargement / reduction target image is input, the collation image and the translation image are enlarged / reduced around the reference points of the collation image and the translation image in accordance with the enlargement / reduction instruction.

For this reason, the collation image and the translation image are enlarged / reduced with the enlargement / reduction centers associated with each other, so that the character information and the translated character information can be displayed in correspondence with each other. Therefore, it becomes easier for the operator to compare the positions of the character information or the translated character information that the operator wants to enlarge or reduce.

In the present embodiment, in the information display device 1, the character position information further indicates the ratio of the sizes of the character ranges, and when the enlargement / reduction instruction is input, the control unit 17 responds to the enlargement / reduction instruction. The enlargement / reduction target image is enlarged / reduced around a reference point on the enlargement / reduction target image, and a matching image or translation image different from the enlargement / reduction target image is associated with each character range on the display unit based on character position information. The image is scaled around the reference point on the collation image or translation image.

For this reason, since the collation image and the translation image are enlarged / reduced with the character range associated with each other, the character information and the translation character information can be displayed in correspondence with each other. Therefore, it becomes easier for the operator to compare the character ranges that the operator wants to see.

In the present embodiment, the information display device 1 further includes a control signal input unit 14, the character position information further indicates a ratio between the size of the character range and the size of the translated character range, and the control unit 17 When a drawing instruction for drawing in the target character range that is the collation character range or the translated character range is input to the control signal input unit 14, a figure corresponding to the drawing instruction is drawn in the target character range. And, based on the character position information, the graphic is drawn in a character range different from the target character range.

For this reason, since the collation image and the translation image are drawn with the character ranges associated with each other, the correspondence between the character information and the translation character information can be clarified.

(Second Embodiment)
FIG. 5 is a diagram illustrating a configuration of the information display device of the present embodiment.

5 includes an input button unit 11, a rear camera unit 12, a display unit 13, a control signal input unit 14, a storage unit 15, an image matching unit 16, and a front camera unit 41. And a control unit 42. The control unit 42 includes a CPU 43 and a calculation unit 19. That is, the information display device 4 is different from the information display device 1 according to the first embodiment shown in FIG. 1 in that a front camera unit 41 is added and the CPU 18 is replaced with a CPU 43.

The front camera unit 41 is a second imaging unit that performs imaging and outputs second captured image data. Specifically, the front camera unit 41 is provided on the front surface of the information display device 4 on which the display unit 13 is provided. The front camera unit 41 captures a person-captured image data obtained by capturing a person on the front side and copying the person. 2 as image data.

The CPU 43 has the same function as the CPU 18 shown in FIG. 1, and further has the following functions.

The CPU 43 specifies the number of persons included as subjects in the person captured image indicated by the person captured image data output from the front camera unit 41. Specifically, the CPU 43 detects the face of a person included as a subject in the person captured image, and specifies the number of faces as the number of persons.

And CPU43 displays either one or both of a translation result image and a collation result image on the display part 13 according to the specified number of people.

Specifically, when the number of persons is one, the CPU 43 sets the display mode of the display unit 13 to the single screen display mode, and displays either the translation result image or the collation result image on the display unit 13.

On the other hand, when there are a plurality of persons, the CPU 43 determines whether or not there is a person facing each other based on the detection result, and displays the collation result image and the translation result image in the direction according to the determination result. To do.

For example, when there is a person facing each other, the CPU 43 sets the display mode of the display unit 13 to the two-screen facing display mode. The two-screen facing display mode is a display mode for displaying the matching result image and the translation result image so that the matching result image and the translation result image are opposite to each other.

When there is no facing person, the CPU 43 sets the display mode of the display unit 13 to the two-screen display mode, and displays both the translation result image and the collation result image in the same direction.

FIG. 6 is a diagram illustrating an example of a user interface screen displayed on the display unit 13 in the two-screen facing display mode.

As shown in FIG. 6, in the two-screen facing display mode, the CPU 43 displays the matching result image and the translation result image on the display unit 13 so that the matching result image and the translation result image are opposite to each other. The collation result image and the translation result image can be displayed in directions that are easy for the operator 201 and the viewer 202 facing the operator 201 to view.

Thus, for example, when a menu is imaged, a translation result image menu is displayed for the operator 201 and a collation result image menu is displayed for the viewer 202 who is a waiter. Therefore, when the operator 201 who is the orderer switches the screen to the pen input mode and underlines the name of the dish to be ordered on the menu in the translation result image, the name of the menu in the matching result image facing the waiter is underlined. Will be drawn and will definitely be able to place an order.

FIG. 7 is a flowchart showing a part of the operation of the information display device 4 of the present embodiment.

In the operation of the information display device 4 of the present embodiment, the processes of steps S401 to S406 shown in FIG. 7 are inserted between the processes of steps S304 and S305 in the flowchart of FIG. However, the processing of the CPU 18 in each step of the flowchart of FIG.

In step S304, if there is search image data in the collation image information group D11, the CPU 43 drives the front camera unit 41 to cause the front camera unit 41 to image a person. The front camera unit 41 transmits person-captured image data indicating a person-captured image obtained by capturing the person to the CPU 43 (step S401).

When receiving the human captured image data, the CPU 43 detects a human face included as a subject in the human captured image indicated by the human captured image data (step S402).

The CPU 43 identifies the number of persons based on the detection result, and determines whether the number is one (step S403).

If the number of persons is one, the CPU 43 proceeds to the process of step S305. On the other hand, when the number of persons is not one, the CPU 43 determines whether or not there is a person facing each other (step S404).

When there is a person facing each other, the CPU 43 displays the translation result image and the collation result image so that the translation result image faces in the direction viewed by the operator 201 and the collation result image faces in the direction viewed by the viewer 202. 13 (step S405).

When there is no facing person, the CPU 43 displays the translation result image and the collation result image on the display unit 13 in the same direction (step S406).

In addition, step S306 is performed when step S405 and step S406 are complete | finished.

As described above, according to the present embodiment, in the information display device 4, the front camera unit 41 images a person and transmits the person captured image data, and the control unit 42 indicates the person captured image data. A person's face included as a subject is detected in the person-captured image, the number of persons is specified based on the detection result, and either one of the translated image and the collation image is displayed on the display unit 13 according to the number of persons. Display both.

Therefore, since the display mode can be switched according to the number of operators or the like, the character information or the translated character information can be displayed according to the number of operators and can be used according to the usage of the operator. .

In the present embodiment, in the information display device 4, when there are a plurality of persons, the control unit 42 determines the orientation of the translated image or the collation image depending on whether or not there are a plurality of persons facing each other. Instead, the translated image and the collation image are displayed.

For this reason, the character information and the translated character information are displayed in an easy-to-read manner depending on whether the operator or the viewer is facing the user, so that the convenience of the operator or the viewer to communicate can be improved. It becomes possible.

(Third embodiment)
FIG. 8 is a diagram showing the configuration of the information display system of this embodiment.

The information display system shown in FIG. 8 includes a collation device 3 and an information display device 5. In FIG. 8, the same components as those in FIG. 1 are denoted by the same reference numerals, and description thereof may be omitted.

The collation device 3 includes a communication unit 30, a storage unit 31, a CPU 32, and an image collation unit 33. The information display device 5 includes an input button unit 11, a rear camera unit 12, a display unit 13, a control signal input unit 14, a communication unit 51, a storage unit 52, and a control unit 53. The control unit 53 includes a CPU 54 and a calculation unit 19.

The communication unit 51 of the information display device 5 communicates with the verification device 3. For example, the communication unit 51 transmits the captured image data output from the rear camera unit 12 to the verification device 3. Further, the communication unit 51 receives search information corresponding to the transmitted captured image data from the verification device 3. The search information is, for example, collation image data corresponding to the captured image data, translation image data, and character position information.

The storage unit 52 temporarily stores the search information received by the communication unit 51.

The CPU 54 displays the collation result image data and the translation result image data based on the search information stored in the storage unit 52.

The collation device 3 collates the captured image data with the image data stored in advance in the database, and searches for image data having character information that matches the character information in the captured image data.

The communication unit 30 of the verification device 3 communicates with the information display device 5. For example, the communication unit 30 receives captured image data from the information display device 5.

The storage unit 31 stores the storage information D1 similarly to the storage unit 15 of the first embodiment.

The image collation unit 33 collates the captured image data received by the communication unit 30 with each of the collation image data X1 to Xn in the storage information D1 stored in the storage unit 31, and character information included in the captured image data Search for collation image data having the same character information. Then, the image collation unit 33 acquires the retrieved collation image data from the storage unit 31 and outputs it to the CPU 32.

CPU32 will acquire the translation image data and character position information corresponding to the collation image data from the memory | storage part 31, if collation image data is received. And CPU32 transmits the collation image data, translation image data, and character position information to the information display apparatus 5 via the communication part 30 as search information.

FIG. 9 is a sequence diagram showing a part of the operation of the information display device 5 of the present embodiment.

The operation of the information display device 5 of the present embodiment shown in FIG. 9 is the same as that in the flowchart shown in FIG. 4 except that the processing between step S301 and step S304 is shown in the sequence diagram shown in FIG. 4 is the same as the operation shown in the flowchart shown in FIG. However, CPU 18 in the flowchart shown in FIG. 4 is replaced with CPU 54, and storage unit 15 is replaced with storage unit 53.

In the flowchart shown in FIG. 4, the CPU 54 proceeds to the process of step S501 after the process of step S301 is completed.

The rear camera unit 12 transmits the captured image data to the CPU 54. The CPU 54 receives the captured image data and transmits it to the communication unit 60 (step S501).

The communication unit 60 receives the captured image data and transmits it to the communication unit 30 of the verification device 3 (step S502).

The communication unit 30 receives the captured image data and transmits it to the image collation unit 33 via the CPU 32 (step S503).

The image collation unit 33 collates the captured image data generated by the rear camera unit 12 with each of the collation image data X1 to Xn in the storage information D1 stored in the storage unit 31, and the characters included in the captured image data The collated image data having the character information that matches the information is searched. The image collation unit 33 then outputs the retrieved collation image data to the CPU 32 (step S504).

CPU32 will acquire the translation image data and character position information corresponding to the collation image data from the memory | storage part 31, if collation image data is received. Then, the CPU 32 transmits the collation image data, translation image data, and character position information as search information to the communication unit 30 (step S505).

The communication unit 30 receives the search information and transmits the search information to the communication unit 51 of the information display device 5. The communication unit 51 receives the search information (step S506).

The communication unit 51 transmits the search information to the CPU 54. The CPU 54 receives the search information and temporarily stores the search information in the storage unit 52 (step S507).

CPU 54 proceeds to the process of step S304 after the process of step S507 is completed.

As described above, according to the present embodiment, the collation device 3 different from the information display device 5 stores the stored information D and collates the captured image data and the collation image data. It becomes possible to collate data with more image data.

The functions of the information display devices 1, 4, 5 and the collation device 3 described above are recorded on a computer-readable recording medium by recording a program for realizing the functions. The program may be read by a computer and executed.

In each embodiment described above, the illustrated configuration is merely an example, and the present invention is not limited to the configuration.

This application claims priority based on Japanese Patent Application No. 2012-22087 filed on Feb. 3, 2012, the entire disclosure of which is incorporated herein.

Claims (10)

A display unit;
An imaging unit that performs imaging and outputs captured image data;
Collation image data indicating a collation image including character information, translation image data indicating a translation image including translation character information obtained by translating the character information, and a collation character range in which the character information on the collation image exists. A storage unit for storing character position information indicating a correspondence relationship between a reference point and a reference point within a translated character range where the translated character information on the translated image exists;
A collation unit that collates the captured image data with the collation image data and determines whether or not the character information is included in the captured image indicated by the captured image data;
When the captured image includes the character information, the reference image and the translated image are displayed on the display by associating the reference points of the collated character range and the translated character range with each other based on the character position information. And an information display device. The information display device according to claim 1,
A control signal input unit;
When the movement instruction to move the movement target image that is the displayed collation image or the translated image is input to the control signal input unit, the control unit moves the movement target according to the movement instruction. The image is moved, and based on the character position information, the collation image or the translation image different from the movement target image is moved so that the reference points of the collation character range and the translation character range correspond to each other. Information display device. The information display device according to claim 1 or 2,
A control signal input unit;
When the enlargement / reduction instruction to enlarge / reduce the enlargement / reduction image that is the collation image or the translation image displayed on the display unit is input to the control signal input unit, the control unit, according to the enlargement / reduction instruction, An information display device that enlarges / reduces each of the collation image and the translation image around a reference point of the collation image and the translation image. The information display device according to claim 3,
The character position information further indicates a size ratio of the collation character range and the translated character range,
The control unit enlarges / reduces the enlargement / reduction object image according to the enlargement / reduction instruction, and, based on the character position information, converts the collation image or the translation image different from the enlargement / reduction object image, the collation character range and the An information display device that expands and contracts so that each translated character range corresponds. The information display device according to any one of claims 1 to 4,
A control signal input unit;
The character position information further indicates a size ratio of the collation character range and the translated character range,
When a drawing instruction for drawing within the target character range that is the collation character range or the translated character range is input to the control signal input unit, the control unit displays a graphic corresponding to the drawing instruction as the target character. An information display device that draws within a range and draws the figure in the collated character range or the translated character range different from the target character range based on the character position information. The information display device according to any one of claims 1 to 5,
A second imaging unit that performs imaging and outputs second captured image data;
The control unit specifies the number of persons included as a subject in the second captured image indicated by the second captured image data, and either one or both of the translated image and the collation image according to the number of persons. Is displayed on the display unit. The information display device according to claim 6,
The control unit determines whether there is a person facing each other when there are a plurality of persons, and displays each of the translated image and the matching image in a direction according to the determination result. apparatus. An information display system having a verification device and an information display device,
The verification device is
Collation image data indicating a collation image including character information, translation image data indicating a translation image including translation character information obtained by translating the character information, and a collation character range in which the character information on the collation image exists. A storage unit for storing character position information indicating a correspondence relationship between a reference point and a reference point within a translated character range where the translated character information on the translated image exists;
A first communication unit that receives captured image data from the information display device;
A collation unit that collates the captured image data with the collation image data and determines whether or not the character information is included in the captured image indicated by the captured image data;
When the character information is included in the captured image, a first control that acquires the collation image data, the translated image data, and the character position information from the storage unit and transmits the information as search information to the information display device. And
The information display device includes:
A display unit;
An imaging unit that performs imaging and outputs the captured image data;
A second communication unit that transmits the captured image data to the verification device and receives the search information from the verification device;
Based on the search information received by the second communication unit, the reference image and the translated image are displayed on the display unit in correspondence with the reference points of the collated character range and the translated character range. And an information display system. Collation image data indicating a collation image including character information, translation image data indicating a translation image including translation character information obtained by translating the character information, and a collation character range in which the character information on the collation image exists. Storing character position information indicating a correspondence relationship between a reference point and a reference point within a translated character range where the translated character information exists on the translated image;
Take an image, output the captured image data,
Collating the captured image data with the collated image data to determine whether the character information is included in the captured image indicated by the captured image data;
When the character information is included in the captured image, based on the character position information, the reference image and the translated image are displayed in correspondence with the reference points of the collated character range and the translated character range. Information display method. Collation image data indicating a collation image including character information, translation image data indicating a translation image including translation character information obtained by translating the character information, and a collation character range in which the character information on the collation image exists. A computer connected to a storage unit for storing a reference point and character position information indicating a correspondence relationship between a reference point in a translated character range where the translated character information exists on the translated image,
A procedure for performing imaging and outputting captured image data;
A procedure for collating the captured image data with the collation image data and determining whether or not the character information is included in a captured image indicated by the captured image data;
When the character information is included in the captured image, based on the character position information, the reference image and the translated image are displayed in correspondence with the reference points of the collated character range and the translated character range. A program for executing the procedure.

Images