WO2025127023A1 - Information processing device and information processing system - Google Patents

Abstract

[Problem] To perform training to facilitate switching of a sensor modality, or, to execute information processing using training results. [Solution] This information processing device comprises a calculation unit. The calculation unit: acquires first data output by a first sensor and second data output by a second sensor that acquires information of a type different from that of the first sensor; inputs the first data to a first converter to acquire a first intermediate expression; inputs the second data to a second converter to acquire a second intermediate expression; and trains the second converter on the basis of the first intermediate expression and the second intermediate expression.

Description

Information processing device and information processing system

This disclosure relates to an information processing device and an information processing system.

Neural network models trained by various machine learning techniques are used in various fields. The trained models may be used to process and analyze sensor data acquired from various sources, e.g., various sensors. This analysis typically involves a specific processing algorithm tuned to the respective sensor modality, and the trained models are trained to infer the results of this specific processing algorithm.

Methods for integrating sensor data from different modalities require the development of both custom algorithms to align and combine the data, as well as pre-processing techniques. This development often requires domain expertise and extensive manual engineering. In addition, because the intermediate representation of the data is different, it is virtually impossible to change only the sensor and use the same trained model on the same processor for analysis. To address this, incorporating a new sensor or changing the sensor modality will likely require significant retraining of the neural network model with a combined (labeled) dataset to adapt to the new input data, making robust processing difficult to achieve.

JP 2023-062217 A

Therefore, one non-limiting problem that the embodiments of the present disclosure aim to solve is information processing that learns to facilitate switching between sensor modalities or utilizes the learning results. The problem that the embodiments of the present disclosure aim to solve can also be, as some further non-limiting examples, a problem that corresponds to the effects described in the embodiments. In other words, a problem that corresponds to at least one of the effects described in the description of the embodiments of the present disclosure can be the problem that the present disclosure aims to solve.

According to one embodiment, the information processing device includes a calculation unit.
The calculation unit is
Acquire first data output from a first sensor and second data output from a second sensor that acquires information of a different type from that of the first sensor;
inputting the first data into a first transformer to obtain a first intermediate representation;
inputting the second data into a second transformer to obtain a second intermediate representation;
The second transformer is trained based on the first intermediate representation and the second intermediate representation.
According to this embodiment, the information processing device can utilize a model trained using a data set of the first sensor for data acquired from the second sensor, making it possible to perform processing by simply replacing the sensor without changing the configuration of the information processing device.

The first intermediate representation may be an intermediate representation capable of executing a first process when input to a first model. The intermediate representation may also be referred to as a latent representation, for example. The above embodiment can be implemented by learning a converter that results in the same intermediate representation for outputs from a first sensor and a second sensor that have the same characteristics.

The first model may be a model trained to execute the first processing when the first intermediate representation is input. The first processing may be, for example, image processing or signal processing such as object detection, classification, segmentation, etc.

The first intermediate representation may be an intermediate representation capable of executing a second process when input to a second model. The first intermediate representation can also be used as an input to a second model that executes a second process different from the first process. According to the above embodiment, it is also possible to input the second intermediate representation to the second model and obtain the results of executing the second process. In other words, it is possible to replace the sensor, or to replace the model that performs the process.

The second model may be a model trained to execute the second process when the first intermediate representation is input. Like the first model, the second model may also be a model trained using a data set of output from a first sensor.

The calculation unit may obtain the first data and the second data by acquiring information of the same object in the same environment using the first sensor and the second sensor, respectively, or may input the first data to the first converter to acquire the first intermediate representation, or may input the second data to the second converter to acquire the second intermediate representation, and may calculate the loss between the first intermediate representation and the second intermediate representation, update the parameters of the second converter, and optimize the second converter. By acquiring information of the same object in the same environment using sensors that acquire different types of information, it is possible to acquire multiple types of sensory information that may have the same characteristics. By optimizing the converter so that the difference between the intermediate representations of the acquired information is reduced, it is possible to refine a converter that converts sensor outputs having the same characteristics into the same intermediate representation.

The calculation unit may input the first data to a third converter to obtain a third intermediate representation, may input the second data to a fourth converter to obtain a fourth intermediate representation, and may perform learning of the fourth converter based on the third intermediate representation and the fourth intermediate representation. In this way, the converter is not limited to being of one type.

The third intermediate representation may be an intermediate representation capable of executing a third process when input to a third model. For example, a converter different from the converter that obtains data to be input to the first model may be used for a third model that executes a third process different from the first process.

The third model may be a model trained to execute the third process when the third intermediate representation is input. Like the first model, the third model may also be a model trained using a data set of output from the first sensor.

The calculation unit may obtain information of the same object in the same environment using the first sensor and the second sensor, respectively, to obtain the first data and the second data, may input the first data to the third converter to obtain the third intermediate representation, may input the second data to the fourth converter to obtain the fourth intermediate representation, may calculate the loss between the third intermediate representation and the fourth intermediate representation, update the parameters of the fourth converter, and optimize the fourth converter. For the third model, multiple types of sensing information that will likely have the same characteristics can be obtained. By optimizing the converter so that the difference between the intermediate representations of this obtained information is reduced, a converter can be refined to convert sensor outputs having the same characteristics into the same intermediate representation.

The first converter and the second converter may be models related to domain adaptation. By using a domain adaptation model, it is possible to align data in different domains, such as sensor sensing information, into an equivalent intermediate representation.

The first converter and the second converter may be the same model. Converters that convert data acquired from different types of sensors into intermediate representations may use the same model. In this case, the model related to the converter may be optimized so that only the second intermediate representation changes, without changing the first intermediate representation. Furthermore, if the first intermediate representation changes, the first model and/or the second model may be subsequently re-learned.

According to one embodiment, the information processing device includes a calculation unit.
The calculation unit is
acquiring second data output by a second sensor that acquires information of a different type from that of the first sensor;
converting the second data to obtain a second intermediate representation;
The second intermediate representation is
a first model that executes a first process when a first intermediate representation obtained by converting first data output by the first sensor is input;
and acquiring a result of executing the first process on the second data.
Information processing device.
This information processing device is capable of performing processing (including inference) using a transducer obtained by any of the methods described above.

The computing unit may input the second intermediate representation to a second model that executes a second process when the first intermediate representation is input, and obtain a result of executing the second process on the second data. As described above, a converter can convert the second intermediate representation into a first intermediate representation that has the same characteristics in processing.

The calculation unit may convert the second data to obtain a fourth intermediate representation, input the fourth intermediate representation to a third model that executes a third process when a third intermediate representation obtained by converting the first data is input, and obtain a result of executing the third process on the second data. As above, different converters may be used for models that execute different processes.

According to one embodiment, an information processing system includes a first sensor, a second sensor that acquires a different type of information from the first sensor, and an information processing device that optimizes a converter described in any of the above.
The information processing device includes:
A converter that converts first data acquired from the first sensor into a first intermediate representation and second data acquired from the second sensor into a second intermediate representation is optimized based on the first intermediate representation converted from the first data and the second intermediate representation converted from the second data.
The information processing system includes a plurality of sensors that obtain different types of information, and an information processing device that optimizes the converter, and the converter can be trained by the information processing device.

The transformation to obtain the first intermediate representation and the transformation to obtain the second intermediate representation may be models related to domain application.

The transformation to obtain the first intermediate representation and the transformation to obtain the second intermediate representation may be performed using the same model.

According to one embodiment, an information processing device includes a first sensor, a second sensor that acquires a different type of information from the first sensor, and an information processing device capable of using a model for the different sensor described above.
The information processing device includes:
A first model executes a first process when a first intermediate representation obtained by converting first data acquired from the first sensor is input, and a second intermediate representation obtained by converting second data acquired from the second sensor is input to a first model to obtain a result of executing the first process on the second data.
The information processing system may include a plurality of sensors that acquire different types of information, and an information processing device having a converter trained by the information processing system or information processing device. This information processing system can realize processing (including inference) using information acquired from the different sensors.

1 is a schematic block diagram showing an example of the overall configuration of an information processing system according to an embodiment. A block diagram showing an example of the configuration of each system in an information processing system according to one embodiment, which registers or downloads an artificial intelligence (AI) model or AI application through the operation of an information processing device on the cloud side. 1 is a flowchart showing an example of at least a part of a process of an information processing system according to an embodiment. FIG. 1 is a block diagram showing an example of the internal configuration of a camera as an imaging apparatus according to an embodiment. FIG. 1 is a schematic diagram showing an example of the configuration of an image sensor as an imaging device according to an embodiment. FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment. 10 is a flowchart showing a process of an information processing device according to an embodiment. FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment. FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment. FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment.

Below, an embodiment of the present disclosure will be explained with reference to the drawings. The drawings are used for explanatory purposes, and the shape, size, or size ratio of each component in the actual device to other components does not necessarily have to be as shown in the drawings. In addition, since the drawings are simplified, components necessary for implementation other than those shown in the drawings are assumed to be appropriately included.

(First embodiment)

[Information processing system configuration]

(1) Overall configuration of the information processing system

FIG. 1 shows an example of a schematic configuration of an information processing system 100 that constitutes an imaging device system according to the first embodiment. In other words, the information processing system 100 can be an example of a system to which the present invention is applied.

As shown in FIG. 1, the information processing system 100 according to the first embodiment includes at least a cloud server 1, a user terminal 2, a plurality of cameras 3 as imaging devices, a fog server 4, and a management server 5. Here, at least the cloud server 1, the user terminal 2, the fog server 4, and the management server 5 are configured to be able to communicate with each other via a network 6, such as the Internet.

The cloud server 1, user terminal 2, fog server 4 and management server 5 are all configured as information processing devices equipped with a microcomputer having a CPU, ROM (Read Only Memory) and RAM (Random Access Memory).

The camera 3 as an imaging device is equipped with an image sensor such as a CCD (Charge Coupled Device) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor. These image sensors form an imaging section (see reference numeral 41 in Figure 4). The camera 3 captures an image of a subject and obtains image information (captured image information) as digital data. The camera 3 also has the function of performing processing of the captured image using AI. Examples of this processing include image recognition processing and image detection processing.

In the following explanation, various types of processing on images, such as image recognition processing and image detection processing, will simply be referred to as "image processing." For example, various types of processing on images using AI or AI models will be referred to as "AI image processing."

The multiple cameras 3 are configured to be able to communicate data with the fog server 4. For example, various data such as processing result information showing the results of image processing using AI is transmitted from the cameras 3 to the fog server 4. In addition, the cameras 3 receive various data from the fog server 4.

Here, the information processing system 100 is assumed to be used in the following ways, for example. First, the fog server 4 or cloud server 1 generates analytical information about the subject based on the processing result information obtained by image processing of the multiple cameras 3. This generated analytical information can be viewed by the user via the user terminal 2.

In this case, the multiple cameras 3 are used as surveillance cameras. For example, they can be used as surveillance cameras for monitoring the interior of stores, offices, homes, etc., or for monitoring the exterior of parking lots, city streets, etc. Examples of surveillance cameras for monitoring the exterior include traffic surveillance cameras for monitoring traffic conditions. They can also be used as surveillance cameras for monitoring manufacturing lines such as FA (Factory Automation) and IA (Industrial Automation). Furthermore, they can also be used as surveillance cameras for monitoring the interior or exterior of automobiles, trains, etc.

In addition, for use as a surveillance camera in a store, multiple cameras 3 can be placed at predetermined locations within the store. Using multiple cameras 3 allows the user to check the customer demographics (gender, age group, etc.) of customers visiting the store and their behavior (traffic flow) within the store.
In this case, information on the customer demographics of customers, the flow of customers within the store, and congestion at the cash registers (for example, waiting times at the cash registers) can be generated as analysis information.

Furthermore, for traffic monitoring purposes, multiple cameras 3 can be placed at various locations near the road. Using multiple cameras 3 allows the user to recognize information such as the license plate number (vehicle number), color, and model of the vehicle of passing vehicles.

In this case, information such as license plate number, vehicle color, model, etc. can be generated.

Furthermore, if the camera is intended to be used as a surveillance camera in a parking lot, the camera 3 can be placed in a position where it can monitor parked vehicles. The camera 3 can be used to monitor, for example, whether there are any suspicious individuals behaving suspiciously around the vehicles. Furthermore, if a suspicious individual is detected, a notification device can be provided that notifies the user of the presence of the suspicious individual and their attributes (gender or age group), etc.

In addition, if the camera is used for monitoring available spaces in a city or parking lot, it can notify the user of the location of spaces where they can park their car.

For example, in the above-mentioned store surveillance application, the fog server 4 is placed in the store to be monitored together with multiple cameras 3. In other words, a fog server 4 is placed for each monitored object.

In this way, when a fog server 4 is placed for each monitored object such as a store, the cloud server 1 does not need to directly receive data transmitted from multiple cameras 3 at the monitored object. As a result, the processing burden on the cloud server 1 can be reduced.

In addition, if there are multiple stores to be monitored and all of the multiple stores belong to the same chain, it is preferable to place a fog server 4 for each set of stores, rather than for each individual store. In other words, it is not limited to placing one fog server 4 for each monitored object, and it is possible to place one fog server 4 for each set of monitored objects.

In addition, if the cloud server 1 or the multiple cameras 3 have processing capabilities, the cloud server 1 or the multiple cameras 3 can have the functions of the fog server 4. This allows the information processing system 100 to omit the fog server 4 and connect the multiple cameras 3 directly to the network 6, allowing the cloud server 1 to directly receive data transmitted from the multiple cameras 3.

The above various devices are broadly divided into cloud-side information processing devices and edge-side information processing devices.

The information processing devices on the cloud side include cloud server 1 and management server 5. The information processing devices on the cloud side are a group of devices that provide services that are expected to be used by multiple users.

In addition, the edge-side information processing devices include the camera 3 and the fog server 4. The edge-side information processing devices are a group of devices prepared by users who use cloud services and placed within the environment.

However, both the cloud-side information processing device and the edge-side information processing device may be placed in an environment prepared by the same user.

Fog server 4 may be an on-premise server.

(2) Registration of AI models and AI applications

As described above, in the information processing system 100, AI image processing is performed in the camera 3, which is an edge-side information processing device. Then, in the cloud server 1, which is a cloud-side information processing device, advanced application functions are realized using the result information of the AI image processing on the edge side. The result information of the AI image processing is, for example, result information of image recognition processing using AI.

Here, the various methods for registering application functions in the cloud server 1, which is an information processing device on the cloud side, or the cloud server 1 including the fog server 4, are as follows:

Figure 2 shows an example of the configuration of each device in the information processing system 100 that registers or downloads AI models and AI applications via the marketplace function provided in the cloud-side information processing device.

In FIG. 2, the fog server 4 is not shown, but the fog server 4 may be provided. In this case, the fog server 4 may take on some of the functions of the edge side.

The above-mentioned cloud server 1 and management server 5 are information processing devices that make up the cloud environment.

Camera 3 is an information processing device that constitutes the edge environment.

The camera 3 can be constructed as a device equipped with a control unit that performs overall control of the camera 3. The camera 3 can also be constructed as a device equipped with an image sensor IS that has an arithmetic processing unit that performs various types of processing, including AI image processing, on captured images. In other words, the camera 3, which is an edge-side information processing device, may be equipped with an image sensor IS, which is another edge-side information processing device.

Furthermore, the user terminals 2 used by users who use various services provided by the cloud-side information processing device include an application developer terminal 2A, an application user terminal 2B, an AI model developer terminal 2C, etc.

The application developer terminal 2A is used by users who develop applications used in AI image processing. The application user terminal 2B is used by users who use applications. The AI model developer terminal 2C is used by users who develop AI models used in AI image processing.

In addition, the application developer terminal 2A may be used by a user who develops applications that do not use AI image processing.

The information processing device on the cloud side has prepared learning data sets for AI learning. A user developing an AI model communicates with the information processing device on the cloud side using the AI model developer terminal 2C and downloads these learning data sets.

In this case, the training dataset may be provided for a fee. For example, an AI model developer may purchase a training dataset in a state where they can purchase various functions and materials registered in a marketplace (electronic marketplace) by registering personal information in the marketplace provided as a function on the cloud side.

After developing an AI model using the learning dataset, the AI model developer uses the AI model developer terminal 2C to register the developed AI model in the marketplace. This may result in an incentive being paid to the AI model developer when the AI model is downloaded.

Furthermore, a user who develops an application can use the application developer terminal 2A to download an AI model from the marketplace and develop an application that uses this AI model (hereinafter simply referred to as an "AI application"). At this time, as described above, an incentive may be paid to the AI model developer.

A user who develops an application registers the developed AI application in the marketplace using the application developer terminal 2A. This may allow an incentive to be paid to the user who developed the AI application when the AI application is downloaded.

A user who uses an AI application uses the application user terminal 2B to perform operations to deploy the AI application and AI model from the marketplace to a camera 3 as an edge-side information processing device that the user manages. At this time, an incentive may be paid to the AI model developer.

This makes it possible for Camera 3 to carry out AI image processing using AI applications and AI models. Specifically, in addition to capturing images, it will be possible to use AI image processing to detect customers and vehicles.

Here, deployment of an AI application and an AI model refers to installing the AI application or AI model in a target (device) acting as an execution subject so that the target can use the AI application or AI model. Furthermore, deployment includes installation in a target acting as an execution subject so that at least a portion of the program as an AI application can be executed.

In addition, in camera 3, attribute information of customers may be extracted from images captured by camera 3 using AI image processing.

This attribute information is sent from the camera 3 via the network 6 to an information processing device on the cloud side.

Cloud applications are deployed on the information processing device on the cloud side. Each user is able to use the cloud applications via network 6. Among the cloud applications are applications that analyze the movement of customers using their attribute information and captured images. Such cloud applications are uploaded by application development users, etc.

　A user of the application uses the cloud application for flow analysis using the application user terminal 2B. This enables the user to analyze the flow of customers visiting their own store and view the analysis results of this analysis. Viewing the analysis results refers to viewing the flow of customers visiting the store that is presented graphically on a map of the store, for example.

The results of the traffic flow analysis may also be displayed in the form of a heat map, and the density of customers visiting the store may be presented, allowing the analysis results to be viewed.

In addition, the information may be displayed in a categorized manner according to the attributes of customers.

In the marketplace on the cloud side, AI models optimized for each user may be registered. For example, images captured by a camera 3 installed in a store managed by a certain user are uploaded and stored in an information processing device on the cloud side as appropriate.

In the information processing device on the cloud side, each time a certain number of uploaded captured images are accumulated, a re-learning process is performed for the AI model, and the AI model is updated and re-registered in the marketplace.

In addition, the re-learning process for the AI model may be made available as an option to users on the marketplace, for example.

For example, an AI model that has been retrained using dark images from a camera 3 placed inside a store is deployed to that camera 3. This makes it possible to improve the recognition rate, etc. of image processing for images captured in dark places. Also, an AI model that has been retrained using bright images from a camera 3 placed outside the store is deployed to that camera 3. This makes it possible to improve the recognition rate, etc. of image processing for images captured in bright places.

In other words, users of the application can always obtain optimized processing result information by redeploying the updated AI model to Camera 3 again.

The re-learning process for the AI model will be explained later.

In addition, if personal information is included in the information (e.g., information on captured images, etc.) uploaded from the camera 3 to the information processing device on the cloud side, the data may be uploaded with the privacy information deleted from the perspective of protecting privacy. The data with the privacy information deleted may be made available to users developing AI models and users developing applications.

The AI model developer terminal 2C is an information processing device used by the AI model developer.

The software developer terminal 7 is an information processing device used by the developer of the AI application.

Figure 3 is a flowchart showing an example of the process described above.

The information processing device on the cloud side corresponds to the cloud server 1, management server 5, etc. shown in Figure 1.

The AI model developer uses an AI model developer terminal 2C having a display unit which may include an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel to view a list of datasets registered in the marketplace. When the AI model developer selects the desired dataset, in response to this selection, the AI model developer terminal 2C transmits a download request for the selected dataset to an information processing device on the cloud side (step S21).

The information processing device on the cloud side accepts the request (step S1). The information processing device on the cloud side performs processing to send the requested data set to the AI model developer terminal 2C (step S2).

The AI model developer terminal 2C performs a process to receive the dataset (step S22). This enables the AI model developer to develop an AI model using the dataset.

After the AI model developer has finished developing the AI model, the AI model developer performs an operation to register the developed AI model in the marketplace. This operation is, for example, an operation of specifying the name of the AI model and the address where the AI model is located. As a result, the AI model developer terminal 2C transmits a request to register the AI model in the marketplace to the information processing device on the cloud side (step S23).

The information processing device on the cloud side receives the registration request (step S3). The information processing device on the cloud side performs registration processing for the AI model (step S4). The information processing device on the cloud side can, for example, display the AI model on a marketplace. This allows users other than the AI model developer to download the AI model from the marketplace.

For example, an application developer who wishes to develop an AI application uses application developer terminal 2A to view a list of AI models registered in the marketplace. In response to an operation by the application developer, application developer terminal 2A transmits a download request for the selected AI model to an information processing device on the cloud side (step S31). The operation here is, for example, an operation to select one of the AI models on the marketplace.

The information processing device on the cloud side accepts the request (step S5) and transmits the AI model to the application developer terminal 2A (step S6).

The application developer terminal 2A receives the AI model (step S32). This enables the application developer to develop an AI application that uses an AI model developed by another person.

After the application developer has finished developing the AI application, he or she performs an operation to register the AI application in the marketplace. This operation involves, for example, specifying the name of the AI application and the address where the AI model is located. As a result, the application developer terminal 2A sends a registration request for the AI application to the information processing device on the cloud side (step S33).

The information processing device on the cloud side accepts the registration request (step S7). The information processing device on the cloud side registers the AI application (step S8). The information processing device on the cloud side can, for example, display the AI application on a marketplace. This allows users other than the application developer to select and download the AI application on the marketplace.

(3) System Functionality Overview

In the first embodiment, a service using the information processing system 100 is envisioned in which a user as a customer can select a function type for AI image processing of multiple cameras 3. For example, an image recognition function, an image detection function, etc. may be selected as the function type, or a more detailed type may be selected to perform an image recognition function, an image detection function, etc. for a specific subject.

For example, as a business model, a service provider sells cameras 3 and fog servers 4 equipped with AI image recognition functions to users, and has the users install the cameras 3 and fog servers 4 in locations to be monitored. The service provider then develops a service that provides users with analytical information such as that described above.

At this time, each customer has a different need for the system, such as store monitoring or traffic monitoring. For this reason, the AI image processing functions of Camera 3 can be selectively set to obtain analytical information that corresponds to the customer's desired use.

In the first embodiment, the management server 5 has a function for selectively setting the AI image processing function of such a camera 3.

In addition, the functions of the management server 5 may be provided by the cloud server 1 or the fog server 4.

(4) Configuration of the imaging device

Figure 4 shows an example of the internal configuration of camera 3.

As shown in FIG. 4, the camera 3 comprises an imaging optical system 31, an optical system driving section 32, an image sensor IS, a control section 33, a memory section 34, and a communication section 35. The image sensor IS, control section 33, memory section 34, and communication section 35 are each connected via a bus 36, enabling data communication between them.

The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, as well as an aperture (iris) mechanism. This imaging optical system 31 guides light from the subject (incident light), and the light is focused on the light receiving surface of the image sensor IS.

The optical system driving unit 32 collectively refers to the driving units for the zoom lens, focus lens, and aperture mechanism of the imaging optical system 31. Specifically, the optical system driving unit 32 has actuators and actuator driving circuits for driving the zoom lens, focus lens, and aperture mechanism.

The control unit 33 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and performs overall control of the camera 3 by the CPU executing various processes according to programs stored in the ROM or programs loaded into the RAM.

The control unit 33 also issues drive instructions to the optical system drive unit 32 to drive the zoom lens, focus lens, aperture mechanism, etc. In response to these drive instructions, the optical system drive unit 32 moves the focus lens and zoom lens, and drives the opening and closing of the aperture blades of the aperture mechanism, etc.

The control unit 33 also controls the writing and reading of various data to the memory unit 34.

The memory unit 34 includes a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory device. The memory unit 34 is used as a storage destination (recording destination) for image data output from the image sensor IS.

Furthermore, the control unit 33 performs various data communications with external devices via the communication unit 35. The communication unit 35 in the first embodiment is capable of data communications at least with the fog server 4 (or cloud server 1) shown in FIG. 1.

The image sensor IS is configured as, for example, a CCD type, CMOS type, or other image sensor.

In the present disclosure, the image sensor IS is not limited to the above-mentioned CCD, CMOS, etc. devices, but may be a sensor for acquiring various types of information, such as a sensor including ToF (Times of Flight) pixels, a sensor for acquiring other distance images, a sensor for acquiring X-ray images, a sensor for acquiring temperature images, a sensor for acquiring ultrasonic images, an infrared sensor, an ultraviolet sensor, etc. Furthermore, the image sensor IS is not limited to one unit, but one or more types of sensors may be combined and used as multiple image sensors IS.

The image sensor IS comprises an imaging section 41, an image signal processing section 42, an internal sensor control section 43, an AI image processing section 44, a memory section 45, and a communication interface (hereinafter referred to as communication I/F 46). These are connected via a bus 47, and are capable of mutual data communication.

The imaging section 41 includes a pixel array section in which multiple pixels are arranged two-dimensionally, and a readout circuit. The pixels include photoelectric conversion elements such as photodiodes. The readout circuit reads out electrical signals obtained by photoelectric conversion from each pixel in the pixel array section. The imaging section 41 outputs the obtained electrical signals as captured image signals.

The readout circuit performs processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the electrical signal obtained by photoelectric conversion, and then performs A/D (Analog/Digital) conversion.

The image signal processing unit 42 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, etc. on the captured image signal as digital data after A/D conversion processing.

The pre-processing involves processes such as clamping the R, G, and B black levels of the captured image signal to a specified level, and correction between the R, G, and B color channels.

In the synchronization process, color separation is performed so that the image data for each pixel contains all the color components R, G, and B. For example, in the case of an image sensor that uses a Bayer array color filter, demosaic processing is performed as the color separation process.

In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion process, resolution conversion is performed on image data that has undergone various signal processing.

In codec processing, image data that has been subjected to the various processes described above is encoded, for example for recording or communication, and a file is generated. In codec processing, it is possible to generate video file formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264. It is also possible to generate still image files in compressed formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), and GIF (Graphics Interchange Format), or in uncompressed formats such as TIFF (Tagged Image File Format) and raw data.

The sensor internal control unit 43 issues instructions to the imaging unit 41 and controls the execution of imaging operations. Similarly, the sensor internal control unit 43 also controls the execution of processing in the image signal processing unit 42.

The AI image processing unit 44 performs image recognition processing as AI image processing on the captured image.
Image recognition functions using AI can be realized using programmable processing devices such as CPUs, FPGAs (Field-Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), and DSPs (Digital Signal Processors).

The image recognition function that can be realized in the AI image processing unit 44 can be switched by changing the algorithm of the AI image processing. In other words, the function type of the AI image processing can be switched by switching the AI model used in the AI image processing. The function types of the AI image processing are, for example, as follows:
・Class Identification ・Semantic Segmentation ・Person Detection ・Vehicle Detection ・Target Tracking ・Optical Character Recognition (OCR)

Among the above functional types, class identification is a function that identifies the class of a target. This "class" is information that represents the category of an object. For example, classes distinguish between "people," "cars," "airplanes," "ships," "trucks," "birds," "cats," "dogs," "deer," "frogs," "horses," etc.

Target tracking is a function that tracks a targeted subject. In other words, target tracking is a function that obtains historical information about the subject's position.

The memory unit 45 is used as a storage destination for various data such as captured image data obtained by the image signal processing unit 42. In the first embodiment, the memory unit 45 is also used for temporary storage of data used by the AI image processing unit 44 in the AI image processing process.

In addition, the memory unit 45 stores information about the AI applications and AI models used in the AI image processing unit 44.

In addition, information about AI applications and AI models may be deployed in the memory unit 45 as a container or the like using the container technology described below. In addition, information about AI applications and AI models may be deployed using microservices technology. By deploying an AI model used for AI image processing in the memory unit 45, it is possible to change the function type of the AI image processing and to change to an AI model whose performance has been improved by re-learning.

The above-mentioned first embodiment is explained based on examples of AI models and AI applications used for image recognition. The present technology is not limited to this, and may also target programs executed using AI technology.

In addition, when the capacity of the memory unit 45 is small, information about the AI application or AI model may be expanded into memory outside the image sensor IS as a container, etc. using container technology, such as the memory unit 34, and then only the AI model may be stored in the memory unit 45 within the image sensor IS via the communication I/F 46 described below.

The communication I/F 46 is an interface for communicating with the control unit 33, memory unit 34, etc., which are external to the image sensor IS. The communication I/F 46 communicates to acquire from the outside the program executed by the image signal processing unit 42, the AI application used by the AI image processing unit 44, the AI model, etc. This information is stored in the memory unit 45 provided in the image sensor IS.
As a result, the AI model, etc. are stored in part of the memory section 45 of the image sensor IS and become available to the AI image processing section 44.

The AI image processing unit 44 performs a predetermined image recognition process using the AI application or AI model obtained in this manner, thereby recognizing the subject according to the purpose.

The recognition result information of the AI image processing is output externally from the image sensor IS via the communication I/F 46.

In other words, the communication I/F 46 of the image sensor IS outputs not only the image data output from the image signal processing unit 42 but also the recognition result information of the AI image processing.

Furthermore, only either image data or recognition result information can be output from the communication I/F 46 of the image sensor IS.

For example, when using the re-learning function of the AI model described above, the captured image data used in the re-learning function is uploaded from the image sensor IS to an information processing device on the cloud side via the communication I/F 46 and the communication unit 35.

In addition, when inference is performed using an AI model, the recognition result information of the AI image processing is output from the image sensor IS to another information processing device outside the camera 3 via the communication I/F 46 and the communication unit 35.

The image sensor IS can be configured in a variety of structures. The first embodiment describes the configuration of an image sensor IS having a two-layer stacked structure.

Figure 5 shows an example of the configuration of an image sensor IS as an imaging device.

As shown in Figure 5, the image sensor IS is formed by stacking two semiconductor chips, die D1 and die D2, into a single semiconductor chip.

The die D1 has the functions of the imaging unit 41 shown in Figure 4. The die D2 has the functions of an image signal processing unit 42, an internal sensor control unit 43, an AI image processing unit 44, a memory unit 45, and a communication I/F 46.

Each of the die D1 and the die D2 has a terminal on the opposing surface. The terminal is formed, for example, using copper (Cu) as a wiring material. In other words, each of the die D1 and the die D2 is electrically connected by a Cu-Cu joint that joins the terminals together.

The technology disclosed herein is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the invention.

For example, solid-state imaging devices according to two or more embodiments may be combined.

(Second embodiment)

The AI processing including the AI image processing in the first embodiment described above will be explained in more detail. In this embodiment, the processing when sensors of various modalities are used as the image sensor IS will be explained in detail.

The configuration according to this embodiment is shown, for example, in FIG. 2. In FIG. 2, the AI model developer terminal 2C or the software developer terminal 7 may be an information processing device that learns the model and converter according to this embodiment, and the application developer terminal 2A and the application user terminal 2B may be information processing devices that execute processes such as estimation and analysis using the trained model according to this embodiment.

Furthermore, the multiple cameras 3 are equipped with sensors that acquire various different types of information, as described above. As another example, the cameras 3 are not limited to optical sensing. The sensors included in the cameras 3 may be sensors that acquire information other than light.

Examples of sensors included in the multiple cameras 3 include, but are not limited to, sensors that acquire RGB information, sensors that acquire grayscale or brightness information, sensors that acquire raw data, sensors that acquire depth information, sensors that acquire event information, sensors that acquire polarization information, multispectral sensors, and the like.

Each of the application developer terminal 2A, the application user terminal 2B, the AI model developer terminal 2C and/or the software developer terminal 7 may, for example, have a processor such as a CPU or a GPU (Graphics Processing Unit) and a processing circuit as a computing unit. Furthermore, they may, for example, have temporary and/or non-temporary memory circuits and storage devices as a memory unit. Furthermore, at least a part of the memory unit may be provided in the cloud server 1 or the management server 5, and each information processing device may obtain data by accessing these servers.

In FIG. 2, the camera 3 is connected to each terminal via the cloud, but this is not limited to the above. The camera 3 may be connected to the AI model developer terminal 2C or the software developer terminal 7 during learning in a manner that allows direct data transmission and reception, and may be connected to the application developer terminal 2A or the application user terminal 2B during inference in a manner that allows direct data transmission and reception.

Processes that the trained model may perform include, but are not limited to, person detection, detection of the orientation of a person (or all or part of a person), detection of two-dimensional barcodes, image classification, counting of objects (e.g., people or animals), object detection, semantic segmentation, and feature point detection.

FIG. 6 is a diagram showing an outline of the processing according to one embodiment. An information processing device, for example, an AI model developer terminal 2C or a software developer terminal 7, optimizes the processing using the first model that executes the first processing, based on the information output from the first sensor 3A.

The first model 22A is, for example, a model that outputs the result of executing a first process on the first data output from the first sensor 3A. The first model 22A may be a model in any format. The first model 22A may be, for example, a model that has been trained by machine learning.

More specifically, the first model 22A is a model trained to output the result of executing the first processing when it receives as input the first intermediate representation into which the first data is converted by the first converter 21A. In other words, the first intermediate representation is an intermediate representation (latent representation) that, when input to the first model 22A, is capable of outputting the result of executing the first processing on the first data.

An information processing device (e.g., an AI model developer terminal 2C or a software developer terminal 7) that executes the learning of the converter converts the second data output from the second sensor 3B into a second intermediate representation using the second converter 21B, and executes learning (optimization) of the second converter 21B based on this second intermediate representation.

Here, the first converter 21A and the second converter 21B may be models formed based on a neural network model such as a CNN or a transformer. Furthermore, the first converter 21A and the second converter 21B may be models related to domain application.

By training the second converter 21B in this manner, when the first intermediate representation, which is an intermediate representation of the first data acquired from the first sensor 3A, and the second intermediate representation, which is an intermediate representation of the second data acquired from the second sensor 3B, are both input to the first model 22A, the result of executing the first process can be obtained.

As described above, the first model 22A is a model that realizes the first processing for the output of the first sensor 3A, but according to this embodiment, it is also possible to use the same first model 22A to execute the first processing for the second data, which is the output of the second sensor 3B that outputs a different type of information from the first sensor 3A.

Note that although two sensors are shown, similar processing can be performed with sensors that acquire three or more different types of information. This also applies to the embodiments described below.

FIG. 7 is a flowchart showing information processing according to this embodiment. Note that the first model 22A (and the first converter 21A) has been trained in advance using the first data set. This training may be performed based on any method. Furthermore, this training may be supervised, semi-supervised, or unsupervised.

The calculation unit of the information processing device acquires the first data output by the first sensor 3A and the second data output by the second sensor 3B (S100). As an example, the first sensor 3A and the second sensor 3B acquire information of the same object in the same environment and output the first data and the second data, respectively. The calculation unit acquires information of the same object in the same environment, for example captured image data, as the first data and the second data.

As described above, the first data and the second data are different types of information, for example, an RGB image and a depth image. It is difficult to directly input such different types of information into the same model and obtain the same processing results. By optimizing the converter, it becomes possible to perform the same processing on different types of information using the same model.

Next, the calculation unit obtains a first intermediate representation and a second intermediate representation (S102). For example, the calculation unit inputs the first data to the first converter 21A to obtain the first intermediate representation, and inputs the second data to the second converter 21B to obtain the second intermediate representation.

Next, the calculation unit obtains the loss between the first intermediate representation and the second intermediate representation (S104). This loss may be calculated using any method for calculating error. For example, the loss may be an arbitrary norm, KL divergence, or other quantity.

Then, the calculation unit optimizes the converter by updating the parameters of the second converter 21B based on the loss calculated in S104 (S106). If necessary, the processes from S102 to S106 may be repeated, and if necessary, the processes from S100 to S106 may be repeated. The optimization may be completed based on a general termination condition, for example, a predetermined number of epochs have been processed, or the evaluation value has fallen below a predetermined threshold value.

The calculation unit transmits the parameters of the second converter 21B optimized in S106 to a server such as the cloud server 1 or the management server 5, outputs them (S108), and completes the process. As another example, the calculation unit may store the parameters in a memory unit within an information processing device such as the AI model developer terminal 2C or the software developer terminal 7, instead of outputting them.

The second converter 21B optimized in this way makes it possible to convert the second data into a second intermediate representation with a distribution similar to that of the intermediate representation of the first data that obtains information of the same subject in the same environment. Therefore, by inputting this second intermediate representation into the first model 22A that accepts as input the first intermediate representation, which is an intermediate representation of the first data, it becomes possible to obtain the results of appropriately executing the first process using the second data.

In other words, by using the above converter, the application developer terminal 2A, the application user terminal 2B or the software developer terminal 7 can realize the first processing by having the calculation unit acquire the second data and convert the second data into a second intermediate representation, and inputting this second intermediate representation into the first model 22A. The first model 22A is a model that has been trained to execute the first processing when it receives as input the first intermediate representation obtained by converting the first data output from the first sensor 3A. In this way, it becomes possible to apply a model optimized for the first data to the second data via the converter.

For example, the application developer terminal 2A, the application user terminal 2B, or the software developer terminal 7 can achieve appropriate processing by changing the converter for obtaining the intermediate representation calculated as the front end of the model for input from different sensors, without changing the configuration of the model executed in the calculation unit.

(Third embodiment)

FIG. 8 is a diagram showing an outline of an information processing system according to one embodiment. The information processing system can use the same converter 21 rather than having a converter for each sensor. In this case, the first data is converted to a first intermediate representation by the converter 21, and the second data is converted to a second intermediate representation by the same converter 21. As above, the converter 21 can be a model that realizes domain application.

The process flow is generally the same as in Figure 7. The calculation unit can optimize the converter 21 by updating the parameters related to the second intermediate representation so that the first intermediate representation does not change.

The converter 21 can be configured, for example, to obtain an intermediate representation based on data automatically input in the input layer without taking into account the type of sensor.

The converter 21 can be configured, for example, to receive input from different neurons for each sensor in the input layer and obtain an intermediate representation.

The converter 21 can be configured, for example, so that there is a neuron that inputs the type of sensor, and the type of connected sensor is explicitly given to the input layer. As an application of this, the converter 21 can be configured to input a one-hot vector indicating the sensor, in addition to inputting image data, etc.

In this way, the converter 21 does not need to be prepared for each individual sensor, but may be implemented as a single converter. In this case, it is possible to use a model that performs appropriate processing on the input from the sensor without any processing on the calculation unit side, simply by switching the sensor connection.

In the embodiment described below, the processing using converter 21 will be explained, but it can of course also be applied to a case where there are multiple converters as shown in Figure 6, as long as no contradictions arise.

(Fourth embodiment)

FIG. 9 is a diagram showing an outline of an information processing system according to one embodiment. The information processing system includes a converter 21 for executing the first model 22A of FIG. 8, and is also capable of executing different processes based on the intermediate representation converted by this converter 21.

The calculation unit can input the first intermediate representation and/or the second intermediate representation output from the converter 21 to the second model 22B to obtain the result of executing a second process that is different from the first process.

This second model 22B is a model trained to perform a second process on the first data. The first data can be converted to a first intermediate representation by the aforementioned converter 21 to generate input data for the second model 22B.

The converter 21 is the same in both Figures 8 and 9. That is, for both the data acquired from the first sensor 3A and the second sensor 3B, the intermediate representation acquired via the converter 21 can be used to obtain the result of a first process using a first model, or the result of a second process using a second model 22B. In other words, it is also possible to replace only the model following the converter 21 with a model that realizes other processes.

More specifically, the calculation unit inputs the second intermediate representation to a second model 22B that has been trained to execute a second process when the first intermediate representation is input, thereby making it possible to execute a second process on the output of the second sensor 3B.

As described above, according to this embodiment, it is possible not only to replace the sensor, but also to change the model that executes the processing. This configuration makes it possible to realize more robust analysis using various types of sensors, using a model trained using output data from a specific sensor.

However, this is not limited to this implementation.

FIG. 10 is a diagram showing another aspect of this embodiment. As shown in this figure, the information processing system can also use a third model 22C that has been trained to obtain a third intermediate representation for the first data using a third converter 21C and to be able to obtain the result of the third processing by inputting this third intermediate representation. In other words, the intermediate representation for the first model 22A and the intermediate representation for the third model 22C that performs different processing may be different even for the same sensor. The third intermediate representation is an input representation of the first data for the third model 22C.

In this case, the calculation unit can be configured to convert the first data acquired from the first sensor 3A into a third intermediate representation using the third converter 21C, convert the second data acquired from the second sensor 3B into a fourth intermediate representation using the fourth converter 21D, and perform learning of the fourth converter 21D based on the third intermediate representation and the fourth intermediate representation to perform optimization.

In this way, when changing the model that executes the process, it is also possible to change the converter. Of course, the converter may be shared and used for both the first data and the second data, as shown in FIG. 9, etc.

The calculation unit of the AI model developer terminal 2C, for example, acquires first data and second data obtained by using a first sensor 3A and a second sensor 3B, respectively, which are information on the same object in the same environment, inputs the first data to a third converter 21C to acquire a third intermediate representation, inputs the second data to a fourth converter 21D to acquire a fourth intermediate representation, and calculates the losses of these intermediate representations, thereby updating and optimizing the parameters of the fourth converter 21D.

The calculation unit of the application developer terminal 2A, the application user terminal 2B or the software developer terminal 7 can, for example, convert the data output from the second sensor 3B using the fourth converter 21D to obtain an intermediate representation, and input this intermediate representation to the third model 22C, thereby realizing the third processing using the third model 22C that has been trained to execute the third processing using the output of the first sensor 3A on the output from the second sensor 3B.

All of the above embodiments have been described with a focus on information processing devices, but they can also of course operate as an information processing system including sensors.

The information processing system, which serves as a learning system for the converter, is equipped with sensors that acquire different types of information, and the information processing device can acquire intermediate representations for each of these sensors and optimize the converter from these intermediate representations. With the optimized converter, it becomes possible to execute converter learning to realize processing such as analysis using the same model for outputs from different sensors acquired inside or outside the information processing system.

As above, this converter may correspond to a different converter for each sensor, which is, for example, a model related to the domain application, or the same converter may correspond to each sensor.

According to all embodiments from the second embodiment onwards, it is possible to implement appropriate processing for outputs from sensors that produce different types of output without retraining the model that realizes the processing, i.e., without incurring the cost of retraining. As an example, training time can be shortened. Also, when supervised learning is realized for a certain sensor (labels are prepared), it becomes possible to apply unsupervised data (data without labels) from other sensors to the same trained model.

The modular system of interchangeable sensors and domain adaptation allows for seamless integration of diverse sensor data within the same trained neural network processing pipeline. The modular system and ability to use common input feature generation reduces the cost of hardware development and production.

It also provides an efficient and flexible approach to processing sensor data, allowing users to adapt the device to different sensors without significant changes to the implementation or time-consuming training. This also saves time and resources. For example, it effectively eliminates the need for pre-processing before inputting the model for each sensor.

By using this information processing device or system, it becomes easy to make modifications to the modular design, such as incorporating additional components such as different mechanisms for sensor compatibility, different architectures, and different algorithms. To achieve the same effect, a trained neural network model may be prepared for each sensor. In this case, the information processing device can change the model according to the sensor, but as mentioned above, retraining for each sensor requires time and resources.

This system allows for greater freedom in sensor replacement, allowing users to seamlessly change sensors without changing the system contents. It is also advantageous in scenarios where different sensor modalities are required for different applications or environments.

Furthermore, processing suitable for output data from a certain sensor can be learned using a data set from that sensor. Furthermore, since it is possible to align intermediate representations of data output from other sensors, it is possible to achieve highly accurate processing using the same trained model for those other sensors. From another perspective, even for a sensor for which it is difficult to obtain a large amount of data for learning, if there is an abundance of data from other sensors for learning the model, it is possible to achieve that processing for that sensor by learning a model that uses a sensor with an abundance of data.

In addition, when developing a new sensor or applying it to a new sensor, it is possible to use the same model to execute the processing without having to learn a new model to execute the processing, by learning a model that aligns the intermediate representation.

The above-described embodiment may be modified as follows:

(1)
A calculation unit,
The calculation unit is
Acquire first data output from a first sensor and second data output from a second sensor that acquires information of a different type from that of the first sensor;
inputting the first data into a first transformer to obtain a first intermediate representation;
inputting the second data into a second transformer to obtain a second intermediate representation;
performing training of the second converter based on the first intermediate representation and the second intermediate representation;
Information processing device.

(2)
the first intermediate representation is an intermediate representation capable of executing a first process when inputted into a first model;
An information processing device as described in (1).

(3)
The first model is a model trained to execute the first process when the first intermediate representation is input.
An information processing device according to (2).

(4)
the first intermediate representation is an intermediate representation capable of executing a second process when inputted to a second model;
An information processing device according to (2) or (3).

(5)
The second model is a model trained to execute the second process when the first intermediate representation is input.
(4) An information processing device.

(6)
The calculation unit is
acquiring information of a same object in a same environment using the first sensor and the second sensor, respectively, to obtain the first data and the second data;
inputting the first data into the first converter to obtain the first intermediate representation; inputting the second data into the second converter to obtain the second intermediate representation;
calculating a loss between the first intermediate representation and the second intermediate representation to update parameters of the second converter and optimize the second converter;
An information processing device according to any one of (1) to (5).

(7)
The calculation unit is
inputting the first data into a third transformer to obtain a third intermediate representation;
inputting the second data into a fourth transformer to obtain a fourth intermediate representation;
performing training of the fourth converter based on the third intermediate representation and the fourth intermediate representation;
An information processing device according to any one of (1) to (6).

(8)
the third intermediate representation is an intermediate representation capable of executing a third process when inputted into a third model;
(7) An information processing device.

(9)
The third model is a model trained to execute the third process when the third intermediate representation is input.
(8) An information processing device.

(10)
The calculation unit is
acquiring information of a same object in a same environment using the first sensor and the second sensor, respectively, to obtain the first data and the second data;
inputting the first data into the third converter to obtain the third intermediate representation; inputting the second data into the fourth converter to obtain the fourth intermediate representation;
calculating a loss between the third intermediate representation and the fourth intermediate representation to update parameters of the fourth converter and optimize the fourth converter;
An information processing device according to any one of (7) to (9).

(11)
The first converter and the second converter are models related to domain application.
An information processing device according to any one of (1) to (10).

(12)
The first converter and the second converter are of the same model.
An information processing device according to (11).

(13)
A calculation unit,
The calculation unit is
acquiring second data output by a second sensor that acquires information of a different type from that of the first sensor;
converting the second data to obtain a second intermediate representation;
The second intermediate representation is
a first model that executes a first process when a first intermediate representation obtained by converting first data output by the first sensor is input;
and acquiring a result of executing the first process on the second data.
Information processing device.
The converter that obtains the intermediate representation may be a model optimized by any of the information processing devices (1) to (12).

(14)
The calculation unit is
The second intermediate representation is
a second model that executes a second process when the first intermediate representation is input;
and acquiring a result of executing the second process on the second data.
An information processing device according to (13).

(15)
The calculation unit is
Transforming the second data to obtain a fourth intermediate representation;
The fourth intermediate representation is
a third model that executes a third process when a third intermediate representation obtained by converting the first data is input;
and acquiring a result of executing the third process on the second data.
The information processing device according to (13) or (14).

(16)
A first sensor;
A second sensor that acquires a type of information different from that of the first sensor;
An information processing device according to any one of (1) to (11);
Equipped with
The information processing device includes:
optimizing a converter that converts first data acquired from the first sensor into a first intermediate representation and second data acquired from the second sensor into a second intermediate representation;
Information processing system.

(17)
the transformation to obtain the first intermediate representation and the transformation to obtain the second intermediate representation are models related to domain application;
An information processing system according to (16).

(18)
the transformation to obtain the first intermediate representation and the transformation to obtain the second intermediate representation are performed using the same model;
(17) An information processing system according to (17).

(19)
A first sensor;
A second sensor that acquires a type of information different from that of the first sensor;
An information processing device according to any one of (12) to (14);
Equipped with
The information processing device includes:
a first model that executes a first process when a first intermediate representation obtained by converting first data acquired from the first sensor is input, and a second intermediate representation obtained by converting second data acquired from the second sensor is input to a first model, and a result of executing the first process on the second data is obtained;
Information processing system.

The aspects of this disclosure are not limited to the above-described embodiments, but include various conceivable modifications, and the effects of this disclosure are not limited to the above-described contents. The components in each embodiment may be appropriately combined and applied. In other words, various additions, modifications, and partial deletions are possible within the scope that does not deviate from the conceptual idea and intent of this disclosure derived from the contents defined in the claims and their equivalents.

100: Information processing systems,
1: Cloud server,
2: User terminal,
2A: Application developer terminal,
2B: Application user terminal,
2C: AI model developer terminal,
21: Converter,
21A: 1st converter,
21B: a second converter;
21C: 3rd converter,
21D: 4th converter,
22A: 1st model,
22B: 2nd model,
22C: 3rd model,
3: Camera,
31: Imaging optical system,
32: Optical system drive unit,
33: control section,
34: Memory section,
35: Communications Department,
36: Bus,
IS: Image sensor,
41: imaging unit,
42: Image signal processing section;
43: Sensor internal control unit,
44: AI image processing unit,
45: Memory section,
46: Communication I/F,
47: Bus,
D1, D2: Die,
3A: 1st sensor,
3B: second sensor,
4: Fog Server,
5: Management server,
6: Network,
7: Software Developer Terminal

Claims (19)

A calculation unit,
The calculation unit is
Acquire first data output from a first sensor and second data output from a second sensor that acquires information of a different type from that of the first sensor;
inputting the first data into a first transformer to obtain a first intermediate representation;
inputting the second data into a second transformer to obtain a second intermediate representation;
performing training of the second converter based on the first intermediate representation and the second intermediate representation;
Information processing device. the first intermediate representation is an intermediate representation capable of executing a first process when inputted into a first model;
The information processing device according to claim 1. The first model is a model trained to execute the first process when the first intermediate representation is input.
The information processing device according to claim 2. the first intermediate representation is an intermediate representation capable of executing a second process when inputted to a second model;
The information processing device according to claim 3. The second model is a model trained to execute the second process when the first intermediate representation is input.
The information processing device according to claim 4. The calculation unit is
acquiring information of a same object in a same environment using the first sensor and the second sensor, respectively, to obtain the first data and the second data;
inputting the first data into the first converter to obtain the first intermediate representation; inputting the second data into the second converter to obtain the second intermediate representation;
calculating a loss between the first intermediate representation and the second intermediate representation to update parameters of the second converter and optimize the second converter;
The information processing device according to claim 1. The calculation unit is
inputting the first data into a third transformer to obtain a third intermediate representation;
inputting the second data into a fourth transformer to obtain a fourth intermediate representation;
performing training of the fourth converter based on the third intermediate representation and the fourth intermediate representation;
The information processing device according to claim 1. the third intermediate representation is an intermediate representation capable of executing a third process when inputted into a third model;
The information processing device according to claim 7. The third model is a model trained to execute the third process when the third intermediate representation is input.
The information processing device according to claim 8. The calculation unit is
acquiring information of a same object in a same environment using the first sensor and the second sensor, respectively, to obtain the first data and the second data;
inputting the first data into the third transformer to obtain the third intermediate representation;
inputting the second data into the fourth transformer to obtain the fourth intermediate representation;
calculating a loss between the third intermediate representation and the fourth intermediate representation to update parameters of the fourth converter and optimize the fourth converter;
The information processing device according to claim 7. The first converter and the second converter are models related to domain application.
The information processing device according to claim 1. The first converter and the second converter are of the same model.
The information processing device according to claim 11. A calculation unit,
The calculation unit is
acquiring second data output by a second sensor that acquires information of a different type from that of the first sensor;
converting the second data to obtain a second intermediate representation;
The second intermediate representation is
a first model that executes a first process when a first intermediate representation obtained by converting first data output by the first sensor is input;
and acquiring a result of executing the first process on the second data.
Information processing device. The calculation unit is
The second intermediate representation is
a second model that executes a second process when the first intermediate representation is input;
and acquiring a result of executing the second process on the second data.
The information processing device according to claim 13. The calculation unit is
Transforming the second data to obtain a fourth intermediate representation;
The fourth intermediate representation is
a third model that executes a third process when a third intermediate representation obtained by converting the first data is input;
and acquiring a result of executing the third process on the second data.
The information processing device according to claim 13. A first sensor;
A second sensor that acquires a type of information different from that of the first sensor;
An information processing device according to claim 1;
Equipped with
The information processing device includes:
optimizing a converter that converts first data acquired from the first sensor into a first intermediate representation and second data acquired from the second sensor into a second intermediate representation;
Information processing system. the transformation to obtain the first intermediate representation and the transformation to obtain the second intermediate representation are models related to domain application;
17. An information processing system according to claim 16. the transformation to obtain the first intermediate representation and the transformation to obtain the second intermediate representation are performed using the same model;
18. An information processing system according to claim 17. A first sensor;
A second sensor that acquires a type of information different from that of the first sensor;
An information processing device according to claim 12;
Equipped with
The information processing device includes:
a first model that executes a first process when a first intermediate representation obtained by converting first data acquired from the first sensor is input, and a second intermediate representation obtained by converting second data acquired from the second sensor is input to a first model, and a result of executing the first process on the second data is obtained;
Information processing system.