WO2025206282A1 - Information processing system, prediction device, information processing method, control program, and recording medium - Google Patents

Abstract

This information processing system is provided with a prediction unit that uses a prediction model to output prediction information from a first image and first data, the first image showing at least a part of a first subject. The prediction model is generated by machine learning using, as an explanatory variable, a third image and second data, the third image showing at least a part of a second subject, and using, as an objective variable, abnormality information relating to an abnormality that has occurred in a bone of the second subject at a second time point, which is a time point different from a first time point at which the third image was captured. The prediction information is information indicating the possibility of an abnormality occurring in a bone of the first subject.

Description

Information processing system, prediction device, information processing method, control program, and recording medium

This disclosure relates to an information processing system, a prediction device, an information processing method, a control program, and a recording medium.

Technology for using neural networks to estimate disease-related information from medical images is known. For example, Patent Document 1 discloses a configuration for estimating whether a patient has osteoporosis based on their X-ray images.

Japanese Patent Application Publication No. 2023-143875

An information processing system according to one aspect of the present disclosure includes a prediction unit that uses a prediction model to output predictive information from a first image and first data that show at least a portion of a first subject. The predictive model is generated by machine learning using a third image and second data that show at least a portion of a second subject as explanatory variables, and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point when the third image was captured as a dependent variable. The predictive information is information that indicates the possibility of an abnormality occurring in the bones of the first subject.

An information processing method according to one aspect of the present disclosure is an information processing method executed by one or more computers, and includes a prediction step of outputting predicted information using a prediction model from a first image and first data that show at least a portion of a first subject. The prediction model is generated by machine learning using a third image and second data that show at least a portion of a second subject as explanatory variables, and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point when the third image was captured as a target variable. The predicted information is information that indicates the possibility of an abnormality occurring in the bones of the first subject.

1 is a block diagram illustrating an example of a configuration of an information processing system according to a first embodiment of the present disclosure. FIG. 2 is a block diagram for explaining the operation of a prediction model according to the first embodiment. 10 is a flowchart illustrating an example of a learning process flow by a learning unit of the prediction device according to the first embodiment. 10 is a flowchart illustrating an example of the flow of a prediction process by a prediction unit of the prediction device according to the first embodiment. FIG. 3 is a diagram illustrating an example of a first image of a first subject according to the first embodiment. FIG. 10 is a diagram illustrating an example of segmentation of a second image according to the first embodiment. FIG. 10 is a block diagram illustrating an example of a configuration of an information processing system according to a second embodiment. FIG. 10 is a block diagram for explaining the operation of an estimation model according to a second embodiment. FIG. 10 is a block diagram for explaining the operation of a prediction model according to a second embodiment. 10 is a flowchart illustrating an example of a learning process performed by a learning unit according to the second embodiment. FIG. 10 is a diagram showing a plain X-ray image of the chest of a first subject according to the second embodiment. 10 is a flowchart illustrating an example of the flow of a prediction process performed by a prediction device according to a second embodiment. 10A and 10B are diagrams showing the results of prediction of a subject's fracture risk by a prediction device according to embodiment 2 and the effects of each countermeasure.

Factors that lead to fractures include a decrease in bone strength and increased stress on bones due to muscle atrophy. In order to accurately estimate the likelihood of a fracture, it is desirable to take into account not only bone strength but also the condition of the muscles. There is room for improvement in the accuracy of technology that can predict the occurrence of fractures from medical images.

One aspect of the present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an information processing system, information processing method, prediction device, control program, and recording medium that can improve the prediction accuracy of predicting information about abnormalities from medical images.

According to one aspect of the present disclosure, it is possible to improve the accuracy of predicting information about abnormalities from medical images.

[Embodiment 1]
An information processing system 1 according to a first embodiment of the present disclosure will be described below with reference to FIGS.

[Configuration of information processing system]
First, the configuration of the information processing system 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the information processing system 1 in embodiment 1. Fig. 1 shows the information processing system 1 including a prediction device 10, an image management device 40, an electronic medical record management device 50, and a presentation device 60. Note that the configuration of the information processing system 1 is not limited to the configuration shown in Fig. 1.

The prediction device 10 is a device that acquires a first image G1 and first data of a first subject, which is the subject of the prediction of information related to an abnormality, and outputs prediction information from the acquired first image G1 and first data using a prediction model 32. In embodiment 1, the first data is, as an example, a second image G2 of the first subject (see Figure 2).

Here, the first subject is, for example, a human. However, the first subject may be a non-human, such as an animal such as a dog, cat, or horse. The first image G1 may show at least some of the bones in a predetermined region of the first subject. The second image G2 may show the muscles in a region of the first subject corresponding to the predetermined region. The second image G2 may show the same region of the first subject as the region corresponding to the predetermined region, or it may show a region of the first subject that is different from the region corresponding to the predetermined region of the first subject.

The first image G1 is, for example, a plain X-ray image showing bones of an area including at least one of the head, neck, chest, lower back, temporomandibular joints, spinal intervertebral joints, hip joints, sacroiliac joints, knee joints, ankle joints, feet, toes, shoulder joints, acromioclavicular joints, elbow joints, wrist joints, hands, fingers, and temporomandibular joints of the first subject. The first image G1 may also show areas of the first subject other than areas where abnormalities such as fractures are expected. Note that the plain X-ray image may include, for example, a panoramic X-ray image used for dentistry. A panoramic X-ray image is, for example, an image that includes multiple teeth or all of the teeth.

The first image G1 may be, for example, an image captured at a medical facility at a third time point. The first image includes at least one of a frontal image showing a specified area from the front (for example, an image obtained by irradiating the specified area with X-rays in the front-to-back direction) and a lateral image showing the specified area from the side (for example, an image obtained by irradiating the specified area with X-rays in the left-to-right direction).

The first image G1 is not limited to a simple X-ray image, but may be any medical image containing information about bones, such as a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, an image obtained by DXA (Dual Energy X-ray Absorptiometry), an image obtained by DES (Dual Energy Subtraction), or an ultrasound image. The first image G1 may be an image that includes a phantom, or an image that does not include a phantom.

The second image G2 is, for example, an echo image showing muscles in areas corresponding to the head, chest, waist, feet, hands, etc. of the first subject (see Figure 6). The second image G2 may show only one area of the first subject, or multiple areas. The second image G2 may be a still image or a video.

Furthermore, the second image G2 is not limited to an echo image, and may be any medical image containing information about muscles, such as a CT image, an MRI image, an image obtained by DXA, an image obtained by DES, or an ultrasound image. In other words, the second image G2 may be of a different type from the first image G1, or the same type. Furthermore, the second image G2 may be an image captured using a medium with a different wavelength from that of the first image G1.

The second image G2 may be, for example, an image taken at a medical facility at a third time point. The difference in the time at which the first image G1 and the second image G2 were taken may be within a predetermined period, such as two weeks, one month, six months, or one year. The third time point may be set based on the date on which the first image G1 was taken, or the date on which the second image G2 was taken. The reference date may be either the later or earlier of the dates on which the first image G1 and the second image G2 were taken.

The second image G2 also includes at least one of a front image showing the specified area from the front (for example, an image obtained by irradiating the specified area with X-rays in the front-to-back direction) and a side image showing the specified area from the side (for example, an image obtained by irradiating the specified area with X-rays in the left-to-right direction).

If the first image G1 or the second image G2 is a CT image, information about the bone trabeculae based on a three-dimensionally constructed image may be used, or information about the bone trabeculae based on a two-dimensionally captured image may be used. Furthermore, for example, at least one of a three-dimensional image, a cross-sectional image perpendicular to the body axis connecting the head and legs (e.g., horizontal section), and a cross-sectional image parallel to the body axis (e.g., sagittal section or coronal section) may be used.

The prediction information output by the prediction device 10 is information indicating the possibility that an abnormality will occur in the bone of the first subject shown in the first image G1 at the fourth time point. In this embodiment, the prediction information is a fracture risk indicating the possibility that a fracture will occur in the bone of the first subject at the fourth time point.

Fractures are an example of musculoskeletal diseases, and fragility fractures are assumed to be the fractures. Musculoskeletal diseases also include osteoporosis, osteoarthritis, spondylosis deformans, neurological disorders, sarcopenia, etc.

The fourth point in time is a point in time different from the third point in time. The third point in time is the point in time when the first image G1 was captured. The fourth point in time refers to any point in time in the future or the past of the third point in time. The fourth point in time may include multiple points in time, such as one year before, one year after, five years after, ten years after, and thirty years after the third point in time.

The image management device 40 is a computer that functions as a server for managing the third and fourth images. The third image is a plain X-ray image showing the bones of a specific area of the second subject. The fourth image is an echo image showing the muscles of a part of the second subject corresponding to the specific area.

The fourth image may depict a region of the second subject that is different from the region corresponding to the above-mentioned predetermined region. The third image and the fourth image may be stored in separate image management devices. The prediction device 10 may also be configured to acquire the third image and the fourth image from the imaging device without going through the image management device 40.

Furthermore, the third image is not limited to a simple X-ray image, but may be any medical image containing information about bones, such as a CT image, MRI image, DXA image, DES image, or ultrasound image. Furthermore, the fourth image is not limited to an echo image, but may be any medical image containing information about muscles, such as a CT image, MRI image, DXA image, DES image, or ultrasound image. In other words, the fourth image may be the same or different from the third image. The third and fourth images may have the same or different combinations of the types of first image G1 and second image G2.

The third image may be, for example, an image taken at a medical facility at a first time point. The difference in the time at which the third image and the fourth image were taken may be within a predetermined period, such as two weeks, one month, six months, or one year. The first time point may be set based on the date the third image was taken, or the date the fourth image was taken. The reference date may be either the later or earlier of the dates at which the third and fourth images were taken.

The electronic medical record management device 50 is a computer that functions as a server for managing electronic medical record information of a first subject who has undergone a medical examination or test at a medical facility, etc. The image management device 40 and the electronic medical record management device 50 are connected to the acquisition unit 21 of the prediction device 10.

The electronic medical record information includes attribute information of the first subject. The attribute information includes at least one of the following: the first subject's age, sex, height, weight, muscle quality, race, lifestyle information, medication information, occupational information, blood test information, urine test information, saliva test information, information on existing diseases, medical history, medical history of the first subject's family, surgery information, genetic information, childbirth information, menopausal information, items from the Fracture Risk Assessment Tool (FRAX (registered trademark)), and information regarding estimated menopause based on hormone information.

Birth information may include at least one of whether or not a woman has given birth, the number of children born, etc. The lifestyle habits may be, for example, sleep time, wake-up time, sleep duration, daily exercise amount, meal contents, meal times, meal duration, and blood glucose level. Meal contents may include, for example, at least one of the name of the dish, the ingredients consumed, and the intake amount. Meal contents may be, for example, an estimated intake of at least one of calcium, vitamin B, vitamin D, and vitamin K. The blood glucose level may be, for example, a designated value estimated from parameters acquired by a wearable device. Medication information may include, for example, information such as the name of the medication, the amount taken, and the duration of medication. Information regarding medications taken may include information regarding the steroid medication being used. Blood test information may be, for example, information regarding the results of at least one of a biochemical test, a glucose metabolism test, and an endocrine system test.

The presentation device 60 is a device for presenting information output by the prediction device 10. The presentation device 60 is a computer used by medical personnel, such as doctors, affiliated with a medical facility. The presentation device 60 is, for example, a personal computer with an LCD display or an organic EL display, a tablet terminal, a smartphone, etc. The presentation device 60 is controlled by the presentation control unit 26 to present fracture risk and other predictive information. The presentation device 60 may also be a device that prints the predictive information on paper or the like and outputs it.

[Configuration of prediction device]
Next, the configuration of the prediction device 10 will be described in detail with reference to Fig. 1. As shown in Fig. 1, the prediction device 10 includes a control unit 2 and a storage unit 3. The control unit 2 has, for example, a CPU (Central Processing Unit), and manages the operation of the prediction device 10 by comprehensively controlling each unit of the prediction device 10.

The control unit 2 of the prediction device 10 includes an acquisition unit 21, an analysis unit 22, a correction unit 23, a learning unit 24, a prediction unit 25, and a presentation control unit 26. The control unit 2 and the memory unit 3 are electrically connected to each other.

The acquisition unit 21 acquires a first image G1 and a second image G2 of the first subject from the image management device 40. The acquisition unit 21 also acquires a third image and a fourth image of the second subject from the image management device 40. The acquisition unit 21 may also acquire the first image G1 and the second image G2 input via an input device (not shown). The acquisition unit 21 may also extract attribute information from the first image G1 and the second image G2 if attribute information has been added to the acquired first image G1 and the second image G2.

Here, the third and fourth images of the multiple people are stored in the memory unit 3 as learning data 33. Furthermore, abnormality information regarding bone abnormalities that occurred in the multiple people at the second time point is stored in the memory unit 3 as training data 34.

The analysis unit 22 segments the second image G2 to identify which bone, muscle, etc. area each pixel in the second image G2 corresponds to, thereby dividing the area. Segmentation can be performed using, for example, a convolutional neural network (CNN), a full convolutional network (FCN), a U-Net, a V-Net, etc. The analysis unit 22 identifies the soft tissue area. The analysis unit 22 analyzes information including at least one of the amount, thickness, amount of atrophy, and flexibility of the muscle and fat of the first subject.

The correction unit 23 performs a predetermined correction on the first image G1. Specifically, the correction unit 23 performs a correction to remove the soft tissue area identified by the analysis unit 22 from the first image G1. Here, soft tissue refers to tissue other than bone, such as muscle and fat. Note that the third image described above may also be corrected by the correction unit 23 in the same way as the first image G1.

The learning unit 24 performs learning processes to generate the prediction model 32. Note that if a prediction model 32 generated by another device is stored in advance in the storage unit 3, the learning unit 24 may not be necessary.

The memory unit 3 is a computer-readable, non-transitory recording medium that stores the control program 31. The memory unit 3 is configured to include ROM (Read Only Memory), RAM (Random Access Memory), etc.

The control unit 2 controls the prediction device 10 by executing the control program 31. In other words, the control program 31 is a control program for causing the computer to function as the information processing system 1, and causes the computer to function as the prediction unit 25.

In addition to the control program 31, the memory unit 3 also stores a prediction model 32. The prediction model 32 has AI (Artificial Intelligence). The prediction model 32 is generated by machine learning using the third image captured at the first time point and the second data as explanatory variables, and abnormality information related to a fracture that occurred in the bone of the second subject at the second time point as a target variable. The memory unit 3 also stores the above-mentioned learning data 33 and teacher data 34. In embodiment 1, the second data is, as an example, the above-mentioned fourth image.

The prediction model 32 may be generated by machine learning using the third image, the fourth image, bone information obtained by inputting the third image into the first estimation model, and muscle information obtained by inputting the fourth image into the second estimation model as explanatory variables, and abnormality information related to a fracture that occurred in the bone of the second subject at the second time point as a dependent variable. The prediction unit 25 may then use the prediction model to output prediction information from the first image G1, the second image G2, bone information obtained by inputting the first image G1 into the first estimation model, and muscle information obtained by inputting the second image G2 into the second estimation model.

[Predictive Model Operation]
Next, the operation of the prediction model will be described with reference to Fig. 2. Fig. 2 is a block diagram for explaining the operation of the prediction model 32 in embodiment 1. The prediction model 32 is, for example, a convolutional neural network (CNN). Note that the prediction model 32 may be configured using a neural network other than a convolutional neural network.

As shown in FIG. 2, the prediction model 32 has, for example, an input layer 32a, a hidden layer 32b, and an output layer 32c. The prediction model 32 was generated by machine learning using the third and fourth images captured at the first time point as explanatory variables and abnormality information regarding an abnormality that occurred in the bones of the second subject at the second time point as the objective variable.

The first point in time is the point in time when the third image is captured. The second point in time is a point in time different from the first point in time. The second point in time refers to, for example, any point in time in the future or the past of the first point in time when the third image is captured. The second point in time may include multiple points in time, such as one year before, one year after, five years after, ten years after, and thirty years after the first point in time. Furthermore, the period between the first point in time and the second point in time may be the same length as the period between the third point in time and the fourth point in time, or it may be different. Furthermore, the period between the third point in time and the fourth point in time may be shorter or longer than the period between the first point in time and the second point in time.

The second point in time may be the same as the third point in time described above. That is, for example, a third image taken five years ago may be used to train the prediction model 32 to predict the risk of fracture occurring five years later. In this case, the first point in time is five years ago, the second and third points in time are the present, and the fourth point in time is five years in the future.

In the prediction model 32, the first image G1 and the second image G2 are input to the input layer 32a, and a fracture risk Y indicating the possibility of a fracture occurring in the bone of the first subject at the fourth time point is output.

The prediction model 32 may output information indicating the possibility of osteoporosis occurring in the bones of the first subject at the fourth time point. The possibility of osteoporosis may be classified, for example, based on at least one of the presence or absence of a fracture, the possibility of a fracture, and a change in bone density. The possibility of osteoporosis includes "no osteoporosis," "suspected osteoporosis," or "possible osteoporosis." More specifically, the possibility of osteoporosis may be indicated as primary osteoporosis when there is no disease that reduces bone mass, secondary osteoporosis is not observed, and there is a fracture or a high possibility of a fracture. Osteoporosis may also be determined when the YAM indicated by the bone density estimate, which is information indicating the bone density of the first subject, is less than 80% and the measurement results indicate a fracture other than the vertebral body or proximal femur. Osteoporosis may also be determined when the YAM indicated by the bone density estimate is 70% or less.

[Learning process flow]
Next, the flow of the learning process by the learning unit 24 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the flow of the learning process by the learning unit 24 of the prediction device 10. The learning unit 24 executes the learning process shown in Fig. 3 and stores the prediction model 32 in the storage unit 3 before the prediction process by the prediction unit 25, which will be described later.

In the flowchart of FIG. 3, first, the learning unit 24 acquires a third image of the second subject via the acquisition unit 21 (S1). Here, the second subject is, for example, multiple people. Note that the second subject may be a non-human, such as an animal such as a dog, cat, or horse. It may be the same species as the first subject, or a different species. Furthermore, the second subject does not have to be multiple people, and may be the same person. Furthermore, the third image is a simple X-ray image taken at a first time point, showing the bones of a specified region of the second subject. Note that multiple pieces of data taken at different times for the same person may be used as the third image.

In S1, the learning unit 24 acquires attribute information for each second subject from the electronic medical record management device 50 and associates the attribute information with each second image G2.

After S1, the learning unit 24 acquires a fourth image of the second subject via the acquisition unit 21 (S2). The fourth image may be an echo image of the muscles of the second subject at a location corresponding to the above-mentioned specified location, imaged at the first time point. The third image may at least depict the same location as the location from which the fourth image was taken. The third image may also depict a location different from the location from which the fourth image was taken. Note that S1 and S2 may be taken in the opposite order, or at the same timing.

The third image may be at least one of a CT image, an MRI image, an image obtained by DXA, an image obtained by DES, and an ultrasound image. The fourth image may be at least one of a CT image, an MRI image, an image obtained by DXA, an image obtained by DES, and an ultrasound image.

After S2, the learning unit 24 acquires abnormality information regarding an abnormality that occurred in the bones of the second subject at the second time point via the acquisition unit 21 (S3). Here, the abnormality may be a musculoskeletal disorder, such as a fracture. That is, the information is regarding a fracture that occurred in the second subject at the second time point. Note that abnormalities include, in addition to fractures, osteoporosis, osteoarthritis, spondylosis deformans, nerve disorders, sarcopenia, etc.

After S3, the learning unit 24 generates a prediction model 32 through machine learning using the third and fourth images as explanatory variables and the abnormality information as a target variable. After S4, the learning unit 24 stores the generated prediction model 32 in the storage unit 3 (S5). This completes the learning process by the learning unit 24.

[Prediction process flow]
Next, the flow of the prediction process by the prediction device 10 will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is a flowchart showing an example of the flow of the prediction process by the prediction device 10. Fig. 5 is a diagram showing an example of a first image G1 of a first subject.

The following describes a case where, as shown in Figure 5, an image of the chest bones of a first subject is captured as a first image G1, and an echo image of the back muscles of the first subject is captured as a second image G2 in a medical facility or the like.

In the flowchart of FIG. 4, first, the acquisition unit 21 acquires a first image G1 of the first subject from the image management device 40 (S11). The first image G1 may be, for example, a simple X-ray image showing a chest bone B of the first subject, as shown in FIG. 5.

Next, the acquisition unit 21 acquires a second image G2 of the first subject from the image management device 40 (S12). The second image G2 may be an echo image showing the muscles of the area corresponding to the chest of the first subject. Note that S11 and S12 may be performed in the opposite order, or at the same time.

After S12, the analysis unit 22 segments the second image G2 and analyzes information about the muscles and fat of the first subject (S13). In S13, the analysis unit 22 segments the second image G2, i.e., divides the multiple types of muscles, fat, etc. that appear in the second image G2 into regions.

Here, FIG. 6 is a diagram showing an example of segmentation of the second image G2. FIG. 6 shows an echo image of the back of the first subject. In the example shown in FIG. 6, the back of the first subject is divided into the regions of subcutaneous fat F, first muscle M1, second muscle M2, and bone B. As a segmentation method, division into the regions of muscle grouping multiple types of muscle, subcutaneous fat F, and bone B may also be used.

The analysis unit 22 can analyze the amount of the first muscle M1 by determining the thickness of the first muscle M1, for example, as shown by the open arrow in Figure 6. The analysis unit 22 can also analyze the amount of fat, the amount of muscle atrophy, flexibility, etc. of the first subject. For example, the analysis unit 22 can calculate the amount of fat of the first subject by segmenting the second image G2 to calculate the fat area or by determining the brightness of the echo image. The analysis unit 22 can also determine the amount of muscle atrophy of the first subject by comparing the average muscle thickness measurements of people of the same age as the first subject with the muscle thickness of the first subject. The analysis unit 22 can also determine the flexibility of the first subject's muscles by capturing a video of the echo image of the first subject and analyzing the muscle movement.

After S13, the correction unit 23 performs a predetermined correction to remove the soft tissue identified by the analysis unit 22, i.e., tissue areas other than bone, from the first image G1 (S14). Specifically, the correction unit 23 removes muscle and/or fat, etc., other than bone identified by the analysis unit 22, from the first image G1. It is preferable that the second image G2 is of a region corresponding to the first image G1.

After S14, the prediction unit 25 reads the prediction model 32 from the memory unit 3, inputs the first image G1 and the second image G2 corrected by the correction unit 23 to the input layer 32a of the prediction model 32, and outputs the fracture risk Y from the output layer 32c (S15: prediction step).

Furthermore, in S15, the prediction unit 25 may use a prediction model 32 to output, as prediction information, an influence level indicating the degree of influence that each of the first image G1 and the second image G2 has on the fracture of the first subject from the first image G1 and the second image G2. For example, information may be output indicating that the influence level of the bone condition shown in the first image G1 is 70% and that of the muscle condition shown in the second image G2 is 30%. The prediction model 32 is generated by machine learning using the third image and the fourth image as explanatory variables and the influence level indicating the degree of influence on the fracture of the second subject at the second time point as the objective variable.

Furthermore, in S15, the prediction unit 25 may use the prediction model 32 to output a predicted fracture time, which is the time when a fracture is likely to occur. In this case, the prediction model 32 is generated by machine learning using the third and fourth images as explanatory variables and the time when a fracture occurs in the bone of the second subject as the objective variable. The predicted fracture time may be in years, for example, 5 years or 10 years, or may be in months, for example, 5 years and 6 months or 10 years and 6 months.

The fracture risk Y, impact level, predicted fracture time, etc. output by the prediction unit 25 are transmitted to the presentation control unit 26. The presentation control unit 26 then presents prediction information including at least one of the fracture risk Y, impact level, predicted fracture time, etc. on the presentation device 60 (S16).

In S16, for example, the presentation control unit 26 presents a message to the presentation device 60 such as, "Fracture risk Y will be 80% or higher in 10 years." The presentation control unit 26 may also display a diagram showing the fracture risk Y at multiple points in time. The presentation control unit 26 may also present a graph showing the progress of the fracture risk Y to the presentation device 60. The presentation control unit 26 may also present the fracture risk Y, which is the estimated result, and the actual measurement result to the presentation device 60. This completes the prediction process by the prediction device 10 shown in FIG. 4.

In the information processing system 1 of embodiment 1 described above, the prediction unit 25 uses the prediction model 32 to output the fracture risk Y, impact degree, predicted fracture time, etc. of the first subject as prediction information from the first image G1 showing the bones of the first subject and the second image G2 showing the muscles of the first subject. In other words, the prediction unit 25 uses information about the bones of the first subject as input information, as well as information about the muscles that support the bones, to predict the fracture risk Y, impact degree, predicted fracture time, etc. using the prediction model 32.

With the above configuration, the prediction unit 25 can accurately predict the fracture risk Y, the degree of impact, and the predicted fracture timing of the first subject. This allows, for example, doctors at medical facilities to use the output results of the information processing system 1, such as the fracture risk Y, to diagnose the first subject, who is a patient, and to more appropriately diagnose the patient. Furthermore, even doctors who do not specialize in orthopedics can diagnose patients with an accuracy close to that of an orthopedic surgeon by referring to the output results of the information processing system 1.

Furthermore, in S14 of FIG. 4, the correction unit 23 performs correction to remove areas of tissue other than bone from the first image G1, thereby improving the accuracy of prediction of the fracture risk Y by the prediction unit 25.

[Embodiment 2]
Next, an information processing system 1A according to a second embodiment of the present disclosure will be described with reference to Figures 7 to 13. For ease of explanation, components having the same functions as those described in the first embodiment will be denoted by the same reference numerals, and their description will not be repeated.

[Configuration of information processing system]
The configuration of an information processing system 1A according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the configuration of the information processing system 1A. Fig. 7 shows the information processing system 1A including a prediction device 10A, an image management device 40, an electronic medical record management device 50, and a presentation device 60, but the configuration of the information processing system 1A is not limited to the configuration shown in Fig. 7.

The prediction device 10A is a device that acquires a first image G1a of a first subject, which is the target of prediction, and outputs prediction information from the acquired first image using a prediction model 32A.

Here, the first subject is, for example, a human. However, the first subject may be a non-human, such as an animal such as a dog, cat, or horse. Furthermore, the first image G1a may show either the bones or muscles of the first subject, or may show at least a portion of the bones and muscles of the first subject.

The first image G1a may be a medical image. The first image G1a is, for example, a plain X-ray image showing tissues of an area including at least one of the head, neck, chest, lower back, temporomandibular joints, spinal intervertebral joints, hip joints, sacroiliac joints, knee joints, ankle joints, feet, toes, shoulder joints, acromioclavicular joints, elbow joints, wrist joints, hands, fingers, and temporomandibular joints of the first subject. The tissues are, for example, bones and muscles. Note that the tissues may be either bones or muscles. The plain X-ray image may include, for example, a panoramic X-ray image used for dentistry. A panoramic X-ray image is an image that includes multiple teeth, for example, all of the teeth.

The first image G1a is captured at a third time point. The first image G1a includes at least one of a frontal image captured from the front of the first subject, for example, an image obtained by irradiating the target area with X-rays in the front-to-back direction, and a lateral image captured from the side, for example, an image obtained by irradiating the target area with X-rays in the left-to-right direction. The first image G1a may be, for example, a frontal chest X-ray image including a person's chest, or a frontal lumbar X-ray image including a person's lumbar region. A chest X-ray image is, for example, an image showing at least one of the ribs, clavicle, and sternum. A lumbar X-ray image is, for example, an image showing at least one of the lumbar vertebrae, pelvis, and femur. The first image G1a is not limited to the chest and lumbar region, and may also be an image showing, for example, the teeth, jaw, arm, hand, shoulder joint, knee joint, heel, skull, or foot bones.

The first image G1a is not limited to a simple X-ray image, but may be any image that includes information about at least one of bones and muscles. Alternatively, the first image G1a may be, for example, an MRI (Magnetic Resonance Imaging) image, a CT (Computed Tomography) image, a PET (Positron Emission Tomography) image, or an ultrasound image. If the first image G1a is a CT image, information about the bone trabeculae based on a three-dimensionally constructed image may be used, or information about the bone trabeculae based on a two-dimensionally captured image may be used. If the first image G1a is a CT image, for example, at least one of a three-dimensional image, a cross-sectional image perpendicular to the body axis connecting the head and legs (e.g., horizontal section), and a cross-sectional image parallel to the body axis (e.g., sagittal section or coronal section) may be used. The first image G1a may be an image that shows bones, or an image that does not show bones.

The prediction information output by the prediction device 10A is information indicating the possibility of an abnormality, such as a musculoskeletal disorder, occurring in the area captured in the first image G1a. In embodiment 2, the prediction information is a fracture risk indicating the possibility of a fracture occurring in the bone of the first subject at a fourth time point, which is different from the third time point. Note that a fracture is an example of a musculoskeletal disorder, and fragility fractures are assumed as fractures. In addition to fractures, musculoskeletal disorders include, for example, osteoporosis, osteoarthritis, spondylosis osteoarthritis, nerve disorders, sarcopenia, etc.

The first image G1a may be, for example, an image taken at a medical facility. The first image G1a shows an area of the first subject where an abnormality such as a fracture is expected. Note that the first image G1a may also show an area of the first subject other than an area where an abnormality such as a fracture is expected.

The fourth point in time refers to any point in time in the future or the past of the third point in time. The fourth point in time may include multiple points in time, such as one year, five years, ten years, and thirty years after the third point in time.

The image management device 40 is a computer that functions as a server for managing the first image G1a and the third image captured at a medical facility. The first image G1a may be a plain X-ray image showing at least a portion of the bones and muscles of the first subject. The third image may be a medical image. The third image may be a plain X-ray image showing the tissues of the second subject. The first image G1a and the third image may be stored in separate image management devices. The third image may be the same type of image as the first image, or may be a different type of image. For example, the first image G1a may be a plain X-ray image, and the third image may be a CT image.

The electronic medical record management device 50 is a computer that functions as a server for managing electronic medical record information of a first subject who has undergone a medical examination and/or test at a medical facility, etc. The image management device 40 and the electronic medical record management device 50 are connected to the acquisition unit 21 of the prediction device 10A.

The electronic medical record information includes attribute information of the first subject. The attribute information includes at least one of the following: the first subject's age, sex, height, weight, race, information on lifestyle habits, medication information, occupational information, blood test information, urine test information, saliva test information, information on existing diseases, medical history, medical history of the first subject's family, genetic information, childbirth information, menopausal information, items from the Fracture Risk Assessment Tool (FRAX (registered trademark)), and information regarding estimated menopause based on hormone information.

The birth information may include at least one of whether or not a subject has given birth, the number of children born, etc. The lifestyle habits may include, for example, sleep duration, wake-up time, sleep duration, daily exercise amount, dietary content, meal times, meal durations, and blood glucose levels. The dietary content may include, for example, at least one of the name of a dish, ingested ingredients, and intake amount. The dietary content may be, for example, an estimated intake of at least one of calcium, vitamin B, vitamin D, and vitamin K. The blood glucose level may be, for example, a specified value estimated from parameters acquired by a wearable device. The medication information may include, for example, information such as the name of the medication, the amount taken, and the duration of medication. The information regarding the medication taken may include information regarding the steroid drug being used. The blood test information may be, for example, information regarding the results of at least one of a biochemical test, a glucose metabolism test, and an endocrine system test. The information processing system 1A may acquire attribute information of the first subject from the electronic medical record management device 50, or, if attribute information is linked to the first image G1a, may acquire the attribute information from the first image G1a.

The presentation device 60 is a device for presenting information output by the prediction device 10A. The presentation device 60 is, for example, a liquid crystal display or an organic EL display. The presentation device 60 is controlled by the presentation control unit 26 to present numerical values such as estimated bone density, estimated bone quality, estimated muscle mass, and fracture risk, as well as support information for the first subject. The presentation device 60 may also be a device that prints the prediction information on paper or the like and outputs it.

[Configuration of prediction device]
Next, the configuration of the prediction device 10A will be described in detail with reference to Fig. 7. As shown in Fig. 7, the prediction device 10A includes a control unit 2 and a storage unit 3. The control unit 2 has, for example, a CPU (Central Processing Unit) and manages the operation of the prediction device 10A by comprehensively controlling each unit of the prediction device 10A. The control unit 2 has an acquisition unit 21, a learning unit 24, a prediction unit 25, an estimation unit 27, and a presentation control unit 26. The control unit 2 and the storage unit 3 are electrically connected to each other.

The acquisition unit 21 acquires a first image G1a from the image management device 40, which shows at least a portion of the bones and muscles of the first subject at the third time point. The acquisition unit 21 also acquires attribute information of the first subject from the electronic medical record management device 50. Note that the acquisition unit 21 may also acquire the first image G1a input by an input device (not shown).

The learning unit 24 controls the learning process that generates the estimation model 35 and the prediction model 32A. The estimation unit 27 outputs first estimated information about the bones of the first subject from the first image G1a, which shows at least a portion of the bones and/or muscles of the first subject, using at least one of the first estimation model 351, second estimation model 352, and third estimation model 353, which will be described later.

The first estimation information includes at least one of a bone density estimation value, which is information indicating the bone density of the bones of the first subject output from the first estimation model 351, a bone quality estimation value, which is information indicating the bone quality of the first subject output from the second estimation model 352, and a muscle mass estimation value, which is information indicating the muscle mass of the first subject output from the third estimation model 353.

The memory unit 3 is a computer-readable, non-transitory recording medium that stores the control program 31. The memory unit 3 is configured to include ROM (Read Only Memory), RAM (Random Access Memory), etc.

The control unit 2 controls the prediction device 10A by executing the control program 31. In other words, the control program 31 is a control program for causing the computer to function as the information processing system 1A, and causes the computer to function as the prediction unit 25 and the estimation unit 27.

In addition to the control program 31, the memory unit 3 also stores a prediction model 32A and an estimation model 35. The prediction model 32A and the estimation model 35 have AI (Artificial Intelligence). Specifically, the estimation model 35 has at least one AI from a first estimation model 351, a second estimation model 352, and a third estimation model 353.

Prediction model 32A was generated by machine learning using the third image and information about the bones of the second subject as explanatory variables, and abnormality information about abnormalities that occurred in the bones of the second subject at the second time point as the objective variable.

Here, the third image is an image showing the tissue of the second subject. The second subject may be the same person as the first subject, or a different person from the first subject. The third image may be captured at the same location as the first image G1a, or at a different location from the first image G1a. Furthermore, the third image may be captured by the same imaging device as the first image G1a, or by a different imaging device from the first image G1a. Furthermore, the first image G1a and the third image may be acquired using the same or different methods. For example, the first image G1a may be acquired from the electronic medical record management device 50, and the third image may be acquired from the image management device 40.

Furthermore, the third image may be an image showing the same area as the first image G1a, or an image showing a different area from the first image G1a. For example, the third image may be an image showing at least a part of the chest, the same as the first image G1a, or an image showing at least a part of the waist, different from the first image G1a. The third image may be an image in the same orientation as the first image G1a, or an image in a different orientation from the first image G1a. In other words, the third image may be a frontal image or a lateral image. Information about the bones of the second subject includes, for example, at least one of the following: possibility of osteoporosis, bone density, bone mass, bone quality, muscle mass, etc.

For example, bone-related information may indicate the possibility of osteoporosis, which is classified based on at least one of the presence or absence of a fracture, the possibility of a fracture, and changes in bone density, and may include no osteoporosis, suspected osteoporosis, or present osteoporosis. Specifically, the possibility of osteoporosis may be indicated when there is no disease that reduces bone mass and no secondary osteoporosis is observed, and when a fracture is present or there is a high possibility of a fracture, it may indicate primary osteoporosis.

Bone density can be measured by actually measuring bone density from at least one of the hand, lumbar vertebrae, proximal femur, tibia, heel, and arm (radius, etc.). Bone density can be measured using, for example, single-energy X-ray absorptiometry, dual-energy X-ray absorptiometry, ultrasound, MD (microdensitometry), or quantitative CT (quantitative computed tomography). In a DXA device that measures bone density using the DXA method, when measuring bone density of the lumbar vertebrae, X-rays are irradiated from the front of the subject's lumbar vertebrae. In the MD method, X-rays are irradiated onto the hand, for example.

Bone mineral density is a value related to bone density. Bone mineral density may be expressed by at least one of bone mineral density per unit area (g/cm2), bone mineral density per unit volume (g/cm3), YAM (%), T-score, and Z-score. YAM (%) is an abbreviation for "Young Adult Mean" and is sometimes called the young adult mean percent. For example, bone mineral density may be a value expressed as bone mineral density per unit area (g/cm2) and YAM (%). Bone mineral density may be an index defined in guidelines or a unique index. Bone mineral density may be the value described in osteoporosis guidelines, such as the "2015 Prevention and Treatment Guidelines of the Japan Osteoporosis Society."

Bone mass may be information measured using a bone density measuring device such as DXA, or information obtained by estimating bone density from X-ray images using a first estimation model. Furthermore, bone quality information may be at least one of bone formation markers, bone resorption markers, bone quality markers (e.g., vitamin K levels), cortical bone thickness, trabecular density, trabecular orientation, and trabecular bone score, but is not limited to these. Bone mass is the sum of bone mineral and bone matrix protein. In the present disclosure, bone mass is an index related to bone density, and is the amount of bone tissue in the skeleton.

As muscle mass, for example, the muscle mass [kg] for each body part measured using a body composition analyzer can be used. Furthermore, as muscle mass, for example, the area [cm2] and/or width [cm] of the muscle region imaged by MRI or DXA can be used. Furthermore, as muscle mass, for example, the muscle thickness [cm] of an ultrasound image can be used. Furthermore, as muscle mass, for example, the measured values [kg] of a muscle dynamometer for back muscles and/or grip strength, etc. can be used.

The second point in time refers to any point in time in the future or the past of the first point in time at which the third image was captured. The second point in time may include multiple points in time, such as one year, five years, ten years, and thirty years after the first point in time.

If the machine learning period of the prediction model 32A is, for example, five years, the time of the prediction information predicted by the prediction unit 25 may be five years from now, the same as the learning period, or it may be shorter than the learning period, for example, three years from now, or longer than the learning period, for example, eight years from now. The prediction unit 25 may also predict, for example, the time when the fracture risk will be 80% or more. The prediction unit 25 may also predict the time when the YAM will be 80% or less. In this case, it is sufficient for the prediction model 32A to have learned the time when the fracture risk will be 80% or more. In this case, it is sufficient for the prediction model 32A to have learned the time when the YAM will be 80% or less.

The estimation unit 27 uses at least one of the estimation models 35, which are the first estimation model 351, the second estimation model 352, and the third estimation model 353. The first estimation model 351 and the second estimation model 352 are examples of bone strength estimation models. The third estimation model 353 is an example of a bone load estimation model. The bone strength estimation model is generated by machine learning using bone strength information indicating the measurement results of at least one of the bone mineral density, bone mass, and bone quality of the bones of the second subject as the objective variable.

The first estimation model 351 is generated by machine learning using the third image as an explanatory variable and bone strength information indicating the measurement results of the bone density of the second subject as a target variable. The first estimation model 351 outputs information indicating the bone density of the bones of the first subject from the first image G1a. Here, the bone strength information includes information regarding bone density and information regarding bone quality. Note that the information regarding bone density and information regarding bone quality may be handled separately.

DXA (Dual-energy X-ray Absorptiometry) can be used as a method for measuring the bone density of the second subject. A DXA device that uses DXA to measure bone density irradiates the lumbar vertebrae with X-rays, specifically two types of X-rays, from the front when measuring the bone density of the lumbar vertebrae, for example. The DXA device may also measure the bone density of the lumbar vertebrae by irradiating the measurement area with X-rays from the side. Furthermore, the measurement area only needs to show at least a portion of the chest, proximal femur, knee joint, etc.

For example, when bone density of the proximal femur is measured using a DXA device, X-rays are irradiated from the front of the proximal femur of the second subject. Here, "front of the proximal femur" refers to the direction that correctly faces the imaging area, such as the proximal femur, and may be the ventral side of the body of the second subject, or the dorsal side of the second subject. The proximal femur includes, for example, at least one of the neck, trochanter, shaft, and the entire proximal femur (neck, trochanter, shaft, etc.).

The bone density of the second subject may be measured using ultrasound. In a device that measures bone density using ultrasound, for example, ultrasound is applied to the calcaneus to measure the bone density of the chest.

The second estimation model 352 is generated by machine learning using the third image as an explanatory variable and bone strength information indicating the measurement results of the bone quality of the second subject as a target variable. The second estimation model 352 outputs information indicating the bone quality of the first subject from the first image G1a.

The third estimation model 353 is generated by machine learning using the third image as an explanatory variable and bone load information indicating the measurement results of the muscle mass of the second subject as a dependent variable. The third estimation model 353 outputs information indicating the muscle mass of the first subject from the first image G1a. The bone load information is information indicating the results of measuring at least one of the muscle mass of the second subject and the posture of the second subject. Note that there is a relationship between muscle mass and bone load such that, for example, if the mass of muscles involved in maintaining posture, such as the rectus abdominis and/or erector spinae muscles, decreases, the load on the bones increases in order to maintain posture. Here, posture is indicated by the reference state of the second subject, for example, the degree of inclination of the body from an upright position.

The third estimation model 353 may be generated by machine learning using the third image as an explanatory variable and a fall risk objective variable indicating the possibility of the second subject falling, and may output information indicating the fall risk of the first subject from the first image G1a.

Muscle mass can be measured using at least one of the following methods: physical function measurement, measurement using a body composition scale, locomotive syndrome test, sarcopenia diagnosis, center of gravity sway measurement, lower limb muscle strength measurement, standing speed measurement, muscle thickness measurement using MRI, DXA, or ultrasound imaging diagnosis, etc.

[Operation of the estimation model]
Next, the operation of the estimation model 35 will be described with reference to Fig. 8. Fig. 8 is a block diagram for explaining the operation of the estimation model 35. The first estimation model 351, the second estimation model 352, and the third estimation model 353 constituting the estimation model 35 are, for example, convolutional neural networks (CNN). Note that the estimation model 35 may be configured using a neural network other than a convolutional neural network.

As shown in FIG. 8, the first estimation model 351 has, for example, an input layer 351a, a hidden layer 351b, and an output layer 351c. The first estimation model 351 includes first learned parameters that use the third image as an explanatory variable and bone strength information indicating the bone density measurement results of the second subject as a target variable.

In the first estimation model 351, a first image G1a is input to the input layer 351a, and a bone mineral density estimate E1 is output from the output layer 351c. Note that the hidden layer 351b may include, for example, multiple convolutional layers, multiple pooling layers, and a fully connected layer.

The bone mineral density estimate E1 is expressed by at least one of bone mineral density per unit area [g/cm ² ], bone mineral density per unit volume [g/cm ³ ], YAM (Young Adult Mean), T-score, and Z-score.

The second estimation model 352 has, for example, an input layer 352a, a hidden layer 352b, and an output layer 352c. The second estimation model 352 includes second learned parameters that use the third image as an explanatory variable and bone strength information indicating the measurement results of the bone quality of the second subject as an objective variable.

In the second estimation model 352, the first image G1a is input to the input layer 352a, and a bone quality estimate E2 is output from the output layer 352c. Note that the hidden layer 352b may include, for example, multiple convolutional layers, multiple pooling layers, and a fully connected layer.

The third estimation model 353 has, for example, an input layer 353a, a hidden layer 353b, and an output layer 353c. The third estimation model 353 includes third learned parameters that use the third image as an explanatory variable and bone load information indicating the measurement results of the muscle mass of the second subject as an objective variable. The first learned parameters, second learned parameters, and third learned parameters correspond to learned parameters for estimation.

In the third estimation model 353, the first image G1a is input to the input layer 353a, and a muscle mass estimate E3, which is information indicating the muscle mass of the first subject, is output from the output layer 353c. Note that the hidden layer 353b may include, for example, multiple convolutional layers, multiple pooling layers, and a fully connected layer.

[Predictive Model Operation]
Next, the operation of the prediction model 32A will be described with reference to Fig. 9. Fig. 9 is a block diagram for explaining the operation of the prediction model 32A in embodiment 2. The prediction model 32A is, for example, a convolutional neural network. Note that the prediction model 32A may be configured using a neural network other than a convolutional neural network.

As shown in FIG. 9, the prediction model 32A has, for example, an input layer 32a, a hidden layer 32b, and an output layer 32c. The prediction model 32A includes learned prediction parameters that use the third image captured at the first time point and information about the bones of the second subject as explanatory variables, and abnormality information about abnormalities that occurred in the bones of the second subject at the second time point as the objective variable. Note that bone abnormalities may include fractures, bone loss, primary osteoporosis, secondary osteoporosis, osteophyte formation, osteomalacia, bone metastasis of malignant tumors, multiple myeloma, vertebral hemangioma, spinal caries, pyogenic spondylitis, Paget's disease of bone, fibrous dysplasia, ankylosing spondylitis, etc.

In the prediction model 32A, a first image G1a captured at a third time point, an estimated bone density value E1, an estimated bone quality value E2, and an estimated muscle mass value E3 are input to the input layer 32a, and a fracture risk Y indicating the probability of a fracture occurring in the bone of the first subject at a fourth time point is output. Note that the first image G1a does not have to be input to the input layer 32a. The estimated bone density value E1, estimated bone quality value E2, and estimated muscle mass value E3, which are the first estimated information, are examples of first data.

In this case, the bone density estimate E1, bone quality estimate E2, and muscle mass estimate E3 input to the input layer 32a may be weighted based on the strength of the causal relationship with the occurrence of a fracture. For example, taking into account that bone strength is more influenced by bone density than by bone quality, the bone density estimate E1 may be weighted more heavily than the bone quality estimate E2. Note that weighting may be done according to predetermined standards such as guidelines, or original standards.

The estimated muscle mass value E3 is input into the prediction model 32A because it takes into account that a person's muscle mass is correlated with the strength that supports their bones, and affects the risk of fractures when they fall, for example.

[Learning process flow]
Next, the flow of the learning process by the learning unit 24 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the learning process by the learning unit 24. The learning unit 24 executes the learning process before the prediction device 10A performs a prediction process (to be described later) and stores a prediction model 32A and an estimation model 35 in the storage unit 3.

In the flowchart shown in FIG. 10, first, the learning unit 24 acquires a third image of the second subject via the acquisition unit 21 (S21). Here, the second subjects are multiple people who are different from each other. The third image is an image of the bones and muscles of the second subject captured at a first time point. Note that in S1, the learning unit 24 acquires attribute information of each second subject from the electronic medical record management device 50 and links this attribute information to each third image. However, if attribute information has been linked to the third image in advance, the learning unit 24 extracts the attribute information from the third image.

After S21, the learning unit 24 acquires information about the bones of the second subject via the acquisition unit 21 (S22). The information about the bones of the second subject includes information indicating the measurement results of the bone density and bone quality of the bones of the second subject, and information indicating the measurement results of the muscle mass of the second subject.

After S22, the learning unit 24 generates an estimation model 35 by machine learning using the third image acquired in S21 as an explanatory variable and the information about the bones of the second subject acquired in S22 as a target variable (S23).

In S23, the learning unit 24 performs machine learning using the third image as an explanatory variable and bone strength information indicating the bone density measurement results of the bones of the second subject as a target variable, to generate a first estimation model 351. The learning unit 24 may input multiple third images into the first estimation model 351, compare the output bone density estimates with the bone density measurement results, and adjust the first estimation model 351 using an error backpropagation method or the like so as to reduce the error between them.

The learning unit 24 also performs machine learning using the third image as an explanatory variable and bone strength information indicating the bone quality measurement results of the second subject as a target variable, to generate a second estimation model 352. The learning unit 24 then inputs multiple third images into the second estimation model 352, compares the output bone quality estimates with the bone quality measurement results, and adjusts the second estimation model 352 using backpropagation or the like to reduce the errors between them.

The learning unit 24 also performs machine learning using the third image as an explanatory variable and bone load information indicating the muscle mass measurement results of the second subject as a target variable, to generate a third estimation model 353. The learning unit 24 then inputs multiple third images into the third estimation model 353, compares the output muscle mass estimates with the muscle mass measurement results, and adjusts the third estimation model 353 using backpropagation or the like to reduce the error between them.

After S23, the learning unit 24 generates a prediction model 32A by machine learning using the third image captured at the first time point and information about the bones of the second subject as explanatory variables, and abnormality information about abnormalities that have occurred in the bones of the second subject at the second time point as a target variable (S24). Here, the abnormality information about abnormalities that have occurred in the bones is information about fractures. Note that the abnormality information may include bone loss, osteophyte formation, bone atrophy, bone sclerosis, and the like, in addition to fractures.

After S24, the learning unit 24 stores the generated estimation model 35 and prediction model 32A in the memory unit 3 (S25).

[Prediction process flow]
Next, the flow of the prediction process by the prediction device 10A will be described with reference to Fig. 11 to Fig. 13. Fig. 11 is a diagram showing a plain X-ray image of the chest of a first subject. Fig. 12 is a flowchart showing an example of the flow of the prediction process by the prediction device 10A.

The following describes a case where a plain X-ray image is taken from the front of the chest of a first subject as the first image G1a in a medical facility, as shown in Figure 11. Figure 11 shows the bone B and muscle M of the first subject. Note that in the plain X-ray image, bone B appears white and muscle M appears gray, making it possible to distinguish between bone B and muscle M based on differences in color and/or brightness in the plain X-ray image. This makes it possible to estimate the size, shape, etc. of bone B and muscle M.

In the flowchart shown in FIG. 12, first, the acquisition unit 21 acquires a first image G1a of the first subject from the image management device 40 (S31). The first image G1a is a simple X-ray image showing the bones B and muscles M of the first subject.

After S31, the estimation unit 27 reads out the first estimation model 351 from the storage unit 3, inputs the acquired first image G1a into the first estimation model 351, and outputs the bone mineral density estimate E1 (S32: first estimation step). The output bone mineral density estimate E1 is transmitted to the prediction unit 25. The estimation unit 27 may output the bone mineral density estimate E1 of the first subject at multiple future and past time points. The estimation unit 27 may also output the progression of the bone mineral density estimate E1 of the first subject from a past time point to a future time point.

The estimation unit 27 then reads out the second estimation model 352 from the storage unit 3, inputs the acquired first image G1a into the second estimation model 352, and outputs the bone quality estimation value E2 (S33: second estimation step). The output bone quality estimation value E2 is sent to the prediction unit 25.

Next, the estimation unit 27 reads out the third estimation model 353 from the memory unit 3, inputs the acquired first image G1a into the third estimation model 353, and outputs a muscle mass estimation value E3 (S34: third estimation step). The output muscle mass estimation value E3 is sent to the prediction unit 25. Here, the first estimation step S32, the second estimation step S33, and the third estimation step S34 correspond to estimation steps.

The estimation unit 27 may use at least one of the first estimation model 351, the second estimation model 352, and the third estimation model 353 to output at least one of the bone density estimate E1, the bone quality estimate E2, and the muscle mass estimate E3.

Furthermore, the estimation unit 27 may output a bone mass estimate using a fourth estimation model that outputs information indicating the bone mass of the first subject from the first image G1a. The fourth estimation model is generated by machine learning using the third image as an explanatory variable and bone strength information indicating the measurement results of the bone mass of the second subject as a target variable.

The estimation unit 27 may also output a posture estimation value using a fifth estimation model that outputs information indicating the posture of the first subject from the first image G1a. The fifth estimation model is generated by machine learning using the third image as an explanatory variable and bone load information indicating the measurement results of the posture of the second subject as a target variable. The posture estimation value may be the thoracic spine kyphotic angle (TKA), the lumbar forearm angle (LLA), the sacral inclination angle (SIA), or a combined value of these.

After S34, the prediction unit 25 reads out the prediction model 32A from the memory unit 3, inputs the first image G1a, the estimated bone density value E1, the estimated bone quality value E2, and the estimated muscle mass value E3 into the prediction model 32A, and outputs the fracture risk Y (S35: prediction step). The fracture risk Y corresponds to prediction information indicating the possibility of a fracture occurring in the bone of the first subject at a fourth time point, which is different from the third time point when the first image G1a was captured. Note that the prediction unit 25 may output the fracture risk Y for each case, taking into account whether or not the subject has undergone menopause and/or whether or not the subject has given birth.

The bone density estimate E1, bone quality estimate E2, and muscle mass estimate E3 output by the estimation unit 27, and the fracture risk Y output by the prediction unit 25 are transmitted to the presentation control unit 26. The presentation control unit 26 then presents the bone density estimate E1, bone quality estimate E2, muscle mass estimate E3, and fracture risk Y on the presentation device 60 (S36).

In S36, the presentation control unit 26 may present support information for supporting the first subject to the presentation device 60. In this case, the prediction unit 25 outputs support information for supporting the first subject to the presentation control unit 26 from the first image and the first estimated information using a prediction model 32A corresponding to the attribute information of the first subject.

The prediction model 32A is generated by machine learning using a third image corresponding to the attribute information of the second subject and information about the second subject's bones, including an estimated bone density, an estimated bone quality, and an estimated muscle mass, as explanatory variables, and support information that supports reducing the second subject's fracture risk as a target variable.

For example, if the estimated bone density value E1 is lower than the average bone density of people with the same or similar attribute information, such as age and gender, as the first subject, the presentation control unit 26 will display on the presentation device 60 a message encouraging the first subject to take in more calcium, get more sunlight, exercise, etc. Furthermore, if the estimated muscle mass value E3 is lower than the average muscle mass of people with the same or similar attribute information, such as age, gender, height, and weight, as the first subject, the presentation control unit 26 will display on the presentation device 60 a message encouraging the first subject to increase the amount of exercise.

The support information may be output by the prediction unit 25. For example, if the fracture risk Y output by the prediction unit 25 is high, the presentation control unit 26 displays support information on the presentation device 60 urging the first subject to avoid strenuous exercise.

。 In addition, in S36, the presentation control unit 26 may present to the presentation device 60 information indicating the time when a fracture is likely to occur in bone B of the first subject. In addition, in S36, the presentation control unit 26 may present to the presentation device 60 information indicating the probability of a fracture occurring in bone B of the first subject within a predetermined period of time. In this case, the prediction unit 25 may predict the predicted fracture time when a fracture is likely to occur, or the probability of a fracture occurring in bone B of the first subject within a predetermined period of time, taking into account the fracture risk Y, etc. The predicted fracture time may be in years, for example, 5 years or 10 years, or may be in months, for example, 5 years and 6 months or 10 years and 6 months. The predetermined period may be in years, for example, 3 years or 5 years, or may be in months, for example, 3 years and 6 months or 5 years and 6 months. In S36, the presentation control unit 26 may present to the presentation device 60 a message, for example, "Fracture risk Y will be 80% or more in 10 years." Furthermore, in S36, the presentation control unit 26 may display on the presentation device 60 a message stating, "The fracture risk Y within three years is 60%."

Furthermore, in S36, the presentation control unit 26 may present on the presentation device 60 the fracture risk Y at multiple points in time and a graph showing the progression of the fracture risk Y.

In the information processing system 1A in the second embodiment described above, the estimation unit 27 can estimate, as first estimated information, an estimated bone density value E1, an estimated bone quality value E2, and an estimated muscle mass value E3 of the bone B of the first subject from the first image G1a showing the bone B and muscle M of the first subject using three estimation models: a first estimation model 351, a second estimation model 352, and a third estimation model 353.

The prediction model 32A uses the first image G1a and at least one of the bone density estimate E1, bone quality estimate E2, and muscle mass estimate E3 of bone B to output the fracture risk Y of the first subject as prediction information. Specifically, the prediction unit 25 can accurately predict the fracture risk Y of a fracture occurring in the chest, which is the area captured in the first image G1a.

As a result, for example, a doctor at a medical facility can use the output results of information processing system 1A, such as fracture risk Y, to diagnose the first subject, who is a patient, allowing them to make a more appropriate diagnosis of the patient and provide the patient with more accurate support information. Furthermore, even a doctor who does not specialize in orthopedics can diagnose the patient with an accuracy close to that of an orthopedic surgeon by referring to the output results of information processing system 1A.

As support information, the doctor etc. may propose a treatment plan to increase bone density for the first subject, for example, whose bone density estimate E1 has a large impact on fracture risk Y. Furthermore, the doctor etc. may propose a treatment plan to improve bone quality as support information for the first subject, whose bone quality estimate E2 has a large impact on fracture risk Y. Furthermore, the doctor etc. may propose measures to increase muscle mass related to maintaining posture, or measures to improve posture as support information for the first subject, whose muscle mass estimate E3 has a large impact on fracture risk Y. Furthermore, based on the support information, the doctor etc. can decide whether to recommend a diet or exercise therapy to the first subject, whether to prescribe medication, or the type of medication to use, etc.

Here, Figure 13 shows the results of the prediction of the fracture risk Y of the first subject by the prediction device 10A and the effects of each countermeasure. Figure 13 shows an example in which the estimation unit 27 estimates that the bone density estimate E1 of the first subject is "0.985", the bone quality estimate E2 is "1.123", and the muscle mass estimate E3 is "20.54". Note that the higher the estimated values, the greater the strength of the bone B of the first subject and the greater the muscle mass. Figure 13 also shows an example in which the prediction unit 25 estimates that the fracture risk Y of the first subject is "0.42". The higher the value of fracture risk Y, the higher the likelihood of a fracture occurring.

In this way, the estimated bone density value E1, estimated bone quality value E2, estimated muscle mass value E3, and fracture risk Y are quantified and presented on the presentation device 60, allowing doctors and others at medical facilities to communicate more specific diagnosis results to the first subject.

Figure 13 also shows that when "Measure 1" of increasing bone density by 5% is implemented on the first subject, the fracture risk Y is reduced by "-12%." It also shows that when "Measure 2" of increasing bone quality by 5% is implemented on the first subject, the fracture risk Y is reduced by "-7%." It also shows that when "Measure 3" of increasing muscle mass by 7% is implemented on the first subject, the fracture risk Y is reduced by "-10%."

As shown in measures 1 to 3 above, by appropriately changing the values of the estimated bone density value E1, estimated bone quality value E2, and estimated muscle mass value E3, it is possible to identify the factors that are increasing the fracture risk Y. In this case, it is possible to identify that the bone density of the first subject is the factor that is increasing the fracture risk Y. Therefore, by recommending that the first subject increase their bone density, doctors and others can expect to effectively reduce the fracture risk Y of the first subject.

Other embodiment 1
In the information processing system 1 of the first embodiment described above, the learning unit 24 of the prediction device 10 generates the prediction model 32. However, this is not limiting, and the prediction model 32 may be generated by a device other than the prediction device 10. In this case, the prediction model 32 generated by the other device may be stored in the storage unit 3. The prediction model 32 generated by the other device may be received by a communication unit (not shown) via a communication network, and the control unit 2 may store the received prediction model 32 in the storage unit 3. With this configuration, there is no need to store the learning data 33 and the teacher data 34 in the storage unit 3.

Furthermore, in the information processing system 1 of the above-described first embodiment, the control unit 2 and the storage unit 3 are provided in the prediction device 10, but this is not limited to this. The prediction device 10 may be a cloud-based device installed on the cloud. In this case, the first image G1 and the second image G2 are transmitted to the prediction device 10 on the cloud via a communications network, and the prediction information predicted by the prediction device 10 is received by the presentation device 60 via the communications network. Furthermore, the prediction device 10 may be an on-premise device installed in a medical facility or a company that provides analysis services.

Furthermore, while the information processing system 1 of the first embodiment described above predicts the fracture risk Y of a fracture occurring in the bones of the first subject, this is not limited to this. The information processing system 1 may also predict the risk of developing osteoporosis, scoliosis, spinal stenosis, intervertebral disc degeneration, ankylosing spondylitis, spinal cord injury, cartilage damage, osteomyelitis, osteophytes, muscular atrophy, spinal muscular atrophy, osteoarthritis, bone and soft tissue tumors, etc. in the first subject.

Furthermore, in the information processing system 1 of the above-described first embodiment, the prediction device 10 outputs the fracture risk Y, which is the possibility of an abnormality occurring in a region shown in the first image G1, as prediction information, but this is not limited to this. The prediction information may also indicate the possibility of an abnormality occurring in a region not shown in the first image G1.

For example, the prediction unit 25 may use a prediction model 32 to predict the risk Y of a lumbar or femur fracture from a first image G1 showing the chest of a first subject and a second image G2 of a region corresponding to the chest. In this case, the prediction model 32 is generated by machine learning using a third image showing the chest of a second subject taken at a first time point and a fourth image showing a region corresponding to the chest as explanatory variables, and abnormality information related to a fracture that occurred in the lumbar or femur of the second subject at a second time point as a response variable.

Furthermore, in the information processing system 1 of the above-described first embodiment, the first image G1 and the second image G2 are input to the input layer 32a of the prediction model 32. However, this is not limited to this, and the results of the muscle strength measurement of the first subject may be input instead of the second image G2. In this case, the prediction model 32 may be generated by machine learning using the third image of the second subject taken at the first time point and the results of the muscle strength measurement of the second subject at the first time point as explanatory variables, and abnormality information regarding an abnormality that occurred in the bones of the second subject at the second time point as the objective variable.

Furthermore, in the information processing system 1 of the first embodiment described above, the prediction device 10 inputs the first image G1 corrected in S14 of FIG. 4 to the prediction model 32 in S15, but this is not limited to this. The prediction device 10 may not perform S13 and S14 of FIG. 4, and may input the first image that has not been corrected in S15 to the prediction model 32.

Furthermore, in the information processing system 1 of the first embodiment described above, the prediction device 10 uses a neural network as the prediction model 32, but this is not limited to this, and other models such as a linear regression model may also be used.

Other embodiment 2
In the prediction device 10 of the first embodiment described above, the prediction model 32 having one AI is stored in the storage unit 3, but this is not limiting, and multiple AIs may be stored in the storage unit 3. For example, the storage unit 3 may store a first estimation model that outputs information indicating the bone density and/or bone quality of the first subject, and a second estimation model that outputs information indicating the muscle mass of the first subject.

The first estimation model is generated by machine learning using the first image G1 of the second subject as an explanatory variable and bone density information indicating the measurement results of the bone density and/or bone quality of the second subject as a dependent variable. The second estimation model is generated by machine learning using the second image G2 of the second subject as an explanatory variable and muscle mass information indicating the measurement results of the muscle mass of the second subject as a dependent variable.

In S15 of FIG. 4, the prediction unit 25 inputs the first image G1 of the first subject into the first estimation model, and outputs a bone density estimate, which is information indicating the bone density of the first subject, and a bone quality estimate, which is information indicating the bone quality of the first subject.

Here, bone quality refers to a property based on at least one of the statistical properties of bone, the geometric properties of bone, the mechanical properties of bone, and the chemical properties of bone. Bone quality may also include information regarding the attribute information of the first subject.

Bone quality can be based on at least one of, for example, bone metabolism markers, sex, race, whether or not the patient has undergone menopause, age, cortical bone condition, cancellous bone condition, cancellous bone trabecular condition, disease information, bone evaluation information, medication information, presence or absence of fracture, number of fractures, location of fracture, and fracture history. More specifically, bone quality can be based on at least one of, for example, bone formation markers, bone resorption markers, bone quality markers (e.g., vitamin K level), cortical bone thickness, trabecular density, trabecular orientation, and trabecular bone score.

The disease information may include, for example, at least one of osteoporosis, rheumatism, osteonecrosis (e.g., femoral head necrosis, etc.), systemic sclerosis, kidney disease, and osteopetrosis. The bone assessment information may include information evaluated using a fracture risk assessment tool (FRAX (registered trademark): Fracture Risk Assessment Tool). The drug information may include, for example, at least one of the trade name, generic name, dosage, administration period, and administration method (e.g., oral, intravenous injection, intramuscular injection, subcutaneous injection, etc.) for drugs including at least one of drugs that inhibit bone resorption, drugs that promote bone formation, and other drugs (e.g., calcium preparations, vitamin preparations, female hormone preparations, etc.).

The bone quality may also include, for example, the type of medullary cavity shape. For example, the Dorr classification can be used for the medullary cavity shape. The medullary cavity shape can be classified as follows using at least one of the thickness of the cortical bone and the shape of the medullary cavity.
- Type A: The cortical bone is thick and the medullary cavity is narrow and thin.
- Type B: A type that is between Type A and Type C, with a medullary cavity that is neither narrow nor wide.
- Type C: A type in which the cortical bone is thin and the medullary cavity is wide.

The bone density estimate may be a value related to bone density. The bone density estimate is expressed, for example, by at least one of bone mineral density per unit area (g/cm2), bone mineral density per unit volume (g/cm3), YAM (%), T-score, and Z-score. YAM (%) is an abbreviation for "Young Adult Mean" and is sometimes called the young adult average percent. The bone density estimate may use an index used in osteoporosis guidelines, such as the "2015 Prevention and Treatment Guidelines of the Japan Osteoporosis Society," or may use an original index.

Furthermore, the prediction unit 25 inputs the second image G2 of the first subject into the second estimation model, thereby outputting a muscle mass estimate, which is information indicating the muscle mass of the first subject. Then, in S16 of FIG. 4, the presentation control unit 26 presents the bone density estimate, bone quality estimate, and muscle mass estimate on the presentation device 60, in addition to the fracture risk Y.

The first estimation model was generated by machine learning using the first image G1 of the second subject as an explanatory variable and bone density information indicating the measurement results of the second subject's bone density and bone quality as a dependent variable. The second estimation model was generated by machine learning using the second image G2 of the second subject as an explanatory variable and muscle mass information indicating the measurement results of the second subject's muscle mass as a dependent variable.

The bone density of the second subject can be measured using, for example, DXA, ultrasound, MD (Micro Densitometry), and CT (Quantitative Computed Tomography). In a DXA device that measures bone density using DXA, when measuring bone density of the lumbar vertebrae, X-rays are irradiated from the front of the subject's lumbar vertebrae. In addition, when measuring bone density of the proximal femur, X-rays are irradiated from the front of the subject's proximal femur. Here, "front of the lumbar vertebrae" and "front of the proximal femur" refer to the direction that correctly faces the imaging site, such as the lumbar vertebrae and proximal femur, and may be on the ventral side of the subject's body or on the back side of the subject. The proximal femur includes, for example, at least one of the neck, trochanter, shaft, and the entire proximal femur (neck, trochanter, and shaft). In the MD method, for example, X-rays are irradiated onto the hand.

The bone quality of the second subject can be measured by calculating the concentration of a bone metabolism marker in the urine or blood of the second subject. Examples of bone metabolism markers that can be used include type I collagen cross-linked N-telopeptide (NTX), type I collagen cross-linked C-telopeptide (CTX), tartrate-resistant acid phosphatase (TRACP-5b), and deoxypyridinoline (DPD).

Furthermore, the muscle mass of the second subject can be measured by, for example, physical function measurement, measurement using a body composition scale, locomotive syndrome test, sarcopenia diagnosis, center of gravity sway measurement, lower limb muscle strength measurement, standing speed measurement, muscle thickness measurement using ultrasound imaging diagnosis, etc.

According to the information processing system 1 of the above-described alternative embodiment 2, it is possible to output the estimated bone density, bone quality, and muscle mass of the first subject from the first image G1 and second image G2 of the first subject. This allows doctors and others at medical facilities to refer to the estimated bone density, bone quality, and muscle mass presented on the presentation device 60 and provide the first subject, who is a patient, with more specific diagnostic results that take into account each estimated value.

Other embodiment 3
In the above-described information processing system 1, in S16 of FIG. 4 , the presentation control unit 26 presents to the presentation device 60 the bone density estimate, the bone quality estimate, and the muscle mass estimate in addition to the fracture risk Y. However, the presentation control unit 26 may also present to the presentation device 60 support information for supporting the first subject.

In this case, the prediction unit 25 outputs support information to support the first subject by comparing the fracture risk Y, estimated bone density, estimated bone quality, and estimated muscle mass with reference information indicating bone density, bone quality, and muscle mass according to the age and/or sex of the first subject. Here, the estimated bone density, bone quality, and muscle mass correspond to the estimated information of the first subject.

For example, if the estimated bone density value is low compared to the average bone density of people of the same or similar age and sex as the first subject, the prediction unit 25 outputs support information to the presentation control unit 26 that encourages the first subject to take in more calcium, get more sunlight, exercise, etc. Furthermore, if the estimated muscle mass value is low compared to the average muscle mass of people of the same or similar age and sex as the first subject, the prediction unit 25 outputs support information to the presentation control unit 26 that encourages the first subject to increase the amount of exercise.

The prediction unit 25 may predict attribute information of the first subject by analyzing the brightness of the echo image, which is the second image G2, using the analysis unit 22. The attribute information includes at least one of the age, sex, and muscle quality of the first subject.

Other embodiment 4
In the information processing system 1 of the first embodiment described above, the prediction unit 25 outputs the fracture risk Y of the bones in the chest, which is a specific region, but this is not limiting. The prediction unit 25 may output the fracture risk for each of multiple regions of the bones of the first subject from the first image G1 and the second image G2 using the prediction model 32.

In this case, the first image G1 shows bones in multiple locations on the first subject. The second image G2 shows muscles in multiple locations on the first subject. The prediction model 32 is generated by machine learning using the third image, which shows bones in multiple locations on the second subject, and the fourth image, which shows muscles in multiple locations on the second subject, as explanatory variables, and abnormality occurrence information for each location that occurred within a specified period after the third image and/or the fourth image were captured as the objective variable. Note that the third image and the fourth image may be captured at different times or the same. If the third image and the fourth image are captured at different times, the date on which either image was captured can be used as the reference date. The reference date may also be a date halfway between the date on which the third image and the date on which the fourth image were captured.

With the above-described configuration, a doctor or other medical professional can inform the first subject of the fracture risk Y for each part of bone B, enabling a more detailed diagnosis. The prediction unit 25 may also combine the fracture risks Y for each part of bone B of the first subject to predict the fracture risk Y for all of the first subject's bones.

Furthermore, the prediction unit 25 may use the prediction model 32 to output the fracture risk Y, bone mineral density estimate, bone quality estimate, and muscle mass estimate for each of multiple bone regions of the first subject from the first image G1 and the second image G2. "By region" may be, for example, divided into regions such as the cervical vertebrae, thoracic vertebrae, and lumbar vertebrae, or may be divided into vertebral bodies such as the lumbar vertebrae L1, L2, L3, and L4.

The prediction unit 25 may then identify a region of interest that is highly related to fracture from among multiple regions of the bones of the first subject based on the fracture risk Y, the estimated bone density value, the estimated bone quality value, and the estimated muscle mass value, and present the region of interest via the presentation control unit 26. This allows doctors and other medical professionals to provide more appropriate treatment by focusing their examination on the region of interest when diagnosing the first subject.

Other embodiment 5
In the information processing system 1A of the second embodiment described above, the learning unit 24 of the prediction device 10A generates the prediction model 32A and the estimation model 35. However, this is not limiting, and the prediction model 32A and the estimation model 35 may be generated by a device other than the prediction device 10A. In this case, the prediction model 32A and the estimation model 35 generated by the other device may be stored in the storage unit 3, and the learning unit 24 may be omitted. Note that the prediction model 32A and the estimation model 35 generated by the other device may be received by a communication unit (not shown) via a communication network, and the control unit 2 may store the received prediction model 32A and the estimation model 35 in the storage unit 3. Alternatively, the prediction model 32A and the estimation model 35 generated by the other device may be recorded on a recording medium such as a USB memory or a DVD, and then the prediction model 32A and the estimation model 35 may be stored in the storage unit 3 via the recording medium.

Furthermore, in the information processing system 1A of the above-described second embodiment, the control unit 2 and the memory unit 3 are provided in the prediction device 10A, but this is not limited to this. The prediction device 10A may be a cloud-based device installed on the cloud. In this case, estimated information regarding the future bone condition of the first subject is transmitted to the prediction device 10A on the cloud via a communications network, and the prediction information predicted by the prediction device 10A is received by the presentation device 60 via the communications network. Furthermore, the prediction device 10A may be an on-premise device installed in a medical facility or a company that provides analysis services.

Furthermore, the information processing system 1A may be configured such that the imaging device that captures the first image G1 and the third image and the prediction device 10A are integrated into one unit. In this case, the image management device 40 and the electronic medical record management device 50 are not required.

In the information processing system 1A of the second embodiment described above, the prediction information indicates the possibility of an abnormality occurring in an area shown in the first image G1a, but this is not limited to this. The prediction information may also indicate the possibility of an abnormality occurring in an area not shown in the first image G1a.

For example, the prediction unit 25 may predict the risk Y of a lumbar or femur fracture from a first image G1a showing the chest of a first subject using a prediction model 32A. In this case, the prediction model 32A is generated by machine learning using a second image showing the chest of a second subject and information about the bones of the second subject as explanatory variables, and abnormality information about a fracture that has occurred in the lumbar or femur of the second subject as a response variable.

In the information processing system 1A of the second embodiment described above, the presentation device 60 presents information indicating the bone density of the first subject's bones, information indicating the bone quality, information indicating the muscle mass, and information regarding the fracture risk Y in numerical format, but this is not limited to this. For example, each piece of information may be presented in heat map format. Furthermore, the information indicating the bone quality may be a feature quantity obtained by texture analysis of at least a portion of the third image.

Other embodiment 6
In the information processing system 1A of the second embodiment described above, a neural network is used as the prediction model 32A and the estimation model 35, but this is not limiting, and other models such as a linear regression model may also be used.

In the above-described second embodiment, the prediction unit 25 outputs the fracture risk Y of the chest bone B using the first image G1a of the chest, which is a specific region, but this is not limited to this. The prediction unit 25 may also use the prediction model 32A to output abnormalities for each region of the bone B of the first subject. Here, outputting abnormalities for each region means, for example, outputting the fracture risk Y, etc. for each region, such as the cervical vertebrae, thoracic vertebrae, and lumbar vertebrae, or for each vertebra in each region (e.g., thoracic vertebrae T1-T12, lumbar vertebrae L1-L5, etc.).

The prediction model 32A is generated by machine learning using the third image and information about each part of bone B of the second subject as explanatory variables, and abnormality information about fractures that occurred in each part of bone B of the second subject at the second time point as the objective variable. With this configuration, a doctor or other medical professional can inform the first subject of the fracture risk Y for each part of bone B, allowing for a more detailed diagnosis. The prediction unit 25 may also combine the fracture risk Y for each part of bone B of the first subject to predict the fracture risk Y for all bones of the first subject.

[Software implementation example]
The functions of the prediction devices 10, 10A can be realized by a program that causes a computer to function as the prediction devices 10, 10A, and a program that causes a computer to function as each control block (particularly, the prediction unit 25 and the presentation control unit 26) of the prediction devices 10, 10A.

In this case, the prediction devices 10 and 10A include a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the programs. By executing the programs using this control device and storage device, the functions described in each of the above embodiments are realized.

The above program may be stored non-temporarily on one or more computer-readable storage media. These storage media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

Furthermore, some or all of the functions of each of the above control blocks can be realized by logic circuits. For example, integrated circuits incorporating logic circuits that function as each of the above control blocks are also included in the scope of this disclosure. In addition, the functions of each of the above control blocks can also be realized by, for example, a quantum computer.

The invention according to the present disclosure has been described above based on various drawings and examples. However, the invention according to the present disclosure is not limited to the above-described embodiments. In other words, the invention according to the present disclosure can be modified in various ways within the scope set forth in this disclosure, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the invention according to the present disclosure. In other words, it should be noted that a person skilled in the art would easily be able to make various modifications or corrections based on this disclosure. It should also be noted that these modifications or corrections are included in the scope of this disclosure.

[Summary 1]
An information processing system according to a first aspect of the present disclosure includes a prediction unit that outputs prediction information using a prediction model based on a first image and first data showing at least a portion of a first subject. The prediction model is generated by machine learning using a third image and second data showing at least a portion of a second subject as explanatory variables, and abnormality information regarding an abnormality occurring in the bones of the second subject at a second time point different from a first time point when the third image was captured as a dependent variable. The prediction information is information indicating the possibility of an abnormality occurring in the bones of the first subject.

In the information processing system according to aspect 2 of the present disclosure, in aspect 1 above, the first data is data including a second image. The second data may also be data including a fourth image.

In the information processing system according to aspect 3 of the present disclosure, in aspect 2 above, the first image is an image depicting a predetermined region of the first subject. The second image is an image depicting a region of the first subject corresponding to the predetermined region. The third image is an image depicting a predetermined region of the second subject. The fourth image may be an image depicting a region of the second subject corresponding to the predetermined region.

In the information processing system according to aspect 4 of the present disclosure, in any of aspects 1 to 3 above, the prediction information may be information indicating the possibility of an abnormality occurring in the bone of the first subject at a fourth time point that is different from the third time point at which the first image was captured.

In the information processing system according to aspect 5 of the present disclosure, in any one of aspects 2 to 4 above, the first image may show bones at multiple locations on the first subject, the second image may show muscles at multiple locations on the first subject, the third image may show bones at multiple locations on the second subject, and the fourth image may show muscles at multiple locations on the second subject.

In the information processing system according to aspect 6 of the present disclosure, in any of aspects 2 to 5 above, the prediction model may be generated by machine learning using the third image and the fourth image as explanatory variables and the abnormality information for each of the parts that has occurred within a predetermined period since the third image and/or the fourth image was captured as a target variable, and the prediction unit may use the prediction model to output the prediction information for each of the multiple parts of the bones of the first subject from the first image and the second image.

In the information processing system according to aspect 7 of the present disclosure, in aspect 5 or 6 above, the body parts may include at least one of the chest, waist, feet, and hands.

In the information processing system according to aspect 8 of the present disclosure, in any of aspects 2 to 7 above, the second image may show one region or multiple regions of the first subject.

In the information processing system according to aspect 9 of the present disclosure, in any of aspects 2 to 8 above, the second image may include at least one of a still image and a video.

In the information processing system according to aspect 10 of the present disclosure, in any of aspects 2 to 9 above, the first image and the second image may include at least one of a plain X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a DXA (Dual Energy X-ray Absorptiometry) image, an echo image, and an image obtained by DES (Dual Energy Subtraction).

In the information processing system according to aspect 11 of the present disclosure, in any of aspects 2 to 10 above, the second image may be a different image type from the first image. The fourth image may be a different image type from the third image.

In the information processing system according to aspect 12 of the present disclosure, in aspects 3 to 11 above, the prediction information may be information indicating the possibility that the abnormality will occur in the specified area shown in the first image.

In the information processing system according to aspect 13 of the present disclosure, in any of aspects 3 to 12 above, the prediction information may be information indicating the possibility that the abnormality will occur in a region other than the predetermined region that is not captured in the first image.

In the information processing system according to aspect 14 of the present disclosure, in any of aspects 2 to 13 above, the abnormality may be a musculoskeletal disorder.

In an information processing system according to aspect 15 of the present disclosure, in any of aspects 2 to 14 above, the prediction unit may output the bone density and/or bone quality of the first subject from the first image of the first subject using a first estimation model, and output the muscle mass of the first subject from the second image of the first subject using a second estimation model, the first estimation model being generated by machine learning using the third image of the second subject as an explanatory variable and bone information indicating the bone density and/or bone quality of the second subject as an objective variable, and the second estimation model being generated by machine learning using the fourth image of the second subject as an explanatory variable and muscle information indicating the muscle mass of the second subject as an objective variable.

In the information processing system according to aspect 16 of the present disclosure, in aspect 15 above, the prediction model may be generated by machine learning using at least one of the third image, the fourth image, the bone information obtained by inputting the third image into the first estimation model, and the muscle information obtained by inputting the fourth image into the second estimation model as explanatory variables, and the abnormality information as a target variable.

The information processing system according to aspect 17 of the present disclosure is in any one of aspects 2 to 16 above, further comprising an analysis unit that uses the second image to analyze information including at least one of the amount, thickness, amount of atrophy, and flexibility of muscle and fat of the first subject, and a correction unit that performs a predetermined correction on the first image. The analysis unit may identify a soft tissue region by segmenting the second image, the correction unit may perform a correction on the first image to remove the soft tissue region identified by the analysis unit, and the prediction unit may output the prediction information using the prediction model from the first image corrected by the correction unit and the second image.

In the information processing system according to aspect 18 of the present disclosure, in accordance with aspect 17 above, the second image may include an echo image, and the prediction unit may predict attribute information of the first subject by analyzing the brightness of the echo image using the analysis unit.

In the information processing system according to aspect 19 of the present disclosure, in aspect 18 above, the attribute information may include at least one of the age, sex, and muscle quality of the first subject.

In the information processing system according to aspect 20 of the present disclosure, in any of aspects 2 to 19 above, the prediction unit outputs support information to support the first subject using at least one of the predicted information, estimated information about the first subject, and reference information, and the estimated information may be at least one of the bone density, bone quality, and muscle mass of the first subject, and the reference information may be at least one of the bone density, bone quality, and muscle mass according to the age and/or sex of the first subject.

In the information processing system according to aspect 21 of the present disclosure, in aspect 20 above, the prediction unit may identify a region of interest that is highly related to the abnormality based on the prediction information and/or the estimation information.

In an information processing system according to aspect 22 of the present disclosure, in any of aspects 2 to 21 above, the prediction information may include an influence degree indicating the degree to which each of the first image and the second image affects the abnormality.

In the information processing system according to aspect 23 of the present disclosure, in any of aspects 1 to 22 above, the prediction information may include information indicating a time when the abnormality is likely to occur in the first subject.

The information processing system according to aspect 24 of the present disclosure may be any of aspects 1 to 23 above, and may include a presentation control unit that causes the presentation device to present the prediction information.

The information processing system according to aspect 25 of the present disclosure is in accordance with aspect 1 above, further comprising an estimation unit that outputs the first data including first estimated information related to the bones of the first subject from the first image using an estimation model. The estimation model is generated by machine learning using the third image as an explanatory variable and the second data including information related to the bones of the second subject as a target variable.

The information processing system according to aspect 26 of the present disclosure is in accordance with aspect 1 above, further comprising an estimation unit that outputs the first data including multiple pieces of first estimated information related to the bones of the first subject from the first image using multiple estimation models. The multiple estimation models are generated by machine learning using the third image as an explanatory variable and the second data including multiple pieces of information related to the bones of the second subject as a target variable.

In the information processing system according to aspect 27 of the present disclosure, in aspect 26 above, the prediction information may be information indicating the possibility of an abnormality occurring in the tissue of the first subject at a fourth time point that is different from the third time point at which the first image was captured.

In the information processing system according to aspect 28 of the present disclosure, in any of aspects 25 to 27 above, the prediction information may be information indicating the possibility that the abnormality will occur in the area shown in the first image.

In the information processing system according to aspect 29 of the present disclosure, in any of aspects 25 to 27 above, the prediction information may be information indicating the possibility that the abnormality will occur in an area not captured in the first image.

In the information processing system according to aspect 30 of the present disclosure, in any of aspects 25 to 29 above, the first image may be a plain X-ray image showing at least a portion of the bones and/or muscles of the first subject, and the third image may be a plain X-ray image showing at least a portion of the bones and/or muscles of the second subject.

In the information processing system according to aspect 31 of the present disclosure, in any of aspects 25 to 30 above, the first image may be a front image or a side image, and the third image may be an image oriented in the same direction as the first image.

In the information processing system according to aspect 32 of the present disclosure, in any of aspects 25 to 31 above, the estimation unit may use at least one of a bone strength estimation model generated by machine learning using the third image as an explanatory variable and bone strength information indicating at least one measurement result of the bone mineral density, bone mass, and bone quality of the bones of the second subject as an objective variable, and a bone stress estimation model generated by machine learning using the third image as an explanatory variable and bone stress information indicating at least one measurement result of the muscle mass and posture of the second subject as an objective variable.

In the information processing system according to aspect 33 of the present disclosure, in the above-mentioned aspect 32, the bone strength estimation model includes a first estimation model that outputs information indicating the bone density of the bone of the first subject from the first image, and a second estimation model that outputs information indicating the bone quality of the first subject from the first image. The bone load estimation model may include a third estimation model that outputs information indicating the muscle mass of the first subject from the first image.

In the information processing system according to aspect 34 of the present disclosure, in the above aspect 33, the first estimated information includes two or more pieces of information from the group consisting of information indicating the bone density of the bone of the first subject output from the first estimation model, information indicating the bone quality of the first subject output from the second estimation model, and information indicating the muscle mass of the first subject output from the third estimation model, and the prediction unit may weight each of the two or more pieces of information based on the strength of the causal relationship with the occurrence of the abnormality and input the weighted information into the prediction model.

In the information processing system according to aspect 35 of the present disclosure, in aspect 33 above, the information indicating the bone density of the first subject may be expressed by at least one of bone mineral density per unit area, bone mineral density per unit volume, YAM (Young Adult Mean), T-score, and Z-score.

In the information processing system according to aspect 36 of the present disclosure, in the above aspect 32, the bone strength information is information measured using a method including at least one of DXA (Dual-energy X-ray Absorptiometry), ultrasound, and a method of calculating the concentration of a bone metabolic marker in the urine or blood of the second subject, and the bone load information may be information indicating the results of measuring at least one of the muscle mass of the second subject and the posture of the second subject.

In the information processing system according to aspect 37 of the present disclosure, in any of aspects 25 to 36 above, the abnormality may be a musculoskeletal disorder.

In the information processing system according to aspect 38 of the present disclosure, in aspect 27 above, the prediction model may be generated by machine learning using the third image and/or information for each bone region of the second subject as explanatory variables and the abnormality information regarding the abnormality that occurred for each bone region of the second subject at the second time point as a target variable, and the prediction information may be information indicating the possibility of the abnormality occurring for each tissue region of the first subject at the fourth time point.

In the information processing system according to aspect 39 of the present disclosure, in aspect 27 above, the prediction information may include information indicating a time when the abnormality is likely to occur in the tissue of the first subject.

In the information processing system according to aspect 40 of the present disclosure, in any of aspects 25 to 39 above, the prediction unit may output support information for supporting the first subject from the first image and/or the first estimated information using the prediction model corresponding to attribute information of the first subject.

The prediction device according to aspect 41 of the present disclosure may be any of aspects 25 to 40 above, and may include a presentation control unit that causes the presentation device to present the prediction information.

A prediction device according to aspect 42 of the present disclosure includes the prediction unit in the information processing system of any one of aspects 1 to 24 above.

A prediction device according to aspect 43 of the present disclosure includes the estimation unit and the prediction unit in the information processing system of any one of aspects 25 to 41 above.

An information processing method according to aspect 44 of the present disclosure is an information processing method executed by one or more computers, and includes a prediction step of outputting predicted information using a prediction model from a first image and first data that show at least a portion of a first subject. The prediction model is generated by machine learning using a third image and second data that show at least a portion of a second subject as explanatory variables, and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point when the third image was captured as a target variable. The predicted information is information that indicates the possibility of an abnormality occurring in the bones of the first subject.

In the information processing method according to aspect 45 of the present disclosure, in aspect 44 above, the first data is data including a second image. The second data may also be data including a fourth image.

The information processing method according to aspect 46 of the present disclosure is in accordance with aspect 44 above, further comprising an estimation step of outputting, from the first image, the first data including first estimated information related to the bones of the first subject using an estimation model. The estimation model may be generated by machine learning using the third image as an explanatory variable and the second data including information related to the bones of the second subject as a target variable.

The information processing method according to aspect 47 of the present disclosure is in accordance with aspect 44 above, further comprising an estimation step of outputting the first data, including multiple first estimated pieces of information related to the bones of the first subject, from the first image using multiple estimation models. The multiple estimation models may be generated by machine learning using the third image as an explanatory variable and the second data, including multiple pieces of information related to the bones of the second subject, as an objective variable, and the prediction model may be generated by machine learning using the second data as an explanatory variable and abnormality information related to an abnormality that occurred in the bones of the second subject at a second time point different from the first time point when the third image was captured as an objective variable.

The control program according to aspect 48 of the present disclosure may be a control program for causing a computer to function as any of the information processing systems of aspects 1 to 24 above, and may be a control program for causing the computer to function as the prediction unit.

The control program according to aspect 49 of the present disclosure may be a control program for causing a computer to function as any of the information processing systems of aspects 25 to 41 above, and may be a control program for causing the computer to function as the estimation unit and the prediction unit.

The recording medium according to aspect 50 of the present disclosure may be a computer-readable, non-transitory recording medium on which the control program according to aspect 48 above is recorded.

The recording medium according to aspect 51 of the present disclosure may be a computer-readable, non-transitory recording medium on which the control program according to aspect 49 above is recorded.

[Summary 2]
An information processing system according to aspect A1 of the present disclosure includes a prediction unit that uses a prediction model to output predictive information from a first image and a second image that show at least a portion of a first subject, the predictive model being generated by machine learning using a third image and a fourth image that show at least a portion of a second subject as explanatory variables and abnormality information relating to an abnormality that occurred in the bones of the second subject at a second time point that is different from a first time point when the third image was captured as a dependent variable, and the predictive information being information indicating the possibility of an abnormality occurring in the bones of the first subject.

In the information processing system according to aspect A2 of the present disclosure, in aspect A1 above, the first image may be an image depicting a predetermined region of the first subject, the second image may be an image depicting a region of the first subject corresponding to the predetermined region, the third image may be an image depicting a predetermined region of the second subject, and the fourth image may be an image depicting a region of the second subject corresponding to the predetermined region.

In the information processing system according to aspect A3 of the present disclosure, in aspect A1 or A2 above, the prediction information may be information indicating the possibility of an abnormality occurring in the bone of the first subject at a fourth time point that is different from the third time point at which the first image was captured.

In the information processing system according to aspect A4 of the present disclosure, in any of aspects A1 to A3 above, the first image may show bones at multiple locations on the first subject, the second image may show muscles at multiple locations on the first subject, the third image may show bones at multiple locations on the second subject, and the fourth image may show muscles at multiple locations on the second subject.

In the information processing system according to aspect A5 of the present disclosure, in aspect A4 above, the prediction model may be generated by machine learning using the third image and the fourth image as explanatory variables and the abnormality information for each of the parts that has occurred within a predetermined period since the third image and/or the fourth image was captured as a target variable, and the prediction unit may use the prediction model to output the prediction information for each of the multiple parts of the bones of the first subject from the first image and the second image.

In the information processing system according to aspect A6 of the present disclosure, in aspect A4 or A5 above, the body parts may include at least one of the chest, waist, feet, and hands.

In the information processing system according to aspect A7 of the present disclosure, in any of aspects A1 to A6 above, the second image may show one region or multiple regions of the first subject.

In the information processing system according to aspect A8 of the present disclosure, in any of aspects A1 to A7 above, the second image may include at least one of a still image and a video.

In the information processing system according to aspect A9 of the present disclosure, in any of aspects A1 to A8 above, the first image and the second image may include at least one of a plain X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a DXA (Dual Energy X-ray Absorptiometry) image, an echo image, and an image obtained by DES (Dual Energy Subtraction).

In the information processing system according to aspect A10 of the present disclosure, in any of aspects A1 to A9 above, the second image may be a different image type from the first image, and the fourth image may be a different image type from the third image.

In the information processing system according to aspect A11 of the present disclosure, in aspect A2 above, the prediction information may be information indicating the possibility that the abnormality will occur in the specified area shown in the first image.

In the information processing system according to aspect A12 of the present disclosure, in aspect A2 above, the prediction information may be information indicating the possibility of the abnormality occurring in a region other than the predetermined region that is not captured in the first image.

In the information processing system according to aspect A13 of the present disclosure, in any of aspects A1 to A12 above, the abnormality may be a musculoskeletal disorder.

In the information processing system according to aspect A14 of the present disclosure, in any of aspects A1 to A13 above, the prediction unit may output the bone density and/or bone quality of the first subject from the first image of the first subject using a first estimation model, and output the muscle mass of the first subject from the second image of the first subject using a second estimation model, the first estimation model being generated by machine learning using the third image of the second subject as an explanatory variable and bone information indicating the bone density and/or bone quality of the second subject as an objective variable, and the second estimation model being generated by machine learning using the fourth image of the second subject as an explanatory variable and muscle information indicating the muscle mass of the second subject as an objective variable.

In the information processing system according to aspect A15 of the present disclosure, in any of aspects A1 to A14 above, the prediction model may be generated by machine learning using at least one of the third image, the fourth image, the bone information obtained by inputting the third image into the first estimation model, and the muscle information obtained by inputting the fourth image into the second estimation model as explanatory variables, and the abnormality information as a target variable.

The information processing system according to aspect A16 of the present disclosure is any of aspects A1 to A15 above, further comprising an analysis unit that uses the second image to analyze information including at least one of the amount, thickness, amount of atrophy, and flexibility of muscle and fat of the first subject, and a correction unit that performs a predetermined correction on the first image. The analysis unit identifies soft tissue regions by segmenting the second image. The correction unit performs correction on the first image to remove the soft tissue regions identified by the analysis unit. The prediction unit may output the prediction information using the prediction model from the first image corrected by the correction unit and the second image.

In the information processing system according to aspect A17 of the present disclosure, in the above-mentioned aspect A16, the second image includes an echo image. The prediction unit may predict attribute information of the first subject by analyzing the brightness of the echo image using the analysis unit.

In the information processing system according to aspect A18 of the present disclosure, in aspect A17 above, the attribute information may include at least one of the age, sex, and muscle quality of the first subject.

In the information processing system according to aspect A19 of the present disclosure, in any of aspects A1 to A18 above, the prediction unit outputs support information to support the first subject using at least one of the predicted information, estimated information about the first subject, and reference information. The estimated information may be at least one of the bone density, bone quality, and muscle mass of the first subject, and the reference information may be at least one of the bone density, bone quality, and muscle mass according to the age and/or sex of the first subject.

In the information processing system according to aspect A20 of the present disclosure, in aspect A19 above, the prediction unit may identify a region of interest that is highly related to the abnormality based on the prediction information and/or the estimation information.

In the information processing system according to aspect A21 of the present disclosure, in any of aspects A1 to A20 above, the prediction information may include an influence degree indicating the degree to which each of the first image and the second image affects the abnormality.

In the information processing system according to aspect A22 of the present disclosure, in any of aspects A1 to A21 above, the prediction information may include information indicating a time when the abnormality is likely to occur in the first subject.

The information processing system according to aspect A23 of the present disclosure may be any of aspects A1 to A22 above, and may include a presentation control unit that causes the presentation device to present the prediction information.

A prediction device according to aspect A24 of the present disclosure includes the prediction unit in the information processing system of any one of aspects A1 to A23 above.

An information processing method according to aspect A25 of the present disclosure is an information processing method executed by one or more computers, and includes a prediction step of outputting predicted information using a prediction model from a first image and a second image that show at least a portion of a first subject, wherein the prediction model is generated by machine learning using a third image and a fourth image that show at least a portion of a second subject as explanatory variables and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point when the third image was captured as a target variable, and the predicted information is information that indicates the possibility of an abnormality occurring in the bones of the first subject.

The control program according to aspect A26 of the present disclosure may be a control program for causing a computer to function as any of the information processing systems of aspects A1 to A23 described above, and may be a control program for causing the computer to function as the prediction unit.

The recording medium according to aspect A27 of the present disclosure may be a computer-readable, non-transitory recording medium on which the control program of aspect A26 above is recorded.

[Summary 3]
An information processing system according to aspect B1 of the present disclosure includes an estimation unit that outputs first estimated information regarding a bone of a first subject using an estimation model from a first image capturing at least a portion of the tissue of the first subject, and a prediction unit that outputs predicted information from the first image and the first estimated information using a prediction model. The estimation model is generated by machine learning using a second image capturing the tissue of a second subject as an explanatory variable and information about the bone of the second subject as a dependent variable. The prediction model is generated by machine learning using the second image and information about the bone of the second subject as explanatory variables and anomaly information regarding an abnormality that occurred in the bone of the second subject at a second time point different from the first time point when the second image was captured as a dependent variable. The predicted information is information indicating the possibility of an abnormality occurring in the tissue of the first subject.

An information processing system according to aspect B2 of the present disclosure includes an estimation unit that uses a plurality of estimation models to output a plurality of first estimated information items related to the bones of a first subject from a first image that shows at least a portion of the tissue of the first subject, and a prediction unit that uses a prediction model to output predicted information from the plurality of first estimated information items. The plurality of estimation models are generated by machine learning using a second image that shows the tissue of a second subject as an explanatory variable and a plurality of pieces of information related to the bones of the second subject as an objective variable. The prediction models are each generated by machine learning using a plurality of pieces of information related to the bones of the second subject as an explanatory variable and abnormality information related to an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point when the second image was captured as an objective variable. The predicted information is information indicating the possibility of an abnormality occurring in the tissue of the first subject.

In the information processing system according to aspect B3 of the present disclosure, in aspect B1 or B2 above, the prediction information may be information indicating the possibility of an abnormality occurring in the tissue of the first subject at a fourth time point that is different from the third time point at which the first image was captured.

In the information processing system according to aspect B4 of the present invention, in any of aspects B1 to B3 above, the prediction information may be information indicating the possibility that the abnormality will occur in the area shown in the first image.

In the information processing system according to aspect B5 of the present invention, in any of aspects B1 to B3 above, the prediction information may be information indicating the possibility that the abnormality will occur in an area not captured in the first image.

In the information processing system according to aspect B6 of the present disclosure, in any of aspects B1 to B5 above, the first image may be a plain X-ray image showing at least a portion of the bones and/or muscles of the first subject, and the second image may be a plain X-ray image showing at least a portion of the bones and/or muscles of the second subject.

In the information processing system according to aspect B7 of the present disclosure, in any of aspects B1 to B6 above, the first image may be a front image or a side image, and the second image may be an image oriented in the same direction as the first image.

In the information processing system according to aspect B8 of the present disclosure, in any of aspects B1 to B7 above, the estimation unit may use at least one of a bone strength estimation model generated by machine learning using the second image as an explanatory variable and bone strength information indicating at least one measurement result of the bone mineral density, bone mass, and bone quality of the bones of the second subject as an objective variable, and a bone stress estimation model generated by machine learning using the second image as an explanatory variable and bone stress information indicating at least one measurement result of the muscle mass and posture of the second subject as an objective variable.

In the information processing system according to aspect B9 of the present disclosure, in the above-mentioned aspect B8, the bone strength estimation model includes a first estimation model that outputs information indicating the bone density of the bone of the first subject from the first image, and a second estimation model that outputs information indicating the bone quality of the first subject from the first image. The bone load estimation model may include a third estimation model that outputs information indicating the muscle mass of the first subject from the first image.

In the information processing system according to aspect B10 of the present disclosure, in the above aspect B9, the first estimated information includes two or more pieces of information from the following: information indicating the bone mineral density of the bone of the first subject output from the first estimation model; information indicating the bone quality of the first subject output from the second estimation model; and information indicating the muscle mass of the first subject output from the third estimation model. The prediction unit may weight each of the two or more pieces of information based on the strength of the causal relationship with the occurrence of the abnormality, and input the weighted information into the prediction model.

In the information processing system according to aspect B11 of the present disclosure, in aspect B9 above, the information indicating the bone density of the first subject may be expressed by at least one of bone mineral density per unit area, bone mineral density per unit volume, YAM (Young Adult Mean), T-score, and Z-score.

In the information processing system according to aspect B12 of the present disclosure, in aspect B8 above, the bone strength information is information measured using a method including at least one of DXA (Dual-energy X-ray Absorptiometry), ultrasound, and a method of calculating the concentration of a bone metabolic marker in the urine or blood of the second subject. The bone load information may be information indicating the results of measuring at least one of the muscle mass of the second subject and the posture of the second subject.

In the information processing system according to aspect B13 of the present disclosure, in any of aspects B1 to B12 above, the abnormality may be a musculoskeletal disorder.

In the information processing system according to aspect B14 of the present disclosure, in any of aspects B3 to B13 above, the prediction model is generated by machine learning using the second image and/or information for each bone region of the second subject as explanatory variables, and the abnormality information regarding the abnormality that occurred for each bone region of the second subject at the second time point as a target variable. The prediction information may be information indicating the possibility of the abnormality occurring for each tissue region of the first subject at the fourth time point.

In the information processing system according to aspect B15 of the present disclosure, in any of aspects B1 to B14 above, the prediction information may include information indicating a time when the abnormality is likely to occur in the tissue of the first subject.

In the information processing system according to aspect B16 of the present disclosure, in any of aspects B1 to B15 above, the prediction unit may output support information for supporting the first subject from the first image and/or the first estimated information using the prediction model corresponding to attribute information of the first subject.

The information processing system according to aspect B17 of the present disclosure may be any of aspects B1 to B16 above, and may include a presentation control unit that causes the presentation device to present the prediction information.

A prediction device according to aspect B18 of the present disclosure is any of aspects B1 to B17 above, and includes the estimation unit and the prediction unit in the information processing system.

An information processing method according to aspect B19 of the present disclosure is an information processing method executed by one or more computers, and includes an estimation step of outputting first estimated information regarding the bones of a first subject from a first image that shows at least a portion of the tissue of the first subject using an estimation model, and a prediction step of outputting predicted information from the first image and the first estimated information using a prediction model. The estimation model is generated by machine learning using a second image that shows the tissue of a second subject as an explanatory variable and information about the bones of the second subject as a dependent variable. The prediction model is generated by machine learning using the second image and information about the bones of the second subject as explanatory variables and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point when the second image was captured as a dependent variable. The predicted information is information indicating the possibility of an abnormality occurring in the tissue of the first subject.

An information processing method according to aspect B20 of the present disclosure is an information processing method executed by one or more computers, and includes an estimation step of outputting, from a first image showing at least a portion of the tissue of a first subject, a plurality of first estimated information items related to the bones of the first subject using a plurality of estimation models, and a prediction step of outputting predicted information from the plurality of first estimated information items using a prediction model. The plurality of estimation models are each generated by machine learning using a second image showing the tissue of a second subject as an explanatory variable and a plurality of pieces of information related to the bones of the second subject as an objective variable. The prediction model is generated by machine learning using a plurality of pieces of information related to the bones of the second subject as an explanatory variable and abnormality information related to an abnormality that occurred in the bones of the second subject at a second time point different from the first time point when the second image was captured as an objective variable. The predicted information is information indicating the possibility of an abnormality occurring in the tissue of the first subject.

The control program according to aspect B21 of the present disclosure may be a control program for causing a computer to function as any of the information processing systems of aspects B1 to B17 above, and may be a control program for causing the computer to function as the estimation unit and the prediction unit.

The recording medium according to aspect B22 of the present disclosure may be a computer-readable, non-transitory recording medium on which the control program of aspect B21 described above is recorded.

REFERENCE SIGNS LIST 1, 1A Information processing system 2 Control unit 3 Storage unit 10, 10A Prediction device 21 Acquisition unit 22 Analysis unit 23 Correction unit 24 Learning unit 25 Prediction unit 26 Presentation control unit 27 Estimation unit 31 Control program 32, 32A Prediction model 35 Estimation model 60 Presentation device 351 First estimation model 352 Second estimation model 353 Third estimation model E1 Bone density estimated value E2 Bone quality estimated value E3 Muscle mass estimated value G1, G1a First image G2 Second image

Claims (51)

a prediction unit that outputs prediction information using a prediction model from a first image in which at least a part of the first object is captured and the first data;
the prediction model is generated by machine learning using a third image in which at least a part of a second subject is captured and the second data as explanatory variables, and abnormality information regarding an abnormality occurring in a bone of the second subject at a second time point that is different from a first time point at which the third image is captured as a dependent variable;
the prediction information is information indicating a possibility that an abnormality will occur in the bone of the first subject.
Information processing system. the first data is data including a second image,
the second data is data including a fourth image;
The information processing system according to claim 1 . the first image is an image of a predetermined region of the first subject,
the second image is an image showing a region of the first subject corresponding to the predetermined region,
the third image is an image of a predetermined region of the second subject,
the fourth image is an image showing a region of the second subject corresponding to the predetermined region;
The information processing system according to claim 2 . the prediction information is information indicating a possibility that an abnormality will occur in the bone of the first subject at a fourth time point that is different from a third time point at which the first image is captured.
4. The information processing system according to claim 2 or 3. the first image shows bones at a plurality of sites of the first subject;
the second image shows muscles at a plurality of sites of the first subject;
the third image shows bones at a plurality of sites of the second subject;
the fourth image shows muscles at a plurality of sites of the second subject;
The information processing system according to any one of claims 2 to 4. The predictive model is
the third image and the fourth image are used as explanatory variables, and the abnormality information for each of the parts that has occurred within a predetermined period since the third image and/or the fourth image was captured is used as a target variable; and
The prediction unit
outputting the prediction information for each of the plurality of bone regions of the first subject from the first image and the second image using the prediction model;
The information processing system according to claim 5 . The part includes at least one of a chest, a waist, a foot, and a hand.
7. The information processing system according to claim 5 or 6. The second image shows one or more parts of the first subject.
The information processing system according to any one of claims 2 to 7. The second image includes at least one of a still image and a video image.
The information processing system according to any one of claims 2 to 8. The first image and the second image include at least one of a plain X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a DXA (Dual Energy X-ray Absorptiometry) image, an echo image, and an image by DES (Dual Energy Subtraction),
The information processing system according to any one of claims 2 to 9. the second image is a different image type from the first image,
the fourth image is a different image type from the third image;
The information processing system according to any one of claims 2 to 10. The information processing system of claim 3, wherein the prediction information indicates the possibility that the abnormality will occur in the specified area shown in the first image. The information processing system of claim 3, wherein the prediction information indicates the possibility that the abnormality will occur in a region other than the predetermined region not captured in the first image. The abnormality is a musculoskeletal disorder.
The information processing system according to any one of claims 2 to 13. The prediction unit
outputting a bone mineral density and/or a bone quality of the first subject from the first image of the first subject using a first estimation model;
outputting a muscle mass of the first subject from the second image of the first subject using a second estimation model;
the first estimation model is generated by machine learning using the third image of the second subject as an explanatory variable and bone information indicating a bone density and/or a bone quality of the second subject as a response variable;
the second estimation model is generated by machine learning using the fourth image of the second subject as an explanatory variable and muscle information indicating a muscle mass of the second subject as a dependent variable.
15. The information processing system according to claim 2. The predictive model is
generated by machine learning using at least one of the third image, the fourth image, the bone information obtained by inputting the third image into the first estimation model, and the muscle information obtained by inputting the fourth image into the second estimation model as explanatory variables, and the abnormality information as a target variable;
16. The information processing system according to claim 15. an analysis unit that analyzes information including at least one of the amount, thickness, amount of atrophy, and flexibility of muscle and fat of the first subject using the second image;
a correction unit that performs a predetermined correction on the first image,
The analysis unit identifies a soft tissue region by segmenting the second image;
the correction unit performs correction to remove the soft tissue region identified by the analysis unit from the first image;
the prediction unit outputs the prediction information using the prediction model from the first image corrected by the correction unit and the second image.
17. The information processing system according to claim 2. the second image includes an echo image;
the prediction unit predicts attribute information of the first subject by analyzing brightness of the echo image using the analysis unit.
18. The information processing system according to claim 17. The attribute information is information including at least one of the age, sex, and muscle quality of the first subject.
19. The information processing system according to claim 18. The prediction unit
outputting support information for supporting the first subject using at least one of the predicted information, the estimated information of the first subject, and the reference information;
the estimated information is at least one of bone mineral density, bone quality, and muscle mass of the first subject;
The reference information is at least one of bone mineral density, bone quality, and muscle mass according to the age and/or sex of the first subject.
20. The information processing system according to claim 2. the prediction unit identifies a region of interest that is highly related to the abnormality based on the prediction information and/or the estimation information.
21. The information processing system according to claim 20. the prediction information includes an influence degree indicating a degree of influence that each of the first image and the second image has on the abnormality,
22. The information processing system according to any one of claims 2 to 21. the prediction information includes information indicating a time when the abnormality is likely to occur in the first subject;
23. The information processing system according to any one of claims 1 to 22. a presentation control unit that causes a presentation device to present the prediction information;
24. The information processing system according to any one of claims 1 to 23. an estimation unit that outputs the first data including first estimated information on bones of the first subject from the first image using an estimation model;
the estimation model is generated by machine learning using the third image as an explanatory variable and the second data including information about bones of the second subject as a target variable.
The information processing system according to claim 1 . an estimation unit that outputs the first data including a plurality of first estimated information related to bones of the first subject from the first image using a plurality of estimation models;
the plurality of estimation models are generated by machine learning using the third image as an explanatory variable and the second data including a plurality of pieces of information related to bones of the second subject as a dependent variable.
The information processing system according to claim 1 . the prediction information is information indicating a possibility that an abnormality will occur in the tissue of the first subject at a fourth time point that is different from a third time point at which the first image is captured.
27. The information processing system according to claim 26. the prediction information is information indicating a possibility that the abnormality will occur in the part shown in the first image.
28. An information processing system according to any one of claims 25 to 27. the prediction information is information indicating a possibility that the abnormality will occur in a part not captured in the first image.
28. An information processing system according to any one of claims 25 to 27. the first image is a plain X-ray image showing at least a part of a bone and/or a muscle of the first subject;
the third image is a plain X-ray image showing at least a part of a bone and/or a muscle of the second subject;
30. An information processing system according to any one of claims 25 to 29. the first image is a front image or a side image,
The third image is an image in the same orientation as the first image.
31. The information processing system according to any one of claims 25 to 30. The estimation unit
a bone strength estimation model generated by machine learning using the third image as an explanatory variable and bone strength information indicating at least one measurement result of bone mineral density, bone mass, and bone quality of the second subject as a dependent variable;
a bone stress estimation model generated by machine learning using the third image as an explanatory variable and bone stress information indicating at least one measurement result of the muscle mass and posture of the second subject as a dependent variable;
At least one of the following is used:
32. The information processing system according to any one of claims 25 to 31. the bone strength estimation model includes a first estimation model that outputs information indicating a bone density of the bone of the first subject from the first image, and a second estimation model that outputs information indicating a bone quality of the first subject from the first image;
the bone load estimation model includes a third estimation model that outputs information indicating a muscle mass of the first subject from the first image.
33. The information processing system according to claim 32. the first estimation information includes two or more pieces of information among information indicating the bone density of the bone of the first subject output from the first estimation model, information indicating the bone quality of the first subject output from the second estimation model, and information indicating the muscle mass of the first subject output from the third estimation model;
The prediction unit
weighting each of the two or more pieces of information based on the strength of a causal relationship with the occurrence of the abnormality, and inputting the weighted pieces of information into the prediction model;
34. The information processing system of claim 33. The information indicating the bone mineral density of the first subject is expressed by at least one of bone mineral density per unit area, bone mineral density per unit volume, YAM (Young Adult Mean), T-score, and Z-score.
34. The information processing system of claim 33. the bone strength information is information measured using a method including at least one of a DXA (Dual-energy X-ray Absorptiometry) method, an ultrasound method, and a method for calculating a concentration of a bone metabolic marker in the urine or blood of the second subject;
the bone load information is information indicating a result of measuring at least one of a muscle mass of the second subject and a posture of the second subject;
33. The information processing system according to claim 32. The abnormality is a musculoskeletal disorder.
37. An information processing system according to any one of claims 25 to 36. the prediction model is generated by machine learning using the third image and/or information for each bone site of the second subject as an explanatory variable, and the abnormality information regarding the abnormality occurring for each bone site of the second subject at the second time point as a target variable;
the prediction information is information indicating a possibility of the abnormality occurring for each site of the tissue of the first subject at the fourth time point;
28. The information processing system according to claim 27. the prediction information includes information indicating a time when the abnormality is likely to occur in the tissue of the first subject.
28. The information processing system according to claim 27. the prediction unit outputs support information for supporting the first subject from the first image and/or the first estimated information by using the prediction model corresponding to attribute information of the first subject.
40. An information processing system according to any one of claims 25 to 39. a presentation control unit that causes a presentation device to present the prediction information;
41. The information processing system according to any one of claims 25 to 40. A prediction device comprising the prediction unit in the information processing system described in any one of claims 1 to 24. A prediction device comprising the estimation unit and the prediction unit in the information processing system described in any one of claims 25 to 41. 1. An information processing method executed by one or more computers, comprising:
a prediction step of outputting prediction information using a prediction model from a first image showing at least a part of the first object and the first data;
the prediction model is generated by machine learning using a third image in which at least a part of a second subject is captured and the second data as explanatory variables, and abnormality information regarding an abnormality occurring in a bone of the second subject at a second time point that is different from a first time point at which the third image is captured as a dependent variable;
the prediction information is information indicating a possibility that an abnormality will occur in the bone of the first subject.
Information processing methods. the first data is data including a second image,
the second data is data including a fourth image;
45. The information processing method according to claim 44. an estimation step of outputting the first data including first estimated information about bones of the first subject from the first image using an estimation model;
the estimation model is generated by machine learning using the third image as an explanatory variable and the second data including information about bones of the second subject as a target variable.
45. The information processing method according to claim 44. an estimation step of outputting the first data including a plurality of first estimated information related to bones of the first subject from the first image using a plurality of estimation models;
the plurality of estimation models are generated by machine learning using the third image as an explanatory variable and the second data including a plurality of pieces of information related to bones of the second subject as a dependent variable;
the prediction model is generated by machine learning using the second data as an explanatory variable and abnormality information regarding an abnormality occurring in the bone of the second subject at a second time point that is different from the first time point at which the third image was captured as a dependent variable.
45. The information processing method according to claim 44. A control program for causing a computer to function as the information processing system described in any one of claims 1 to 24, the control program causing the computer to function as the prediction unit. A control program for causing a computer to function as the information processing system described in any one of claims 25 to 41, the control program causing the computer to function as the estimation unit and the prediction unit. A computer-readable non-transitory recording medium having the control program described in claim 48 recorded thereon. A computer-readable non-transitory recording medium having the control program described in claim 49 recorded thereon.