KR20250085177A - Method for leveraging multi-modal data to predict risk events - Google Patents

Abstract

According to one embodiment of the present disclosure, a method for predicting a dangerous situation using multi-modal data, performed by a computing device, is disclosed. The method may include the steps of: obtaining a first type of data corresponding to image data; obtaining location information on an object included in the image data; obtaining a second type of data generated by a mobile device associated with the object based on the location information; and predicting a dangerous situation for the object based on the first type of data and the second type of data.

Description

{METHOD FOR LEVERAGING MULTI-MODAL DATA TO PREDICT RISK EVENTS}

The present invention relates to a method for predicting a dangerous situation, and more specifically, to a technique for predicting a dangerous situation using multi-modal data.

Recently, as the installation of Closed Circuit Television (CCTV) increases, interest in intelligent video analysis technology for efficient monitoring is increasing. Intelligent video analysis technology is a technology that analyzes video information to automatically detect abnormal behavior and sends an alarm to the manager, which allows accidents to be prevented in advance and, when an accident occurs, to respond quickly and reduce damage. In addition, intelligent video analysis technology is a technology that analyzes video information to automatically detect abnormal behavior, and consists of actions such as object identification, object tracking, and event detection based on predefined rules.

In addition, mobile devices mainly refer to portable digital devices that are easy to carry and can be used while on the move. It mainly refers to smartphones and tablets, but in addition, portable laptops, smart watches, etc. can also be classified as mobile devices. For example, the motion sensor included in the mobile device can detect the user's movement and measure acceleration, gravity, rotational speed, rotation vector value, drift, etc. The accelerometer measures acceleration changes, and the gyroscope detects the rotation of the device, and by collecting such sensor data in real time, movement patterns can be analyzed and activities such as the user's walking, running, climbing stairs, and falling can be predicted.

Korean Patent Publication No. 10-2022-0063280 (May 17, 2022) discloses a method and device for predicting crowding.

The present disclosure aims to provide a method for predicting a risk situation by utilizing multi-modal data capable of obtaining location information on an object included in an image, obtaining sensor data generated by a mobile device associated with the object based on the location information, and predicting a risk situation of the object.

Meanwhile, the technical problems to be achieved by the present disclosure are not limited to the technical problems mentioned above, and various technical problems may be included within a scope obvious to those skilled in the art from the contents to be described below.

According to one embodiment of the present disclosure for achieving the above-described task, a method for predicting a dangerous situation by utilizing multi-modal data performed by a computing device is disclosed. The method may include the steps of: obtaining a first type of data corresponding to image data; obtaining location information on an object included in the image data; obtaining a second type of data generated by a mobile device associated with the object based on the location information; and predicting a dangerous situation for the object based on the first type of data and the second type of data.

In one embodiment, the second type of data may include sensor data generated by a sensor included in the mobile device.

In one embodiment, the step of predicting a dangerous situation for the object may include the step of obtaining sensor data of another mobile device associated with another object included in the image data; and the step of predicting that the object and the other object are in a dangerous situation when the amount of change in the sensor data of the mobile device and the amount of change in the sensor data of the other mobile device are synchronized and exceed a first threshold.

In one embodiment, the step of predicting a dangerous situation for the object may further include the step of analyzing a time pattern in which a change amount of sensor data of the mobile device and a change amount of sensor data of the other mobile device occur; and the step of predicting a dangerous situation for the object further based on a result of the time pattern analysis.

In one embodiment, the step of predicting that the object and the other object are in a dangerous situation may include the step of predicting that the object and the other object are in a dangerous situation when the number of mobile devices whose changes in the sensor data are synchronized and exceed the first threshold exceeds a second threshold.

In one embodiment, when the number of mobile devices exceeding the first threshold by being synchronized with the change amount of the sensor data exceeds a second threshold, the step of predicting that the object and the other object are in a dangerous situation may include the step of calculating spatial information about where the object can move based on location information about the object; and the step of dynamically determining the second threshold based on the calculated spatial information.

In one embodiment, the step of predicting a dangerous situation for the object may include the step of extracting physical feature information for the object based on features on an image of the object; and the step of predicting a dangerous situation for the object by further considering the physical feature information for the object.

In one embodiment, the step of predicting a dangerous situation for the object by additionally considering physical characteristic information about the object may include the step of predicting a dangerous situation for the first object based on physical characteristic information about the first object included in the image data and sensor data of a mobile device of the first object; and the step of predicting a dangerous situation for the second object based on physical characteristic information about the second object included in the image data and sensor data of a mobile device of the second object.

In one embodiment, even if the sensor data of the mobile device of the first object and the sensor data of the mobile device of the second object are identical to each other, if the physical characteristic information of the first object and the physical characteristic information of the second object are different from each other, a dangerous situation for the first object and a dangerous situation for the second object can be predicted differently from each other.

In one embodiment, the method further includes a step of acquiring a third type of data corresponding to the thermal image data, the third type of data including thermal image data corresponding to the image data, and the step of predicting a dangerous situation for the object may include a step of predicting a dangerous situation for the object based on the second type of data and the third type of data.

According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, it causes the one or more processors to perform the following operations for predicting a dangerous situation by utilizing multi-modal data, wherein the operations may include: an operation of obtaining a first type of data corresponding to image data; an operation of obtaining location information on an object included in the image data; an operation of obtaining a second type of data generated by a mobile device associated with the object based on the location information; and an operation of predicting a dangerous situation for the object based on the first type of data and the second type of data.

In one embodiment, the second type of data may include sensor data generated by a sensor included in the mobile device.

In one embodiment, the operation of predicting a dangerous situation for the object may include: obtaining sensor data of another mobile device associated with another object included in the image data; and predicting that the object and the other object are in a dangerous situation when a change amount in the sensor data of the mobile device and a change amount in the sensor data of the other mobile device are synchronized and exceed a first threshold.

In one embodiment, the operation of predicting a dangerous situation for the object may include an operation of analyzing a time pattern in which a change amount of sensor data of the mobile device and a change amount of sensor data of the other mobile device occur; and an operation of predicting a dangerous situation for the object further based on a result of the time pattern analysis.

In one embodiment, the action of predicting that the object and the other object are in a dangerous situation may include predicting that the object and the other object are in a dangerous situation when the number of mobile devices whose changes in the sensor data are synchronized and exceed the first threshold exceeds a second threshold.

In one embodiment, the operation of predicting a dangerous situation for the object may include an operation of extracting physical feature information for the object based on features in an image of the object; and an operation of predicting a dangerous situation for the object by further considering the physical feature information for the object.

In one embodiment, the operation of predicting a dangerous situation for the object by additionally considering physical characteristic information about the object may include: an operation of predicting a dangerous situation for the first object based on physical characteristic information about the first object included in the image data and sensor data of a mobile device of the first object; and an operation of predicting a dangerous situation for the second object based on physical characteristic information about the second object included in the image data and sensor data of a mobile device of the second object.

In one embodiment, the operation further includes an operation of acquiring a third type of data corresponding to the thermal image data, the third type of data including thermal image data corresponding to the image data, and the operation of predicting a dangerous situation for the object may include an operation of predicting a dangerous situation for the object based on the second type of data and the third type of data.

According to one embodiment of the present disclosure for achieving the above-described task, a computing device is disclosed. The device includes at least one processor and a memory, and the at least one processor is configured to: obtain a first type of data corresponding to image data; obtain location information on an object included in the image data; obtain a second type of data generated by a mobile device associated with the object based on the location information; and predict a dangerous situation for the object based on the first type of data and the second type of data.

In one embodiment, the second type of data may include sensor data generated by a sensor included in the mobile device.

In one embodiment, the at least one processor may be configured to obtain sensor data of another mobile device associated with another object included in the image data; and if a change amount of the sensor data of the mobile device and a change amount of the sensor data of the other mobile device are synchronized and exceed a first threshold, predict that the object and the other object are in a dangerous situation.

In one embodiment, the at least one processor may be further configured to analyze a time pattern in which a change in sensor data of the mobile device and a change in sensor data of the other mobile device occur; and to predict a risk situation for the object further based on a result of the time pattern analysis.

In one embodiment, the at least one processor may be configured to predict that the object and the other object are in a dangerous situation if the number of mobile devices whose changes in the sensor data are synchronized to exceed the first threshold exceeds a second threshold.

In one embodiment, the at least one processor may be configured to extract physical feature information about the object based on features in an image of the object; and further consider the physical feature information about the object to predict a dangerous situation for the object.

In one embodiment, the at least one processor may be configured to predict a dangerous situation for a first object based on physical characteristic information about the first object included in the image data and sensor data of a mobile device of the first object; and to predict a dangerous situation for a second object based on physical characteristic information about the second object included in the image data and sensor data of a mobile device of the second object.

In one embodiment, the at least one processor may be configured to obtain a third type of data corresponding to thermal image data, the third type of data including thermal image data corresponding to the image data, and to predict a dangerous situation for the object based on the second type of data and the third type of data.

The present disclosure can prevent accidents in advance by detecting potential risks in advance by additionally utilizing sensor data generated from a mobile device carried by a user in addition to image data.

Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within a range obvious to those skilled in the art from the contents described below.

FIG. 1 is a block diagram of a computing device for predicting a risk situation using multi-modal data according to one embodiment of the present disclosure.
FIG. 2 is a conceptual diagram illustrating a neural network according to one embodiment of the present disclosure.
FIG. 3 is a flowchart illustrating a method for predicting a risk situation using multi-modal data according to one embodiment of the present disclosure.
FIG. 4 is a schematic diagram illustrating an operation for calculating movable space information according to one embodiment of the present disclosure.
FIG. 5 is a schematic diagram illustrating physical characteristic information for an object according to one embodiment of the present disclosure.
FIG. 6 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are set forth to provide an understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific descriptions.

The terms "component," "module," "system," and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, an application running on a computing device and the computing device may both be components. One or more components may reside within a processor and/or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, such components may execute from various computer-readable media having various data structures stored therein. The components may communicate via local and/or remote processes, for example, by a signal comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and/or data transmitted via a network such as the Internet to another system via the signal).

Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

Also, the terms "comprises" and/or "comprising" should be understood to mean the presence of the features and/or components. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, components, and/or groups thereof. Also, unless otherwise specified or clear from the context to refer to the singular form, the singular form as used in the specification and claims should generally be construed to mean "one or more."

And, the term "at least one of A or B" should be interpreted to mean "including only A", "including only B", or "combined in the composition of A and B".

Those skilled in the art should additionally recognize that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as combinations of electronic hardware, computer software, or both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to a person skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments disclosed herein. The present invention is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

In the present disclosure, network function, artificial neural network and neural network can be used interchangeably.

FIG. 1 is a block diagram of a computing device for predicting a risk situation using multi-modal data according to one embodiment of the present disclosure.

The configuration of the computing device (100) illustrated in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing a computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

A computing device (100) may include a processor (110), a memory (130), and a network unit (150).

The processor (110) may be composed of one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in the memory (130) and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process learning of a network function. For example, CPU and GPGPU can be used together to process learning of network functions and data classification using network functions. In addition, in one embodiment of the present disclosure, processors of multiple computing devices can be used together to process learning of network functions and data classification using network functions. In addition, a computer program executed in a computing device according to one embodiment of the present disclosure can be a CPU, GPGPU or TPU executable program.

According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, an SD or XD memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read-Only Memory (ROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Programmable Read-Only Memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may also operate in relation to web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example, and the present disclosure is not limited thereto.

The network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

In addition, the network unit (150) presented in this specification can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

In the present disclosure, the network unit (150) may be configured regardless of the communication mode, such as wired or wireless, and may be configured with various communication networks, such as a local area network (LAN), a personal area network (PAN), and a wide area network (WAN). In addition, the network may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth.

The techniques described in this specification can be used in other networks as well as the networks mentioned above.

FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

Throughout this specification, the terms artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) that make up the neural network may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link can form a relationship of input node and output node relatively. The concept of input node and output node is relative, and any node in an output node relationship to one node can be in an input node relationship in a relationship to another node, and vice versa. As described above, the input node-to-output node relationship can be created based on a link. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between input nodes and output nodes connected through a single link, the data of the output node can have its value determined based on the data input to the input node. Here, the link interconnecting the input nodes and the output nodes can have a weight. The weight can be variable and can be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values input to the input nodes connected to the output node and the weight set for the link corresponding to each input node.

As described above, a neural network is a network in which one or more nodes are interconnected through one or more links to form input node and output node relationships within the neural network. Depending on the number of nodes and links within the neural network, the relationship between the nodes and links, and the weight values assigned to each link, the characteristics of the neural network can be determined. For example, if there are two neural networks with the same number of nodes and links and different weight values for the links, the two neural networks can be recognized as being different from each other.

A neural network can be composed of a set of one or more nodes. A subset of the nodes constituting the neural network can form a layer. Some of the nodes constituting the neural network can form a layer based on distances from the initial input node. For example, a set of nodes that are n in distance from the initial input node can form an n layer. The distance from the initial input node can be defined by the minimum number of links that must be passed through to reach the node from the initial input node. However, this definition of a layer is arbitrary for the purpose of explanation, and the order of a layer in a neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by the distance from the final output node.

In one embodiment of the present disclosure, a set of neurons or nodes may be defined as a layer.

The initial input node may refer to one or more nodes to which data is directly input without going through a link in a relationship with other nodes in the neural network. Or, in a neural network, in a relationship between nodes based on a link, it may refer to nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in a relationship with other nodes in the neural network. In addition, a hidden node may refer to nodes that constitute the neural network other than the initial input node and the final output node.

According to one embodiment of the present disclosure, a neural network may be a neural network in which the number of nodes in an input layer may be the same as the number of nodes in an output layer, and the number of nodes decreases and then increases as it progresses from the input layer to the hidden layer. In addition, according to another embodiment of the present disclosure, a neural network may be a neural network in which the number of nodes in an input layer may be less than the number of nodes in an output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. In addition, according to another embodiment of the present disclosure, a neural network in which the number of nodes in an input layer may be greater than the number of nodes in an output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. In accordance with another embodiment of the present disclosure, a neural network may be a neural network in a combined form of the neural networks described above.

A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a deep neural network, it is possible to identify latent structures of data. That is, latent structures of photos, text, videos, voices, protein sequence structures, gene sequence structures, peptide sequence structures, latent structures of music (for example, what objects are in photos, what the content and emotion of text are, what the content and emotion of voice are, etc.), and/or binding affinities between peptides and MHC. A deep neural network can include a convolutional neural network (CNN), a recurrent neural network (RNN), an auto encoder, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, a generative adversarial network (GAN), a transformer, etc. The description of the deep neural network described above is only an example and the present disclosure is not limited thereto.

The artificial intelligence-based model of the present disclosure can be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

The neural network that can be used in the artificial intelligence-based model of the present disclosure can be trained by at least one of supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training of the neural network can be a process of applying knowledge to the neural network for performing a specific operation.

A neural network can be trained in the direction of minimizing the error of the output. In the training of a neural network, training data is repeatedly input into the neural network, the output of the neural network for the training data and the target error are calculated, and the error of the neural network is backpropagated from the output layer of the neural network to the input layer in the direction of reducing the error, thereby updating the weights of each node of the neural network. In the case of supervised learning, training data in which the correct answer is labeled for each training data is used (i.e., labeled training data), and in the case of unsupervised learning, the correct answer may not be labeled for each training data. That is, for example, in the case of supervised learning for data classification, the training data may be data in which categories are labeled for each training data. Labeled training data is input into the neural network, and the error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the output of the neural network. The calculated error is backpropagated in the backward direction (i.e., from the output layer to the input layer) in the neural network, and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined according to the learning rate. The calculation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of repetitions of the learning cycle of the neural network. For example, a high learning rate can be used in the early stage of learning of the neural network so that the neural network can quickly acquire a certain level of performance, thereby increasing efficiency, and a low learning rate can be used in the later stage of learning to increase accuracy.

In learning neural networks, learning data can generally be a subset of real data (i.e., data to be processed using the learned neural network), and therefore, there can be a learning cycle in which errors for learning data decrease but errors for real data increase. Overfitting is a phenomenon in which excessive learning on learning data increases errors for real data. For example, a neural network that learned cats by showing yellow cats cannot recognize cats when it sees cats other than yellow, which can be a type of overfitting. Overfitting can act as a cause of increasing errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. To prevent overfitting, methods such as increasing learning data, regularization, dropout that deactivates some nodes of the network during the learning process, and utilization of batch normalization layers can be applied.

FIG. 3 is a flowchart illustrating a method for predicting a dangerous situation using multi-modal data according to one embodiment of the present disclosure. The method for predicting a dangerous situation using multi-modal data can be performed by a computing device (100). For reference, multi-modal data is data in which various types of different data are combined. For example, multi-modal data can include text data, image data, video data, audio data, depth data, infrared data, and inertial measurement unit (IMU) data. The computing device (100) can predict a dangerous situation more accurately by integrating various types of data to interact, extract patterns, and extract meaningful information.

According to one embodiment of the present disclosure, the computing device (100) can obtain a first type of data corresponding to image data (S110). For example, the computing device (100) can obtain image data captured by a closed-circuit television (CCTV). CCTV cameras are installed in various places such as public places, stores, buildings, and roads, and can monitor situations in real time through them. In addition, the computing device (100) can also obtain the first type of data corresponding to the image data through a device that is installed in various places such as public places, stores, buildings, and roads and captures the movement of an object (person).

According to one embodiment of the present disclosure, the computing device (100) can obtain location information on an object included in image data (S120). The location information refers to a geographical location where a specific object or thing is located. In addition, the location information is expressed as coordinates such as latitude, longitude, and altitude, and the exact location of a specific location can be identified through this. For example, the computing device (100) can obtain location information on an object based on location information of a CCTV from which the image data was obtained. In addition, the computing device (100) can also obtain location information on an object through meta data included in the image data. For example, the computing device (100) can obtain the location information based on meta data stored in an image file in a format called Exif (Exchangeable Image File Format). The Exif data can include various information such as the date and time when the photo was taken, the camera model, the shooting conditions, and GPS information. The GPS information records the latitude and longitude of the location where the photo was taken, and through this, the specific location where the photo was taken can be confirmed on a map.

According to one embodiment of the present disclosure, the computing device (100) may obtain a second type of data generated by a mobile device associated with an object based on location information (S130). For example, the second type of data may include sensor data generated by a sensor included in the mobile device. For example, the computing device (100) may obtain the second type of data generated by a mobile device carried by the object. For reference, the mobile device may include a small-sized electronic device that can be used while moving as a portable device. For example, the mobile device may include a smartphone, a tablet, a smartwatch, etc. In addition, the sensor included in the mobile device may include an accelerometer and a gyroscope sensor. For example, the computing device (100) may obtain sensor data that detects movement of the device and measures acceleration generated by an accelerometer sensor included in the mobile device. In addition, the computing device (100) may obtain sensor data for measuring rotation in three-dimensional space by detecting rotation and inclination of the device generated by a gyroscope included in the mobile device. The computing device (100) may obtain second type data generated by a mobile device associated with an object to analyze human actions such as falling, walking, and running. The computing device (100) may obtain second type data generated by a mobile device of at least one object located at a corresponding location based on previously obtained location information.

According to one embodiment of the present disclosure, the computing device (100) can predict a dangerous situation for an object based on the first type of data and the second type of data (S140). For example, the computing device (100) can predict a dangerous situation for an object based on image data and sensor data. The computing device (100) can predict actions (e.g., walking, running, sitting, falling), etc. of an object (person) more accurately based on sensor data as well as image data. When an abnormal pattern or dangerous situation is predicted, the computing device (100) can provide the predicted information to a mobile device carried by the object and equipment equipped in a public office (e.g., police, firefighters, local government).

In the following, we would like to describe in more detail the operation of predicting a dangerous situation for an object by utilizing i) the amount of change of the second type, ii) spatial information about the object's movement, or iii) physical characteristics of the object, and iv) the time pattern in which the amount of change of the second type occurs.

According to one embodiment of the present disclosure, the computing device (100) can obtain sensor data of another mobile device associated with another object included in the image data. In addition, the computing device (100) can obtain sensor data of a mobile device associated with all of a plurality of objects included in the image data. For example, the computing device (100) can recognize at least one object included in the image data. For example, the computing device (100) can perform object recognition on the image data through various methods such as classification, object detection, and instance segmentation. In addition, the computing device (100) can recognize an object included in the image data by using at least one of a plurality of classification algorithms. In addition, the computing device (100) can recognize an object included in the image data from image retrieval, image annotation, face detection, image classification, and the like performed in object detection. In addition, the computing device (100) can perform object determination included in the environmental information through computer vision, image processing, machine learning models, etc. For example, computer vision can recognize objects included in an image by including technologies such as object classification, object detection and localization, object segmentation, image captioning, object tracking, and action classification. For example, the computing device (100) can perform object recognition (identification) by utilizing deep learning-based convolutional neural network (CNN), YOLO (You Only Look Once), Faster R-CNN, SSD (Single Shot MultiBox Detector), semantic segmentation, instance segmentation, SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), HOG (Histogram of Oriented Gradients), etc. The computing device (100) can obtain sensor data of the mobile device corresponding to the number of identified objects. For example, referring to FIG. 5, if there are six objects included in the image data, the computing device (100) can obtain sensor data of the mobile device for each of the first object (1) to the sixth object (6) included in the image data.

In addition, the computing device (100) can predict that the object and other objects are in a dangerous situation if the amount of change in the sensor data of the mobile device and the amount of change in the sensor data of another mobile device are synchronized and exceed the first threshold. For example, the amount of change in the sensor data of the mobile device may mean a change in a value measured by a plurality of sensors mounted on the mobile device. For example, the amount of change in the sensor data of the mobile device may be a change in a value measured by a gyroscope mounted on the mobile device by measuring the rotational speed of the mobile device to detect the inclination and rotation. For example, the computing device (100) can synchronize the amount of change in the sensor data and the amount of change in the sensor data of another mobile device by utilizing a timestamp included in the sensor data. In addition, the computing device (100) can also process the sensor data of the mobile device in real time to synchronize the amount of change in the sensor data and the amount of change in the sensor data of another mobile device. In addition, the mobile device may provide sensor data when a specific event (e.g., fall detection) occurs in the sensor data, and the computing device (100) may synchronize the amount of change in the acquired sensor data of the mobile device and the amount of change in the sensor data of another mobile device. For example, when the amount of change in the sensor data of the mobile devices associated with each of the objects located at the same location exceeds the first threshold, it may mean that a change in the behavior of the objects located at the corresponding location has occurred. In addition, the computing device (100) may determine the first threshold by considering the amount of change in the sensor data that occurs when a specific behavior is taken. For example, the computing device (100) may determine the first threshold by analyzing the characteristic movement of a person when he or she falls and the amount of change in the gyroscope (e.g., a sudden change in inclination or a sudden increase in rotation speed). For example, referring to FIG. 5, the computing device (100) may synchronize the amount of change in the sensor data of the mobile devices of each of the first object (1) to the sixth object (6). In addition, the computing device (100) can predict that the first object (1) to the sixth object (6) are in a dangerous situation if the amount of change in the sensor data of the mobile devices associated with the synchronized first object (1) to the sixth object (6) exceeds the first threshold. For example, in a situation where the first object (1) to the sixth object (6) are located at the very front of a narrow alley and a crowd gathers and the first object (1) to the sixth object (6) falls down, the amount of change in the sensor data of the mobile devices of the synchronized first object (1) to the sixth object (6) can exceed the first threshold. The computing device (100) can predict that a plurality of objects are in a dangerous situation at the corresponding location if the amount of change in the sensor data of a plurality of mobile devices at the same location information exceeds the first threshold. On the other hand, the computing device (100) can predict that only the first object (1) is in a dangerous situation if the amount of change in sensor data of the mobile device of the first object (1) exceeds the first threshold.

According to one embodiment, the computing device (100) can predict that the object and the other object are in a dangerous situation when the number of mobile devices that are synchronized with the change in the sensor data exceeds the first threshold exceeds the second threshold. For example, when the change in the sensor data of the first mobile device associated with the first object exceeds the first threshold, since only the behavior of the first object is judged, only the dangerous situation for the first object can be predicted. Considering this, the computing device (100) can predict that both the object and the other object are in a dangerous situation when the number of mobile devices exceeds the second threshold. For example, the second threshold can be dynamically determined according to the objects included in the image data. For example, the computing device (100) can determine the number of objects by performing object separation included in the image data. For example, when there are 6 objects included in the image data, the computing device (100) can determine the second threshold as 4. On the other hand, if the number of objects included in the image data is 100, the computing device (100) may determine the second threshold as 80. In addition, the second threshold may be dynamically determined according to the number of sensor data acquired of the mobile device. For example, sensor data generated by a mobile device associated with an object based on location information may mean the number of objects. In consideration of this, the computing device (100) may dynamically determine the second threshold according to the number of sensor data acquired. However, the criterion for determining the second threshold is not limited to the operation described above. For example, referring to FIG. 5, the computing device (100) may synchronize the amount of change in sensor data of each of the mobile devices of the first object (1) to the sixth object (6). In addition, the computing device (100) can predict that the first object (1) to the sixth object (6) are in a dangerous situation if the number of mobile devices exceeding the first threshold by synchronizing the amount of change in the six sensor data of each of the mobile devices of the first object (1) to the sixth object (6) exceeds the second threshold. On the other hand, the computing device (100) can predict that only the objects associated with the mobile devices exceeding the first threshold are in a dangerous situation if the number of mobile devices exceeding the first threshold by synchronizing the amount of change in the six sensor data of each of the mobile devices of the first object (1) to the sixth object (6) does not exceed the second threshold.

FIG. 4 is a schematic diagram illustrating an operation for calculating movable space information according to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device (100) can calculate spatial information about an object that can move based on location information about the object. In addition, the computing device (100) can dynamically determine the second threshold based on the calculated spatial information. For example, the computing device (100) can obtain surrounding terrain, road information, or map information based on the location information. The computing device (100) can calculate spatial information about an object that can move based on the surrounding terrain, road information, or map information. In addition, the computing device (100) can also calculate spatial information about an object that can move using image data. For example, the computing device (100) can segment an image into a person and a background at the pixel level using segmentation technology and extract an area where a person can move. In addition, the computing device (100) can calculate spatial information about an object that can move and dynamically determine the second threshold related to the number of mobile devices based on the calculated spatial information. For example, the spatial information about an object that can move may include information related to the area of a road used for pedestrian traffic. For example, referring to FIG. 4, the first region may be a narrow alley as illustrated in (a) of FIG. 4. Additionally, the second region may be a wide road as illustrated in (b) of FIG. 4. The computing device (100) may calculate space information for which objects can move for each of the first region and the second region. The computing device (100) may determine that more objects can move in the second region than in the first region based on the calculation result of the space information for which objects can move. In other words, the computing device (100) may estimate that the area of a road intended for pedestrian traffic is wider in the second region than in the first region. For example, in a first area (a narrow alley) such as in (a) of FIG. 4, if the amount of change in sensor data of mobile devices of a predetermined number of objects (e.g., 10 people) exceeds the first threshold, a greater risk may occur compared to a second area (a wide road) such as in (b) of FIG. 4. In other words, if the spatial information in which an object can move is relatively narrow, if the amount of change in sensor data of mobile devices of a predetermined number of objects exceeds the first threshold, a greater risk may occur. On the other hand, if the spatial information in which an object can move is relatively wide, even if the amount of change in sensor data of mobile devices of a predetermined number of objects (e.g., 10 people) exceeds the first threshold, the risk may be less than in a narrow area. In this regard, since a greater risk may occur if the amount of change in sensor data of a smaller number of people exceeds the first threshold when the area of the area is smaller than the regional average, the computing device (100) may dynamically determine a second threshold related to the number of mobile devices based on the calculated spatial information.

FIG. 5 is a schematic diagram illustrating physical characteristic information for an object according to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device (100) can extract physical feature information about an object based on features of an image of the object. For example, the physical feature information can include volume, weight, movement status, etc. The computing device (100) can identify the outline of the object by utilizing a technology for detecting and classifying a person in an image, and estimate the volume of the object through this. In addition, the computing device (100) can recognize and extract features such as a face, body, skin color, hairstyle, etc. by utilizing a neural network such as a CNN (Convolutional Neural Network). However, the operation of extracting physical feature information about the object is not limited thereto, and various technologies that have been developed or will be developed in the future can be applied.

In addition, the computing device (100) can predict a dangerous situation for the object by additionally considering the physical characteristic information for the object. The computing device (100) can predict a dangerous situation for the object based on the first type of data, the second type of data, and the physical characteristic information for the object. Even if the same dangerous situation occurs, the intensity of the danger may differ depending on the physical characteristic information for the object. The computing device (100) can predict a dangerous situation for each object by additionally considering the physical characteristic information for the object. In addition, the computing device (100) can predict a dangerous situation for the first object based on the physical characteristic information for the first object included in the image data, and the sensor data of the mobile device of the first object. In addition, the computing device (100) can predict a dangerous situation for the second object based on the physical characteristic information for the second object included in the image data, and the sensor data of the mobile device of the second object. According to one embodiment, even if the sensor data of the mobile device of the first object and the sensor data of the mobile device of the second object are identical to each other, if the physical characteristic information of the first object and the “physical characteristic information” of the second object are “different from each other”, a dangerous situation for the first object and a dangerous situation for the second object may be predicted differently from each other. For example, referring to FIG. 5, in a space where a crowd is densely packed in a narrow alley, in a situation where the first object (1) to the sixth object (6) are pushed from behind to advance forward, a relatively bulky object may have a greater strength to withstand. For example, when comparing the first object (1) and the second object (3), it may be estimated that the difference in volume between the first object (1) and the second object (3) is twice based on the first physical characteristic information of the first object and the second physical characteristic information of the second object (3) estimated by the computing device (100). The computing device (100) may predict that even if the sensor data of the mobile device of the first object (1) and the sensor data of the mobile device of the second object (2) have the same amount of change in angle, the physical characteristic information of the first object and the “physical characteristic information of the second object are different”, and therefore, a dangerous situation for the first object and a dangerous situation for the second object may be predicted differently. The computing device (100) may predict that the second object is in a more dangerous situation than the first object.

For example, if the computing device (100) estimates the first physical characteristic information for the first object and classifies the first object as a “child”, and if the computing device (100) estimates the second physical characteristic information for the second object and classifies the second object as an “adult”, even if the sensor data of the mobile device of the first object and the sensor data of the mobile device of the second object are identical to each other, if the “physical characteristic information of the first object and the physical characteristic information of the second object are different from each other”, a dangerous situation for the first object and a dangerous situation for the second object may be predicted differently from each other. The computing device (100) may predict that the first object is in a more dangerous situation than the second object.

In addition, in a situation where a first object (1) to a sixth object (6) are pushed in order to advance from behind in a space where a crowd is densely packed in a narrow alley, the computing device (100) estimates the first physical characteristic information about the first object and, as a result, finds that only the first object is moving in the first direction (backward) and the second object to the sixth object (6) are moving in the second direction (forward), even if the sensor data of the mobile device of the first object and the sensor data of the mobile devices of the second object to the sixth object (6) are identical to each other, in the case where the “physical characteristic information of the first object and the physical characteristic information of the second object are different from each other”, the dangerous situation of the first object and the dangerous situation of the second object can be predicted differently from each other. The computing device (100) can predict that the first object having a different moving direction is in a more dangerous situation. However, the above-described matters are merely examples and the present disclosure is not limited thereto.

According to one embodiment of the present disclosure, the computing device (100) can analyze a time pattern in which the amount of change in sensor data of the mobile device and the amount of change in sensor data of the other mobile device occur. In addition, the computing device (100) can predict a dangerous situation for the object based on the result of the time pattern analysis. The computing device (100) can predict a dangerous situation for the object based on the first type of data, the second type of data, and the result of the time pattern analysis. For example, the computing device (100) can analyze the time pattern by considering the periodicity of the amount of change in sensor data of the mobile device and the amount of change in sensor data of the other mobile device, the frequency of occurrence of events, etc. For example, the computing device (100) can obtain location information for an object included in image data, and obtain a second type of data generated by a mobile device associated with the object based on the location information. In addition, the computing device (100) can obtain sensor data of another mobile device associated with another object included in the image data. In addition, the computing device (100) can analyze a time pattern when the amount of change in the sensor data of the mobile device and the amount of change in the sensor data of the other mobile device are synchronized and exceed the first threshold. In addition, the computing device (100) can predict that the object and the other object are in a dangerous situation when the number of mobile devices in which the amount of change in the sensor data is synchronized and exceeds the first threshold as a result of the time pattern analysis exceeds the second threshold. For example, in the case of "a large number of people falling over in a crowd", a situation in which the amount of change in the sensor data exceeds the first threshold can occur within a short period of time (e.g., 5 minutes). For example, in a situation in which the first object (1) to the sixth object (6) are located at the very front of a narrow alley as shown in (a) of FIG. 4, a situation in which the first object (1) to the sixth object (6) fall over due to crowding occurs, the amount of change in the sensor data of each mobile device of the synchronized first object (1) to the sixth object (6) can exceed the first threshold. In addition, when the first object (1) to the sixth object (6) located at the front end falls, a situation occurs where multiple objects following also fall, and the amount of change in sensor data of mobile devices associated with multiple objects following may also exceed a first threshold. When the number of mobile devices exceeding the first threshold exceeds a second threshold, the computing device (100) can predict that multiple objects included in the image data are in a dangerous situation. In other words, the computing device (100) can predict, based on location information, that an object carrying a mobile device associated with at least one object included in the image data is in a dangerous situation. On the other hand, on an "icy road," a situation where the amount of change in sensor data exceeds 1 threshold may occur for a relatively long period of time (for example, 1 hour). When a situation where the amount of change in sensor data exceeds the first threshold occurs continuously or intermittently for a relatively long period of time in the same location, the computing device (100) can predict a dangerous situation for the object.

According to one embodiment of the present disclosure, the computing device (100) can obtain a third type of data corresponding to the thermal image data. For example, the computing device (100) can obtain the third type of data from a thermal image camera provided at the same location as the device from which the image data was obtained. In addition, the computing device (100) can predict a dangerous situation for an object based on the second type of data and the third type of data. For example, the computing device (100) can obtain location information for an object included in the thermal image data. In addition, the computing device (100) can obtain a second type of data generated by a mobile device associated with the object based on the location information. In addition, the computing device (100) can obtain sensor data of another mobile device associated with another object included in the thermal image data. In addition, the computing device (100) can predict that the object and the other object are in a dangerous situation if the amount of change in the sensor data of the mobile device and the amount of change in the sensor data of the other mobile device are synchronized and exceed a first threshold. At this time, the computing device (100) can predict that the object and the other object are in a dangerous situation if the number of mobile devices for which the amount of change in the sensor data is synchronized and exceeds the first threshold exceeds a second threshold. The first threshold and the second threshold can be determined in the same manner as the method described above. Meanwhile, the computing device (100) can supplement the problem of not being able to acquire the first type of data from intelligent CCTV, etc. during battlefield environment verification by obtaining the third type of data corresponding to thermal image data and predicting the dangerous situation.

The steps mentioned in the above description may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. In addition, some steps may be omitted as needed, and the order between the steps may be changed.

Meanwhile, a computer-readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data to solve a specific problem (e.g., data retrieval in the shortest time, data storage, data modification). A data structure can also be defined as a physical or logical relationship between data elements designed to support a specific data processing function. A logical relationship between data elements can include a connection relationship between user-defined data elements. A physical relationship between data elements can include an actual relationship between data elements that are physically stored in a computer-readable storage medium (e.g., a persistent storage device). A data structure can specifically include a collection of data, a relationship between data, and a function or command that can be applied to the data. An effectively designed data structure enables a computing device to perform operations while using minimal resources of the computing device. Specifically, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and searching through an effectively designed data structure.

Data structures can be divided into linear data structures and nonlinear data structures depending on the form of the data structure. A linear data structure can be a structure in which only one piece of data is connected after another piece of data. Linear data structures can include lists, stacks, queues, and deques. A list can mean a series of data sets that have an internal order. A list can include a linked list. A linked list can be a data structure in which data is connected in a way that each piece of data has a pointer and is connected in a single line. In a linked list, a pointer can include information on the connection to the next or previous piece of data. A linked list can be expressed as a singly linked list, a doubly linked list, or a circular linked list depending on its form. A stack can be a data listing structure that allows limited access to data. A stack can be a linear data structure in which data can be processed (e.g., inserted or deleted) only at one end of the data structure. Data stored in a stack can be a data structure in which the later it enters, the sooner it comes out (LIFO - Last in First Out). A queue is a data list structure that allows limited access to data, and unlike a stack, it can be a data structure that comes out later (FIFO - First in First Out) as data is stored later. A deck can be a data structure that can process data at both ends of the data structure.

A nonlinear data structure can be a structure in which multiple data are connected behind one data. A nonlinear data structure can include a graph data structure. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. A graph data structure can include a tree data structure. A tree data structure can be a data structure in which there is only one path connecting two different vertices among multiple vertices included in the tree. In other words, it can be a data structure that does not form a loop in a graph data structure.

Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they are collectively described as neural networks. The data structure may include a neural network. And the data structure including the neural network may be stored in a computer-readable medium. The data structure including the neural network may also include preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. The data structure including the neural network may include any of the components among the above-described configurations. That is, the data structure including the neural network may be configured to include all or any combination of preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. In addition to the above-described configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. In addition, the data structure may include all forms of data used or generated in the computational process of the neural network, and is not limited to the above-described matters. The computer-readable medium may include a computer-readable recording medium and/or a computer-readable transmission medium. The neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network is composed of at least one node.

The data structure may include data input to the neural network. The data structure including the data input to the neural network may be stored in a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and/or input data input to the neural network after training is completed. The data input to the neural network may include data that has undergone preprocessing and/or data that is the target of preprocessing. The preprocessing may include a data processing process for inputting data to the neural network. Accordingly, the data structure may include data that is the target of preprocessing and data generated by the preprocessing. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

The data structure may include weights of the neural network. (In this specification, the terms weight and parameter may be used interchangeably.) And the data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weights may be variable and may be variable by a user or an algorithm so that the neural network may perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node may determine a data value output from the output node based on values input to the input nodes connected to the output node and weights set for links corresponding to each of the input nodes. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

By way of example and not limitation, the weights may include weights that are variable during the neural network learning process and/or weights that have completed neural network learning. The weights that are variable during the neural network learning process may include weights at the start of a learning cycle and/or weights that are variable during a learning cycle. The weights that have completed neural network learning may include weights that have completed a learning cycle. Accordingly, the data structure including the weights of the neural network may include the data structure including the weights that are variable during the neural network learning process and/or weights that have completed neural network learning. Therefore, the above-described weights and/or combinations of each weight are included in the data structure including the weights of the neural network. The above-described data structures are merely examples and the present disclosure is not limited thereto.

The data structure including the weights of the neural network can be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be a process of converting the data structure into a form that can be stored in the same or different computing devices and reconstructed and used later. The computing device can serialize the data structure to transmit and receive data over a network. The data structure including the weights of the serialized neural network can be reconstructed in the same computing device or another computing device through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weights of the neural network can include a data structure (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in nonlinear data structures) for increasing the efficiency of computation while minimizing the use of computing device resources. The above are examples and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of the neural network. And the data structure including the hyper-parameters of the neural network may be stored in a computer-readable medium. The hyper-parameters may be variables that are changed by the user. The hyper-parameters may include, for example, a learning rate, a cost function, a number of repetitions of a learning cycle, weight initialization (e.g., setting a range of weight values to be weight initialization targets), and the number of Hidden Units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The above-described data structure is only an example, and the present disclosure is not limited thereto.

FIG. 6 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may also be implemented in combination with computer-executable instructions and/or other program modules that may be executed on one or more computers and/or as a combination of hardware and software.

In general, program modules include routines, programs, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

The described embodiments of the present disclosure can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Any media that can be accessed by a computer can be a computer-readable media, and such computer-readable media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An exemplary environment for implementing various aspects of the present disclosure is illustrated, including a computer (1102) including a processing unit (1104), a system memory (1106), and a system bus (1108). The system bus (1108) couples system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

The system bus (1108) may be any of several types of bus structures that may additionally be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input/output system (BIOS) is stored in nonvolatile memory (1110), such as ROM, EPROM, EEPROM, and the BIOS contains basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. An interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide nonvolatile storage of data, data structures, computer-executable instructions, and the like. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above has referred to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those of ordinary skill in the art will appreciate that other types of media readable by a computer, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and/or data may also be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in a variety of commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer (1102) via one or more wired/wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is coupled to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148), via wired and/or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and may generally include many or all of the components described for the computer (1102), but for simplicity, only the memory storage device (1150) is illustrated. The logical connections illustrated include wired/wireless connections to a local area network (LAN) (1152) and/or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a worldwide computer network, such as the Internet.

When used in a LAN networking environment, the computer (1102) is connected to the local network (1152) via a wired and/or wireless communications network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may also include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means for establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules described for the computer (1102) or portions thereof may be stored in a remote memory/storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

The computer (1102) is configured to communicate with any wireless device or object that is configured and operates in wireless communication, such as a printer, a scanner, a desktop and/or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows you to connect to the Internet and other things without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data, indoors and outdoors, anywhere within the coverage area of a base station, similar to a cell phone. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein for convenience as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The various embodiments presented herein can be implemented as a method, an apparatus, or an article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. It is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

Claims (26)

A method for predicting risk situations by utilizing multi-modal data performed by a computing device,
A step of obtaining a first type of data corresponding to image data;
A step of obtaining location information for an object included in the above image data;
A step of obtaining a second type of data generated by a mobile device associated with the object based on the location information; and
A step of predicting a risk situation for the object based on the first type of data and the second type of data.
Including,
method.
In paragraph 1,
The second type of data includes sensor data generated by a sensor included in the mobile device.
method.
In paragraph 1,
The step of predicting a risk situation for the above object is:
A step of acquiring sensor data of another mobile device associated with another object included in the image data; and
A step of predicting that the object and the other object are in a dangerous situation when the amount of change in the sensor data of the mobile device and the amount of change in the sensor data of the other mobile device are synchronized and exceed a first threshold.
Including,
method.
In the third paragraph,
The step of predicting a risk situation for the above object is:
A step of analyzing a time pattern in which the amount of change in sensor data of the mobile device and the amount of change in sensor data of the other mobile device occur; and
A step for predicting a risk situation for the object based further on the results of the above time pattern analysis.
Including more,
method.
In the third paragraph,
The step of predicting that the above object and the other object are in a dangerous situation is:
A step of predicting that the object and the other object are in a dangerous situation when the number of mobile devices exceeding the first threshold exceeds the second threshold due to the synchronization of the change in the above sensor data,
method.
In paragraph 5,
If the number of mobile devices exceeding the first threshold by synchronizing the change in the above sensor data exceeds the second threshold, the step of predicting that the object and the other object are in a dangerous situation is,
A step of calculating spatial information in which the object can move based on location information about the object; and
A step of dynamically determining the second threshold based on the spatial information generated above.
Including,
method.
In the second paragraph,
The step of predicting a risk situation for the above object is:
A step of extracting physical feature information about the object based on features on the image of the object; and
A step of predicting a risk situation for the object by additionally considering physical characteristic information for the object.
Including,
method.
In paragraph 7,
In addition, considering the physical characteristic information of the above object, the step of predicting a risk situation for the above object is:
A step of predicting a dangerous situation for a first object based on physical characteristic information for the first object included in the image data and sensor data of a mobile device of the first object; and
A step of predicting a dangerous situation for a second object based on physical characteristic information for the second object included in the image data and sensor data of a mobile device of the second object.
Including,
method.
In Article 8,
Even if the sensor data of the mobile device of the first object and the sensor data of the mobile device of the second object are identical to each other, if the physical characteristic information of the first object and the physical characteristic information of the second object are different from each other, the risk situation of the first object and the risk situation of the second object are predicted differently from each other.
method.
In paragraph 1,
The above method,
Further comprising a step of acquiring a third type of data corresponding to the thermal image data,
The third type of data includes thermal image data corresponding to the image data,
The step of predicting a risk situation for the above object is:
A step of predicting a risk situation for the object based on the second type of data and the third type of data.
Including,
method.
A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, causes the one or more processors to perform the following operations for predicting a risk situation by utilizing multi-modal data, the operations being:
An operation of obtaining a first type of data corresponding to image data;
An action of obtaining location information for an object included in the above image data;
An operation of obtaining a second type of data generated by a mobile device associated with the object based on the location information; and
An operation for predicting a risk situation for the object based on the first type of data and the second type of data
Including,
A computer program stored on a computer-readable storage medium.
In Article 11,
The second type of data includes sensor data generated by a sensor included in the mobile device.
A computer program stored on a computer-readable storage medium.
In Article 11,
The action of predicting a risk situation for the above object is:
An operation of acquiring sensor data of another mobile device associated with another object included in the image data; and
An operation for predicting that the object and the other object are in a dangerous situation when the amount of change in sensor data of the mobile device and the amount of change in sensor data of the other mobile device are synchronized and exceed a first threshold.
Including,
A computer program stored on a computer-readable storage medium.
In Article 13,
The action of predicting a risk situation for the above object is:
An operation of analyzing a time pattern in which a change in sensor data of the mobile device and a change in sensor data of the other mobile device occur; and
An action to predict a risk situation for the object based further on the results of the above time pattern analysis.
Including,
A computer program stored on a computer-readable storage medium.
In Article 13,
Actions that predict that the above object and the other object are in a dangerous situation,
If the number of mobile devices exceeding the first threshold by synchronizing the change amount of the above sensor data exceeds the second threshold, an action is included to predict that the object and the other object are in a dangerous situation.
A computer program stored on a computer-readable storage medium.
In Article 12,
The action of predicting a risk situation for the above object is:
An operation of extracting physical feature information about the object based on features on the image of the object; and
An action to predict a risk situation for the object by additionally considering physical characteristic information about the object.
Including,
A computer program stored on a computer-readable storage medium.
In Article 16,
In addition to considering the physical characteristic information of the above object, the operation of predicting a risk situation for the above object is as follows:
An operation of predicting a dangerous situation for a first object based on physical characteristic information for the first object included in the image data and sensor data of a mobile device of the first object; and
An operation for predicting a dangerous situation for a second object based on physical characteristic information about the second object included in the image data and sensor data of a mobile device of the second object.
Including,
A computer program stored on a computer-readable storage medium.
In Article 11,
The above actions are,
Further comprising an operation of acquiring a third type of data corresponding to the thermal image data,
The third type of data includes thermal image data corresponding to the image data,
The action of predicting a risk situation for the above object is:
An operation for predicting a risk situation for the object based on the second type of data and the third type of data.
Including,
A computer program stored on a computer-readable storage medium.
As a computing device,
at least one processor; and
memory;
Including,
At least one processor of the above,
Obtaining a first type of data corresponding to the image data;
Obtain location information for an object included in the above image data;
Based on the above location information, obtaining a second type of data generated by a mobile device associated with the object; and
Based on the first type of data and the second type of data, configured to predict a risk situation for the object,
device.
In Article 19,
The second type of data includes sensor data generated by a sensor included in the mobile device.
device.
In Article 19,
At least one processor of the above,
Acquire sensor data of another mobile device associated with another object included in the above image data; and
If the amount of change in sensor data of the mobile device and the amount of change in sensor data of the other mobile device are synchronized and exceed a first threshold, it is configured to predict that the object and the other object are in a dangerous situation.
device.
In Article 21,
At least one processor of the above,
Analyze the time pattern in which the amount of change in sensor data of the mobile device and the amount of change in sensor data of the other mobile device occur; and
Further configured to predict a risk situation for the object based on the results of the above time pattern analysis,
device.
In Article 21,
At least one processor of the above,
If the number of mobile devices exceeding the first threshold by synchronizing the change in the above sensor data exceeds the second threshold, it is configured to predict that the object and the other object are in a dangerous situation.
device.
In Article 20,
At least one processor of the above,
Extracting physical feature information about the object based on the image features of the object; and
By additionally considering the physical characteristic information of the above object, it is configured to predict a risk situation for the above object.
device.
In paragraph 24,
At least one processor of the above,
Predicting a risk situation for a first object based on physical characteristic information for the first object included in the image data and sensor data of a mobile device of the first object; and
A device configured to predict a dangerous situation for a second object based on physical characteristic information about the second object included in the image data and sensor data of a mobile device of the second object,
device.
In Article 25,
At least one processor of the above,
Acquire a third type of data corresponding to thermal image data,
The third type of data includes thermal image data corresponding to the image data,
Based on the second type of data and the third type of data, configured to predict a risk situation for the object,
device.

Images