KR102883210B1 - Listenable heatmap analysis method and appratus for explaining neural network model - Google Patents

Abstract

A heatmap auditoryization method and device for analyzing a neural network model are disclosed. According to an embodiment of the present invention, a method for analyzing a neural network model may include the steps of: converting target audio data to be analyzed into an audio spectrogram composed of a plurality of pixels and time-frequency data including phase information; inputting the audio spectrogram into a neural network model that analyzes the audio spectrogram to obtain an analysis result; determining the color of each pixel according to the contribution of each pixel to the analysis result to generate a heatmap image corresponding to the audio spectrogram; adjusting the heatmap image to a size corresponding to the time-frequency data and binarizing pixels with high and low contributions according to a set threshold to generate a heatmap mask; masking a pixel area with low contribution from the time-frequency data to generate masked time-frequency data; and restoring audio data capable of audibly representing sounds that have contributed to the analysis result of the neural network model from the masked time-frequency data.

Description

LISTENABLE HEATMAP ANALYSIS METHOD AND APPRATUS FOR EXPLAINING NEURAL NETWORK MODEL

The present invention relates to a method and device for audibly analyzing a heatmap for analyzing a neural network model, and more specifically, to a method and device for audibly analyzing sounds that have influenced the output value of a neural network model by converting a heatmap region representing the contribution of audio data in an analysis result for audio data of a neural network model into audio data.

Recent advancements in neural network technology, a key area of artificial intelligence, have led to research into neural network models capable of human-level recognition and classification performance in image and video applications. These models are also being actively utilized in audio data processing. However, effective research and utilization of neural network models requires the ability to analyze their predictions in a human-readable manner. Without this analysis technology, it becomes difficult to trust and utilize neural network models in areas requiring high reliability, such as autonomous vehicles. Furthermore, even when problems arise, root cause analysis and remediation become challenging.

In the image domain, various methods have been proposed to visualize the predictive basis of neural network models as heatmaps. Furthermore, methods for analyzing neural network models using heatmaps have also been attempted in the audio domain.

However, in the audio domain, unlike in the image domain, such visual analysis makes it difficult to pinpoint the specific sounds that actually influenced the neural network model's predictions. Therefore, the present invention proposes a heatmap audibility method that allows audible analysis of the sounds that the neural network model used to make predictions.

The present invention provides a method and device for listening to sounds that have contributed to the analysis results of a neural network model by converting an area with a high contribution to model prediction in a heat map using the analysis results of a neural network model into audio data in order to effectively study and utilize a neural network model in the field of audio data processing.

In addition, the present invention provides a method and device capable of judging the reliability of a neural network model by extracting and listening to audio data that contributed to the analysis results of a neural network model, thereby determining whether the analysis results of the neural network model are accurate.

According to an embodiment of the present invention, a method for analyzing a neural network model may include the steps of: converting original audio data into an audio spectrogram composed of a plurality of pixels and time-frequency data including phase information; inputting the audio spectrogram into a neural network model to obtain an analysis result; determining a color of each pixel according to a contribution of each pixel to the analysis result to generate a heatmap image corresponding to the audio spectrogram; adjusting the heatmap image to a size corresponding to the time-frequency data and binarizing it into pixels with high contribution and pixels with low contribution based on a threshold value to generate a heatmap mask; masking a pixel area with low contribution from the time-frequency data to generate masked time-frequency data, and restoring audio data capable of hearing a sound that has contributed to the analysis result of the neural network model from the masked time-frequency data.

In the step of converting into the above audio spectrogram, the original audio data is converted into an audio spectrogram corresponding to the input form of the neural network model, and in the process of converting into the audio spectrogram, time-frequency data including the phase information can be obtained.

The step of generating the above heatmap mask can generate a heatmap image by setting the contribution of each pixel to the analysis result as a color, and a detailed method of calculating the contribution includes a method of utilizing a gradient obtained by backpropagation or a method of weighted summing feature maps of a specific layer of the neural network model, and a heatmap mask divided into a high contribution area and a low contribution area can be generated by adjusting the heatmap image to a size corresponding to the time-frequency data and binarizing each pixel value based on a threshold value.

The step of restoring the above audio data may include generating masked time-frequency data by masking an area with low contribution in the time-frequency data through a bit operation of the heat map mask and the time-frequency data, and restoring audio data capable of hearing sounds that contributed to the analysis results of the neural network model from the masked time-frequency data.

A method for analyzing an acoustic scene classification model according to an embodiment of the present invention may include the steps of: converting original audio data into a mel spectrogram composed of a plurality of pixels and time-frequency data including phase information; inputting the mel spectrogram into an acoustic scene classification model to obtain an acoustic scene classification result; determining a color of each pixel according to a contribution of each pixel to the classification result to generate a heatmap image corresponding to the audio spectrogram; adjusting the heatmap image to a size corresponding to the time-frequency data and binarizing each pixel value based on a threshold value to generate a heatmap mask; masking the time-frequency data with the heatmap mask and restoring the masked time-frequency data into audio data to obtain audio data capable of hearing sounds that have contributed to the acoustic scene classification of the acoustic scene classification model.

In the step of converting to the above audio spectrogram, the original audio data is converted to a mel spectrogram corresponding to the input of an acoustic scene classification model, and time-frequency data including the phase information can be obtained in the process of converting to the mel spectrogram.

The step of generating the above heat map mask can generate a heat map image by weighting feature maps of a specific layer of the above acoustic scene classification model, and can generate a heat map mask divided into areas with high and low contribution to acoustic scene classification by adjusting the size corresponding to the time-frequency data and binarizing each pixel value based on a threshold value.

The step of obtaining the above audio data may include generating masked time-frequency data by masking a contribution or low area in the time-frequency data through a bit operation of the heat map mask and the time-frequency data, and obtaining audio data capable of hearing sounds that contributed to the acoustic scene classification from the masked time-frequency data.

In a heatmap auditory device of a neural network model that performs a method for analyzing a neural network model according to an embodiment of the present invention, the heatmap auditory device performs a process of generating time-frequency data and an audio spectrogram, a process of generating a heatmap image, and a process of masking time-frequency data and restoring a signal.

In the above time-frequency data and audio spectrogram generation process, the processor converts the original audio data input to the heatmap auditory device into time-frequency data in a complex domain consisting of multiple pixels through Fourier transform and processes the same in the time-frequency data masking and signal restoration process, and simultaneously converts the original audio data into an audio spectrogram of a neural network model consisting of multiple pixels and processes the same in the heatmap image generation process.

In the process of generating the above heatmap image, the processor analyzes the audio spectrogram using a pre-trained neural network model, generates a heatmap image having a binarized value according to the contribution of each pixel to the generated analysis result, and adjusts the size of the generated heatmap to correspond to the time-frequency data.

In the above time-frequency data masking and signal restoration process, the processor masks an area in the time-frequency data except an area adjacent to pixels with the highest contribution in the heat map image, and restores the masked time-frequency data into audio data using an inverse Fourier transform, thereby obtaining masked audio data that can hear sounds that contributed to the analysis results of the neural network model.

The neural network model of the above heatmap image generation process can utilize various audio features in the audio spectrogram domain as input, includes multiple layers each consisting of at least one weight, and can analyze the audio spectrogram.

In a heatmap auditory device of a neural network model that performs a method for analyzing an acoustic scene classification model according to an embodiment of the present invention, the heatmap auditory device performs a process of generating time-frequency data and an audio spectrogram, a process of generating a heatmap image, and a process of masking time-frequency data and restoring a signal.

In the above time-frequency data and audio spectrogram generation process, the processor converts the input original audio data into time-frequency data composed of multiple pixels by Fourier transforming the input original audio data and processes it in the time-frequency data masking and signal restoration process, and at the same time, converts the original audio data into a mel spectrogram, which is an input form of an acoustic scene classification model composed of multiple pixels, and processes it in the heatmap image generation process.

In the process of generating the above heatmap image, the processor includes a pre-trained acoustic scene classification model, analyzes the mel spectrogram, obtains the contribution of each pixel to the generated acoustic scene classification result by weighting feature maps of a specific layer, generates a heatmap image having a binarized value according to the contribution of each pixel, and resizes the size of the generated heatmap to correspond to the time-frequency data.

In the above time-frequency data masking and signal restoration process, the processor masks an area in the time-frequency data except an area adjacent to pixels with the highest contribution in the heat map image, and restores the masked time-frequency data into audio data by inverse Fourier transform, thereby obtaining audio data that can hear sounds that contributed to the acoustic scene classification of the acoustic scene classification model.

The acoustic scene classification model of the above heat map image generation process can utilize various audio features of the audio spectrogram domain as model input, includes multiple layers composed of at least one weight, and can perform acoustic scene classification by analyzing the audio spectrogram.

According to one embodiment of the present invention, in order to effectively study and utilize a neural network model in the field of audio data processing, a heat map image is generated using the analysis results of a neural network model, thereby extracting audio data that contributed to the analysis results of the neural network model.

In addition, according to one embodiment of the present invention, by extracting audio data that contributed to the analysis results of the neural network model, it is possible to hear whether the analysis results of the neural network model were extracted accurately, thereby determining the reliability of the neural network model.

FIG. 1 is a diagram illustrating input and output targets of a heat map auditory device of a neural network model according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating an example of a process for generating masked audio data from audio data according to one embodiment of the present invention.
FIG. 3 is a diagram illustrating an example of a Mel spectrogram and a heat map image according to one embodiment of the present invention.
FIG. 4 is a diagram illustrating an example of time-frequency data visualized as a spectrogram, a heatmap mask, and a masked spectrogram according to one embodiment of the present invention.
FIG. 5 is a diagram illustrating a flow chart of a method for analyzing a neural network model according to an embodiment of the present invention.

Hereinafter, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be modified in various ways, and the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, or alternatives to the embodiments are included within the scope of the patent application.

The terms used in the examples are for illustrative purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood to not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiments pertain. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

In addition, when describing with reference to the attached drawings, identical components will be assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted. When describing embodiments, if a detailed description of a related known technology is judged to unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

FIG. 1 is a diagram illustrating input and output targets of a heat map auditory device of a neural network model according to an embodiment of the present invention.

The present invention discloses a heatmap auralization method and a heatmap auralization device (101) for auralizing audio data that influences analysis results in a heatmap representing analysis results of a neural network model to measure the reliability of a neural network model that analyzes audio data. The heatmap auralization device (101) of the present invention corresponds to a processor, and the processor performs the heatmap auralization method.

To this end, the heatmap auditory device (101) of the present invention receives audio data and generates time-frequency data including phase information and an audio spectrogram.

Specifically, time-frequency data is a two-dimensional data in the complex domain obtained by Fourier transforming audio data, representing changes in each frequency axis over time. A spectrogram is a two-dimensional flat image that visualizes audio or wave data, combining the characteristics of the waveform and spectrum.

The form of the audio spectrogram is determined by the neural network model included in the heatmap generation device (101), and the mainly utilized forms include a spectrogram that takes the amplitude of time-frequency data, and a Mel spectrogram that expresses the frequency axis of the spectrogram in a Mel scale. A Mel spectrogram is a spectrogram that compresses the frequency axis in a Mel scale that reflects a human hearing model that is sensitive to low-frequency ranges, and is a spectrogram image that is nonlinearly compressed along the frequency axis.

The neural network model utilized in the present invention may include multiple layers, each composed of at least one parameter, and receives an audio spectrogram or two-dimensional audio feature and outputs analysis results. For example, the neural network model may be implemented using deep learning technology and may be a neural network model comprising multiple layers, such as a convolutional neural network (CNN).

Neural network models that analyze audio data can be various types of neural network models that perform analysis of audio data, such as recognizing acoustic events or acoustic scenes from various audio spectrograms.

In the present invention, a neural network model is included in a heatmap generation device (101) in a pre-trained state and is used to generate a heatmap image by analyzing an audio spectrogram.

There are two main ways to create a heatmap image: one is to utilize the gradient backpropagated from the prediction results of the neural network model, and the other is to weight the feature maps of a specific layer of the neural network model.

Specifically, the backpropagation method can obtain the contribution of each pixel of the image input to the neural network model as a gradient for the predicted result class. Each pixel Heatmap consisting of contributions to ) is the input image (as shown in the mathematical expression 1 below) ) and class Prediction score for ( ) can be expressed as.

Unlike the backpropagation method, the method of weighting the feature maps of a specific layer of the model constructs a heatmap by utilizing the feature maps generated during the forward propagation process of the neural network model.

Classes constructed using Grad-CAM (Gradient-weighted Class Activation Mapping) Heatmap for ( ) is a specific layer as shown in the mathematical expression 2 below. The th feature map ( ) and weights ( ) can be expressed as, where the weight is the predicted score ( Feature map of a specific layer for The gradient of the feature map pixel count ( ) is obtained by dividing by .

The heatmap generation device (101) generates a heatmap image for the analysis results of the neural network model in the above manner, adjusts the size of the heatmap image so that it can correspond to time-frequency data including phase information, and binarizes each pixel of the heatmap image based on a threshold value.

The size of the heatmap image is the same as the audio spectrogram input to the neural network model or the feature map of a specific layer, depending on the generation method. Accordingly, the heatmap generation device (101) can adjust (resize) the size of the heatmap image so that the size of the time-frequency data corresponds to the size of the heatmap image.

Resizing of heatmap images can be done using linear interpolation, but when nonlinear transformations such as Mel frequency compression are applied during the process of extracting audio spectrograms, resizing using the transpose matrix of the Mel filter bank or non-negative least square estimation is required.

A heatmap image is an image that contains values based on the contribution of each pixel in the audio spectrogram to the analysis results. For example, pixels with a high contribution to the analysis results may have a higher value.

In the present invention, in order to facilitate listening to the area with high contribution, this value is set as a threshold ( ) and scale correction value ( ) for the binarized heatmap image using the sigmoid function as in the mathematical expression 3 below. ) is proposed to be utilized.

A heat map generation device (101) obtains masked time-frequency data using a heat map mask and time-frequency data generated using a heat map image and an audio spectrogram, and performs an inverse Fourier transform on the masked time-frequency data to restore audio data, thereby obtaining masked audio data that can hear sounds that have affected a neural network model.

The heatmap audio device can generate a heatmap mask that emphasizes pixels that contribute to the analysis result by binarizing each pixel value of the heatmap image and adjusting the size corresponding to the time-frequency data (221). In the heatmap mask, an area composed of pixels that exhibit a contribution greater than a threshold value is an area with a pixel value of 1, and the rest are binarized to 0.

Masked time-frequency data ( can be obtained by masking the areas that do not correspond to the areas with high contribution, and the time-frequency data (as in mathematical expression 4 below) ) and binarized heatmap ( ) is calculated by element-wise multiplication.

Masked time-frequency data contains amplitude and frequency and can be restored to audio data through inverse Fourier transform.

FIG. 2 is a diagram illustrating an example of a process for extracting masked audio data from original audio data according to one embodiment of the present invention.

A heatmap auditory device receives audio data (201) as input and generates time-frequency data (221) including phase information and an audio spectrogram (211) that is input to a neural network model. Then, the heatmap auditory device can input the audio spectrogram (211) to a trained neural network model (212) to obtain a heatmap (213) representing the analysis result. Depending on the type of neural network model (212), the analysis result may be a result of classifying an acoustic event, an acoustic scene, or a music genre included in the audio data.

The heatmap auditory device can generate a heatmap image (213) corresponding to the audio spectrogram (211) by determining the color of each pixel based on its contribution to the analysis result of each pixel in the audio spectrogram (211). For example, the heatmap auditory device can generate a heatmap image (213) corresponding to the audio spectrogram (211) by weighting feature maps generated in a specific layer of a neural network model.

In addition, the heatmap audio device can generate a heatmap mask (214) by binarizing each pixel value of the heatmap image (213) and adjusting it to a size corresponding to time-frequency data (221). 210 in FIG. 2 illustrates the process of generating the heatmap mask (214).

Specifically, the heatmap auditory device can generate a heatmap mask (214) that emphasizes pixels that contributed to the analysis results for a specific class output from the neural network model (212).

The heatmap audio device adjusts the heatmap image (213) to a size corresponding to the time-frequency data. The size of the heatmap image (213) varies depending on the generation method, but is usually generated in a smaller form than the time-frequency data. Linear interpolation is used to increase the size of the heatmap. However, if nonlinear transformation is applied during the audio spectrogram extraction process, this must be taken into account when adjusting the size.

For example, if Mel frequency compression is applied during the process of extracting the audio spectrogram, the size must be restored taking this into account.

A heatmap audibility device binarizes pixels of a heatmap image (213) based on a threshold value to generate a heatmap mask that is divided into high-contribution pixel areas and low-contribution pixel areas. The heatmap audibility device can extract audio data (222) that influences a neural network model (212) from time-frequency data (221) by using the high-contribution areas in the heatmap mask.

In FIG. 2, 220 illustrates a process of extracting audio data that has influenced a neural network model (212) from time-frequency data (221) using a heat map mask (214) and obtaining masked audio data (223) that can be heard as sound.

The heatmap auditory device can distinguish between high-contribution pixels and low-contribution pixels based on a threshold using activation functions such as the sigmoid function and the unit step function.

A heatmap auditory device can mask regions other than high-contribution regions. Specifically, the heatmap auditory device can perform masking by binarizing regions with high and low contributions.

For example, a heatmap auditory device can generate a heatmap mask by binarizing pixels with high contribution to values closer to 1 and the remaining areas to 0.

The heatmap auditory device can mask time-frequency data (221) according to pixel areas with high contribution. The heatmap auditory device can perform a bitwise operation on the heatmap mask (214) and time-frequency data (221) to generate masked time-frequency data (222).

For example, the heatmap auditory device can obtain an area (223) corresponding to an area (215) with a high contribution to the heatmap mask (214) in the time-frequency data (221) by not changing pixels in an area corresponding to an area with a high contribution to the heatmap mask (214) in the time-frequency data (221) and masking the remaining areas.

For example, for a heatmap mask (214) in which a region with a high contribution is 1 and the remaining regions are 0, a heatmap auditory device can perform an AND operation between each pixel of the heatmap mask (214) and the pixel corresponding to that pixel in the time-frequency data (221) to extract a region (223) corresponding to a region with a contribution higher than a threshold value in the time-frequency data (221).

In addition, the heatmap auditory device can mask areas in the time-frequency data (221) whose contribution is lower than a threshold value to 0. The masked areas in the masked time-frequency data (222) correspond to areas mapped to 0 in the heatmap mask (214).

The heatmap auditory device can obtain audio data (224) that contributes to the analysis results of the neural network model (212) by transforming the masked time-frequency data (222). The heatmap auditory device can convert the masked time-frequency data (222) into audio data (224) by performing an inverse Fourier transform. If the time-frequency data has lost phase information, the heatmap auditory device can use the Griffin-Lim algorithm (GLA) or the like as a method to restore the phase information.

When analyzing an acoustic scene classification model, a heatmap auditory device inputs an audio spectrogram into a neural network model to obtain an acoustic scene classification result for audio data, and generates a heatmap image that emphasizes pixels that contributed to the classification result.

A heatmap auditory device can generate a heatmap image corresponding to an audio spectrogram by weighting feature maps generated from a specific layer of an acoustic scene classification model. The specific process for generating the heatmap image is identical to the process described in 210.

In addition, the heatmap auditory device can mask the time-frequency data with a heatmap mask. Based on the masked result, the heatmap auditory device can obtain audio data that can audibly identify sounds that contributed to the acoustic scene classification of the acoustic scene classification model.

FIG. 3 is a diagram illustrating an example of a Mel spectrogram and a heat map image according to one embodiment of the present invention.

Fig. 3 illustrates an example of a Mel Spectrogram (301) and a heat map image (302) corresponding to a neural network model audio spectrogram. Each horizontal axis represents time, and the vertical axis represents frequency. However, while the color of a pixel in the Mel Spectrogram (301) represents a difference in amplitude, the color of a pixel in the heat map image (302) represents the contribution of each pixel to the analysis result.

In the heat map image (302), a larger pixel value indicates a larger contribution, and a smaller pixel value indicates a smaller contribution. In addition, in the heat map image (302) of Fig. 3, a larger pixel value indicates a redder color, and a smaller pixel value indicates a bluer color.

FIG. 4 is a diagram illustrating an example of time-frequency data visualized as a spectrogram, masked time-frequency data visualized as a heatmap mask, and log spectrogram according to one embodiment of the present invention.

FIG. 4 illustrates an example of time-frequency data (401) visualized as a log spectrogram, a heatmap mask (402), and time-frequency data (403) masked according to the heatmap mask (402).

Since time-frequency data is information in the complex domain, an image (401) converted to a log spectrogram is used as an example for diagramming it. Similarly, since masked time-frequency data is also information in the complex domain, an image (403) converted to a log spectrogram is used for diagramming it.

The heatmap auditory device can resize the heatmap image obtained in Fig. 3 to a size that corresponds to time-frequency data, and binarize a region with a high contribution and the remaining region based on a threshold value.

The heatmap audio device can binarize high contribution areas and remaining areas using activation functions such as the sigmoid function and unit step function that binarize high and low contribution areas based on a preset threshold.

For example, a heatmap audio device creates a heatmap mask by setting pixels with contributions below a threshold to 0 and turning them black, and by setting pixels with contributions above the threshold to have values greater than 0.

In addition, the heatmap auditory device can obtain masked time-frequency data (403) by masking the time-frequency data (401) with a heatmap mask (402).

Specifically, when the heatmap image is binarized into 0 and 1, the heatmap audio device can mask areas with low contribution in the time-frequency data (401) to 0 by performing an AND operation on pixels having the same time and frequency in the time-frequency data (401) and the heatmap mask (402).

The heatmap auditory device can obtain masked audio data that can hear sounds that have influenced the analysis results of the neural network model by inverse Fourier transforming the masked time-frequency data (403).

FIG. 5 is a diagram illustrating a flow chart of a method for analyzing a neural network model according to an embodiment of the present invention.

In step (501), the heatmap auditory device extracts an audio spectrogram from the audio data to be analyzed and transmits it to the heatmap generation device, and preserves time-frequency data including phase information generated in the spectrogram extraction process and transmits it to the activation area masking and signal restoration device.

In step (502), the heatmap auditory device analyzes the audio spectrogram and generates a heatmap image corresponding to the audio spectrogram. Specifically, the heatmap auditory device can generate a heatmap image representing the pixel-by-pixel contribution to the class-specific analysis results of a layer that outputs the analysis results in a neural network model.

In step (503), the heatmap auditory device can resize the heatmap image so that it can correspond to time-frequency data and binarize each pixel value based on a preset threshold.

For example, a heatmap auditory device can generate a heatmap mask by using an activation function such as a sigmoid function or a unit step function to binarize an area whose contribution to the analysis result is higher than a threshold value into 1 and an area whose contribution is lower than the threshold value into 0.

In step (504), the heatmap auditory device masks the time-frequency data according to the heatmap mask. The heatmap auditory device can mask the remaining areas except for areas with high contribution. Specifically, the heatmap auditory device can mask the time-frequency data of areas with contributions lower than a threshold value, thereby generating masked time-frequency data that includes only areas with contributions higher than the threshold value.

In step (505), the activation area masking and signal restoration device of the heat map auditory device performs an inverse Fourier transform on the masked time-frequency data to obtain audio data that can hear sounds that contributed to the analysis results of the neural network model.

Meanwhile, the method according to the present invention can be written as a program that can be executed on a computer and implemented in various recording media such as a magnetic storage medium, an optical reading medium, and a digital storage medium.

Implementations of the various technologies described herein may be implemented as digital electronic circuitry, or as computer hardware, firmware, software, or combinations thereof. Implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., a machine-readable storage medium (computer-readable medium) or a radio signal, for processing by the operation of a data processing device, e.g., a programmable processor, a computer, or multiple computers, or for controlling the operation thereof. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or multiple computers at a single site, or to be distributed across multiple sites and interconnected by a communications network.

Processors suitable for processing a computer program include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random-access memory, or both. Components of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer may include, or be coupled to receive data from, transmit data to, or both, one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data. Information carriers suitable for embodying computer program instructions and data include, for example, semiconductor memory devices, magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as compact disk read only memory (CD-ROM), digital video disks (DVD), magneto-optical media such as floptical disks, read only memory (ROM), random access memory (RAM), flash memory, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), etc. The processor and memory may be supplemented by, or included in, special purpose logic circuitry.

Additionally, the computer-readable medium may be any available medium that can be accessed by a computer, and may include both computer storage media and transmission media.

While this specification contains details of a number of specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of particular inventions. Certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. Furthermore, although features may operate in a particular combination and may initially be described as being claimed as such, one or more features from a claimed combination may in some cases be excluded from that combination, and the claimed combination may be modified into a subcombination or variation of a subcombination.

Likewise, while operations are depicted in the drawings in a particular order, this should not be construed as requiring that those operations be performed in the particular or sequential order depicted to achieve desired results, or that all depicted operations be performed. In certain instances, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various device components of the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the program components and devices described may generally be integrated together in a single software product or packaged into multiple software products.

Meanwhile, the embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to aid understanding and are not intended to limit the scope of the present invention. It will be apparent to those skilled in the art that other modifications based on the technical concepts of the present invention are possible in addition to the embodiments disclosed herein.

101: Heatmap Auditory Device

Claims (20)

In a heatmap auditory method performed by a heatmap auditory device including a processor,
A step of converting audio data into an audio spectrogram composed of a plurality of pixels and time-frequency data including phase information;
A step of obtaining analysis results by inputting the audio spectrogram into a neural network model that analyzes the audio spectrogram;
A step of generating a heatmap image representing the contribution of each pixel to the analysis results;
A step of masking the time-frequency data according to an area composed of pixels whose contribution is higher than a threshold value in the heat map image; and
A step of restoring audio data capable of hearing sounds that contributed to the analysis results of the neural network model from the above masked time-frequency data.
A heatmap auditory method including:
In the first paragraph,
The steps for generating the above heatmap image are:
A heatmap auditoryization method for generating a heatmap image by determining the color of pixels constituting the heatmap image according to their contribution to the above analysis results.
In the first paragraph,
A step of adjusting the heatmap image to a size corresponding to the time-frequency data and generating a heatmap mask by binarizing it into a high contribution pixel area and a low contribution pixel area based on a threshold value.
Including more,
The above masking step is,
A heatmap auditoryization method for masking the time-frequency data according to an area composed of pixels with high contribution based on the threshold value in the heatmap mask.
In the third paragraph,
The above masking step is,
A heatmap auditoryization method that generates masked time-frequency data by masking an area corresponding to a pixel whose contribution of the heatmap mask is lower than a threshold value from the time-frequency data through a bit operation of the heatmap mask and the time-frequency data.
In a heatmap auditory method performed by a heatmap auditory device including a processor,
A step of converting audio data into a mel spectrogram composed of a plurality of pixels and time-frequency data including phase information;
A step of obtaining an acoustic scene for the mel spectrogram by inputting the mel spectrogram into an acoustic scene classification model that classifies an acoustic scene through an audio spectrogram;
A step of generating a heatmap image representing the contribution of each pixel to the acquired acoustic scene;
A step of masking the time-frequency data according to an area composed of pixels whose contribution is higher than a threshold value in the heat map image; and
A step of restoring audio data capable of hearing sounds that contributed to the acoustic scene classification of the acoustic scene classification model from the above-mentioned masked time-frequency data.
A heatmap auditory method including:
In paragraph 5,
The steps for generating the above heatmap image are:
A heatmap auditoryization method for generating a heatmap image by determining the color of pixels constituting the heatmap image according to their contribution to the above acoustic scene classification.
In paragraph 5,
A step of adjusting the heatmap image to a size corresponding to the time-frequency data and generating a heatmap mask by binarizing it into a high contribution pixel area and a low contribution pixel area based on a threshold value.
Including more,
The above masking step is,
A heatmap auditoryization method for masking the time-frequency data according to an area composed of pixels with high contribution based on the threshold value in the heatmap mask.
In paragraph 7,
The above masking step is,
A heatmap auditoryization method that generates masked time-frequency data by masking an area corresponding to a pixel whose contribution of the heatmap mask is lower than a threshold value from the time-frequency data through a bit operation of the heatmap mask and the time-frequency data.
delete delete delete delete delete delete delete delete delete delete delete delete