WO2024218836A1 - Associative memory device - Google Patents

Abstract

This novel associative memory device models, by a mathematical expression, a neural network that is described with Potts-type interactions, and operates on the basis of a novel model of neural network. An associative memory device 100 comprises: an N-bit multilayer neuron 101 (x_i, where i=1, ..., N), an N×M-bit memory holding unit 102, and an interaction computation unit 103. The N-bit multilayer neuron 101 means the time evolution of an N-bit neuron 111. It is also possible to reduce the calculation load in the whole process of associative memory by combining arithmetic processing of associative memory based on a Hopfield-type neural network of the prior art with a computation for minimizing a Potts Hamiltonian.

Description

Associative Memory

The present invention relates to an associative memory device, and more specifically to an associative memory device based on a neural network.

Humans can recall the face of an acquaintance from their name, and can also visualize the overall features of an acquaintance's face when they are wearing a mask, for example. When humans see an object, they can recall the entire object even if part of it is hidden, or recall another object related to the object. This is one of the intellectual processes in the human brain called association. This association process can be thought of as an associative memory system that recalls the most relevant data from stored data corresponding to input data. A neural network is a network that constructs associative memory by expressing basic building blocks called neurons using a mathematical model and connecting these together. It can model associative memory. As an example suitable for application to associative memory, it is known that it can be simply modeled using a Hopfield-type neural network, etc.

FIG. 9 is a diagram explaining a Hopfield type neural network. (a) of FIG. 9 conceptually shows the configuration of a Hopfield type neural network. In neural network 50, a certain neuron 51 is connected to all other neurons, making it a completely interconnected network. This network maps the spin of particles in physics to the state of neurons, and the interactions between spins to the connections of neurons, and uses a definition formula in physics to simulate a human neural network and realize associative memory processing. Associative memory using a Hopfield type neural network is known to be described as a minimization problem of the Ijin Hamiltonian, as described below (Non-Patent Document 1).

Patent No. 625507 specification

J. J. Hopfield, Proceedings of National Academy of Sciences of the United States of America Vol79, No.8, pp 2554-2558, (1982) H. K. Sulehria and Y. Zhang, Information Technology Journal Vol7 No.4 pp 684-688 (2008) Krotov, D. & Hopfield, J. Dense associative memory for pattern recognition. 325 Neural Information Processing Systems 29, 1172-1180 (2016) Ramsauer, H. et al. Hopfield networks is all you need. Arxiv:2008.02217 D. Ventura and T. Martinez, Proceeding of the International Joint Conference on Neural Networks, 4-9May, IEEE, Alaska, USA, pp 509-513 T. Inagaki, Y. Haribara, etal, “A coherent ising machine for 2000-node optimization problems,” Science 354, 603-606 (2016)

However, the prior art associative memory using a Hopfield neural network had the problem of recalling incorrect patterns (spurious memories), resulting in a reduced memory capacity. As will be described later, in a Hopfield neural network, the connections between neurons are determined so that the energy function for a given solution becomes a minimum value. In reality, as shown in Figure 9(b), the value of the energy function for the spin state has many extreme values corresponding to false solutions, i.e., spurious attractors 53a, 53b, around the minimum value 52.

When the spurious attractors increase, the probability of making incorrect associations increases, and the memory capacity decreases. In the conventional associative memory, the memory capacity is limited to a very small value of about 0.12N, where N is the number of neurons (Non-Patent Document 1). Various methods have been proposed to improve the capacity, but even with these improvements, the capacity has only increased to about 5N at most (Non-Patent Document 2). In the latest research, a method has been proposed to improve the Hopfield network and expand the capacity to ^2N/2 (Non-Patent Document 3, Non-Patent Document 4). A method has also been proposed to increase the capacity to about ^2N by using high-cost quantum mechanics and quantum calculation, but it is difficult to actually realize this with current technology (Non-Patent Document 5).

The present invention was made in consideration of these problems, and its purpose is to provide a configuration for an associative memory device that significantly expands memory capacity.

One embodiment of the present invention is an associative memory device that includes a calculation unit that executes a feedback calculation to minimize the Potts Hamiltonian H _potts and a memory holding unit that stores in advance information ξ to be used for association, and is configured to output, as an association result, stored information that is closest to input information x, based on the result of the feedback calculation to minimize the Potts Hamiltonian H _potts .

Another embodiment of the present invention is an associative memory device comprising an input/output unit accessible to information ξ configured as a learning input, and a calculation unit that executes a feedback calculation to minimize the Potts Hamiltonian H _potts , and configured to output, as an associative result, information within the information ξ that is closest to information x input to the input/output unit based on a result of the feedback calculation.

The present invention provides a configuration for an associative memory device that significantly expands memory capacity.

FIG. 2 is a conceptual diagram of a neural network corresponding to the associative memory device of the present disclosure. FIG. 2 is a diagram showing more specific functions and a configuration of the content addressable memory device of the present disclosure. FIG. 2 illustrates an algorithm for implementing associations in the content addressable memory device of the present disclosure. 13 is a graph showing an example of numerical calculation of the accuracy rate of associative memory in comparison with the prior art. FIG. 11 is a diagram showing the configuration of a content addressable memory device according to a second embodiment. FIG. 11 is a diagram showing an algorithm for performing association in the content addressable storage device of the second embodiment. FIG. 1 is a diagram illustrating an example of the configuration of an associative memory device including a physically coherent Ising machine. FIG. 11 is a diagram showing the configuration of a content addressable memory device according to a third embodiment. FIG. 1 is a diagram illustrating a Hopfield type neural network.

The associative memory device of the present disclosure is based on a new neural network configuration that minimizes the Potts Hamiltonian, and can be easily realized by a general-purpose computer, etc. The currently realized memory capacity of 2 ^{N /2} is significantly improved, and expanded to a capacity of about 2 ^N . At first glance, this may seem like a minor change, but even when the number of bits N of stored information is 100, this is an improvement of about 100 million times, and when the number of bits N is even larger, the improvement is an astronomical multiple.

Below, we will first go into more detail about the problems with the prior art associative memory device using a Hopfield neural network that minimizes the Ijin Hamiltonian. We will then explain the configuration of the associative memory device of the present disclosure and the concept of the neural network on which it is based.

The Hopfield-type associative memory of the prior art is described as a minimization problem of the Ising Hamiltonian H described below. The Ising model is originally a model of a lattice model that statistically analyzes the interactions of spins arranged at each site of a lattice point of a magnetic material. This Ising model is applied to modeling of neural networks, and is used not only for associative memories but also for solving combinatorial optimization problems, for example.

The Ising Hamiltonian H in equation (1) represents the energy of the Ising model system. The coupling constant J _ij is given by the following equation, and σ _i is the ith (i = 1...N) Ising spin. The Ising spin is a binary spin that takes any value between ±1.

The i-th (i = 1...N) bit of information ξ in the stored information shown in definition (3) included in equation (2) is expressed as ±1.

In an associative memory device, "associating" means outputting a recalled result for an input that corresponds to the input data. As shown in FIG. 9(a), a Hopfield neural network 50 based on the Ising model is represented as having neurons 51 interconnected by lines that correspond to coupling coefficient constants. A state corresponding to the input data is given as the initial state to the neural network 50, and the final equilibrium state in which the Ising Hamiltonian H is minimized corresponds to the recalled result.

The minimum energy state of the Ising Hamiltonian H in equation (1) may take a state different from the stored information ξ as the number of stored pieces of information M increases. This is the spurious attractor shown in Figure 9(b). In an associative memory device, as convergence to the spurious attractor increases, the probability of making incorrect associations increases, resulting in a reduction in memory capacity. As mentioned above, the memory capacity (the number of pieces of information that can be stored M) is limited to a very small value.

The associative memory device disclosed herein is based on a neural network configuration described by Potts-type interactions, rather than the Hopfield-type neural network of the prior art. Below, an embodiment of the associative memory device disclosed herein will be described in detail with reference to the drawings.

[Embodiment 1]
Fig. 1 is a conceptual diagram showing the associative memory device of the present disclosure abstractly with a corresponding neural network. As described later, the associative memory device 100 of the present disclosure is physically implemented by arithmetic processing using a general computer equipped with a memory, a central processing unit (CPU), etc. A neural network described by Potts-type interactions is modeled by mathematical expressions, and the associative memory device operates based on the new model of the neural network. The associative memory device 100 of Fig. 1 is abstractly represented for comparison with the Hopfield-type neural network of Fig. 9(a), and the more specific operation of the neural network is represented by a series of mathematical expressions described later.

1, an associative memory device 100 includes an N-bit multi-layer neuron 101 (x _i :i=1,...,N), an NxM-bit memory holding unit 102, and an interaction calculation unit 103. The N-bit multi-layer neuron 101 means the time evolution of an N-bit neuron 111, as described later. The time-evolving neuron configuration in FIG. 1 is in contrast to the interconnected type configuration of a Hopfield neural network shown in FIG. 9(a). Each element of the memory holding unit 102 corresponds to the i-th (i=1...N) bit information ξ of the m-th (m=1...M) stored information defined in equation (3), and represents a memory holding bit 112.

The interaction calculation unit 103 represents the interaction between the memory storage bits 112 and the neurons 111, and is described by a complex mathematical formula described below. The associative memory device 100 in FIG. 1 visualizes the action of a neural network described by Potts-type interactions and the mathematical formulas that define it. Next, we will formulate the neural network described by Potts-type interactions on which the associative memory device 100 of the present disclosure is based.

The following describes the Potts Hamiltonian H _Potts , the cost function F, and related definitions that are minimized in the neural network on which the associative memory device 100 shown in Figure 1 is based. The Potts Hamiltonian H _Potts is expressed by the following equation.

The Σ in equation (4) is a Kronecker delta function, which, as shown in definition (5), is 1 only when the two vectors x and y in the parentheses are equal, and is 0 in other cases. The ξ ^m in the parentheses in equation (4) represents the stored information ξ as an N-dimensional vector. The Hamiltonian H _Potts in equation (4) can be written as follows using each component of the vector:

Here, consider a cost function F as follows:

In equation (8), β is a hyperparameter having a large value, and when β→∞, the cost function F has the following proportional relationship:

As is clear from equation (9), minimizing the cost function F is equivalent to minimizing the Potts Hamiltonian H _Potts defined in equation (4). In other words, minimizing the cost function F is equivalent to minimizing the Potts Hamiltonian H _Potts , and minimizing the Potts Hamiltonian H _Potts is equivalent to minimizing the cost function F.

Here, for the cost function F shown in formula (8), the spin σ (=±1) is replaced with a variable x _i that takes real numbers from -1 to 1 as shown below. This replacement is a measure for the convenience of numerical calculation, and is intended to allow the number x _i to change gradually with time evolution by considering a continuous variable x _i , thereby enabling good "association."

For equation (10), the cost function F is further rewritten as follows. This rewriting is a device for convergence in the minimization process for the cost function F, which will be described later, and is necessary to replace the spin with a real number x _i .

In the rewritten formula (11), the replaced x _i is returned to σ _i again. Then, x _i x _i →σ _i σ _i = 1, so it can be seen that formula (11) is identical to the original formula (8) that defines the cost function F, except for the constant term in parentheses.

As is clear from the above examination of equations (8) to (11), the Potts Hamiltonian H _{Potts is reproduced by setting β to the limit of infinity in the cost function F of equation (10). In reality, for convenience of numerical calculation, β is stopped at a relatively large finite value (β>1) and the Potts Hamiltonian H Potts of} equation (11) is minimized. β can be set to 5, for example. This minimization can be achieved by updating x _k by the following equation.

The update equation for x _k in equation (12) describes the time evolution of the neuron. In equation (12), α is a hyperparameter and is set to an appropriate small value (0<α<1). The differential term of the second term on the right side of equation (12) is expressed by the following equation. α can be set to 0.5, for example.

The process of minimizing the Potts Hamiltonian H _Potts using the above-mentioned equations (8) to (13) corresponds to the interaction calculation unit 103 in the conceptual diagram of the associative memory device 100 shown in FIG.

In formula (13), x _k is the input information 104 to the multi-layer neuron 101 in the conceptual diagram of the associative memory device 100 in FIG. 1, and is a real number ranging from −1 to 1. In reality, it is N-bit input data, and association is performed on the input information 104. Since FIG. 1 is an abstract model, four neuron circles are shown, and this is an example where N=4. In formula (13), ξ with subscripts m and k is information stored in the memory retention unit 102, and is the k-th (k=1...N) bit information of the m-th (m=1...M) stored information as in the above definition (6).

The interaction calculation unit 103 performs minimization processing given by equations (10) and (11) using the memory holding unit 102 and input information 104. Specifically, it performs updates in equations (12) and (13) and outputs x ^new (with →) as the final output 105 after repeating the update of x (with →) a certain number of times. This series of processes associates the stored ξ ^μ that is closest to the initial input x (with →). Next, a more specific possible configuration and algorithm of the content addressable memory device 100 of the present disclosure will be described.

Fig. 2 is a diagram showing more specific functions and configurations of the associative memory device of the present disclosure. Fig. 2(a) shows an associative memory device 100 represented by functional blocks, and corresponds to the abstract configuration diagram represented as a neural network in Fig. 1. The reference numerals of each block in Fig. 2(a) correspond to those in Fig. 1. In Fig. 2(a), a neuron 101 represents the multi-layer neuron described in conjunction with Fig. 1, and a novel neural network represented based on the Potts Hamiltonian H _Potts defined by equation (4) is shown as one functional block.

Therefore, the associative memory device of the present disclosure can be implemented as one that includes calculation units 101, 103 that execute a feedback calculation to minimize the Potts Hamiltonian H _potts , and a memory holding unit 102 that stores in advance information ξ to be used for association, and is configured to output, as the association result, stored information 105 that is closest to input information x 102, based on the result of the feedback calculation to minimize the Potts Hamiltonian H _potts .

2(b) shows a specific configuration for implementing the associative storage device 100, which is implemented by the arithmetic processing of the computer 100-1. The computer 100-1 includes a CPU 120 and a memory 121, and also has an interface unit (I/O) 123 for inputting and outputting information to and from the outside. The memory 122 can be various storage devices capable of temporary or semi-permanent storage, and may include multiple memories of different types. When the CPU 120 operates, an arithmetic program 122 for implementing associative storage according to an algorithm described below is loaded into the memory 122. The program 122 may be stored in a memory within the computer, or may be provided from outside the computer 100-1. It may be provided to the computer 100-1 via a network. Also, the information stored in the memory retention unit 102 in FIG. 1 and FIG. 2(a) may be stored in a storage means outside the computer 121, different from the memory of the computer 121.

The neuron 101 and the interaction calculation unit 103 in the functional block of FIG. 2(a) correspond to the calculation processing based on the program 122 in the CPU 120 in FIG. 2(b), and the memory holding unit 102 corresponds to the physical memory 121 in FIG. 2(b). As long as the processing of the algorithm described below can be carried out, it is not limited to the configuration of the computer 100-1 described above, and some of the calculation processing can be carried out by dedicated hardware, or can be carried out by distributed processing using multiple computers.

FIG. 3 is a diagram showing an algorithm for performing association in the associative memory device of the present disclosure. A flow chart 300 shows the processing shown in formulas (4) to (13) of the neural network based on the Potts Hamiltonian H _Potts in FIG. 1. The algorithm will be described below with reference to the associative memory device 100 in FIG. 2(a) and the above-mentioned formulas. The flow chart 300 starts with step S301 of initial input to the associative memory device 100 and ends with step S305 of outputting the associative result x. By going through all the steps of the flow chart 300, the associative result 105 is output for the input data 104, and the associative processing in the associative memory device 100 is performed.

In S302, data x _i is input to the device 100, which corresponds to the time-evolving neuron 101 being set to its initial state in Fig. 1. In other words, the four neurons lined up vertically on the left side of Fig. 1 can be considered as the initial state.

S303 and S304 are repeated in a loop until the convergence condition is satisfied. For the sake of simplicity, it is assumed that the input data has not converged in the initial state. x _k is calculated and updated using equations (12) and (13). Convergence is judged again in S303 using x _k (x ^new ) updated in S304.

In S303, it is determined that convergence has occurred when the value of dF/dx _k in equation (13) becomes 0. If the determination result is No (false), x _k is updated again in S304. Note that the method of determining convergence can be varied in various ways.

If the determination result is Yes (true), the latest X _k is output as the association result in S305.

In the above-described computation algorithm of flow 300, the repetition of steps S303 and S304 to update x _k according to equation (12) corresponds to the time evolution of neuron 111 in the conceptual diagram of associative memory device 100 in Fig. 1. The repetitive computations of steps S303 and S304 execute a feedback calculation to minimize the Potts Hamiltonian H _potts .

In the above explanation, it was described that the initial input data 104 has not converged at the step S302, but if the input data is very similar to the association result, it may converge immediately without updating in step S304.

As described above, the associative memory device 100 of the present disclosure minimizes a cost function represented by an interaction operation between a neural network modeled based on the Potts Hamiltonian H _Potts and memory bits in a memory holding unit. This minimization of the cost function is performed by a repetitive operation described as the time evolution of neurons, and association is performed. This is a relatively complex operation compared to the operation of minimizing the Isin Hamiltonian H, which is expressed only by coupling constants and is modeled by fully interconnected neurons in a Hopfield neural network of the prior art.

The algorithm shown in FIG. 3 can be implemented by a normal computer 100-1 as shown in FIG. 2(b) if a CPU with a certain level of computing power is used. A configuration for reducing the computing power of the CPU will be described later as embodiment 2.

The neural network-based associative memory device 100 modeled on the Potts Hamiltonian H _Potts of the present disclosure has been shown numerically to be free of spurious attractors.

Figure 4 is a graph showing an example of numerical calculation of the accuracy rate of associative memory, comparing the conventional technology with the associative memory device disclosed herein. The horizontal axis of the graph shows the M/N value, where M is the number of stored information and N is the number of bits of information, and the horizontal axis shows the accuracy rate. The accuracy rate was determined as the case where one of the stored items was associated with a randomly given input. The parameters used in the numerical calculation were α = 0.5 and β = 10. It was confirmed that all converged sufficiently with less than 100 updates.

The ▽ (inverted triangle) plot shows the case of a conventional associative memory device using a Hopfield neural network that minimizes the Ising Hamiltonian. It can be seen that the accuracy rate drops when M/N is around 0.12. This drop in accuracy rate is due to spurious attractors, as already mentioned in FIG. 9(b). On the other hand, the ● (circle) plot shows the accuracy rate in the associative memory device 100 of the present disclosure, which is based on a neural network configuration described by Potts-type interactions. Regardless of the value of M/N, there is no drop in accuracy rate at all.

As is clear from the comparison in Figure 4, the accuracy rate does not decrease at all in the associative memory device, suggesting that spurious attractors do not occur when the neural network configuration based on the Potts Hamiltonian shown in Figure 1 is used. The energy function for the spin state of the Potts Hamiltonian has a simple structure in which all points where the Kronecker δ is 1 have a minimum energy value, and all other points are 0. On the other hand, the energy function for the spin state of the Ising Hamiltonian has local minima at various values in various places, as shown in Figure 9(b), and spurious attractors appear.

Setting a cost function that minimizes the Potts Hamiltonian, as shown in equation (8), has not been attempted up until now. Furthermore, as examined in equations (8) to (11), by taking β to the limit of infinity in the cost function F in equation (10), we devised an operation that reproduces the Potts Hamiltonian, and were able to describe the time evolution of neurons with practical calculations.

[Embodiment 2]
In the associative memory device of the first embodiment, the calculation of interaction with stored information by formulas (11) to (13) for minimizing the Potts Hamiltonian requires a greater computing power than the calculation of the coupling constant as in formula (2) of the prior art. If the computing power of the CPU is insufficient, the associative processing may take longer than the prior art using a Hopfield neural network. In this embodiment, a configuration is disclosed that combines the calculation processing of the associative memory based on the Hopfield neural network of the prior art with the calculation for minimizing the Potts Hamiltonian to reduce the computation load in the entire process of the associative memory.

FIG. 5 is a diagram showing the functional blocks and configuration of the associative memory device of embodiment 2. The associative memory device 200 includes a neural network 210 and a weight update unit 220. The neural network 210 may be, for example, a Hopfield-type neural network that performs association by minimizing the Isin Hamiltonian H. In other words, it corresponds to the associative memory device of the prior art. The neural network 210 forms a first calculation loop that minimizes the Isin Hamiltonian H, as described below.

The weight update unit 210 minimizes a cost function using the Potts Hamiltonian H _Potts , and is implemented in the same manner as the calculation in the algorithm in the associative storage device of embodiment 1. As will be described later, the weight update unit 210 forms a second calculation loop that minimizes the Potts Hamiltonian H _Potts .

The details of the algorithm will be described later, but the weighting section 202 and calculation section 203 of the neural network 210, and the calculation section 222 of the weight updating section 210 are all calculations that can be performed by a single CPU. Therefore, the associative memory device 200 of the embodiment of FIG. 5 can also be implemented by a computer 100-1 as shown in FIG. 2(b). Also, a neural network in which the calculations of the weighting section 202 and calculation section 203 of the neural network 210 are implemented using elements that can perform specific calculations at high speed, such as an FPGA, can also be used. In this case, the calculation section 222 of the weight updating section 210 can also be implemented by a CPU. The neural network 210 may also be implemented by a calculation device that uses optical pulses as will be described later.

The input 104, output 105, and learning input 106 of the associative memory device 200 of the second embodiment are the same as those of the first embodiment. The input information 104 to the input/output unit 211 is a real number between -1 and 1, and is actually N-bit input data, and association is performed on the input information 104. By going through all the steps of flow 400 in FIG. 6, which will be described later, the associative result 105 is output for the input data 104, and the associative processing in the associative memory device 200 is performed.

FIG. 6 is a diagram showing an algorithm for performing association in the associative storage device of embodiment 2. Flowchart 400 shows the processing performed in the associative storage device 200 of FIG. 5. Below, the computation algorithm of the associative storage device 200 will be explained with reference to flowchart 400.

First, in S402, initial input information x _i is input to the input/output unit 211. For the initial input, in order to simplify the explanation, the convergence determination step in S403 is bypassed, and the determination in S403 will be described later.

Next, in S404, the weighting unit 212 multiplies the input x _i by the weights J _ij and B _i to obtain y _i as shown in the following equation.

In formula (14), i and j are i=1,...,N, j=1,...,N, and are bit indexes of the input x _i . Also, x ₀ in formula (14) is a special input that is always 1, and its weight B _i is also a kind of special weight called a magnetic field term. y _i in formula (14) is the sum of the Σ term of the coupling coefficient (interaction) J _ij and the magnetic field term, and the form of y _i in formula (14) is the Ising Hamiltonian H _Ising . That is, it means that y _i weighted by formula (14) is expressed by a Hopfield-type neural network by formulas (1) and (2).

Furthermore, in S405, the weighted input y _i is further replaced by an activation function in the calculation unit 213 as shown in the following equation.

Equation (15) corresponds to updating the current x _i to new x _i , and S405 means that "association" is performed in the Hop-Filled neural network 210. Equation (15) may be replaced with another activation function, and another neural network other than the Hop-Filled type may be used. The step of S403 is performed again for the updated x _i .

In S403, a convergence test is performed on the updated x _i . The condition for this convergence test is when the updated x ^new is no longer different from the x ^old before the update, that is, |x ^new -x ^old |=0. If the convergence test result is No (false), the calculation of y _i is repeated again in S404, and the update of x _i is repeated in S404 (first calculation loop). If the convergence test result is Yes (true), the process proceeds to S406, where processing is performed by the calculation unit 222 of the weight update unit 220. Therefore, the above-mentioned steps S402 to S405 are processing in the neural network 210 as shown in FIG. 6, and can be called the first calculation loop processing.

In S406, the last updated x _i for which the convergence judgment in S403 was performed is output to the calculation unit 222 of the weight update unit 220 as a temporary associative output (provisional x output).

Next, in S407, the calculation unit 222 uses _x updated by the neural network 210 to update the weights J _ij and B _i based on the information ξ stored in the memory retention unit 221. The stored information ξ is expressed as M N-dimensional vectors, as in definition (6).

Here, i and j are indices of bits of input x _{i, where i = 1, ..., N, j = 1, ..., N. Similarly, l is an index of bits of input x i, where} l = 1, ..., i-1, i+1, ..., j-1, j+1, ..., N, but does not include i and j. The Ising Hamiltonian H _Ising with the weights updated in the above equations (16) and (17) can be rewritten as follows.

In equation (18), by replacing σ _i with x _i , we obtain the following equation:

From equation (19), it can be seen that the Ising Hamiltonian H _{Ising is equal to the Potts Hamiltonian expressed by equation (7) except for the constant multiple. Therefore, the weighted Ising Hamiltonian H Ising expressed} by equation (14) with weights updated by equations (16) and (17) is equal to the Potts Hamiltonian.

Subsequently, in S408, the calculation unit 222 judges whether the weights J _ij and B _i updated in S407 are the same as the J _ij and B _i before the update, that is, whether the weights have converged. The following equation is the convergence condition.

If the convergence determination result is No (false), the process returns to S406 of the neural network 210, where y _i is calculated again in S404, and x _i is updated in S404, i.e., a new loop operation is repeated. If the convergence determination result is Yes (true), the process proceeds to S406 of the weight update unit 220, where x _i last used in the process of the weight update unit 220 is output as the final output, that is, as the final result 105 associated in S409. Therefore, the above-mentioned steps S407 to S408 become the process in the weight update unit 220 as shown in FIG. 6, and can be called the second operation loop process.

The above-mentioned updating of weights J _ij and B _i in S407 and the convergence test in S408 include the first calculation loop of the convergence test of x _i in S403, and form a large second calculation loop. Referring again to Fig. 5, the associative memory device 200 alternates between the processing in the neural network 210 and the processing in the weight update unit 220, and associates the closest stored data ξ with the initial input information 104, and outputs it as the result 105.

As shown in formula (19), the weights J _ij and B _i are updated in the weight update unit 220 by a calculation process based on the Potts Hamiltonian. Also, as is clear from formulas (16) and (17), the weights J _ij and B _i are updated by an interaction calculation performed by the x _i updated by the neural network 210 and the information ξ stored in the memory holding unit 221. In this respect, it should be noted that the associative memory device 200 of the second embodiment operates in the same manner as the interaction calculation 103 in the conceptual diagram shown in FIG. 1. The difference between the associative memory device 100 of the first embodiment in FIG. 1 is that the N-bit multi-layer neuron 101 is a Hopfield type neural network in FIG. 5. The reason why the associative memory device 200 uses the neural network 210 is to replace a part of the calculation process of the association process with the processing of the Ising Hamiltonian H _Ising of the conventional technology, which has a low calculation load.

The present invention also has an aspect of a method implemented in an associative memory device that sequentially implements the steps of the flowcharts 300 and 400 in Figures 3 and 6. For example, consider an associative memory device that includes a calculation unit that executes a feedback calculation to minimize the Potts Hamiltonian H _potts and a memory holding unit that stores information ξ to be used for association in advance. The method for implementing associative memory can be implemented as a method that includes the steps of receiving an initial input x _i corresponding to the flowchart 300, successively updating the Potts Hamiltonian H _Potts , which is a cost function, determining whether the cost function has been minimized and outputting the association result when minimization has been achieved.

Compared with the flow 300 including one loop in FIG. 3, the flow 500 in FIG. 6 is composed of a first computation loop for x _i based on the Ising Hamiltonian H _Ising , and a second computation loop for weights J _ij and B _i based on the Potts Hamiltonian H _Potts . If the flow 500 in FIG. 6 is arranged in a straight line from the start (S401) to the end (S409), the steps corresponding to the first computation loop and the steps corresponding to the second computation loop will appear alternately. The total number of steps is likely to be increased compared to the flow 300, but the computation load of S403 to S405 of the processing of the Ising Hamiltonian H _Ising does not include an interaction calculation with the information ξ stored in the memory holding unit 221, and the computation load is light. Therefore, the computation load until the associated output is obtained can be lighter than that of the associative storage device 100 of the first embodiment. Even if the associative storage device 200 is implemented on a computer with a small computing power, the association can be performed in a shorter time.

Therefore, the associative storage device of the present disclosure further includes an input/output unit 221 that accepts input information x, a first calculation loop unit 210 that corresponds to a Hopfield neural network and is configured to sequentially update the input information x by a first cost function based on an Ising Hamiltonian H _Ising (S405), determine whether the first cost function has been minimized by the value of dF/dx (S403), and output a temporary associative result (S406), and a second calculation loop unit that sequentially updates each value of weights J ij and B i that represent weighting for the input information x by a second cost function based on a Potts Hamiltonian H _Potts that defines an interaction with the information ξ stored in the memory storage unit (S407), determine whether the second cost function has been minimized (S408), and if it is determined that the information has not been minimized, provides the updated weights J _ij and B i to the first calculation loop unit, and the first calculation loop unit further includes a second calculation loop unit that sequentially updates each value of weights J ij and B _i that represent weighting for the input information x by a second cost function based on an Potts Hamiltonian H Potts that defines an interaction with the information ξ stored in the memory storage unit (S408), determines whether the second cost function has been minimized (S409), and provides the updated weights J _ij and B _i to the first calculation loop unit when it is determined that the information has not been minimized. _ij and B _i to perform a new loop operation for minimizing the first cost function, and when the second cost function is minimized in the second operation loop section, the input/output unit is configured to output the latest temporary association result as an association result 105.

As described above, the weight unit 202 and calculation unit 203 of the neural network 210 of the associative memory device 200 in FIG. 5, and the calculation unit 222 of the weight update unit 210 can all be implemented by a single CPU. The memory holding unit 221 can be a general memory, so the associative memory device 200 can be implemented by a general computer. Associative memory can be realized by executing a program for the calculation processing of the algorithm shown in FIG. 6 on a computer. The neural network 210 is a Hopfield-type neural network of the prior art, and is equivalent to the minimization problem of the Ising model, so it can be implemented in a form other than a configuration that is only a computer.

For example, the neural network 210 can be implemented by a computing device known as a coherent Ising machine. The configuration of the coherent Ising machine is not limited in any way. As an example of a computing device for a coherent Ising machine, a device that injects a pump light pulse into a phase-sensitive amplifier in a ring resonator made of optical fiber and associates the phase of the pulse with the spin of the Ising model can be used (Non-Patent Document 6). The coherent Ising machine may include a dedicated FPGA, GPU, or a combination of these for calculating the coupling coefficients.

FIG. 7 is a diagram showing an example of the configuration of an associative memory device including a physical coherent Ising machine. The associative memory device 500 is a calculation device for an Ising model, and includes an optical element part 501 including a ring-shaped optical fiber and the like, and a computer part 502 that performs calculation processing and controls the optical elements. The optical element part 501 includes a ring resonator consisting of an optical fiber 1 and a phase-sensitive amplifier (PSA) 2. It also includes an external pulse generating light source 5 that generates a pulse train, an optical pulse measuring unit 3 that may be a homodyne detector that detects the phase state of the pulse, and a calculator 4 that calculates the interaction to be applied to the optical pulse. The update calculation of the coupling coefficient in the Ising model is performed by a coupling coefficient setting unit 7 and a calculation result holding unit 6, and the elements of the calculator 4, the coupling coefficient setting unit 7, and the calculation result holding unit 6 can be implemented by the CPU and memory of a computer.

Although a detailed description of the operation of the associative memory device 500 will be omitted here, an Ising model calculation device can be realized by physical hardware including the optical circuit elements described above, and the neural network 210 of embodiment 2 can be replaced with an Ising model calculation device. The CPU and memory implementing the coupling coefficient setting unit 7 and calculation result holding unit 6 can also perform the processing of the calculation unit 222 and memory holding unit 221 of the weight update unit 210 of the associative memory device 200 in FIG. 5. Therefore, the associative memory device 200 in FIG. 5 can be implemented with the same configuration as the physical coherent Ising machine in FIG. 7.

In this way, the neural network 210 of embodiment 2 can be implemented using various physical configurations that enable calculations based on the Ising model of the Hopfield neural network of conventional technology.

[Embodiment 3]
The associative storage device 100 of the first embodiment and the associative storage device 200 of the second embodiment described above both include a memory storage unit 102, 221 in which stored information ξ is stored. Specifically, the memory storage unit 102, 221 may be a storage device such as a computer memory or a hard disk drive. The stored information ξ is information that is input to the memory storage unit as a learning input and is stored therein, and is information ξ configured as a learning input. In either case, it is sufficient that the calculation unit (CPU) can access the stored information ξ so that an interaction calculation between the input information x and the stored information ξ in the algorithms shown in Figures 3 and 5 is possible.

FIG. 8 is a diagram showing the functional blocks and configuration of the associative memory device of the third embodiment. The associative memory device 600 is obtained by omitting the memory retention unit 102 from the associative memory device 100 of the first embodiment shown in FIG. 2(a). Also, an input/output unit 107 is shown as an interface for exchanging information with the outside of the device.

The memory retention unit 102 having the stored information ξ is provided in a device 610 separate from the associative memory device 600. The associative memory device 600 is connected to the memory retention unit 102 of the separate device 610 via a communication path 601 by the input/output unit 106. Therefore, the associative memory device 600 is configured to be able to access the stored information ξ prepared in the memory retention unit 102 of the separate device 610.

The communication path 601 may be wired or wireless, and there are no limitations on the type of communication interface or protocol. Therefore, the other device 610 may be another computer, or may have any function, such as a server, a storage device, or other electronic device, so long as it is different from the computer that implements the associative storage device 600. It may also be a device located in another location via a network. For example, the associative storage device 600 may be implemented on a first server computer, and association may be performed in cooperation with a second server computer in a remote location that has a memory retention unit 102.

In the configuration of embodiment 3, the learning input 106-1 may be input to another device 610, or the learning input 106-2 may be input to the associative memory device 600 and stored in the memory storage unit 102 via a communication path 601. In all cases of embodiments 1 to 3, the input information 104 to the associative memory device and the associated result output 105 are input or output via a speaker/microphone, a keypad, a display/touchpad, etc. The input information 104 and the associated result output 105 may also be input or output from yet another input/output device via another communication path not shown.

8, memory retention unit 102 in another device 610 executes arithmetic processing related to the Potts Hamiltonian H _Potts and feedback calculation for minimizing the Potts Hamiltonian H _potts under the control of the associative memory device 600. That is, it performs the function of interaction calculation unit 103 shown in Fig. 1. Based on the result of the feedback calculation, the associative memory device 600 outputs, as association result 105, information that is closest to information x input to input/output unit 107 among stored information ξ.

Although the associative memory device 600 does not hold stored information ξ, it should be noted that when the associative memory device 600 is implemented by a computer, a physical memory, for example, the memory 121 in Fig. 2(b) is naturally provided in order to execute the calculation program. The associative memory device 200 of the second embodiment can also be configured in such a way that the memory holding unit 221 is located in a separate device, exactly like in Fig. 8. There is no change in the algorithm of the feedback calculation process for minimizing the Potts Hamiltonian H _potts in Figs. 3 and 5.

As explained in detail above, by performing the calculation to minimize the Potts Hamiltonian in equation (4) instead of the Ising Hamiltonian based on the Hopfield neural network of the conventional technology, the storage capacity of the associative memory device can be significantly increased. The calculation to minimize the Potts Hamiltonian can be implemented by arithmetic processing in a general computer. By partially incorporating the configuration and processing corresponding to the Hopfield neural network of the conventional technology, it is also possible to create an associative memory device with a reduced calculation load.

The present invention can be used in devices that implement associative memory using computers and other devices.

Claims (9)

a computation unit that performs a feedback calculation to minimize _{the Potts} Hamiltonian H;
a memory holding unit for storing in advance information ξ used for association;
An associative memory device configured to output, as an associative result, stored information that is closest to input information x, based on the result of a feedback calculation that minimizes the Potts Hamiltonian H _potts . The calculation unit corresponds to an N-bit multi-layer neuron,
Calculating a value of a cost function F expressed by a Potts Hamiltonian that defines an interaction between the input information x and the information ξ stored in the memory storage unit;
Sequentially updating the value of the cost function F;
determining whether the cost function F has been minimized based on the value of dF/dx;
2. The associative memory device according to claim 1, configured to output said associative results once said minimization has been achieved. an input/output unit for receiving the input information x;
a first calculation loop unit corresponding to a Hopfield neural network, configured to sequentially update the input information x by a first cost function based on an Ising Hamiltonian H _Ising , determine whether the first cost function is minimized by a value of dF/dx, and output a temporary association result;
a second calculation loop unit configured to sequentially update each value of weights J _ij and B i representing weighting for the input information x by a second cost function based on a Potts Hamiltonian H _Potts that defines an interaction with the information ξ stored in the memory storage unit, to determine whether the second cost function has been minimized, and if it is determined that the second cost function has not been minimized, to provide the updated _{weights J ij and} B _i to the first calculation loop unit, and to operate as the calculation unit;
the first computation loop unit is further configured to perform a new loop computation using the updated weights J _ij and B _i to minimize the first cost function;
2. The associative memory device according to claim 1, wherein the input/output unit is configured to output the latest temporary associative result as the associative result when the second cost function is minimized in the second computation loop unit. The Potts Hamiltonian H _Potts is expressed as follows, where δ is the Kronecker delta function:

4. The content addressable memory device according to claim 1, wherein the content addressable memory device is represented by the formula: The first calculation loop unit
Coherent Ising Machine,
FPGA, GPU, or a combination thereof, or
4. The associative storage device according to claim 3, which is implemented by a physical Ising machine using any one of Ising model calculation devices that operates with optical elements including a ring resonator through which an optical pulse train circulates. An associative memory device according to any one of claims 1 to 3, wherein the calculation unit is configured with a central control unit (CPU), the memory storage unit is configured with a memory, and the minimization process is implemented in a computer by a software program. An input/output unit accessible to information ξ configured as a learning input;
a calculation unit that performs a feedback calculation to minimize the Potts Hamiltonian H _potts ;
an associative memory device configured to output, based on a result of the feedback calculation, information among the information ξ that is closest to the information x input to the input/output unit as an associative result; The calculation unit corresponds to an N-bit multi-layer neuron,
Calculating a value of a cost function F expressed by a Potts Hamiltonian that defines an interaction between the input information x and the information ξ;
Sequentially updating the value of the cost function F;
determining whether the cost function F has been minimized based on the value of dF/dx;
8. The associative memory device according to claim 7, configured to output the associative result once the minimization has been achieved. a first calculation loop unit corresponding to a Hopfield neural network, configured to sequentially update the input information x by a first cost function based on an Ising Hamiltonian H _Ising , determine whether the first cost function is minimized by a value of dF/dx, and output a temporary association result;
a second calculation loop unit configured to sequentially update each value of weights J _ij and B _i for the input information x by a second cost function based on a Potts Hamiltonian H _Potts that defines an interaction with the information ξ, determine whether the second cost function is minimized, and if it is determined that the second cost function is not minimized, provide the updated weights J _ij and B _i to the first calculation loop unit, and operate as the calculation unit;
the first computation loop unit is further configured to perform a new loop computation using the updated weights J _ij and B _i to minimize the first cost function;
8. The associative memory device according to claim 7, wherein the input/output unit is configured to output the latest temporary associative result as the associative result when the second cost function is minimized in the second computation loop unit.

Images