[go: up one dir, main page]

CN119623510A - An efficient online adaptation method for multi-device collaboration - Google Patents

An efficient online adaptation method for multi-device collaboration Download PDF

Info

Publication number
CN119623510A
CN119623510A CN202411574116.6A CN202411574116A CN119623510A CN 119623510 A CN119623510 A CN 119623510A CN 202411574116 A CN202411574116 A CN 202411574116A CN 119623510 A CN119623510 A CN 119623510A
Authority
CN
China
Prior art keywords
model
online
knowledge
adaptation
shared
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202411574116.6A
Other languages
Chinese (zh)
Inventor
谭明奎
陈国昊
陈德宇
牛帅程
张书海
陈耀佛
邓泽帅
杜卿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Super Robot Research Institute Huangpu
South China University of Technology SCUT
Original Assignee
Super Robot Research Institute Huangpu
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Super Robot Research Institute Huangpu, South China University of Technology SCUT filed Critical Super Robot Research Institute Huangpu
Priority to CN202411574116.6A priority Critical patent/CN119623510A/en
Publication of CN119623510A publication Critical patent/CN119623510A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/042Knowledge-based neural networks; Logical representations of neural networks
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/084Backpropagation, e.g. using gradient descent
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/096Transfer learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention discloses a multi-device collaborative efficient online adaptation method which comprises the steps of processing a first model in a device by adopting an online collaborative adaptation mode based on forward and backward propagation aiming at a device with rich computational power resources, dynamically storing the learned weight of the model into a shared knowledge base when domain offset is detected, capturing new knowledge in an adaptation process by introducing a new parameter matrix into the first model parameter, and processing a second model in the device by adopting the online collaborative adaptation mode only adopting forward propagation aiming at a device with low computational power, wherein the model directly adopts the existing shared knowledge base to carry out online adaptation. The invention realizes the accumulation, sharing and dynamic utilization of knowledge among multiple devices by dynamically updating the shared knowledge base, and processes the knowledge by adopting different online collaborative adaptation modes aiming at devices with different calculation forces, thereby greatly improving the energy efficiency and practicability of online adaptation in practical application. The invention can be widely applied to the technical field of transfer learning.

Description

Efficient online adaptation method for multi-equipment cooperation
Technical Field
The invention relates to the technical field of migration learning, in particular to a multi-device collaborative efficient online adaptation method.
Background
The existing deep learning relies on the assumption that training and test data are distributed independently and uniformly, unknown data distribution deviation of a dynamic change scene is difficult to deal with, and the performance of a depth model is easy to be greatly reduced. Therefore, how to adapt to the test data online and realize the stabilization generalization of the dynamic scene is a key premise of deployment application of the depth model in the scenes such as smart cities, smart traffic and the like.
The online adaptation technology aims at performing scene adaptation by only using non-annotated online data, and the related method mainly comprises 1) self-supervision-based online adaptation and 2) non-supervision-based online adaptation. The self-supervision-based online adaptation method mainly focuses on information reconstruction learning from scene data, namely, depth models are understood and generalized to new application scenes by reducing reconstruction loss of the scenes online, however, the method introduces additional self-supervision branches, so that training and deployment cost of the models is obviously increased, and the efficiency is low. The unsupervised online adaptation method is based on shannon entropy analysis and optimizes the decision uncertainty of the model, and helps the model form a clearer decision boundary in a new scene, so that the cross-scene decision capability of the model is improved. The part of the method further analyzes the influence of different samples on the online adaptation, constructs a reliable and non-redundant active sample screening mechanism, solves the online adaptation problem only on a reserved sample subset, and improves the performance and efficiency of the adaptation.
However, in practical application, there are often multiple cooperative devices, such as smart phones, monitoring cameras, unmanned vehicles, and the like. However, the existing online adaptation method is only limited by independent adaptation of single equipment, efficiency and performance are limited in practical application, and the bottleneck exists that 1) high redundancy exists in adaptation of different equipment to similar scenes, waste of computational resources is serious, the online adaptation efficiency and performance are poor, 2) the existing online adaptation method is prone to catastrophic forgetting, is difficult to accumulate adaptation learned knowledge, cannot support knowledge sharing and utilization across equipment, multi-equipment cooperation is difficult, 3) online adaptation learning relies on a counter-propagation mechanism with high computational cost, however, the computational power of edge equipment is limited, contradiction between the two is prominent, and the online adaptation technology is difficult to apply to the edge equipment.
Disclosure of Invention
In order to solve at least one of the technical problems in the prior art to a certain extent, the invention aims to provide a multi-device collaborative efficient online adaptation method, electronic equipment and a medium.
The first technical scheme adopted by the invention is as follows:
The efficient on-line adaptation method for multi-device cooperation is used for realizing accumulation, sharing and dynamic utilization of knowledge among multiple devices, and different devices share the same shared knowledge base, and comprises the following steps:
aiming at equipment with rich computational power resources, a first model in the equipment is processed by adopting an online collaborative adaptation mode based on forward and backward propagation, when domain offset is detected, the learning weight of the model is dynamically saved to a shared knowledge base, so that the forgetful knowledge accumulation of cross-domain learning is realized;
aiming at the edge equipment with low computational effort, the second model in the equipment is processed by adopting an online collaborative adaptation mode only using forward propagation, and the model directly uses the existing shared knowledge base to carry out online adaptation, so that the problem of high cost of knowledge learning computational effort is avoided.
It should be noted that the first model in the above refers to a model deployed in a device with rich computing power resources, and the second model refers to a model deployed in a device with low computing power, i.e., a model deployed in a device with rich computing power and a device with low computing power are distinguished by "first" and "second".
Further, the expression of the parameters of the first model is:
Wherein, theta 0 is an original parameter of the model; is M pieces of knowledge accumulated in a shared knowledge base shared by multiple devices, alpha i is a learnable weighting coefficient normalized by different knowledge, namely Δθ new is a newly introduced learnable parameter matrix.
Further, the expression of the parameters of the second model is:
wherein, theta 0 is the original parameters of the model, Is M pieces of knowledge accumulated in a shared knowledge base shared by multiple devices, and gamma i is a learnable weighting coefficient normalized by different knowledge, namelyFor supporting shared knowledge multiplexing among devices.
Further, the processing the first model in the device by adopting the online cooperative adaptation mode based on forward and backward propagation comprises the following steps:
A batch of target images to be adapted Inputting a first model in the device;
detecting whether domain offset occurs currently, if so, saving the learned knowledge to a shared knowledge base
And generating a prediction result according to the input image, updating the model by utilizing an online optimization target and a back propagation mechanism until all image data are processed, and stopping online adaptation of the model.
Further, the detection is performed to determine whether a domain shift occurs, and if so, the learned knowledge is saved to a shared knowledge baseComprising the following steps:
measuring the distribution of currently input image data by using the output characteristics of the model backbone network
Calculating the distribution of currently input image dataDifferences from the historical data distribution phi d
If the difference isWhen the acquired knowledge delta theta M+1 is larger than a preset threshold value, judging that the currently input image data has domain offset, and storing the learned knowledge delta theta M+1 and the corresponding domain distribution phi d into a shared knowledge base
Further, the currently input image data distributionThe calculation formula of (2) is as follows:
wherein B (·) is the backbone network output characteristic, x i is the input data, and N is the data amount; And (3) with Is the mean and variance of the features,From the following componentsAnd (3) withComposition is prepared.
Further, the historical data distribution phi d is a moving average of the input image data distribution, and the expression is:
Wherein λ is a balance factor;
measuring image data distribution by KL divergence Differences from the historical data distribution phi d;
the expression of the learned knowledge Δθ M+1 is:
in the formula, Is M pieces of knowledge accumulated in a shared knowledge base shared by multiple devices, alpha i is a learnable weighting coefficient normalized by different knowledge, namelyΔθ new is a newly introduced learnable parameter matrix.
Further, the expression for updating the model by using the online optimization target and the back propagation mechanism is as follows:
in the formula, Representing any online self-supervision or unsupervised optimization target, wherein θ is a parameter of a first model, x is a target data set to be adapted, and α is a normalized learnable weighting coefficient vector;
Different devices can adaptively adjust alpha, delta theta new according to data input by the devices, so that efficient adaptation to an application scene is realized.
Further, the processing the second model in the device by using the online collaborative adaptation mode only using forward propagation for the low-computation-force edge device comprises the following steps:
A batch of target images to be adapted Inputting a second model in the device;
calculating the distribution of currently input image data Distributed in shared knowledge baseSimilarity of (2);
And according to the calculated similarity, acquiring knowledge learned by a similar domain from the shared knowledge base to update the model until all image data are processed, and stopping online adaptation of the model.
Further, the calculation formula of the similarity is as follows:
wherein D (·,) is the difference of data distribution, phi i is the data distribution information phi d when Deltaθ i is learned in the shared knowledge base, and epsilon is the minimum amount for preventing numerical value overflow.
The second technical scheme adopted by the invention is as follows:
An electronic device comprising a processor and a memory having stored therein at least one instruction, at least one program, code set or instruction set loaded and executed by the processor to implement a multi-device collaborative, efficient online adaptation method as described above.
The third technical scheme adopted by the invention is as follows:
A computer readable storage medium having stored therein at least one instruction, at least one program, code set, or instruction set loaded and executed by a processor to implement a multi-device collaborative, efficient online adaptation method as described above.
The fourth technical scheme adopted by the invention is as follows:
a computer program product, or computer program, comprising computer instructions, the computer instructions are stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, cause the computer device to perform the method described above.
The invention has the beneficial effects that the invention provides a multi-device collaborative efficient online adaptation framework, the knowledge base is shared in a dynamic update way, the accumulation, sharing and dynamic utilization of knowledge among the multiple devices are realized, in addition, the energy efficiency and the practicability of online adaptation in practical application are greatly improved by adopting different online collaborative adaptation modes for processing the devices with different calculation forces.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description is made with reference to the accompanying drawings of the embodiments of the present invention or the related technical solutions in the prior art, and it should be understood that the drawings in the following description are only for convenience and clarity of describing some embodiments in the technical solutions of the present invention, and other drawings may be obtained according to these drawings without the need of inventive labor for those skilled in the art.
FIG. 1 is a schematic flow chart of a multi-device collaborative efficient online adaptation method in an embodiment of the invention;
FIG. 2 is a flow diagram of an online collaborative adaptation mode based on forward and backward propagation in an embodiment of the invention;
FIG. 3 is a flow diagram of an online collaborative adaptation mode using only forward propagation in an embodiment of the present invention.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention. The step numbers in the following embodiments are set for convenience of illustration only, and the order between the steps is not limited in any way, and the execution order of the steps in the embodiments may be adaptively adjusted according to the understanding of those skilled in the art.
In the description of the present invention, it should be understood that references to orientation descriptions such as upper, lower, front, rear, left, right, etc. are based on the orientation or positional relationship shown in the drawings, are merely for convenience of description of the present invention and to simplify the description, and do not indicate or imply that the apparatus or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus should not be construed as limiting the present invention.
In the description of the present invention, a number means one or more, a number means two or more, and greater than, less than, exceeding, etc. are understood to not include the present number, and above, below, within, etc. are understood to include the present number. The description of the first and second is for the purpose of distinguishing between technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
In the description of the present invention, unless explicitly defined otherwise, terms such as arrangement, installation, connection, etc. should be construed broadly and the specific meaning of the terms in the present invention can be reasonably determined by a person skilled in the art in combination with the specific contents of the technical scheme.
Aiming at the prior art, the invention provides a multi-device collaborative efficient online adaptation framework, constructs a parameter dynamic storage mechanism based on a shared knowledge base, realizes accumulation, sharing and dynamic utilization of knowledge among multiple devices, and further provides an online collaborative adaptation method based on forward and backward propagation and only based on forward propagation, thereby improving the energy efficiency and practicability of online adaptation in practical application.
Example 1
As shown in fig. 1, the present embodiment provides a multi-device collaborative efficient online adaptation method, which is used for implementing accumulation, sharing and dynamic utilization of knowledge among multiple devices, and different devices share the same shared knowledge base, and includes the following steps:
aiming at equipment with rich computational power resources, a first model in the equipment is processed by adopting an online collaborative adaptation mode based on forward and backward propagation, when domain offset is detected, the learning weight of the model is dynamically saved to a shared knowledge base, so that the forgetful knowledge accumulation of cross-domain learning is realized;
aiming at the edge equipment with low computational effort, the second model in the equipment is processed by adopting an online collaborative adaptation mode only using forward propagation, and the model directly uses the existing shared knowledge base to carry out online adaptation, so that the problem of high cost of knowledge learning computational effort is avoided.
In this embodiment, the online collaborative adaptation mode based on forward and backward propagation is directed to a device with rich computing power resources, and involves learning, accumulation, sharing and utilization processes of cross-domain knowledge. In the mode, a dynamic knowledge storage mechanism based on a knowledge base is constructed, the difference between the historical data distribution and the current test data distribution is measured, the learned weight is dynamically stored in the knowledge base when the domain is shifted, and the knowledge accumulation without forgetting in the cross-domain learning is realized. The same knowledge base is shared by different devices, so that the knowledge sharing and subsequent knowledge utilization of the cross-device are effectively supported, and the problem that disastrous forgetting is easy to occur in the prior art is solved. And secondly, introducing a weighting coefficient to each shared knowledge, and converting the multiplexing problem of the shared knowledge into a self-adaptive learning problem of the weighting coefficient. Further, a new parameter matrix delta theta new is introduced to capture new knowledge in the adaptation process, so that the knowledge multiplexing and new knowledge learning processes are decoupled, and the optimization conflict of the two processes is relieved. The parameters can be optimized by utilizing an online optimization target and a back propagation mechanism, so that the dynamic utilization of knowledge among multiple devices is realized, the adaptive efficiency and performance are improved, and the problems of high redundancy, serious waste of computational power resources, poor online adaptive efficiency and performance and the like in the adaptation of different devices to similar scenes in the prior art are solved.
In this embodiment, only the online collaborative adaptation mode of forward propagation is utilized to face the low-computation-power edge device, and the utilization process of shared knowledge is involved. The main idea is to directly utilize the existing shared knowledge to carry out online adaptation, thereby avoiding the problem of high cost of knowledge learning and calculation. In this mode, the embodiment approximates the weighted coefficient optimization problem of knowledge utilization to a domain matching process, and based on similarity analysis of data distribution, higher utilization weight is given to knowledge learned during similar domain adaptation, so as to realize online non-optimization dynamic adaptation. The method reduces the calculation cost of online adaptation based on the mode, solves the problem that the online adaptation technology is difficult to apply to the edge equipment aiming at the limited calculation of the edge equipment in the prior art, and effectively expands the online adaptation technology to the edge equipment with limited calculation resources such as smart phones and the like.
The embodiment of the invention innovatively expands the independent online adaptation of single equipment to the efficient online adaptation of multi-equipment cooperation, breaks through the disastrous forgetting problem of the online adaptation, realizes the accumulation, sharing and dynamic utilization of knowledge among the multi-equipment, improves the efficiency and performance of the online adaptation, and enables the online adaptation technology with high cost to be applied to the edge equipment with limited resources. The embodiment of the invention has the characteristics of lower communication cost and higher expansibility, and can have important influence on model deployment and online adaptation technology widely applied to the fields of edge computing, internet of things, distributed systems and the like, thereby improving the energy efficiency and practicability of the model online adaptation in application.
The foregoing method is explained in detail below with reference to the drawings and specific embodiments.
(1) On-line collaborative adaptation mode based on forward and backward propagation
The mode is oriented to equipment with rich computing power resources and relates to learning, accumulating, sharing and utilizing processes of cross-domain knowledge. As an example, referring to fig. 2, on a device using this mode, the algorithm comprises the following steps:
Step one, a batch of target images to be adapted is processed The model is input. Formalized expression of model parameters θ is as follows:
wherein, theta 0 is the original parameter of the model, Is M pieces of knowledge accumulated in a knowledge base shared by multiple devices, alpha i is a learnable weighting coefficient normalized by different knowledge, namelyΔθ new is a newly introduced learnable parameter matrix. The scheme decouples the knowledge utilization and updating into the learning process of alpha, delta theta new, and effectively supports the knowledge utilization and the learning of new knowledge across devices.
Step two, detecting whether domain offset occurs currently, if so, saving the learned knowledge to a shared knowledge base
Specifically, the domain detection offset includes the following flow:
1) Measuring the data distribution of the current input by using the output characteristics of the model backbone network The calculation formula is as follows:
Wherein B (·) is the backbone network output characteristic, And (3) withIs the mean and variance of the features,From the following componentsAnd (3) withComposition is prepared.
2) The difference between the current input data distribution and the historical data distribution is measured. The historical data distribution phi d is a moving average of the input data distribution formalized as:
Where λ is a balance factor, which may be set to 0.8 in implementation. The difference metric D (·,) of the data distribution is the KL divergence between the data distributions under the assumption that the data distribution obeys Gaussian distribution, and the calculation formula is as follows:
where H is the dimension of the statistic. Data distribution differences Above a certain threshold, i.eWhen the current input data is considered to have a domain offset. The domain offset detection scheme does not need to save original data, has low calculation cost and storage cost, and can effectively maintain the efficiency of online adaptation.
Specifically, the learned knowledge delta theta M+1 and the corresponding domain distribution phi d are saved to a shared knowledge baseFormalized representation of the learned knowledge is as follows:
The knowledge is shared with other devices immediately after being saved and an additional learnable weighting coefficient alpha M+1 is introduced into the model parameters theta, thereby updating the formal representation of the parameters theta to: Because Deltaθ M+1 is solidified after being stored, the problem of disastrous forgetting caused by continuous online adaptation is effectively relieved, gradual accumulation of knowledge is realized, and the stability of online adaptation is improved. Further, a learning process that decouples knowledge utilization and update to alpha, deltaθ new has the benefits that a) can be avoided Frequent updating of (1), effectively reduce the cause of cross-equipmentExpensive communication cost generated synchronously, realizing efficient multi-equipment collaborative online adaptation of communication, b) multi-equipment based on knowledge base onlyAny device can join or exit the collaboration group at any time (using the latest or non-latest shared knowledge base) to realize completely asynchronous multi-device collaboration adaptation, thereby improving the practicability and expansibility of the invention patent in the real scene.
Generating a prediction result, and updating a model by using an online optimization target and a back propagation mechanism, wherein the model is expressed in a formalized way as follows:
Wherein the method comprises the steps of Representing any online self-monitoring or unsupervised optimization objective. The patent focuses on breaking through the cooperative bottleneck of online adaptation among multiple devices, and can be flexibly combined with the existing advanced online optimization targets. Preferably, the entropy minimization loss can be used for parameter optimization in the mathematical form ofWherein f θ(xi) is the model output result. Model optimization based on the loss, and sharing the same knowledge base among devicesBut different devices can adaptively adjust alpha, delta theta new according to own input data, so that efficient adaptation to application scenes is realized.
And fourthly, repeating the first step to the third step until all the test data are input, and stopping the online adaptation of the model.
(2) Online collaborative adaptation mode with forward propagation only
The mode is oriented to low-computation-force edge equipment and relates to a shared knowledge utilization process. As an alternative embodiment, referring to fig. 3, on a device using this mode, the algorithm comprises the following steps:
Step one, a batch of target images to be adapted is processed The model is input. Formalized expression of model parameters θ is as follows:
wherein, theta 0 is the original parameter of the model, Is M pieces of knowledge accumulated in a knowledge base shared by multiple devices, and gamma i is a learnable weighting coefficient for normalization of different knowledge, namelyShared knowledge multiplexing across devices is supported.
Measuring the data distribution of the current inputDistributed in shared knowledge baseIs a similarity of (3). Wherein the data distribution of the current inputThe calculation formula is the same as above, phi i is the data distribution information phi d when delta theta i is learned in the shared knowledge base. The calculation formula of the data distribution similarity ρ i is as follows:
Where D (·, ·) is the data distribution difference metric defined above, and ε is the minimum amount to prevent value overflow.
And thirdly, based on the distribution similarity measurement, preferentially selecting knowledge acquired from the similar domains in the shared knowledge base. Specifically, formalization of γ is defined as:
γ=softmax(ρ)
by innovatively converting the online adaptation problem of the model into a dynamic utilization process of shared knowledge and approximating the weighting coefficient optimization problem to a domain matching process, the embodiment effectively avoids the problem of high computational cost caused by knowledge learning and coefficient optimization, reduces the computational cost of online adaptation, and realizes online non-optimized dynamic adaptation, thereby expanding the online adaptation technology with high cost into edge equipment with limited computational resources such as smart phones.
And fourthly, repeating the first step to the third step until all the test data are input, and stopping the online adaptation of the model.
In general, the invention aims to break through the bottleneck that the existing online adaptation technology is limited to single-device independent adaptation, the catastrophic forgetting problem is prominent, the useful knowledge of other devices is difficult to use, the online adaptation performance and efficiency are poor, and the online adaptation technology cannot be applied to edge devices. In contrast, the invention provides a multi-device collaborative efficient online adaptation method, which realizes accumulation, sharing and dynamic utilization of knowledge among multiple devices, and has two collaboration modes, namely 1) an online collaborative adaptation mode based on forward and backward propagation, wherein the mode reconstructs model parameters into a model parameterTherefore, the multiplexing and updating of the knowledge are decoupled into a learning process of alpha, delta theta new, the optimization conflict of the two processes is relieved, and the knowledge utilization of the cross-equipment and the learning of new knowledge are supported. Further, a dynamic knowledge preservation mechanism based on a knowledge base is constructed, learned new knowledge delta theta M+1 is preserved to a shared knowledge base when a domain is shifted, gradual accumulation of knowledge is realized, the problem of catastrophic forgetting under continuous online adaptation is relieved, efficient knowledge sharing across devices is supported, and the stability and efficiency of online adaptation are improved. 2) On-line collaborative adaptation mode with forward propagation only, which reconstructs model parameters intoThe method dynamically utilizes shared knowledge of different devices, converts the gamma i optimization process into domain similarity analysis for solving, and effectively avoids the problem of high computational effort and cost caused by knowledge learning and coefficient optimization, so that the high-cost online adaptation technology is expanded to edge devices with limited computing resources, such as smart phones. In summary, the method innovatively expands the independent online adaptation of the single device to the efficient online adaptation of the multi-device cooperation, breaks through the disastrous forgetting problem of the online adaptation, realizes the accumulation, sharing and dynamic utilization of knowledge among the multi-devices, improves the efficiency and performance of the online adaptation, and enables the online adaptation technology with high cost to be applied to the edge devices with limited resources. The method has the characteristics of lower communication cost and higher expansibility, and can have important influence on model deployment and online adaptation technology widely applied to the fields of edge computing, the Internet of things, a distributed system and the like, and the energy efficiency and the practicability of the model online adaptation in application are improved.
Example 2
The embodiment of the invention also provides electronic equipment, which comprises a processor and a memory, wherein at least one instruction, at least one section of program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one section of program, the code set or the instruction set is loaded and executed by the processor so as to realize the efficient online adaptation method of multi-equipment cooperation as shown in fig. 1.
It is understood that the Memory may include random access Memory (Random Access Memory, RAM) or Read-Only Memory (ROM). Optionally, the memory includes a non-transitory computer readable medium (non-transitory computer-readable storage medium). The memory may be used to store instructions, programs, code sets, or instruction sets. The memory may include a stored program area that may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, and the like, and a stored data area that may store data created according to the use of the server, and the like.
The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various portions of the overall server, perform various functions of the server, and process data by executing or executing instructions, programs, code sets, or instruction sets stored in memory, and invoking data stored in memory. Alternatively, the processor may be implemented in hardware in at least one of digital signal Processing (DIGITAL SIGNAL Processing, DSP), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU) and a modem etc. The CPU is mainly used for processing an operating system, application programs and the like, and the modem is used for processing wireless communication. It will be appreciated that the modem may not be integrated into the processor and may be implemented by a single chip.
Because the electronic device is an electronic device corresponding to the efficient online adaptation method with multi-device cooperation according to the embodiment of the present invention, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device may refer to the implementation process of the embodiment of the method, and the repetition is omitted.
Example 3
Embodiments of the present invention also provide a computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by a processor to implement a multi-device collaborative, efficient online adaptation method as illustrated in fig. 1.
Those of ordinary skill in the art will appreciate that all or part of the steps of the various methods of the above embodiments may be implemented by a program that instructs associated hardware, the program may be stored in a computer readable storage medium including Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), erasable programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), one-time programmable Read-Only Memory (OTPROM), electrically erasable programmable Read-Only Memory (EEPROM), compact disc Read-Only Memory (Compact Disc Read-Only Memory, CD-ROM) or other optical disk Memory, magnetic disk Memory, tape Memory, or any other medium that can be used for carrying or storing data.
Because the storage medium is a storage medium corresponding to the multi-device collaborative efficient online adaptation method according to the embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the method embodiment, and the repetition is omitted.
Example 4
In some possible implementations, aspects of the methods of the embodiments of the present application may also be implemented in the form of a program product comprising program code for causing a computer device to carry out the steps of a multi-device collaborative, efficient online adaptation method according to the various exemplary embodiments of the present application described hereinabove, when the program product is run on a computer device. Wherein executable computer program code or "code" for performing the various embodiments may be written in a high-level programming language, such as C, C ++, c#, SMALLTALK, JAVA, JAVASCRIPT, visual Basic, structured query language (e.g., act-SQL), perl, or in a variety of other programming languages.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of techniques known in the art, discrete logic circuits with logic gates for implementing logic functions on data signals, application specific integrated circuits with appropriate combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement the same, and are not intended to limit the scope of the present invention. All equivalent changes or modifications made in accordance with the essence of the present invention are intended to be included within the scope of the present invention.

Claims (10)

1.一种多设备协同的高效在线适应方法,其特征在于,用于实现多设备间知识的累积、共享与动态利用,不同设备共享同一共享知识库,包括以下步骤:1. An efficient online adaptation method for multi-device collaboration, characterized in that it is used to realize the accumulation, sharing and dynamic utilization of knowledge among multiple devices, and different devices share the same shared knowledge base, including the following steps: 针对算力资源丰富的设备,采用基于前向和反向传播的在线协同适应模式对设备中的第一模型进行处理:在检测到域偏移时,动态将模型所学权重保存到共享知识库;通过在第一模型参数中引入新参数矩阵,以在适应过程中捕捉新知识;For devices with abundant computing resources, an online collaborative adaptation mode based on forward and back propagation is used to process the first model in the device: when domain shift is detected, the weights learned by the model are dynamically saved to the shared knowledge base; a new parameter matrix is introduced into the first model parameters to capture new knowledge during the adaptation process; 针对低算力的设备,采用仅利用前向传播的在线协同适应模式对设备中的第二模型进行处理:模型直接利用共享知识库进行在线适应。For devices with low computing power, the second model in the device is processed in an online collaborative adaptation mode that only uses forward propagation: the model directly uses the shared knowledge base for online adaptation. 2.根据权利要求1所述的一种多设备协同的高效在线适应方法,其特征在于,所述第一模型的参数的表达式为:2. According to the method of claim 1, the efficient online adaptation method of multi-device collaboration is characterized in that the expression of the parameters of the first model is: 式中,θ0为模型原始参数;是多设备共享的共享知识库中累积的M份知识;αi是不同知识归一化的可学习加权系数,即△θnew是新引入可学习的参数矩阵。Where, θ 0 is the original parameter of the model; is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; α i is the learnable weight coefficient for normalizing different knowledge, i.e. △θ new is a newly introduced learnable parameter matrix. 3.根据权利要求1所述的一种多设备协同的高效在线适应方法,其特征在于,所述第二模型的参数的表达式为:3. According to the efficient online adaptation method for multi-device collaboration of claim 1, it is characterized in that the expression of the parameters of the second model is: 式中,θ0为模型原始参数,是多设备共享的共享知识库中累积的M份知识;γi是不同知识归一化的可学习加权系数,即用于支撑跨设备之间的共享知识复用。In the formula, θ 0 is the original parameter of the model, is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; γ i is the learnable weight coefficient for normalizing different knowledge, i.e. Used to support shared knowledge reuse across devices. 4.根据权利要求1所述的一种多设备协同的高效在线适应方法,其特征在于,所述采用基于前向和反向传播的在线协同适应模式对设备中的第一模型进行处理,包括:4. The efficient online adaptation method for multi-device collaboration according to claim 1, wherein the first model in the device is processed using an online collaborative adaptation mode based on forward and backward propagation, comprising: 将一批待适应的目标图像输入设备中的第一模型;A batch of target images to be adapted a first model in the input device; 检测当前是否发生域偏移,若发生域偏移则将所学知识保存至共享知识库 Detect whether domain shift has occurred. If so, save the learned knowledge to the shared knowledge base. 根据输入的图像生成预测结果,利用在线优化目标和反向传播机制进行模型更新,直到处理完所有的图像数据,模型停止在线适应。Generate prediction results based on the input image, and use online optimization objectives and back-propagation mechanism to update the model until all image data are processed and the model stops online adaptation. 5.根据权利要求4所述的一种多设备协同的高效在线适应方法,其特征在于,所述检测当前是否发生域偏移,若发生域偏移则将所学知识保存至共享知识库包括:5. According to claim 4, a highly efficient online adaptation method for multi-device collaboration is characterized in that the detection of whether domain shift currently occurs, if domain shift occurs, the learned knowledge is saved to a shared knowledge base include: 利用模型骨干网络的输出特征,衡量当前输入的图像数据分布 Use the output features of the model backbone network to measure the distribution of the current input image data 计算当前输入的图像数据分布与历史数据分布φd的差异 Calculate the current input image data distribution Difference from historical data distribution φd 若差异大于预设阈值,判定当前输入的图像数据存在域偏移,将所学知识△θM+1与对应域分布φd保存至共享知识库 If the difference If the value is greater than the preset threshold, it is determined that the current input image data has domain shift, and the learned knowledge △θ M+1 and the corresponding domain distribution φ d are saved to the shared knowledge base. 6.根据权利要求5所述的一种多设备协同的高效在线适应方法,其特征在于,所述当前输入的图像数据分布的计算公式如下:6. The efficient online adaptation method for multi-device collaboration according to claim 5, characterized in that the current input image data distribution The calculation formula is as follows: 式中,B(·)为骨干网络输出特征,xi为输入数据,N为数据量;为特征的均值与方差,组成。In the formula, B(·) is the output feature of the backbone network, xi is the input data, and N is the amount of data; and is the mean and variance of the feature, Depend on and composition. 7.根据权利要求5所述的一种多设备协同的高效在线适应方法,其特征在于,所述历史数据分布φd为输入的图像数据分布的移动平均,表达式为:7. The efficient online adaptation method for multi-device collaboration according to claim 5, characterized in that the historical data distribution φd is a moving average of the input image data distribution, expressed as: 式中,λ是平衡因子;Where λ is the balance factor; 通过KL散度来度量图像数据分布与历史数据分布φd之间的差异;Measuring image data distribution through KL divergence The difference between the distribution of historical data φd ; 所学知识△θM+1的表达式为:The expression of the knowledge learned △θ M+1 is: 式中,是多设备共享的共享知识库中累积的M份知识;αi是不同知识归一化的可学习加权系数,即△θnew是新引入可学习的参数矩阵。In the formula, is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; α i is the learnable weight coefficient for normalizing different knowledge, i.e. △θ new is a newly introduced learnable parameter matrix. 8.根据权利要求4所述的一种多设备协同的高效在线适应方法,其特征在于,所述利用在线优化目标和反向传播机制进行模型更新的表达式为:8. According to the efficient online adaptation method for multi-device collaboration of claim 4, it is characterized in that the expression for updating the model using the online optimization target and the back propagation mechanism is: 式中,表示任意在线自监督或无监督的优化目标;θ为第一模型的参数;x为待适应目标数据集;α为归一化的可学习加权系数向量;In the formula, represents any online self-supervised or unsupervised optimization objective; θ is the parameter of the first model; x is the target dataset to be adapted; α is the normalized learnable weight coefficient vector; 其中,不同的设备能够根据自身输入的数据自适应调整α,△θnew,以实现对应用场景的高效适应。Among them, different devices can adaptively adjust α,△θ new according to their own input data to achieve efficient adaptation to application scenarios. 9.根据权利要求1所述的一种多设备协同的高效在线适应方法,其特征在于,所述针对低算力的边缘设备,采用仅利用前向传播的在线协同适应模式对设备中的第二模型进行处理,包括:9. The efficient online adaptation method for multi-device collaboration according to claim 1, characterized in that, for edge devices with low computing power, an online collaborative adaptation mode using only forward propagation is used to process the second model in the device, comprising: 将一批待适应的目标图像输入设备中的第二模型;A batch of target images to be adapted a second model in the input device; 计算当前输入的图像数据分布与共享知识库中分布的相似度;Calculate the current input image data distribution Distributed in a shared knowledge base similarity; 根据计算的相似度,从共享知识库中获取相似域习得的知识进行模型更新,直到处理完所有的图像数据,模型停止在线适应。According to the calculated similarity, the knowledge learned from similar domains is obtained from the shared knowledge base to update the model until all image data are processed and the model stops online adaptation. 10.根据权利要求9所述的一种多设备协同的高效在线适应方法,其特征在于,所述相似度的计算公式如下:10. The efficient online adaptation method for multi-device collaboration according to claim 9, wherein the similarity calculation formula is as follows: 式中,D(·,·)为数据分布的差异;φi为共享知识库中△θi学习时的数据分布信息φd;∈是防止数值溢出的极小量。Where D(·,·) is the difference in data distribution; φ i is the data distribution information φ d when learning △θ i in the shared knowledge base; ∈ is a very small amount to prevent numerical overflow.
CN202411574116.6A 2024-11-06 2024-11-06 An efficient online adaptation method for multi-device collaboration Pending CN119623510A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202411574116.6A CN119623510A (en) 2024-11-06 2024-11-06 An efficient online adaptation method for multi-device collaboration

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202411574116.6A CN119623510A (en) 2024-11-06 2024-11-06 An efficient online adaptation method for multi-device collaboration

Publications (1)

Publication Number Publication Date
CN119623510A true CN119623510A (en) 2025-03-14

Family

ID=94893583

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202411574116.6A Pending CN119623510A (en) 2024-11-06 2024-11-06 An efficient online adaptation method for multi-device collaboration

Country Status (1)

Country Link
CN (1) CN119623510A (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023088916A1 (en) * 2021-11-16 2023-05-25 Five AI Limited Online domain adaptation
CN117540829A (en) * 2023-10-18 2024-02-09 广西壮族自治区通信产业服务有限公司技术服务分公司 Knowledge sharing large language model collaborative optimization method and system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023088916A1 (en) * 2021-11-16 2023-05-25 Five AI Limited Online domain adaptation
US20250037441A1 (en) * 2021-11-16 2025-01-30 Five AI Limited Online domain adaptation
CN117540829A (en) * 2023-10-18 2024-02-09 广西壮族自治区通信产业服务有限公司技术服务分公司 Knowledge sharing large language model collaborative optimization method and system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
张全海, 施鹏飞: "基于本体的多智能体知识共享和协作", 上海交通大学学报, no. 09, 30 September 2003 (2003-09-30) *
牛帅程: "面向动态视觉场景的连续域适应理论与方法研究", 中国博士学位论文全文数据库信息科技辑, 15 February 2025 (2025-02-15) *

Similar Documents

Publication Publication Date Title
US11004012B2 (en) Assessment of machine learning performance with limited test data
CN109344615B (en) A method and device for detecting malicious commands
CN113609948B (en) Method, device and equipment for detecting video time sequence action
US20230117088A1 (en) Method and device for improving performance of data processing model, storage medium and electronic device
US20220129764A1 (en) Anomaly detection apparatus, anomaly detection method, and computer-readable medium
CN113328908A (en) Abnormal data detection method and device, computer equipment and storage medium
CN114254673A (en) Denoising countermeasure self-encoder-based spectrum anomaly detection method
CN114219306A (en) Method, apparatus, medium, and program product for creating a weld quality detection model
JP2019036112A (en) Abnormal sound detection device, abnormality detection device, program
CN112364304A (en) Method and device for detecting solar erosion attack of block chain
US20250077953A1 (en) Managing evolving artificial intelligence models
US12463993B2 (en) System and method for memory-less anomaly detection using anomaly thresholds based on probabilities
US12423419B2 (en) System and method for determining types of anomalies while performing memory-less anomaly detection
CN113055384A (en) SSDDQN network abnormal flow detection method
CN116797973A (en) Data mining method and system applied to sanitation intelligent management platform
CN119623510A (en) An efficient online adaptation method for multi-device collaboration
CN114882414A (en) Abnormal video detection method, abnormal video detection device, electronic equipment, abnormal video detection medium and program product
CN120509775A (en) Training method of detection model and detection method of bolt tightening quality
CN117478434B (en) Edge node network traffic data processing method, device, equipment and media
WO2022264387A1 (en) Training device, training method, and training program
CN113515556A (en) Data processing method, client and electronic device
CN116433050B (en) Abnormal alarm method and system applied to agricultural big data management system
CN117688955A (en) Method, apparatus, electronic device, and computer-readable medium for humidity temperature adjustment
US20240119148A1 (en) System and method for memory-less anomaly detection using anomaly levels
CN116319386A (en) Availability and fault prediction method and device, electronic equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination