TWI879972B - Federated machine-learning system, client device and method for federated machine-learning - Google Patents

Abstract

A federated machine-learning system includes a global server and client devices. The server receives updates of weight factor dictionaries and factor strengths vectors from the clients, and generates a globally updated weight factor dictionary and a globally updated factor strengths vector. A client device selects a group of parameters from a global group of parameters, and trains a model using a dataset of the client device and the group of selected parameters. The client device sends to the server a client-updated weight factor dictionary and a client-updated factor strengths vector. The client device receives the globally updated weight factor dictionary and the globally updated factor strengths vector, and retrains the model using the dataset of the client device, the group of parameters selected by the client device, and the globally updated weight factor dictionary and the globally updated factor strengths vector.

Description

Joint machine learning system, client device, and method for joint machine learning

The subject matter disclosed herein relates to joint machine learning. More particularly, the subject matter disclosed herein relates to a system and method for joint machine learning. [CROSS-REFERENCE TO RELATED APPLICATIONS]

This application claims priority to U.S. Provisional Application No. 63/033,747 filed on June 2, 2020, the disclosure of which is incorporated herein by reference in its entirety.

The development of the Internet of Things (IoT), the growth of smartphones, and the digitization of records have contributed to modern systems that generate increasingly large amounts of data. The data generated can provide a wide range of information about individuals, which on one hand can lead to highly personalized smart applications, but on the other hand can be sensitive and should remain private. Examples of such private data include, but are not limited to, facial images, typing histories, medical records, and survey responses.

An example embodiment provides a client device in a joint machine learning system, wherein the client device may include at least one computing device, a communication interface, and a processor. The processor may be coupled to the at least one computing device and to the communication interface. The processor may select a group of parameters for a client device from a global group of parameters; train a model using a data set of the client device and the group of parameters selected by the client device, wherein the data set may be formed by the output of at least one computing device; update a weight factor dictionary and a factor strength vector after training the model; send the client-updated weight factor dictionary and the client-updated factor strength vector to a global server via a communication interface; receive a globally updated weight factor dictionary and a globally updated factor strength vector from the global server via a communication interface; and retrain the model using the data set of the client device, the group of parameters selected by the client device, and the globally updated weight factor dictionary and the globally updated factor strength vector. In one embodiment, the client device may be part of a group of N client devices, where N is an integer. In another embodiment, the processor selects the group of parameters from the global group of parameters by using three variational parameters that may include a seed value and minimizing a difference between supervised learning of the data set and regularization of the selected group of parameters and the global group of parameters. The processor may select the group of parameters from the global group of parameters by receiving the global group of parameters that has been sent from a global server to a first subset of client devices in the N client devices, the client device being part of the first subset of client devices. The client device may receive a globally updated dictionary of weight factors and a globally updated factor strength vector by receiving the globally updated dictionary of weight factors and the globally updated factor strength vector sent by the global server to a second subset of N client devices, wherein the client device may be part of the second subset of client devices. In yet another embodiment, the processor may send a request for a current version of the global group of parameters to the global server via the communication interface, the model may be updated using the current version of the global group of parameters, and the model updated using the current version of the global group of parameters may be evaluated to form an inference based on a data set of the client device.

An example embodiment provides a joint machine learning system that may include a global server and N client devices. The global server may receive updates of weight factor dictionaries and factor strength vectors from the N client devices, where N is an integer, and may generate a globally updated weight factor dictionary and a globally updated factor strength vector. At least one client device may include at least one computing device, a communication interface, and a processor. The processor may be coupled to the at least one computing device and to the communication interface. The processor can select a group of parameters from a global group of parameters; train a model using a data set of a client device and the group of parameters selected by the client device; update a weight factor dictionary and a factor strength vector after training the model; send the client-updated weight factor dictionary and the client-updated factor strength vector via a communication interface; receive a globally updated weight factor dictionary and a globally updated factor strength vector from a global server via a communication interface; and retrain the model using a data set of a client device, a group of parameters selected by the client device, and a globally updated weight factor dictionary and a globally updated factor strength vector. In one embodiment, the processor may select the group of parameters from the global group of parameters by using three variational parameters that may include a seed value and minimizing the difference between supervised learning of the data set and regularization of the selected group of parameters and the global group of parameters. In another embodiment, the processor may select the group of parameters from the global group of parameters by receiving the global group of parameters that has been sent from a global server to a first subset of client devices in N client devices, where the client devices may be part of the first subset of client devices. In another embodiment, the client device may receive a globally updated dictionary of weight factors and a globally updated factor strength vector by receiving the globally updated dictionary of weight factors and the globally updated factor strength vector sent by the global server to a second subset of N client devices, wherein the client device may be part of the second subset of client devices. In one embodiment, the processor may send a request for a current version of the global group of parameters to the global server via the communication interface, the model may be updated using the current version of the global group of parameters, and the model updated using the current version of the global group of parameters may be evaluated to form an inference based on a data set of the client device.

An example embodiment provides a method for joint machine learning, which may include: selecting a group of parameters from a global group of parameters at a client device, the global group of parameters including a weight factor dictionary and a factor strength vector; training a model at the client device using a dataset of the client device and the group of parameters selected by the client device; updating the weight factor dictionary and the factor strength vector after training the model; and updating the weight factor dictionary and the factor strength vector at the client device. An updated dictionary of weight factors and a client-side updated factor strength vector are sent from a client device to a global server; a globally updated dictionary of weight factors and a globally updated factor strength vector are received at the client device from the global server; and a model is retrained at the client device using a dataset of the client device, a group of parameters selected by the client device, and the globally updated dictionary of weight factors and the globally updated factor strength vector. In one embodiment, the client device may be part of a group of N client devices, where N is an integer. In another embodiment, selecting a group of parameters from a global group of parameters may include: selecting the group of parameters using three variational parameters including a seed value, and minimizing a difference between supervised learning of the dataset and regularization of the selected group of parameters and the global group of parameters. In yet another embodiment, selecting a group of parameters from a global group of parameters may include receiving, at a client device, a global group of parameters that has been sent from a global server to a first subset of client devices among N client devices, wherein the client device may be part of the first subset of client devices. In yet another embodiment, receiving, at a client device, a globally updated dictionary of weight factors and a globally updated factor strength vector from a global server may include receiving, at a client device, a globally updated dictionary of weight factors and a globally updated factor strength vector sent by the global server to a second subset of N client devices, wherein the client device may be part of the second subset of client devices. In one embodiment, the method may further include: requesting, by a client device, a current version of a global group of parameters from a global server; receiving the current version of the global group of parameters; updating a model using the current version of the global group of parameters; and evaluating the model updated using the current version of the global group of parameters to form inferences based on a data set of the client device.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, one of ordinary skill in the art will appreciate that the disclosed aspects can be practiced without these specific details. In other cases, well-known methods, procedures, components, and circuits are not described in detail in order not to obscure the subject matter disclosed herein.

References throughout this specification to "one embodiment" or "an embodiment" mean that the particular features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment disclosed herein. Therefore, the appearance of the phrase "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases of similar meaning) in various places throughout this specification may not necessarily all refer to the same embodiment. In addition, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as being better or advantageous than other embodiments. In addition, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner. In addition, depending on the context of the discussion herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional", "predetermined", "pixel-specific", etc.) may be used interchangeably with corresponding non-hyphenated versions (e.g., "two-dimensional", "predetermined", "pixel-specific", etc.), and capitalized terms (e.g., "counter clock", "row select", "PIXOUT", etc.) may be used interchangeably with corresponding non-capitalized versions (e.g., "counter clock", "row select", "pixout", etc.). Such occasional interchangeable usage should not be considered inconsistent with each other.

In addition, depending on the context of the discussion herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It should be further noted that the various figures (including assembly diagrams) shown and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, various waveforms and timing diagrams are shown for illustrative purposes only. For example, the size of some elements may be exaggerated relative to other elements for clarity. In addition, when deemed appropriate, figure numbers have been repeated in the various figures to indicate corresponding and/or similar elements.

The terms used herein are used only for the purpose of describing some example embodiments and are not intended to be limitations on the claimed subject matter. As used herein, unless the context clearly indicates otherwise, the singular forms "a/an" and "the" are intended to also include the plural forms. It should be further understood that the term "including" when used in this specification specifies the presence of the described features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof. As used herein, the terms "first", "second", etc. are used as labels for the nouns that precede the terms, and do not imply any type of ordering (e.g., space, time, logic, etc.) unless so clearly defined. In addition, the same figure reference numerals may be used to refer to parts, components, blocks, circuits, units, or modules having the same or similar functionality across two or more figures. However, this usage is for simplicity of illustration and ease of discussion only; it does not imply that the construction or architectural details of such components or units are the same across all embodiments, or that such commonly referenced parts/modules are the only way to implement some of the example embodiments disclosed herein.

It should be understood that when an element or layer is referred to as being "on," "connected to," or "coupled to" another element or layer, the element or layer may be directly on, directly connected to, or coupled to the other element or layer, or there may be intervening elements or layers. In contrast, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to," there are no intervening elements or layers. Like numbers refer to like elements throughout. As used herein, the term "and/or" includes any and all combinations of one or more of the associated enumerated items.

As used herein, the terms "first," "second," and the like are used as labels for the nouns that precede the terms, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless expressly defined as such. In addition, the same figure numerals may be used to refer to parts, components, blocks, circuits, units, or modules having the same or similar functionality across two or more figures. However, this usage is for simplicity of illustration and ease of discussion only; it does not imply that the construction or architectural details of such components or units are the same across all embodiments, or that such commonly referenced parts/modules are the only way to implement some of the example embodiments disclosed herein.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the subject matter belongs. It should be further understood that the terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined in this article.

As used herein, the term "module" refers to any combination of software, firmware, and/or hardware configured to provide the functionality described herein in conjunction with the module. For example, software may be embodied as a software suite, program code, and/or instruction set or instructions, and the term "hardware" as used in any implementation described herein may include, for example, an assembly, a solid-wire circuit, a programmable circuit, a state machine circuit, and/or firmware that stores instructions executed by the programmable circuit, alone or in any combination. Modules may collectively or individually be embodied as circuits that form part of a larger system, such as, but not limited to, an integrated circuit (IC), a system on-chip (SoC), an assembly, and the like.

Federated learning has been proposed to provide machine learning that may make it possible to keep personalized data private by keeping user data locally on each client device and only sharing model updates with a global server. Therefore, federated learning represents a possible strategy for training machine learning models on heterogeneously distributed networks in a privacy-preserving manner.

Although the federated machine learning paradigm may provide a way to keep private data private, there are still many challenges for federated machine learning systems. For example, currently used federated machine learning systems include a single global model used by each client. However, for certain subpopulations, the single model approach may not work well because there may be a skewed data distribution across different clients.

To illustrate this, consider client devices, and the i-th client device contains a random number different from other client devices. Changing data distribution In the traditional joint machine learning setting, a single global model is learned that can be deployed on all on client devices. Traditional methods assume that there are layers A multilayer perceptron (MLP) architecture and a set of weights shared across all client devices To meet the global goal, a set of weights can be learned To minimize the average loss across all clients. For example, a traditional joint machine learning system minimizes the following objective: (1) Among them is the index of the client device, N is the number of clients, is the local target function, and For each device weight.

However, given statistical heterogeneity, a one-size-fits-all-type approach can cause the global model to perform poorly on some clients. In general, performance can be translated into how closely a particular client's local distribution matches the distribution of the entire population. Therefore, for this example, the model of a traditional joint machine learning system can be seen as being unfair to clients with data characteristics that are less common among clients.

The subject matter disclosed herein can improve model consistency for federated learning by using Bayesian nonparametric weight factorization, which can provide a personalized federated learning solution that can achieve higher local model performance on many clients.

Compared to traditional federated learning systems, the federated machine learning system disclosed in this article includes at least three improved features. The first improved feature is that the network on which federated learning occurs is divided into two parts. The first part provides server aggregation, and the second part is used for client personalization. The second improved feature is about the reduced amount of data transmitted between the global server and the client device. That is, the data transmission between the global server and the client device is more efficient because core factorization is used in the client device and only a subset of the parameters used for training is transmitted. The third improved feature is about the additional security layer provided by the core factorization and only a subset of the parameters used for training is transmitted.

The joint machine learning system disclosed herein provides a method for effectively using data from a global model to factorize the A federated learning system for training neural networks in local models. Each client model can be personalized based on the local distribution at the client, and all client models share commonly learned components.

FIG. 1 depicts a functional block diagram of an example implementation of a federated learning system 100 according to the subject matter disclosed herein. The federated learning system 100 may include a global server 101 and N clients (i.e., local devices) 102 ₁ to 102 _N . The global server 101 may be located in a cloud at a single location or at distributed locations. The term “global server” as used herein means any server device configured to communicate (wired and/or wirelessly) with two or more client devices via a wide area network (e.g., the Internet), and may be any server device configured to communicate directly with two or more client devices in a federated machine learning system. Clients ₁₀₂₁ to _102N are communicatively coupled to the global server 101 via a communication link 103. The communication link 103 may be a wired communication link and/or a wireless communication link.

2A and 2B depict functional block diagrams of example implementations of a global server 101 and a client 102, respectively, according to the subject matter disclosed herein. The global server 101 may include a processing device 201, such as a central processing unit (CPU), and a communication interface 203, communicatively coupled to a memory 202. The memory 202 may include non-volatile memory and/or volatile memory. The communication interface 203 may be configured to communicate with a network structure, such as but not limited to the Internet. The communication interface 203 may be a wired and/or wireless communication interface. Other configurations of the global server 101 are possible. The global server 101 may be configured to provide joint machine learning functionality as described herein. In one embodiment, the joint machine learning functionality provided by the global server 101 may be provided by one or more modules, which may be any combination of software, firmware, and/or hardware configured to provide the functionality described herein.

The client 102 may include a processing device 251, such as a CPU, communicatively coupled to a memory 252; a communication interface 253; and one or more computing devices 254. The one or more computing devices 254 may include the ability to sense or collect information related to, but not limited to, motion, one or more images, biometrics and/or medical conditions of humans and/or non-human animals and/or plants, sound, voice, location, metadata, application usage (i.e., browsing history), and/or survey responses. In one embodiment, at least one computing device 254 is a sensing device. Other configurations of the client device 102 are possible. The client 102 may be configured to provide joint machine learning functionality as described herein. In one embodiment, the joint machine learning functionality provided by the client 102 may be provided by one or more modules, which may be any combination of software, firmware, and/or hardware configured to provide the functionality described herein.

The client 102 _i may have a local model, which is specific to the data distribution Training The layers have weight matrices Each group of weights Can be as specific to each client as possible Data distribution However, each client typically has limited data, which may not be sufficient to train the full model without overfitting. Therefore, the total number of parameters that must be learned on all clients scales with the number of clients. Individual models may not exploit similarities between client data distributions or shared learning tasks. To use data more efficiently, the joint machine learning system 100 provides a single global model that That is, each client model can be personalized to the local data distribution by sharing all models of commonly learned components. To do this, the client The weight matrix Factoring this out is: (2) (3) Among them and is a rank-1 dictionary of weight factors that can be shared among clients, and For each client The diagonal personalized matrix.

The factorization can be equivalently expressed as: (4) Among them for No. OK, for The kth column of In this way, the rows as weight factors and columns The corresponding interpretation of the dictionary becomes more obvious. and together form a global dictionary of weight factors, and Can be considered as a client The factor of . The difference allows the model to be customized for each client's data distribution, while sharing the base factor and Enable data learning from all clients.

Each client factor can be formed as an element-wise product: (5) Among them indicates the strength of each factor, and is a binary vector indicating the activity factor. As described below, Typically sparse, so typically each client uses only a small subset of the available weight factors. As used herein, missing Superscript (for example, ) refers to all levels across which factorization is performed. The entire set of , and factor strength Learn some estimation.

In the context of joint machine learning with statistical heterogeneity, there are many desirable properties that client-side factors should have in common. As mentioned earlier, Usually sparse and therefore is also sparse, which promotes the consolidation of relevant knowledge while minimizing interference. That is, client A should be able to update the global factor during training without destroying the ability of client B to perform client B's task. On the other hand, the factor should be reused among clients. Although the data can be distributed non-independently and non-identically across clients, there is usually some similarity or overlap in the data. The common factor is learned across all client data distributions, which avoids Independent model scenarios. Additionally, in the distributed settings considered for FML, the total number of nodes is rarely predefined. Therefore, the system is able to scale gracefully to accommodate new clients without reinitializing the entire model. This feature includes both increasing server-side capacity (when necessary) and initializing new clients.

To promote the diagonal personalization matrix The sparsity on , a process similar to the Indian Buffet Process (IBP) can be used to regularize the diagonal vectors. The posterior distribution of the diagonal vectors can be forced to be as close to the prior diagonal vectors as possible by variational inference. Using Bayesian nonparametric methods, the data can be used to indicate client-side factor assignment, factor reuse, and server-side model expansion. The stick-breaking construction can be used with IBP as the prior distribution for factor selection as follows: (6) (7) (8) Among them may be a hyperparameter that controls the expected number of active factors and the rate at which new factors are incorporated, and Index the factors.

For random variables and The posterior distribution is learned. Exact inference of the posterior can be intractable, so variational inference with a mean field approximation can be used to determine the activity factor for each client device using the following variational distribution, which is propagated via backprop using Bayesian learning of the variational parameters (i.e., seed values) for each queried client: : (9) (10) (11)

In order to have a differentiable parameterization, the Kumaraswamy distribution can be used as The goal of each client is to maximize the variational lower bound: (12) where | |For client The first term provides label supervision, and the second term regularizes the posterior distribution to be not too noisy from the IBP prior distribution.

The mean field approximation can be used to allow the second term to expand to: (13)

Before training begins, the server 101 may initialize the global weight factor { , } and factor strength Once initialized, each training round begins with { , , } is sent to a selected subset of the total number of clients 102. Each selected (sampled) client then Use of their own private data distribution within calendar years To train the model, not only update the weight factor dictionary { , } and factor strength , and also updates the client's variational parameters , which controls which factors are used by the client. Data Distribution May include information related to biometric data, medical data, imaging data, location data, application usage data, thermal data, atmospheric data, and/or audio data.

Once local training is complete, each client will { , , } is sent back to the server instead of the variational parameters , the variational parameter is related to the data distribution After server 101 receives updates from all sampled clients, server 101 may aggregate { , , }, which in one embodiment may be a simple averaging step. The process is then repeated by the server selecting a new subset of clients to sample, sending a new updated set of global parameters to the new subset, and so on, until the desired number of communication rounds has occurred. This process is outlined in the pseudo code of Algorithm 1. Algorithm 1 1: Input: Communication rounds , local training epochs , learning rate 2: Server initializes global weight factor dictionaries and , factor strengths 3: Clients each initialize variational parameters 4: for = 1, …, do 5: Server randomly selects subsets of clients and sends { , , } 6: for client in parallel do 7: , , , , , ←CLIENTUPDATE( , , , , , ) 8: Send { , , } to the server. 9: end for 10: Server aggregates and averages updates { , , } 11: end for 12: function CLIENTUPDATE( , , , , , ) 13: for = 1; …, do 14: for minibatch do 15: Update { , , , , , } by minimizing Eq. (12) 16: end for 17: end for 18: Return { , , , , , } 19: end function

When the client 102 enters the evaluation mode, the client can request global parameters { , , If the client has been previously queried for joint training, the local model contains the aggregated global parameters and the local variational parameters of the client. Otherwise, the client uses only the aggregated { , , }. Note that if the client has previously sampled, a recent cached copy of global parameters at the client may be an option when a network connection is unavailable or too expensive. Typically, the client is able to request the latest parameters.

Data security is one of the central aspects of federated machine learning. More simply, if all data is first aggregated at a central server, a more standard method of training a model can be utilized. The very real possibility of sensitive client data being intercepted during transmission or the data repository of server 101 being compromised by an attacker are both major issues and motivating data to remain on the local device 102 for use in federated machine learning. On the other hand, simply keeping the data on the client side may not be sufficient for security purposes. Just as data can be corrupted in transmission or at a central database in a non-federated setting, federated training updates are similarly vulnerable. For example, in one example federated machine learning approach, the update includes all parameters of the model. This effectively means that the trade-off of immediately generating data can be used to reproduce white-box access to the model, which can open the model to a wide range of malicious activities, including exposing the data itself that the joint machine learning is designed to protect.

For the joint machine learning system disclosed in this article, the client sets the weight factor { , The entire dictionary and factor strength of Rather than The client data is transmitted to the server 101. Therefore, the information related to the specific factors used by the client is kept on the local side. or factor selection Therefore, even if the message is intercepted, the enemy may not be able to fully reconstruct the model, thus hindering the enemy's ability to perform attacks to recover the data.

FIG. 3 is a flow chart of an example implementation of a method 300 for joint machine learning at a client device in accordance with the subject matter disclosed herein. The method begins at 301. A selected subset of a total number of global parameters (i.e., a global weight factor dictionary and factor strengths) may be initialized by a global server 101 and sent to a client 102 before training begins. At 302, a group of parameters for a client device is selected by the client device from the global group of parameters. In one embodiment, the client uses variational parameters to form the parameter selection for the client. At 303, the client device trains a model using the client device's dataset and the group of parameters selected by the client device. At 304, after training, the client device sends a client-updated dictionary of weight factors and a client-updated factor strength vector to the global server 101, rather than sending a dataset of variational parameters or client devices used by the client to form a selection of parameters for the client. The global server 101 may aggregate the client-updated dictionary components and factor strength vectors using an averaging step. The global server 101 may select a new subset of clients for sampling and send a new updated set of global parameters to the new subset of clients. For an example embodiment of method 300, the client is selected as part of the new subset of clients. At 305, the client device receives a globally updated dictionary of weight factors and a globally updated factor strength vector from the global server 101. At 306, the client device retrains based on the client data set, the group of parameters selected by the client device, the globally updated weight factor dictionary, and the globally updated factor strength vector. The method may continue until a desired number of training epochs have occurred. The method ends at 307.

FIG. 4 depicts an electronic device 400 including functionality for joint machine learning according to the subject matter disclosed herein. In one embodiment, the electronic device 400 may be a global server operable to provide joint machine learning as disclosed herein. In another embodiment, the electronic device 400 may be a client device operable to provide joint machine learning as disclosed herein. Whether a global server or a client device, the electronic device 400 may also be embodied as, but not limited to, a computing device, a personal digital assistant (PDA), a laptop, a mobile computer, a web tablet, a wireless phone, a cellular phone, a smart phone, a digital music player, or a wired or wireless electronic device. The electronic device 400 may include a controller 410, an input/output device 420 (such as but not limited to a keypad, a keyboard, a display, a touch screen display, a camera and/or an image sensor), a memory 430, an interface 440, a GPU 450, and an imaging processing unit 460 coupled to each other via a bus 470. The controller 410 may include, for example, at least one microprocessor, at least one digital signal processor, at least one microcontroller, or the like. The memory 430 may be configured to store command codes or user data to be used by the controller 410.

Interface 440 may be configured to include a wireless interface configured to transmit data to or receive data from a wireless communication network using RF signals. Wireless interface 440 may include, for example, an antenna. The electronic device 400 may also be used in communication interface protocols of communication systems, such as but not limited to: Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), North American Digital Communications (NADC), Extended Time Division Multiple Access (E-TDMA), Wideband CDMA (WCDMA), CDMA2000, Wi-Fi, Municipal Wi-Fi (Muni Wi-Fi), Bluetooth, Digital Enhanced Cordless Telecommunications (DECT), Wireless Universal Serial Bus (Wireless USB), Fast low-latency access with seamless handoff Orthogonal Frequency Division Multiplexing (Flash-OFDM), IEEE 802.11ac, IEEE 802.11ac, IEEE 802.11ac, IEEE 802.11ac, IEEE 802.11ac, IEEE 802.11ac, IEEE 802.11ac, IEEE 802.11ac 802.20, General Packet Radio Service (GPRS), iBurst, Wireless Broadband (WiBro), WiMAX, Advanced WiMAX, Universal Mobile Telecommunication Service-Time Division Duplex (UMTS-TDD), High Speed Packet Access (HSPA), Evolution Data Optimized (EVDO), Long Term Evolution-Advanced (LTE-Advanced), Multichannel Multipoint Distribution Service (MMDS), fifth generation wireless communication (5G), etc.

Embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuits or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by the data processing device. A computer storage medium may be or be contained in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. In addition, when a computer storage medium is not a propagated signal, the computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be or be contained in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). In addition, the operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

Although this specification may contain many specific implementation details, the implementation details should not be viewed as limitations on the scope of any claimed subject matter, but rather as descriptions of features specific to particular embodiments. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may be deleted from the combination in certain circumstances, and a claimed combination may be directed to subcombinations or variations of subcombinations.

Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

Thus, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

As one of ordinary skill in the art will recognize, the novel concepts described herein may be modified and varied across a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to any of the specific exemplary teachings discussed above, but is instead defined by the following claims.

100: Joint learning system 101: Global server 102, 102 ₁ , 102 ₂ , 102 _N : Client 103: Communication link 201, 251: Processing device 202, 252, 430: Memory 203, 253: Communication interface 254: Computing device 300: Method 301, 302, 303, 304, 305, 306, 307: Step 400: Electronic device 410: Controller 420: Input/output device 440: Interface 450: GPU 460: Image processing unit 470: Bus

In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the drawings, in which: FIG. 1 depicts a functional block diagram of an example embodiment of a joint learning system according to the subject matter disclosed herein. FIG. 2A and FIG. 2B depict functional block diagrams of example embodiments of a global server and a client, respectively, according to the subject matter disclosed herein. FIG. 3 is a flow chart of an example embodiment of a method for joint machine learning at a client device according to the subject matter disclosed herein. FIG. 4 depicts an electronic device including functionality for joint machine learning according to the subject matter disclosed herein.

300:Methods

301, 302, 303, 304, 305, 306, 307: Steps

Claims (20)

A client device in a joint machine learning system, the client device comprising: at least one computing device; a communication interface; and a processor coupled to the at least one computing device and to the communication interface, the processor: selecting a group of parameters of the client device from a global group of parameters using variational parameters, training a model using a data set of the client device and the group of parameters selected by the client device, the data set being formed by outputs of the at least one computing device, updating weights included in the global group of parameters after training the model A weight factor dictionary and a factor strength vector, wherein the weight factor dictionary includes Bayesian non-parametric weight factor decomposition values, sending the weight factor dictionary updated by the client and the factor strength vector updated by the client to a global server via the communication interface, receiving the globally updated weight factor dictionary and the globally updated factor strength vector from the global server via the communication interface, and retraining the model using the data set of the client device, the group of parameters selected by the client device, and the globally updated weight factor dictionary and the globally updated factor strength vector. A client device as described in claim 1, wherein the client device is part of a group of N client devices, where N is an integer equal to or greater than 1. A client device as claimed in claim 2, wherein the processor selects the group of parameters from the global group of parameters using three variational parameters including a seed value and minimizes the difference between supervised learning of the data set and regularization of the group of parameters selected by the client device and the global group of parameters. A client device as claimed in claim 3, wherein the processor selects the group of parameters from the global group of parameters by receiving the global group of parameters that has been sent from the global server to a first subset of client devices among the N client devices, the client devices being part of the first subset of client devices. A client device as claimed in claim 4, wherein the client device receives the globally updated weight factor dictionary and the globally updated factor strength vector by receiving the globally updated weight factor dictionary and the globally updated factor strength vector sent by the global server to a second subset of the client devices, the client device being part of the second subset of the client devices. A client device as described in claim 4, wherein the processor sends a request for a current version of the global group of parameters to the global server via the communication interface, wherein the processor updates the model using the current version of the global group of parameters, and wherein the processor evaluates the model updated using the current version of the global group of parameters to form an inference based on the data set of the client device. A client device as described in claim 1, wherein the data set includes information related to at least one of the following: biometric data, medical data, image data, voice data, location data, application usage data, thermal data, atmospheric data, audio data, and survey data. A joint machine learning system, comprising: a global server that receives updates of a weight factor dictionary and a factor strength vector from N client devices, wherein N is an integer equal to or greater than 1, and generates a globally updated weight factor dictionary and a globally updated factor strength vector; and the N client devices, at least one of which comprises: at least one computing device, a communication interface, and a processor coupled to the at least one computing device and coupled to the communication interface, the processor: selecting a group of parameters from a global group of parameters using a variational parameter, training the group using a data set of the client devices and the parameters selected by the client devices A model, updating a weight factor dictionary and a factor strength vector included in the global group of parameters after training the model, wherein the weight factor dictionary includes Bayesian non-parametric weight factor decomposition values, sending the weight factor dictionary updated by the client and the factor strength vector updated by the client via the communication interface, receiving the globally updated weight factor dictionary and the globally updated factor strength vector from the global server via the communication interface, and retraining the model using the data set of the client device, the group of parameters selected by the client device, and the globally updated weight factor dictionary and the globally updated factor strength vector. A joint machine learning system as described in claim 8, wherein the processor selects the group of parameters from the global group of parameters using three variational parameters including a seed value and minimizes the difference between supervised learning of the data set and regularization of the group of parameters selected by the client device and the global group of parameters. A joint machine learning system as described in claim 9, wherein the processor selects the group of parameters from the global group of parameters by receiving the global group of parameters that have been sent from the global server to a first subset of client devices in the N client devices, the client devices being part of the first subset of client devices. A joint machine learning system as described in claim 10, wherein the client device receives the globally updated weight factor dictionary and the globally updated factor strength vector by receiving the globally updated weight factor dictionary and the globally updated factor strength vector sent by the global server to a second subset of the client devices, the client device being part of the second subset of the client devices. A joint machine learning system as described in claim 10, wherein the processor sends a request for a current version of the global group of parameters to the global server via the communication interface, wherein the processor updates the model using the current version of the global group of parameters, and wherein the processor evaluates the model updated using the current version of the global group of parameters to form an inference based on the data set of the client device. A combined machine learning system as described in claim 8, wherein the data set includes information related to at least one of the following: biometric data, medical data, image data, voice data, location data, application usage data, thermal data, atmospheric data, audio data, and survey data. A method for joint machine learning, the method comprising: selecting a group of parameters from a global group of parameters at a client device by using variational parameters, the global group of parameters comprising a weight factor dictionary and a factor strength vector; training a model at the client device using a data set of the client device and the group of parameters selected by the client device; updating the weight factor dictionary and the factor strength vector comprised by the global group of parameters after training the model, wherein the weight factor dictionary comprises a Bayesian non-parametric weight factorization values; sending a client-updated weight factor dictionary and a client-updated factor strength vector from the client device to a global server; receiving a globally updated weight factor dictionary and a globally updated factor strength vector from the global server at the client device; and retraining the model at the client device using the dataset of the client device, the group of parameters selected by the client device, and the globally updated weight factor dictionary and the globally updated factor strength vector. The method of claim 14, wherein the client device is part of a group of N client devices, where N is an integer equal to or greater than 1. The method of claim 15, wherein selecting the group of parameters from the global group of parameters further comprises selecting the group of parameters using three variational parameters including a seed value; and minimizing the difference between supervised learning of the data set and regularization of the group of parameters selected by the client device and the global group of parameters. The method of claim 16, wherein selecting the group of parameters from the global group of parameters further comprises receiving, at the client device, the global group of parameters that has been sent from the global server to a first subset of client devices among the N client devices, the client device being part of the first subset of client devices. The method of claim 17, wherein receiving the globally updated weight factor dictionary and the globally updated factor strength vector from the global server at the client device further comprises receiving the globally updated weight factor dictionary and the globally updated factor strength vector sent by the global server to a second subset of the client devices at the client device, the client device being part of the second subset of the client devices. The method of claim 17 further comprises: requesting, by the client device, a current version of the global group of parameters from the global server; receiving the current version of the global group of parameters; updating the model using the current version of the global group of parameters; and evaluating the model updated using the current version of the global group of parameters to form an inference based on the data set of the client device. The method of claim 14, wherein the data set includes information related to at least one of the following: biometric data, medical data, image data, voice data, location data, application usage data, thermal data, atmospheric data, audio data, and survey data.