[go: up one dir, main page]

WO2025106737A1 - Apparatuses and methods involving sensor-based sensory extrapolation - Google Patents

Apparatuses and methods involving sensor-based sensory extrapolation Download PDF

Info

Publication number
WO2025106737A1
WO2025106737A1 PCT/US2024/056006 US2024056006W WO2025106737A1 WO 2025106737 A1 WO2025106737 A1 WO 2025106737A1 US 2024056006 W US2024056006 W US 2024056006W WO 2025106737 A1 WO2025106737 A1 WO 2025106737A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
sensors
signals
array
region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/US2024/056006
Other languages
French (fr)
Inventor
Kyun Kyu KIM
Zhenan Bao
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Leland Stanford Junior University
Original Assignee
Leland Stanford Junior University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Leland Stanford Junior University filed Critical Leland Stanford Junior University
Publication of WO2025106737A1 publication Critical patent/WO2025106737A1/en
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/24Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/30Input circuits therefor
    • A61B5/307Input circuits therefor specially adapted for particular uses
    • A61B5/313Input circuits therefor specially adapted for particular uses for electromyography [EMG]
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/68Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/6813Specially adapted to be attached to a specific body part
    • A61B5/6824Arm or wrist
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/24Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316Modalities, i.e. specific diagnostic methods
    • A61B5/389Electromyography [EMG]
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/68Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/683Means for maintaining contact with the body
    • A61B5/6832Means for maintaining contact with the body using adhesives
    • A61B5/6833Adhesive patches
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning

Definitions

  • BACKGROUND 0001 Aspects of the present disclosure are related generally to the field of sensors involving physiological/biological applications, and as may be exemplified by uses, among others, including wearables, prosthetics, motion analysis as used in physical therapy, etc. [0002] Using the domain of wearables as one such technology type for ease of discussion, it has been appreciated that many sensing designs have required high-resolution and high-density 7 sensors to precisely extract information about the user’s body kinematics and physiology. The complex interconnection between body muscles and parts amplifies the necessity for high-density sensor arrays. This requirement adds layers of complexity to the system and also surges computational demands, making it challenging to produce compact, efficient, and affordable wearable devices.
  • physiological signal monitoring such as EMG (electromyography) has attracted significant attention for its capacity to non-invasively monitor muscle activity across different body regions.
  • EMG electromyography
  • HD-sEMG high-density surface electromyography
  • circuit-based apparatus including one more sensor structures and data-processing computing circuitry’ (e.g., including a computer, processor, etc.), configured to collect data via certain sensors and to use the data for sensory extrapolation (e g., collecting more data such as for an expanded region of a user than would be expected from the certain sensors).
  • data-processing computing circuitry e.g., including a computer, processor, etc.
  • an apparatus and/or method is designed for use with an array of sensors to collect user signals indicative of kinematics and/or electrophysiology, corresponding to an observed region of the user.
  • the apparatus (which may be characterized as a computer-implemented method) involves computer processing circuitry to use the signals as collected by the array of sensors bycarrying out certain computer-programmed aspects or steps such as: training and/or using an algorithm to leam and/or identify a pattern of physiological signals manifested by the use; and generating, via the algorithm, data knowledge about relationships between the observed region and at least one unobserved region of the user that does not include the observed region, and therein extrapolating further physiological signals of the user, corresponding to the unobserved regions.
  • Certain other examples of the present disclosure are directed to the computer processing circuitry’ being configured (e.g., programmed) to generate, based on the signals as collected by the array of sensors, output data indicative of the user concerning at least one of kinematics and electrophysiology of the at least one unobserved region of the user.
  • the computer processing circuitry may also or alternatively use the signals as collected by the array of sensors by training the algorithm to leam and/or identify a pattern of physiological signals manifested by the user, wherein in some instance the algorithm training is referred to as generative model training.
  • EMG electrophysiology corresponding to the observed region of the user
  • EEG electrophysiology corresponding to the observed region of the user
  • kinematics corresponding to the observed region of the user.
  • Nonlimiting examples of other collected and/or collectable physiological signals according to specific aspects to the present disclosure are signals corresponding to: EMG (electromyography), EEG (electroencephalogram), ECG (electrocardiography), EGG (electrogastrography), and EOG (electrooculography).
  • aspects of the present disclosure involve methods and related apparatuses concerning a generative multi-array sensory system capable of predicting unseen areas by utilizing a reduced (minimal or optimized) set of sensory data (e.g., data collected via the primary user sensors).
  • a reduced (minimal or optimized) set of sensory data e.g., data collected via the primary user sensors.
  • methods and related apparatuses according to the present disclosure are designed to extrapolate signals associated with unknown regions (which are not being monitored by electrode-based sensors).
  • Particularly for wearables, methods and related apparatuses according to the present disclosure include stretchable sensors used in this regard and these sensors attach seamlessly with the skin, thereby acting to lower undesirable artifacts and to improve signal noise.
  • example algorithms according to the present disclosure intentionally discard over half of the originally-collected (input) data to obtain key (informative) signals for the application, and the model may reconstruct the information that is missing. In this manner, such example algorithms may be used to effectively simulate a high-density' sensory network with significantly fewer sensors.
  • the above type of method further includes training the computer-executable algorithm to identify the pattern of physiological signals by' using a learning model that operates based on an auto encoder framework and that includes an encoder circuit module and a decoder circuit module that are cooperatively arranged to process raw data, from a number of training electrodes used during training, to configure the computer-executable algorithm to identify, after the computer-executable algorithm is trained, the pattern of physiological signals while the array of sensors for said step of collecting signals utilizes less than one half of the number of training electrodes used during training.
  • a learning model that operates based on an auto encoder framework and that includes an encoder circuit module and a decoder circuit module that are cooperatively arranged to process raw data, from a number of training electrodes used during training, to configure the computer-executable algorithm to identify, after the computer-executable algorithm is trained, the pattern of physiological signals while the array of sensors for said step of collecting signals utilizes less than one half of the number of training electrodes used during training.
  • the present disclosure is directed to apparatuses (e g., system, circuitry, components, and/or materials) and methods which provide certain advantages and/or improvements over known approaches. Examples of such advantages and/or improvements, depending on the implementation, may include. 1) Reduced complexity: designs of the present disclosure reduce or diminish the need for high- density sensory networks by leveraging locational correlations, and streamlining both the physical setup and computational processing. Reducing the number of required sensors may be achieved without compromising on data accuracy which can lead to more affordable devices. 2) Enhanced Comfort: the integration of stretchable sensors in apparatuses of the present disclosure offers users an unobtrusive, comfortable fit, making the device more user- friendly and wearable for extended durations.
  • exemplary aspects of the present disclosure involve wearable applications, some of which integrate the benefits of stretchable sensors and low impedance electrodes to enhance the efficacy of the above-characterized system types and then implement a generative algorithm to process signals from one or more of these t pes of sensors.
  • Such sensors offer superior conformity to the skin, and they also facilitate the acquisition of low-noise signals that enables higher performance of the generative network by reducing motion artifacts and contact impedance.
  • the signals collected from such sensors are fed into the generative algorithm, which is trained to intentionally omit a significant portion of the collected data.
  • the array of sensors may be characterized as including at least one of: strain pressure sensors, photo-diode based sensors, and ultra-sonic sensors;
  • the array of sensors can include a body -adhering polymer based substrate layer that is stretchable and elastic, and integrated with a conductive gel that serves to lower impedance;
  • the array of sensors can be protected by a rubber barrier layer to mitigate or prevent solvent induced swelling;
  • the array of sensors can include a liquid metal electrode layer which is both stretchable and micro-patterned, encapsulated by a photo-patterned synthetic rubber which leaves only the electrode areas uncovered to facilitate skin contact and signal measurement.
  • FIGs. 1 A-1B depict a system having a generative electronic sensory network (G- SNET) for kinematic predictions, according to certain exemplary aspects of the present disclosure, with: FIG. 1 A highlighting exemplary sets of raw-channel electrodes and pseudo electrodes (depicting the data from the raw-channel electrodes after processing), and FIG.
  • G- SNET generative electronic sensory network
  • IB showing a first limb-specific example application and embodiment with an exploded view of the limb-specific example embodiment in the form of limb-wearable device (e.g., a strip, band, patch, and/or w ireless smartw atch;
  • limb-wearable device e.g., a strip, band, patch, and/or w ireless smartw atch;
  • FIGs. 2A-2H depict a stretchable sensory array for high-quality dataset generation with: FIGs. 2A. 2B, and 2C showing exemplary sets of electrodes according to one or more specific embodiments, FIG. 2D showing an example composition of an electrode, FIGs. 2E, 2F and FIG. 2G showing graphs depicting performance-related data of one or more arrays shown in FIGs. 2A-2C and FIG. 2H showing an example block layout with an arrangement of circuit components as a system-level example;
  • FIGs. 3A-3E depict post-training of a G-SNET with: FIG. 3 A showing processing of data from one or more arrays shown in FIGs. 2A-2C, FIG. 3B showing exemplary 7 flow' for generative model training, FIG. 3C showing performance-related data of the one or more arrays, FIG. 3D showing a flow diagram of a G-SNET, and FIG. 3E which is a diagram illustrating how post-training is used for prediction;
  • FIGs. 4A-4E depict approaches for predicting various body kinematics through a G-SNET combined device with: FIG. 4A showing a wireless module (e.g., band, patch or other user device such as w atch), FIG. 4B showing a flow diagram of post-training as may be used for prediction, FIG. 4C showing an application of the module of FIG. 4A. FIG. 4D showing a related application of the module of FIG. 4A and with predictive data being plotted, and FIG. 4E show ing output data plotted in a pair of related graphs;
  • FIG. 5 is a more-specific example of a generative model structure for postlearning for a sign-language translation application;
  • FIGs. 6A-6C depict a schematic of a system that is configured to utilize latent vector(s) for post-training American sign-language translation results with: FIG. 6A showing a flow diagram of an example embodiment, FIG. 6B showing known hand/finger signs according to the American sign language, and FIG. 6C showing results of the American sign language as processed by the data flow of FIG. 6A; and
  • FIGs. 7A-7F depict various representative sets of results from exemplary' data- processing. according to certain exemplary' aspects of the present disclosure, and with each set showing live or raw (e.g., EMG-captured) data, masking (e.g., random masking) of the live or raw data, and the full data set (or signals) including data as extrapolated for the unseen regions, also according to experimental example embodiments and aspects of the present disclosure.
  • live or raw e.g., EMG-captured
  • masking e.g., random masking
  • aspects of the present disclosure are believed to be applicable to a variety of different types of apparatuses, systems and methods involving devices characterized at least in part by sensors such as, but not necessarily limited to, user wearable-type sensors, and to using and/or exploiting sensor-based data for increased understanding of sensory regions of a user. While the present disclosure is not necessarily limited to such aspects, an understanding of specific examples in the following description may be understood from discussion of such specific contexts as disclosed hereinbelow.
  • Exemplary contexts of the present disclosure are directed to examples involving generative multi-array sensory systems and methods capable of extrapolating unseen areas based on a minimal set of sensory data.
  • the system implements an algorithm to identify patterns between sensory activations from varied locations, as exemplified by use of bio-signals (e.g., indicative of muscle activities, brain signals, pulses, or the like) which can exhibit locational correlations. Leveraging these correlations, one or more generative algorithms, according to the present disclosure, are used to predict a user’s signal activities in unseen areas (e.g., not having sensory components attached thereto) based on algorithm training about relationships between extrapolated regions.
  • bio-signals e.g., indicative of muscle activities, brain signals, pulses, or the like
  • an apparatus is designed for use with an array of sensors to collect user signals indicative of kinematics and/or electrophysiology, corresponding to an observed region of the user.
  • the apparatus (which may be characterized as a computer-implemented method) includes computer processing circuitry to use the signals as collected by the array of sensors by carrying out certain computer-programmed aspects (or steps) such as: training and/or using an algorithm (that is trained) to identify a pattern of physiological signals manifested by the use; generating, via the algorithm, data knowledge about relationships between the observed region and at least one unobserved region of the user that does not include the observed region; and extrapolating, in response to the generated data knowledge about relationships, further physiological signals of the user, corresponding to the unobserved regions (e g., with the further physiological signals being predicted).
  • a more specific example embodiment is directed to an apparatus which has the computer processing circuitry to train the algorithm based on a set of raw signals acquired from a number of training electrodes which are associated with at least the unobserved region of the user signals.
  • the computer processing circuitry is to use the algorithm to identify the pattern of physiological signals by collecting the user (electro)physiological signals from the user- coupled electrodes of the array of sensors, wherein the number of training electrodes is at least twice the number of user-coupled electrodes.
  • the computer-executable algorithm may be trained to identify the pattern of physiological signals by using: a set of M electrodes to acquire signals associated with said at least one unobserved region of the user; deriving at least one latent vector (corresponding to more pertinent aspects of the collected data) from the carried signals, wherein the computer-executable algorithm is trained to identify' the pattern of physiological signals based on said at least one latent vector; and the step of generating knowledge about relationships between the observed region and at least one unobserved region uses not more than N electrodes in the array of sensors for said step of collecting signals, wherein M and N are positive integers and M is at least three-to- four times greater than N.
  • the ratio of the number (M) of training electrodes to the number (N) of user-coupled electrodes is set based on the design and specific application needs (such ratios may be 32/4, 64/6, 41/4, 48/8, etc.).
  • another more specific example embodiment is an apparatus which includes the computer processing circuitry being associated with a first software-executable module to train the algorithm by responding to raw signals from electrodes associated with the observed region of the user.
  • the computer processing circuitry' is to use the trained algorithm, via a second softwareexecutable module, for the extrapolation of further (electro)physiological signals of the user or information related to the further physiological signals by responding to a set of user- coupled electrodes that are physically coupled to the observed region of the user without concurrently using any further electrodes coupled to the unobserved region of the user.
  • the computer processing circuitry may be implemented as a single integrated computer processing circuit (e.g., with both of the first and second software-executable module), or alternatively, the computer processing circuitry may be implemented as distinct integrated computer processing circuits. As distinct integrated computer processing circuits, one of the computer processing circuits may be configured with the first software-executable module, and another of the computer processing circuits may be configured with the second software-executable module. In this manner, the multiple modules and/or the multiple computer processing circuits, while referred to as computer processing circuitry, may be manufactured and/or configured (e.g.. programmed such as by training) separately (e.g., separated physically and/or in time).
  • wearable applications e.g., spanning the above-noted observed region
  • these types of sensors and/or electrodes offer superior conformity to the skin, and they also facilitate the acquisition of low-noise signals that enables higher performance of the generative network by reducing motion artifacts and contact impedance.
  • example embodiments involve processing the collected signals from such sensors and then feeding the signals into a generative algorithm which, according to the present disclosure, may be implemented (or trained) by intentionally omitting (or masking) a significant portion of the collected data and using, by masking, only the so-called latent data.
  • the computer processing circuitry and the algorithm collectively correspond to a pre-trained encoding network configured based on a defined or an iteratively -refined latent vector derived from a set of raw data carried or derived from a set of electrodes, or signal channels, which are associated with the observed user region and to be used to predict a pattern of physiological signals associated with the observed user region and said at least one separate unobserved region.
  • the generative algorithm may be trained by such latent data by learning to fill in data missing from the multi-array signals.
  • This pre-trained model is then merged with a simplified device having fewer sensing channels, enabling the created low-dimensional vector that provides information equivalent to that from a multi-sensing array.
  • the lowdimensional latent vector can then be used by a generative network, also according to one example of the present disclosure, as a refining or concise feature of spatiotemporal muscle activities and subsequently harnessed for use in various (and sometimes diverse) applications, such as American Sign Language Translation (ASL) and gait phase detection.
  • ASL American Sign Language Translation
  • a circuit- implemented method includes collecting, via an array of sensors, signals (e.g., indicative of kinematics and/or electrophysiology) corresponding to an observed region of a user, and then processing (extrapolating) and/or using the collected signals to identify a pattern of physiological signals manifested by the user, which corresponds to one or more unobserved user regions that does not include the observed region.
  • signals e.g., indicative of kinematics and/or electrophysiology
  • the circuit-implemented method is carried out by one or more computing data-processing (or CPU) circuits (sometimes referred to as computer processing circuitry).
  • the method and computer processing circuitry can include an algorithm collectively configured as a pre-trained encoding network, which may use a G-SNET (generative electronic sensory network), that is to define or iteratively -refine latent vector(s) ("the latent vector information” ) derived from a set of raw data earned from a set of N electrodes (e.g., coupled to N signal channels), coupled to the observed user region.
  • G-SNET generative electronic sensory network
  • the latent vector information is processed via an M-channel encoder and used to generated the relationships between the observed region and at least one unobserved region, and wherein M and N are positive integers and depending on the particular relationships needs, M is several times greater than N (e.g.. M is at least 3-4 times greater than N, and in some instances such as those involving high-speed data processing such as in high-speed neural network, M is in a range of several times greater than N to 10 or 100 times greater than N).
  • M and N are positive integers with M being in a range from 20 to 133. N being in a range from 7 to 64 with M at least 3 times greater than N. It is appreciated that the N electrodes/channels need not be in a specially-formed array, but rather one in which each of the relative locations is known.
  • FIGs. 1 A-1B depict such a G-SNET system for generating kinematic predictions, according to certain exemplary aspects of the present disclosure.
  • FIG. 1 A highlights an exemplary set of raw-channel electrodes (4 EMG pads in the observed area) and pseudo electrodes spanning regions outside the observed regions, and with a latent vector being processed to the right of FIG. 1 A.
  • FIG. IB shows a first limb-specific example application (6 EMG pads in an observed area of the leg or arm) with an exploded view of the limb-specific example device being a limb-wearable device (e.g., a strip, band, patch, and/or wireless smartwatch).
  • the latent vector and related processing of the small-channels raw data is used to create an effective pseudo multi-array EMG with, in these illustrated examples, a range of several tens of pseudo electrodes (more with multiple pseudo EMG arrays).
  • this type of system can be implemented to integrate the benefits of stretchable sensors and low impedance electrodes.
  • These sensors/electrodes not only offer superior conformity to the skin, but they also facilitate the acquisition of low-noise signals that enables higher performance of the generative network by reducing motion artifacts and contact impedance, also facilitate the acquisition of low-noise signals.
  • These collected signals are then fed into the above-characterized type of generative algorithm where, as noted above, the raw-signal information may be intentionally omitted. With repeated learning cycles (e.g., refining the above-discussed vector(s)), the model learns to fill in this missing data from the multi-array signals.
  • This refined, pre-trained model is then merged with a device having a subset of channels (e.g., significantly fewer channels), enabling the created pseudo multiarray signals.
  • the latent vector is then used at the middle of the generative network which is a concise form of the input, emphasizing one or more features specific to the application. This information can then be harnessed for any of different and/or diverse applications, such as American Sign Language translation and gait phase detection.
  • the model next learns to fill in this missing data from the multi-array signals.
  • This pre-trained model is then merged with a simplified device having fewer sensing channels, enabling the created low-dimensional vector that provides information equivalent to that from a multi-sensing array.
  • the low-dimensional latent vector of the generative network is a concise (and in some examples, an important) feature of spatiotemporal muscle activities. Then this information is harnessed for diverse applications, such as American Sign Language Translation (ASL) and gait phase detection (as in FIG. IB).
  • ASL American Sign Language Translation
  • IB gait phase detection
  • This work presents a new paradigm for wearable sensing system development and enables comfortable wear of less complex sensing systems which can produce high-quality data previously requiring high -density sensing arrays. This approach also simplifies data processing which should result in lower power consumption and faster processing time.
  • FIGs. 2A-2H depict a stretchable sensor array for high-quality dataset generation. More specifically, FIGs. 2A-2C show a structure with an exemplary (non-limiting) set of 32 channels, according to one or more specific embodiments, with corresponding electrodes. As illustrated in FIGs. 2A-2C. the electrodes may be viewed as arranged in one or more arrays (e.g., a 4x8 electrode array, a pair of 4x4 electrodes, etc.) for implementation of a 32- electrode / 32-channel EMG structure.
  • arrays e.g., a 4x8 electrode array, a pair of 4x4 electrodes, etc.
  • FIGs. 2D and 2E respectively show: example (molecular) compositions (FIG. 2D) of an electrode (such as in FIG. 2B) with and without a PEDOT layer on the exposed portion of the electrode, and a graph of advantageous performance parameters (FIG. 2E) showing much lower impedance at relevant signal frequencies corresponding to the exposed portion with and without such a PEDOT layer.
  • FIGs. 2F and 2G show respective graphs depicting further performance-related data corresponding to the exposed portion with and without such a PEDOT layer, with the electrode of FIG. 2G having an Ag/AgCl composition.
  • FIG. 2H shows a detailed example of an electrode structure which may be viewed as corresponding to the structure shown in FIGs. 2B and 2C.
  • the electrode structure is shown to include an example block circuit layout having X-channel stretchable EMG structure including a FFC (flexible-flat connector) connecting to wireless module, wherein for this particular experimental example, X is equal to 32 (one channel for each skin-contacting electrode).
  • FFC flexible-flat connector
  • the signals from the electrodes are carried by a cable to circuitry (e.g., embedded as an integrated part of a user- w orn patch) which, for illustrative purposes, is shown to include an MCU (CPU-based master control unit), a multiplexer (MUX) for selectively feeding the channeled signals to the MCU, an amplifier for amplifying the signals before and/or after being processed by the MUX, and where a wireless connection is appropriate, a wireless (e g., Bluetooth) transponder for transmitting the signals.
  • This transmission may be for further processing such as from the user-wom patch to a more robust CPU circuit which is configured to cany' out the processing as discussed above in connection with FIGs. 3A-3E, FIGs. 4A-4E, 5 and FIGs. 6A-6C.
  • the pre-training generative algorithm is used to reconstruct the original detailed temporal and spatial patterns of muscle activity signal with high fidelity’ using minimal sensor inputs.
  • achieving high-accuracy levels from the generative algorithm is contingent upon the quality of the initial training dataset.
  • the creation of a high-quality dataset is facilitated, according to the present disclosure, by engineering a fully stretchable multi-electrode array EMG device, such as shown in FIG. 2A.
  • the layer structure includes several layers: a substrate layer (e.g., PDMS), a protective layer (e.g., NBR), an electrode (e.g., EGain), a gel (e.g.. PEDOT), and an encapsulation layer (e.g., SBS).
  • a substrate layer e.g., PDMS
  • a protective layer e.g., NBR
  • an electrode e.g., EGain
  • a gel e.g... PEDOT
  • an encapsulation layer e.g., SBS
  • the PDMS layer provides a thin substrate with sub-millimeter thickness for flexible handling and easy adherence to various body contours.
  • the NBR layer acts as a barrier to protect the PDMS from solvent-induced swelling during the fabrication of the multiple layers.
  • the PDMS and/or NBR layer(s) may be used as a substrate for the other depicted layers.
  • the EGain liquid metal electrode which is both stretchable and micro-patterned, is then applied, followed by a highly conductive PEDOT gel that serves to lower impedance.
  • the structure may include a gold (Au) layer between the NBR and EGain layers.
  • the device is completed with a photo-patterned SBS layer that leaves uncovered only the electrode areas (e.g., one of which is shown as an upper portion of a PEDOT in FIG. 2B), to facilitate skin contact and signal measurement.
  • FIG. 2B and 2C illustrate various views (including the top and side views) of the exemplary sensor array and the fabricated stretchable array.
  • a chosen material set includes a conductive and adhesive gel composed of acrylamide (AAm) integrated with a PEDOT:PSS physically cross-linked conducting polymer network.
  • AAm acrylamide
  • PEDOT:PSS physically cross-linked conducting polymer network
  • FIGs. 3A-3E show, consistent with the above-discussed aspects of FIGs. 2A-2G, further aspects and methodology of an exemplary sensory and information-processing structure.
  • FIG. 3A show s processing of data from one or more arrays such as shown in FIGs. 2A-2C.
  • FIG. 3B shows exemplary high-level methodology, according to the present disclosure, involving data flow to be carried out by generative model training circuitry. Once trained, the sensory and information-processing structure can predict how pseudo sensors (covering unseen areas) are acting.
  • the methodology deliberately masks a significant (e.g., predominant) amount of the complete sensor data. This masked data is then fed into the G- SNET, where learning parameters are adjusted to reproduce the initial signal from the masked data. To bolster the model's resilience against varying sensor attachment positions and orientations, the methodology can optionally incorporate randomly -distributed masking. [0049] Particular masking involved for this exemplary methodology is shown in FIG.
  • 3C via two example sets of signal-channel images with each set showing: data flow from the originally-collected (raw) electrode-obtained signals, to the masked signals, and on to the expanded set of (pseudo) information (with the darker spots corresponding to information from the originally -collected signals which is selected via the masking for analysis and/or use).
  • the G-SNET successfully reconstructs the original signals using the masked inputs, wherein the particular examples of FIG. 3C have intentionally masked 70% to 90% of the complete sensor data.
  • the latent data may be: generally less than 70% (or 50%) and greater than 10%, in other cases in a range from 40% to 65%, etc.
  • the learning-model architecture uses an autoencoder framework that is principally divided into two sections: an encoder and a decoder.
  • the encoder translates the intentionally random masked (e.g., 70-90%) signals into a latent representation. Subsequently, the decoder revives the original signal from this latent space.
  • FIG. 3D shows a more-detailed example of each of the encoder and the decoder sections of FIG. 3B, with each coding section including layers of logic circuitry' (e.g., a circuit including at least one programmed CPU and/or programmable logic array).
  • the layers include: multi-head attention, normalization, and linear layers.
  • the decoder will then generate reconstructed patches from the output of the encoder.
  • the latent vector is then used in this example by feeding it into another dense layer for further classification (e.g., American Sign language translation, gaiting features, etc.). Utilizing latent vectors helps to avoid reconstruction noise of the decoder, generalization of the input, and computation efficiency of its low-dimensional property.
  • the raw signals captured by the 32-channel sensory- array were subjected to post-processing, which involved the calculation of root mean square (RMS) values across 32-time windows, using a sliding window with a size of 10.
  • RMS root mean square
  • certain of the more-specific/experimental example embodiments employed RMS values as a simplified representation of the EMG signals.
  • This processing technique facilitated the establishment of a temporal correlation within the 32-time window across all 32 EMG channels. This facilitates construction of an EMG signal tensor with dimensions of 32 x 32.
  • the arrangement of 32 channels, and corresponding electrode array is a non-limiting example for experimental/proof-of-concept purposes and other arrangements of channels and corresponding electrode array(s) may be used in a similar manner.
  • FIGs. 4A-4E depict prediction of various body kinematics through a G-SNET combined device with: FIG. 4A showing a wireless module (e.g., band, patch or other user device such as watch), FIG.
  • a wireless module e.g., band, patch or other user device such as watch
  • FIG. 4B showing a flow diagram of post-training as may be used for prediction
  • FIG. 4C showing an application of the module of FIG. 4A
  • FIG. 4D showing a related application of the module of FIG. 4A and with predictive data being plotted
  • FIG. 4E showing output data plotted in a pair of related graphs.
  • aspects of the present disclosure can be implemented with the G-SNET utilized for kinematic prediction.
  • G-SNET utilized for kinematic prediction.
  • a compact 6-channel wireless EMG watch depicted in FIG. 4A, was employed to acquire signals.
  • This reduced-channel data was fed into the encoder to generate a latent vector, which w as subsequently processed for American Sign Language (ASL) gesture prediction.
  • ASL American Sign Language
  • the initial dataset comprised EMG recordings corresponding to the 26 hand gestures representing the ASL alphabet, from A to Z. Utilizing the pre-trained encoder, such experimental modeling is refined to enhance its predictive capabilities for ASL interpretation.
  • the device is attached to the user’s leg (e.g.. at the call) to ascertain various gait phases.
  • Monitoring gait kinematics continuously provides insights into potential musculoskeletal disorders, thereby aiding in the identification of risk factors for conditions such as falls, the need for rehabilitation, preventive measures for injuries, and optimization of athletic performance.
  • GRF ground reaction forces
  • FIG. 5 and FIG. 6A may be configured for the American sign-language, as represented in FIG. 6B.
  • FIG. 5 is another example of a generative model structure for post-learning for a sign-language translation application, as a more specific example of the generative model structure shown in connection with FIGs. 3A-3E.
  • the upper portion of FIG. 5 may be implemented with a particular encoder, as shown in connection with FIGs. 3B and 3D, before the latent vector data is developed at the output of the encoder.
  • raw data from 6 channels (the highlighted rows in the image of FIG. 5) is shown as being used to develop the masked signal, which may be generated by decomposing data from patches with embedded circuity (including the skin-contacting electrodes).
  • the latent vector data may then be developed, for example, using different layered (software-based) CPU modules.
  • such different layered CPU modules may include a first layer for batching and/or normalization of the data derived from the signals carried on each channel, a second layer for data linearization to better reflect continuity and/or relationships in areas between actual electrodes and virtual (or pseudo electrodes), and a third (softmax) layer for normalization of the data into various output classes (e.g., O 1 , O 2 , O 3 , O 4 and O 5 , as shown to the right of FIG. 5.
  • the softmax layer may be used to normalize the output and therein provide a probability distribution over predicted ones of such output classes; for example, for a given set of output values (e.g., O 1 - O 5 ), one for each class in the classification task, the softmax function may be used to normalize the outputs, by converting them from weighted sum values into probabilities that sum to one. Each value in the output of the softmax function is interpreted as the probability of membership for each class.
  • the example schematic/flow diagram of FIG. 6A corresponds to a system configured for utilizing a set of one or more latent vectors for post-training sign-language translation results.
  • a pre-trained encoding netw ork w as adapted to reconstruct a system with a full 32-channel dataset from a 6-channel input (as the masked signal).
  • the 6-channel data was input into an encoder, thereby generating a latent vector.
  • This vector was then introduced to a linear layer for finger spelling prediction. Initially, 15 finger letters (of the American sign-language as shown in FIG. 6B) ranging from A to N were captured by the system.
  • the model was further refined for language prediction.
  • FIG. 6C shows the results of the American sign language, in the form of a confusion matrix, as processed by the data flow and system of FIG. 6A.
  • the confusion matrix has along its vertical axis, as the true object, each of 15 rows respectively corresponding to the 15 finger letters (e.g., as indicated by a unique motion or (near) position that corresponds to one or more of the 15 letters).
  • the horizontal axis of the confusion matrix shows 15 rows also corresponding to the 15 finger letters but as predicted by the system of FIG. 6A.
  • the algorithm may be further trained for refinement towards higher scores (with acceptable levels of accuracy) and/or the system may generate user-feedback (e.g., visual, tactile and/or audible) on such a wearable device to prompt the user to repeat, and where appropriate, continue with signing as inputs to the system.
  • user-feedback e.g., visual, tactile and/or audible
  • Such feedback may be implemented using circuitry integrated into the wearable device (e.g., a vibration circuit or display/audible alert as known in many smartwatches).
  • circuitry integrated into the wearable device e.g., a vibration circuit or display/audible alert as known in many smartwatches.
  • a latent vector Since a latent vector has rich information density and inherent generalization, it offers enhanced resilience to a variety of inputs. Given that biological structures exhibit consistency across individuals, a latent space trained using a select yet varied group of representatives (based on gender, weight, height) is well-positioned to generalize corresponding sets of individual data as one set of representative data that is useful for a plurality of users. Thus, by using a high-resolution map trained on this small and diverse cohort, the system can effectively generalize across different individuals.
  • results for signal reconstruction and ASL prediction are presented.
  • a final reconstructed signal is shown from the masked information (with the masking depicted in the center of each set), with the electrode-captured raw data shown to the left, and the processed and extrapolated data shown on the right.
  • the six sets may be different in various ways; as examples, each set may use: a different number, type and arrangement of raw electrodes; and different types and locations of masks (e.g., with some being randomly chosen and/or placed).
  • One or more of the above aspects may be implemented in a variety of other more specific experimental/more-specific examples.
  • One such example involves implementation for use by different individuals (or users). Since a latent vector has rich information density and inherent generalization, it offers enhanced resilience to a variety of inputs. Given that biological structures exhibit consistency across individuals, a latent space trained using a select yet varied group of representatives (based on gender, weight, height) is well-positioned to generalize. Thus, by using a high-resolution map trained on this small and diverse cohort, the system can effectively generalize across different individuals.
  • EMG electromyography
  • ECG electrocardiography
  • EEG electroencephalogram
  • EEG electrogastrography
  • EOG electrooculography
  • EMG tracks various muscle activities throughout the biological body, encompassing motions in the hands, legs, neck, and back.
  • G-SNET With the assistance of G-SNET, this can be expanded to predict a range of motions for applications like sign-language translation, gait-phase detection, and body pose detection.
  • Key applications include rehabilitation (muscle function monitoring), sports (performance analysis), human-machine interfaces (prosthetic control, gesture recognition, silent speech interface), and health monitoring (fall detection).
  • An ECG especially a multi-array ECG, offers a comprehensive view of the heart's electrical activity from multiple viewpoints.
  • G-SNET can potentially condense the standard 12-lead electrode measurement system into a more compact channel system. Notable applications lie in health monitoring (heart rate, arrhythmia detection) and sports (performance and recovery monitoring).
  • Multi-array EEGs have applications in cognitive neuroscience and braincomputer interfaces. Typically requiring 256 or more channels for better spatial resolution and detailed brain activity mapping, G-SNET streamlines this by leveraging merely 10-20% of the original electrode count, ensuring enhanced user comfort. Applications span health monitoring (sleep disorders, epilepsy monitoring, mental health) to brain-computer interfaces (like prosthetic control), and also for applications involving probes (or electrodes) being inserted for uses in brain neural recording, with the brain neural recording benefiting from this for deep brain electrodes used for stimulation and accurate recording. These and other implementations according to the present disclosure may be applicable to both wearable EEG as well as implanted high-density EEG probes.
  • EGG offers a non-invasive approach to monitor the stomach's muscle activity. With G-SNET, fewer electrodes are needed to predict a broader range of gastrointestinal activities. This technology has applications in health monitoring (detecting gastrointestinal disorders) and in research to study the correlation between gastric activities, stress, and sleep. [0072] EOG captures the comeo-retinal standing potential that exists between the front and the back of the human eye, making it possible to detect eye movements. G-SNET aims to achieve accurate readings with fewer electrodes, eliminating the need for extensive coverage around the eye. The potential applications of this technology extend to the human-computer interface (communication, wheelchair navigation, and gaze tracking in AR/VR), and healthcare (e.g., diagnosing retinal diseases and optic nerve disorders).
  • sensors include, among others, strain/pressure sensors, photo-diode based sensors, and ultra-sonic sensors, and such sensors need not necessarily be wearable (e.g., they may be temporarily applied or coupled to the user for temporary measurements).
  • Certain (e.g., wearable) strain/pressure sensors detect deformations (stretching, bending, twisting) and convert them into electrical signals. These sensors can discern changes in pressure and stretch from various sources. Traditional methods entail placing sensors on each joint and muscle. In contrast, G-SNET will discover areal strain/pressure distribution correlations, simplifying the sensory' complexity.
  • Applications encompass health monitoring (blood pressure, respiration rate, pulse), prosthetics (prosthetic limbs, gait analysis), sports (posture monitoring), organ movement (e.g. stomach, gut), and human-machine interfaces (gesture recognition, facial movement recognition, body movement recognition, virtual reality).
  • such methodology and corresponding structures are applicable to obtaining measurements with one or more probes (having one or more electrodes referring to or at the end of. each probe) inserted in an organ and to measure maps of electrophysiological signals both on a surface of the body as w ell as on a surface (or membrane) of internal organs, such as heart, bladder or uterus.
  • Such methodology' and corresponding structures can also be applied to implantable probes used for electrophysiology measurements.
  • Photo-diode based sensors translate variations in the light absorption reflected by the body. When organized in a multi-array pattern on wearable devices, their sensitivity, spatial resolution, and functionality are elevated. G-SNET allows for higher sensory resolution with few channels when gauging heart rate, oxygen saturation (SpO2), glucose monitoring, and blood pressure. This can also be applied to photo-diode based imaging.
  • SpO2 oxygen saturation
  • glucose monitoring glucose monitoring
  • blood pressure blood pressure
  • This can also be applied to photo-diode based imaging.
  • ultra-sonic sensors ultrasound offers a non-invasive window into the body, facilitating health monitoring and disease diagnosis. Ultrasound waves permeate the body and reflect off internal structures, generating echoes. These are captured and relayed to devices that render them into images or videos. A greater number of ultrasound sensor elements begets enhanced image resolution.
  • G-SNET enables a reduction in transmitter count while preserving image quality.
  • one or more modules are discrete logic circuits or programmable logic circuits configured and arranged for implementing these operations/activities, as in the blocks/modules as shown in and among the figures.
  • a programmable circuit includes one or more computer circuits programmed to execute a set (or sets) of instructions (and/or configuration data).
  • the instructions (and/or configuration data) can be in the form of firmware or software stored in and accessible from a memory (circuit).
  • first and second modules include a combination of a CPU hardware-based circuit and a set of instructions in the form of firmware, in which the first module includes a first CPU hardware circuit with one set of instructions for training an algorithm, and the second module includes a second CPU hardware circuit with another set of instructions for further training of the algorithm and/or algorithm utilization.
  • orientation such as upper/lower, left/right, top/bottom and above/below, may be used herein to refer to relative positions of elements as shown in the figures. Such terminology is used for notational convenience only and that in actual use the disclosed structures may be oriented different from the orientation shown in the figures.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Theoretical Computer Science (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Pathology (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)

Abstract

Certain examples are directed to an apparatus designed for use with an array of sensors to collect user signals indicative of kinematics and/or electrophysiology, corresponding to an observed region of the user. The apparatus (which may be characterized as a computer-implemented method) include a computer processing circuitry to use the signals as collected by the array of sensors by carrying out certain computer-programmed aspects or steps such as: training an algorithm to identify a pattern of physiological signals manifested by the use; generating, via the algorithm, data knowledge about relationships between the observed region and at least one unobserved region of the user that does not include the observed region; and extrapolating, in response to the generated data knowledge about relationships, further physiological signals of the user, corresponding to the unobserved regions.

Description

APPARATUSES AND METHODS INVOLVING SENSOR-BASED SENSORY EXTRAPOLATION
BACKGROUND 0001] Aspects of the present disclosure are related generally to the field of sensors involving physiological/biological applications, and as may be exemplified by uses, among others, including wearables, prosthetics, motion analysis as used in physical therapy, etc. [0002] Using the domain of wearables as one such technology type for ease of discussion, it has been appreciated that many sensing designs have required high-resolution and high-density7 sensors to precisely extract information about the user’s body kinematics and physiology. The complex interconnection between body muscles and parts amplifies the necessity for high-density sensor arrays. This requirement adds layers of complexity to the system and also surges computational demands, making it challenging to produce compact, efficient, and affordable wearable devices.
[0003] As more particular examples, physiological signal monitoring such as EMG (electromyography) has attracted significant attention for its capacity to non-invasively monitor muscle activity across different body regions. The complex interplay between muscles and their corresponding body7 movements calls for sophisticated high-density surface electromyography (HD-sEMG) instruments capable of capturing intricate kinematics like gesture recognition and gait analysis. Despite advancements in electrode design to enhance impedance and flexibility, as well as in computational algorithms to improve signal processing and predictive accuracy, many challenges remain especially7 in terms of practicable implementations. These challenges are evidenced, for example, by the large-area and large electrode counts of sEMG devices, in which there is often an extensive physical dimensionality that compromises the form factors used in such devices, thereby limiting their practicality and broader adoption.
[0004] These and other matters have presented challenges to sensor-based designs and related processes for a variety7 of different types of implementations and applications. SUMMARY OF VARIOUS ASPECTS AND EXAMPLES
[0005] Various aspects and examples according to the present disclosure are directed to issues such as those addressed above and/or others which may become apparent from the following disclosure involving a circuit-based apparatus, including one more sensor structures and data-processing computing circuitry’ (e.g., including a computer, processor, etc.), configured to collect data via certain sensors and to use the data for sensory extrapolation (e g., collecting more data such as for an expanded region of a user than would be expected from the certain sensors).
[0006] In certain specific examples according to the present disclosure, an apparatus and/or method is designed for use with an array of sensors to collect user signals indicative of kinematics and/or electrophysiology, corresponding to an observed region of the user. The apparatus (which may be characterized as a computer-implemented method) involves computer processing circuitry to use the signals as collected by the array of sensors bycarrying out certain computer-programmed aspects or steps such as: training and/or using an algorithm to leam and/or identify a pattern of physiological signals manifested by the use; and generating, via the algorithm, data knowledge about relationships between the observed region and at least one unobserved region of the user that does not include the observed region, and therein extrapolating further physiological signals of the user, corresponding to the unobserved regions.
[0007] Certain other examples of the present disclosure, which are consistent with and may build on the above-discussed aspects, are directed to the computer processing circuitry’ being configured (e.g., programmed) to generate, based on the signals as collected by the array of sensors, output data indicative of the user concerning at least one of kinematics and electrophysiology of the at least one unobserved region of the user. The computer processing circuitry may also or alternatively use the signals as collected by the array of sensors by training the algorithm to leam and/or identify a pattern of physiological signals manifested by the user, wherein in some instance the algorithm training is referred to as generative model training.
[0008] Other specific examples of the present disclosure, also consistent with and may build on the above-discussed aspects, are directed to the array of sensors being configured to collect physiological signals indicative of: electrophysiology corresponding to the observed region of the user; and/or kinematics, corresponding to the observed region of the user. Nonlimiting examples of other collected and/or collectable physiological signals according to specific aspects to the present disclosure are signals corresponding to: EMG (electromyography), EEG (electroencephalogram), ECG (electrocardiography), EGG (electrogastrography), and EOG (electrooculography).
(0009] In certain example embodiments, aspects of the present disclosure involve methods and related apparatuses concerning a generative multi-array sensory system capable of predicting unseen areas by utilizing a reduced (minimal or optimized) set of sensory data (e.g., data collected via the primary user sensors). By understanding the inherent locational correlations in bio-signals, muscle activities, brain waves, and/or pulses, methods and related apparatuses according to the present disclosure are designed to extrapolate signals associated with unknown regions (which are not being monitored by electrode-based sensors). Particularly for wearables, methods and related apparatuses according to the present disclosure include stretchable sensors used in this regard and these sensors attach seamlessly with the skin, thereby acting to lower undesirable artifacts and to improve signal noise. In more particular example implementations (e.g., for experimental proof-of-concept), example algorithms according to the present disclosure intentionally discard over half of the originally-collected (input) data to obtain key (informative) signals for the application, and the model may reconstruct the information that is missing. In this manner, such example algorithms may be used to effectively simulate a high-density' sensory network with significantly fewer sensors.
(0010] In yet a further example, the above type of method further includes training the computer-executable algorithm to identify the pattern of physiological signals by' using a learning model that operates based on an auto encoder framework and that includes an encoder circuit module and a decoder circuit module that are cooperatively arranged to process raw data, from a number of training electrodes used during training, to configure the computer-executable algorithm to identify, after the computer-executable algorithm is trained, the pattern of physiological signals while the array of sensors for said step of collecting signals utilizes less than one half of the number of training electrodes used during training.
[0011] In the above and other example embodiments, the present disclosure is directed to apparatuses (e g., system, circuitry, components, and/or materials) and methods which provide certain advantages and/or improvements over known approaches. Examples of such advantages and/or improvements, depending on the implementation, may include. 1) Reduced complexity: designs of the present disclosure reduce or diminish the need for high- density sensory networks by leveraging locational correlations, and streamlining both the physical setup and computational processing. Reducing the number of required sensors may be achieved without compromising on data accuracy which can lead to more affordable devices. 2) Enhanced Comfort: the integration of stretchable sensors in apparatuses of the present disclosure offers users an unobtrusive, comfortable fit, making the device more user- friendly and wearable for extended durations. 3) Low Noise Signal Acquisition: designs of the present disclosure ensure reduced or minimal interference, motion artifacts, and contact impedance, ensuring cleaner data acquisition. 4) Versatility: designs of the present disclosure are adaptive and can be integrated into many applications ranging from health monitoring to gesture recognition and language translation.
[0012] In more specific examples related to the above methodology and/or devices, exemplary aspects of the present disclosure involve wearable applications, some of which integrate the benefits of stretchable sensors and low impedance electrodes to enhance the efficacy of the above-characterized system types and then implement a generative algorithm to process signals from one or more of these t pes of sensors. Such sensors offer superior conformity to the skin, and they also facilitate the acquisition of low-noise signals that enables higher performance of the generative network by reducing motion artifacts and contact impedance. In such specific examples, the signals collected from such sensors are fed into the generative algorithm, which is trained to intentionally omit a significant portion of the collected data.
[0013] Also according to the present disclosure, other specific aspects relating to the above-discussed examples concern the array of sensors. Regarding such a sensor array (or set of sensors which may not necessarily be geographically-organized on a portion of the user): at least one of the sensors in the array may be characterized as including at least one of: strain pressure sensors, photo-diode based sensors, and ultra-sonic sensors; the array of sensors can include a body -adhering polymer based substrate layer that is stretchable and elastic, and integrated with a conductive gel that serves to lower impedance; the array of sensors can be protected by a rubber barrier layer to mitigate or prevent solvent induced swelling; and the array of sensors can include a liquid metal electrode layer which is both stretchable and micro-patterned, encapsulated by a photo-patterned synthetic rubber which leaves only the electrode areas uncovered to facilitate skin contact and signal measurement.
[0014] Various specific example embodiments and aspects of the present disclosure can be implemented as advancements in the development of wearable sensing systems by enabling comfortable wear of less complex sensing systems which, in turn, can produce high- quality data previously requiring high-density sensing arrays and by simplifying corresponding data processing to result in lower power consumption and faster processing time.
[0015] The above discussion is not intended to describe each aspect, embodiment or every implementation of the present disclosure. The figures and detailed description that follow also exemplify various embodiments.
BRIEF DESCRIPTION OF FIGURES
[0016] Various example embodiments, including experimental examples, may be more completely understood in consideration of the following detailed description in connection with the accompanying drawings, each in accordance with the present disclosure, in which: [0017] FIGs. 1 A-1B depict a system having a generative electronic sensory network (G- SNET) for kinematic predictions, according to certain exemplary aspects of the present disclosure, with: FIG. 1 A highlighting exemplary sets of raw-channel electrodes and pseudo electrodes (depicting the data from the raw-channel electrodes after processing), and FIG. IB showing a first limb-specific example application and embodiment with an exploded view of the limb-specific example embodiment in the form of limb-wearable device (e.g., a strip, band, patch, and/or w ireless smartw atch;
[0018] FIGs. 2A-2H depict a stretchable sensory array for high-quality dataset generation with: FIGs. 2A. 2B, and 2C showing exemplary sets of electrodes according to one or more specific embodiments, FIG. 2D showing an example composition of an electrode, FIGs. 2E, 2F and FIG. 2G showing graphs depicting performance-related data of one or more arrays shown in FIGs. 2A-2C and FIG. 2H showing an example block layout with an arrangement of circuit components as a system-level example;
[0019] FIGs. 3A-3E depict post-training of a G-SNET with: FIG. 3 A showing processing of data from one or more arrays shown in FIGs. 2A-2C, FIG. 3B showing exemplary7 flow' for generative model training, FIG. 3C showing performance-related data of the one or more arrays, FIG. 3D showing a flow diagram of a G-SNET, and FIG. 3E which is a diagram illustrating how post-training is used for prediction;
[0020] FIGs. 4A-4E depict approaches for predicting various body kinematics through a G-SNET combined device with: FIG. 4A showing a wireless module (e.g., band, patch or other user device such as w atch), FIG. 4B showing a flow diagram of post-training as may be used for prediction, FIG. 4C showing an application of the module of FIG. 4A. FIG. 4D showing a related application of the module of FIG. 4A and with predictive data being plotted, and FIG. 4E show ing output data plotted in a pair of related graphs; [0021] FIG. 5 is a more-specific example of a generative model structure for postlearning for a sign-language translation application;
[0022] FIGs. 6A-6C depict a schematic of a system that is configured to utilize latent vector(s) for post-training American sign-language translation results with: FIG. 6A showing a flow diagram of an example embodiment, FIG. 6B showing known hand/finger signs according to the American sign language, and FIG. 6C showing results of the American sign language as processed by the data flow of FIG. 6A; and
[0023] FIGs. 7A-7F depict various representative sets of results from exemplary' data- processing. according to certain exemplary' aspects of the present disclosure, and with each set showing live or raw (e.g., EMG-captured) data, masking (e.g., random masking) of the live or raw data, and the full data set (or signals) including data as extrapolated for the unseen regions, also according to experimental example embodiments and aspects of the present disclosure.
[0024] While various embodiments discussed herein are amenable to modifications and alternative forms, aspects thereof have been shown by way of example in the drawings and will be described in detail. It should be understood, however, that the intention is not to limit the disclosure to the particular embodiments described. On the contrary', the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure including aspects defined in the claims. In addition, the term “example’7 as used throughout this application is only by way of illustration, and not limitation.
DETAILED DESCRIPTION
[0025] Aspects of the present disclosure are believed to be applicable to a variety of different types of apparatuses, systems and methods involving devices characterized at least in part by sensors such as, but not necessarily limited to, user wearable-type sensors, and to using and/or exploiting sensor-based data for increased understanding of sensory regions of a user. While the present disclosure is not necessarily limited to such aspects, an understanding of specific examples in the following description may be understood from discussion of such specific contexts as disclosed hereinbelow.
[0026] Exemplary contexts of the present disclosure are directed to examples involving generative multi-array sensory systems and methods capable of extrapolating unseen areas based on a minimal set of sensory data. In such examples, the system implements an algorithm to identify patterns between sensory activations from varied locations, as exemplified by use of bio-signals (e.g., indicative of muscle activities, brain signals, pulses, or the like) which can exhibit locational correlations. Leveraging these correlations, one or more generative algorithms, according to the present disclosure, are used to predict a user’s signal activities in unseen areas (e.g., not having sensory components attached thereto) based on algorithm training about relationships between extrapolated regions.
[0027] In one type of example embodiment consistent with aspects of the present disclosure, an apparatus is designed for use with an array of sensors to collect user signals indicative of kinematics and/or electrophysiology, corresponding to an observed region of the user. The apparatus (which may be characterized as a computer-implemented method) includes computer processing circuitry to use the signals as collected by the array of sensors by carrying out certain computer-programmed aspects (or steps) such as: training and/or using an algorithm (that is trained) to identify a pattern of physiological signals manifested by the use; generating, via the algorithm, data knowledge about relationships between the observed region and at least one unobserved region of the user that does not include the observed region; and extrapolating, in response to the generated data knowledge about relationships, further physiological signals of the user, corresponding to the unobserved regions (e g., with the further physiological signals being predicted).
[0028] Consistent with the above type of example embodiment, a more specific example embodiment is directed to an apparatus which has the computer processing circuitry to train the algorithm based on a set of raw signals acquired from a number of training electrodes which are associated with at least the unobserved region of the user signals. Once trained and while the array of sensors uses a number of user-coupled electrodes coupled to the observed region, the computer processing circuitry is to use the algorithm to identify the pattern of physiological signals by collecting the user (electro)physiological signals from the user- coupled electrodes of the array of sensors, wherein the number of training electrodes is at least twice the number of user-coupled electrodes. Consistent with the above discussion, the computer-executable algorithm may be trained to identify the pattern of physiological signals by using: a set of M electrodes to acquire signals associated with said at least one unobserved region of the user; deriving at least one latent vector (corresponding to more pertinent aspects of the collected data) from the carried signals, wherein the computer-executable algorithm is trained to identify' the pattern of physiological signals based on said at least one latent vector; and the step of generating knowledge about relationships between the observed region and at least one unobserved region uses not more than N electrodes in the array of sensors for said step of collecting signals, wherein M and N are positive integers and M is at least three-to- four times greater than N. In each of various more-specific more detailed examples, the ratio of the number (M) of training electrodes to the number (N) of user-coupled electrodes is set based on the design and specific application needs (such ratios may be 32/4, 64/6, 41/4, 48/8, etc.).
[0029] Also consistent with the above type of example embodiment, another more specific example embodiment is an apparatus which includes the computer processing circuitry being associated with a first software-executable module to train the algorithm by responding to raw signals from electrodes associated with the observed region of the user. The computer processing circuitry' is to use the trained algorithm, via a second softwareexecutable module, for the extrapolation of further (electro)physiological signals of the user or information related to the further physiological signals by responding to a set of user- coupled electrodes that are physically coupled to the observed region of the user without concurrently using any further electrodes coupled to the unobserved region of the user.
[0030] As may be apparent from the above examples, the computer processing circuitrymay be implemented as a single integrated computer processing circuit (e.g., with both of the first and second software-executable module), or alternatively, the computer processing circuitry may be implemented as distinct integrated computer processing circuits. As distinct integrated computer processing circuits, one of the computer processing circuits may be configured with the first software-executable module, and another of the computer processing circuits may be configured with the second software-executable module. In this manner, the multiple modules and/or the multiple computer processing circuits, while referred to as computer processing circuitry, may be manufactured and/or configured (e.g.. programmed such as by training) separately (e.g., separated physically and/or in time).
[0031] Other (more-specific) exemplary aspects of the present disclosure involve wearable applications (e.g., spanning the above-noted observed region), some of which integrate the benefits of stretchable sensors and low impedance electrodes to enhance the efficacy of the above-characterized system types. Further, these types of sensors and/or electrodes offer superior conformity to the skin, and they also facilitate the acquisition of low-noise signals that enables higher performance of the generative network by reducing motion artifacts and contact impedance.
[0032] Consistent with the above aspects, such a manufactured device or method of such manufacture may involve aspects presented and claimed in U.S. Provisional Application Senal No. 63/599,190 filed on November 15, 2023 (STFD.459P1 / S23-326) with Appendices A-B, to which priority is claimed. To the extent permitted, such subject matter is incorporated by reference in its entirety generally and to the extent that further aspects and examples (such as experimental and/more-detailed embodiments) may be useful to supplement and/or clarify.
[0033] Accordingly, in the following description various specific details are set forth to describe specific examples presented herein. It should be apparent to one skilled in the art, however, that one or more other examples and/or variations of these examples may be practiced without all the specific details given below. In other instances, well known features have not been described in detail so as not to obscure the description of the examples herein. For ease of illustration, the same connotation and/or reference numerals may be used in different diagrams to refer to the same elements or additional instances of the same element. Also, although aspects and features may in some cases be described in individual figures, it will be appreciated that features from one figure or embodiment can be combined with features of another figure or embodiment even though the combination is not explicitly shown or explicitly described as a combination.
[0034] In related experimental efforts, example embodiments involve processing the collected signals from such sensors and then feeding the signals into a generative algorithm which, according to the present disclosure, may be implemented (or trained) by intentionally omitting (or masking) a significant portion of the collected data and using, by masking, only the so-called latent data. In this regard, the computer processing circuitry and the algorithm collectively correspond to a pre-trained encoding network configured based on a defined or an iteratively -refined latent vector derived from a set of raw data carried or derived from a set of electrodes, or signal channels, which are associated with the observed user region and to be used to predict a pattern of physiological signals associated with the observed user region and said at least one separate unobserved region.
[0035] The generative algorithm may be trained by such latent data by learning to fill in data missing from the multi-array signals. This pre-trained model is then merged with a simplified device having fewer sensing channels, enabling the created low-dimensional vector that provides information equivalent to that from a multi-sensing array. The lowdimensional latent vector can then be used by a generative network, also according to one example of the present disclosure, as a refining or concise feature of spatiotemporal muscle activities and subsequently harnessed for use in various (and sometimes diverse) applications, such as American Sign Language Translation (ASL) and gait phase detection.
[0036] As one more-specific example of the above-described embodiment, a circuit- implemented method includes collecting, via an array of sensors, signals (e.g., indicative of kinematics and/or electrophysiology) corresponding to an observed region of a user, and then processing (extrapolating) and/or using the collected signals to identify a pattern of physiological signals manifested by the user, which corresponds to one or more unobserved user regions that does not include the observed region.
[0037] In specific more-detailed embodiments, the circuit-implemented method is carried out by one or more computing data-processing (or CPU) circuits (sometimes referred to as computer processing circuitry). The method and computer processing circuitry can include an algorithm collectively configured as a pre-trained encoding network, which may use a G-SNET (generative electronic sensory network), that is to define or iteratively -refine latent vector(s) ("the latent vector information” ) derived from a set of raw data earned from a set of N electrodes (e.g., coupled to N signal channels), coupled to the observed user region. In certain implementations, the latent vector information is processed via an M-channel encoder and used to generated the relationships between the observed region and at least one unobserved region, and wherein M and N are positive integers and depending on the particular relationships needs, M is several times greater than N (e.g.. M is at least 3-4 times greater than N, and in some instances such as those involving high-speed data processing such as in high-speed neural network, M is in a range of several times greater than N to 10 or 100 times greater than N). In certain experimental proof-of-concept implementations, M and N are positive integers with M being in a range from 20 to 133. N being in a range from 7 to 64 with M at least 3 times greater than N. It is appreciated that the N electrodes/channels need not be in a specially-formed array, but rather one in which each of the relative locations is known.
[0038] FIGs. 1 A-1B depict such a G-SNET system for generating kinematic predictions, according to certain exemplary aspects of the present disclosure. FIG. 1 A highlights an exemplary set of raw-channel electrodes (4 EMG pads in the observed area) and pseudo electrodes spanning regions outside the observed regions, and with a latent vector being processed to the right of FIG. 1 A. FIG. IB shows a first limb-specific example application (6 EMG pads in an observed area of the leg or arm) with an exploded view of the limb-specific example device being a limb-wearable device (e.g., a strip, band, patch, and/or wireless smartwatch). The latent vector and related processing of the small-channels raw data is used to create an effective pseudo multi-array EMG with, in these illustrated examples, a range of several tens of pseudo electrodes (more with multiple pseudo EMG arrays).
[0039] Specific to wearable applications, to enhance the efficacy of the abovecharacterized system, this type of system can be implemented to integrate the benefits of stretchable sensors and low impedance electrodes. These sensors/electrodes not only offer superior conformity to the skin, but they also facilitate the acquisition of low-noise signals that enables higher performance of the generative network by reducing motion artifacts and contact impedance, also facilitate the acquisition of low-noise signals. These collected signals are then fed into the above-characterized type of generative algorithm where, as noted above, the raw-signal information may be intentionally omitted. With repeated learning cycles (e.g., refining the above-discussed vector(s)), the model learns to fill in this missing data from the multi-array signals. This refined, pre-trained model is then merged with a device having a subset of channels (e.g., significantly fewer channels), enabling the created pseudo multiarray signals. The latent vector is then used at the middle of the generative network which is a concise form of the input, emphasizing one or more features specific to the application. This information can then be harnessed for any of different and/or diverse applications, such as American Sign Language translation and gait phase detection.
[0040] Consistent with the above aspects, the model next learns to fill in this missing data from the multi-array signals. This pre-trained model is then merged with a simplified device having fewer sensing channels, enabling the created low-dimensional vector that provides information equivalent to that from a multi-sensing array. As used in this example, the low-dimensional latent vector of the generative network is a concise (and in some examples, an important) feature of spatiotemporal muscle activities. Then this information is harnessed for diverse applications, such as American Sign Language Translation (ASL) and gait phase detection (as in FIG. IB). This work presents a new paradigm for wearable sensing system development and enables comfortable wear of less complex sensing systems which can produce high-quality data previously requiring high -density sensing arrays. This approach also simplifies data processing which should result in lower power consumption and faster processing time.
[0041] FIGs. 2A-2H depict a stretchable sensor array for high-quality dataset generation. More specifically, FIGs. 2A-2C show a structure with an exemplary (non-limiting) set of 32 channels, according to one or more specific embodiments, with corresponding electrodes. As illustrated in FIGs. 2A-2C. the electrodes may be viewed as arranged in one or more arrays (e.g., a 4x8 electrode array, a pair of 4x4 electrodes, etc.) for implementation of a 32- electrode / 32-channel EMG structure.
[0042] FIGs. 2D and 2E respectively show: example (molecular) compositions (FIG. 2D) of an electrode (such as in FIG. 2B) with and without a PEDOT layer on the exposed portion of the electrode, and a graph of advantageous performance parameters (FIG. 2E) showing much lower impedance at relevant signal frequencies corresponding to the exposed portion with and without such a PEDOT layer. FIGs. 2F and 2G show respective graphs depicting further performance-related data corresponding to the exposed portion with and without such a PEDOT layer, with the electrode of FIG. 2G having an Ag/AgCl composition. [0043] FIG. 2H shows a detailed example of an electrode structure which may be viewed as corresponding to the structure shown in FIGs. 2B and 2C. In this detailed example, the electrode structure is shown to include an example block circuit layout having X-channel stretchable EMG structure including a FFC (flexible-flat connector) connecting to wireless module, wherein for this particular experimental example, X is equal to 32 (one channel for each skin-contacting electrode). The signals from the electrodes are carried by a cable to circuitry (e.g., embedded as an integrated part of a user- w orn patch) which, for illustrative purposes, is shown to include an MCU (CPU-based master control unit), a multiplexer (MUX) for selectively feeding the channeled signals to the MCU, an amplifier for amplifying the signals before and/or after being processed by the MUX, and where a wireless connection is appropriate, a wireless (e g., Bluetooth) transponder for transmitting the signals. This transmission may be for further processing such as from the user-wom patch to a more robust CPU circuit which is configured to cany' out the processing as discussed above in connection with FIGs. 3A-3E, FIGs. 4A-4E, 5 and FIGs. 6A-6C.
[0044] In one example, the pre-training generative algorithm is used to reconstruct the original detailed temporal and spatial patterns of muscle activity signal with high fidelity’ using minimal sensor inputs. In many examples and applications, achieving high-accuracy levels from the generative algorithm is contingent upon the quality of the initial training dataset. For such examples, the creation of a high-quality dataset is facilitated, according to the present disclosure, by engineering a fully stretchable multi-electrode array EMG device, such as shown in FIG. 2A.
[0045] Referring again to FIG. 2A and the side view (lower) portion of FIG. 2B, the device's layer structure is depicted in an exploded view. The layer structure includes several layers: a substrate layer (e.g., PDMS), a protective layer (e.g., NBR), an electrode (e.g., EGain), a gel (e.g.. PEDOT), and an encapsulation layer (e.g., SBS). The PDMS layer provides a thin substrate with sub-millimeter thickness for flexible handling and easy adherence to various body contours. The NBR layer acts as a barrier to protect the PDMS from solvent-induced swelling during the fabrication of the multiple layers. The PDMS and/or NBR layer(s) may be used as a substrate for the other depicted layers. The EGain liquid metal electrode, which is both stretchable and micro-patterned, is then applied, followed by a highly conductive PEDOT gel that serves to lower impedance. The structure may include a gold (Au) layer between the NBR and EGain layers. In one example, the device is completed with a photo-patterned SBS layer that leaves uncovered only the electrode areas (e.g., one of which is shown as an upper portion of a PEDOT in FIG. 2B), to facilitate skin contact and signal measurement. FIG. 2B and 2C illustrate various views (including the top and side views) of the exemplary sensor array and the fabricated stretchable array.
[0046] In connection with certain experimental examples, a chosen material set includes a conductive and adhesive gel composed of acrylamide (AAm) integrated with a PEDOT:PSS physically cross-linked conducting polymer network. This enhances the electrical and mechanical properties, improving impedance characteristics and minimizing noise from movement. Surprisingly, such modifications to the AAm poly mer network with the addition of PEDOT:PSS have resulted in a significant reduction in interfacial impedance when interfaced with phosphate buffered saline (PBS), as demonstrated in FIG. 2E. The modified gel's impedance remains low- at 100% strain at 50 Hz (FIG. 2F). Moreover, upon performing Welch's power spectral density estimate, the developed sensor array exhibits a higher power density compared to a standard Ag/AgCl electrode, as shown in FIG. 2G.
[0047] FIGs. 3A-3E show, consistent with the above-discussed aspects of FIGs. 2A-2G, further aspects and methodology of an exemplary sensory and information-processing structure. FIG. 3A show s processing of data from one or more arrays such as shown in FIGs. 2A-2C. and FIG. 3B shows exemplary high-level methodology, according to the present disclosure, involving data flow to be carried out by generative model training circuitry. Once trained, the sensory and information-processing structure can predict how pseudo sensors (covering unseen areas) are acting.
[0048] In one example, the methodology deliberately masks a significant (e.g., predominant) amount of the complete sensor data. This masked data is then fed into the G- SNET, where learning parameters are adjusted to reproduce the initial signal from the masked data. To bolster the model's resilience against varying sensor attachment positions and orientations, the methodology can optionally incorporate randomly -distributed masking. [0049] Particular masking involved for this exemplary methodology is shown in FIG.
3C, via two example sets of signal-channel images with each set showing: data flow from the originally-collected (raw) electrode-obtained signals, to the masked signals, and on to the expanded set of (pseudo) information (with the darker spots corresponding to information from the originally -collected signals which is selected via the masking for analysis and/or use). In connection with each sample set of FIG. 3C, via the masking occurring during the training cycles, the G-SNET successfully reconstructs the original signals using the masked inputs, wherein the particular examples of FIG. 3C have intentionally masked 70% to 90% of the complete sensor data.
[0050] In connection with other examples such as generally depicted in connection with the left side of FIG. 3B, such intentionally omitting or masking the collected data may leave less than a half or one third of the data as becoming the latent data. In some examples, such as iterative processing of the collected data to refine the accuracy of the latent data and/or small targeted (focused) regions, the latent data may be: generally less than 70% (or 50%) and greater than 10%, in other cases in a range from 40% to 65%, etc.
[0051] As depicted in the middle portion of FIG. 3B, the learning-model architecture uses an autoencoder framework that is principally divided into two sections: an encoder and a decoder. The encoder translates the intentionally random masked (e.g., 70-90%) signals into a latent representation. Subsequently, the decoder revives the original signal from this latent space.
[0052] . FIG. 3D shows a more-detailed example of each of the encoder and the decoder sections of FIG. 3B, with each coding section including layers of logic circuitry' (e.g., a circuit including at least one programmed CPU and/or programmable logic array). The layers include: multi-head attention, normalization, and linear layers. The decoder will then generate reconstructed patches from the output of the encoder. After training of the autoencoder, the latent vector is then used in this example by feeding it into another dense layer for further classification (e.g., American Sign language translation, gaiting features, etc.). Utilizing latent vectors helps to avoid reconstruction noise of the decoder, generalization of the input, and computation efficiency of its low-dimensional property.
[0053] Consistent with the above-described experimentation aspects of FIGs. 3A-3E, pre-training of example generative algorithms according to the present disclosure have demonstrated successful outcomes. These experimentation aspects are supported by a sample study conducted on such methodology' involving 32-channel EMG signals attached above the wrist. Due to the high computational cost associated with spectrogram conversion, the RMS values of the EMG signals were used. Collected were 32-length time windows from the 32- channel EMG array, which enabled the system to generate a 32 x 32-sized EMG signal image (e.g., FIGs. 3A-3B). This acquired signal is then further used as inputs into an autoencoder network, such as in FIGs. 3B and/or 3D, for training the generative model.
[0054] In connection with this study, the raw signals captured by the 32-channel sensory- array were subjected to post-processing, which involved the calculation of root mean square (RMS) values across 32-time windows, using a sliding window with a size of 10. To mitigate the high computational demands associated with conventional spectrogram conversion, certain of the more-specific/experimental example embodiments employed RMS values as a simplified representation of the EMG signals. This processing technique facilitated the establishment of a temporal correlation within the 32-time window across all 32 EMG channels. This facilitates construction of an EMG signal tensor with dimensions of 32 x 32. It will be appreciated that the arrangement of 32 channels, and corresponding electrode array, is a non-limiting example for experimental/proof-of-concept purposes and other arrangements of channels and corresponding electrode array(s) may be used in a similar manner.
[0055] For subsequent application-specific post-training, an exemplary device was implemented as a compact, portable wireless EMG device having 6 channels. This device's outputs were then introduced to the pre-trained encoder network as in FIG. 3E. Following the training of the autoencoder, the latent vector was used by channeling it through an additional dense layer to enable signal classification. The utilization of latent vectors sidesteps the potential for reconstruction noise introduced by the decoder, facilitates the generalization of inputs, and enhances computational efficiency due to their lower-dimensional attributes. [0056] FIGs. 4A-4E depict prediction of various body kinematics through a G-SNET combined device with: FIG. 4A showing a wireless module (e.g., band, patch or other user device such as watch), FIG. 4B showing a flow diagram of post-training as may be used for prediction, FIG. 4C showing an application of the module of FIG. 4A, FIG. 4D showing a related application of the module of FIG. 4A and with predictive data being plotted, and FIG. 4E showing output data plotted in a pair of related graphs.
[0057] As one of various application-specific examples, aspects of the present disclosure can be implemented with the G-SNET utilized for kinematic prediction. As a demonstration for practical deployment, a compact 6-channel wireless EMG watch, depicted in FIG. 4A, was employed to acquire signals. This reduced-channel data was fed into the encoder to generate a latent vector, which w as subsequently processed for American Sign Language (ASL) gesture prediction. The initial dataset comprised EMG recordings corresponding to the 26 hand gestures representing the ASL alphabet, from A to Z. Utilizing the pre-trained encoder, such experimental modeling is refined to enhance its predictive capabilities for ASL interpretation.
[0058] To extend such a system's applicability to different body locations, the device is attached to the user’s leg (e.g.. at the call) to ascertain various gait phases. Monitoring gait kinematics continuously provides insights into potential musculoskeletal disorders, thereby aiding in the identification of risk factors for conditions such as falls, the need for rehabilitation, preventive measures for injuries, and optimization of athletic performance. [0059] For the initial training dataset, 6-channel EMG signals are recorded in conjunction with ground reaction forces (GRF) throughout a normal gait cycle, as illustrated in FIG. 4C. Key gait states (e g., heel strike, mid-stance, and toe-off) are identified using GRF data, as shown in FIG. 4D. These states were then input into the pre-trained G-SNET encoder with a 32-channel stretchable EMG array for additional post-training. As demonstrated in FIG. 4E, the above-characterized model successfully predicted continuous gait phases throughout the gait cycle.
[0060] In this above experimentation, results for signal reconstruction w ere gathered with the final reconstructed signal from the masked information. For 25% of the information (75% masked, 8-channel information), the reconstructed signal was successful in accurately reconstructing the original signal waveforms and amplitude. Interestingly, even with only 10% of the information (90% masked, 3~4 channel information), the reconstructed signal was able to capture and reconstruct the original w aveforms.
[0061] Various applications of the present disclosure may be used to more specifically- configure (application-specific) different ones of the exemplary systems disclosed herein. Among many such applications is sign-language (e.g., translation and/or prediction). As a specific example, the systems of FIG. 5 and FIG. 6A may be configured for the American sign-language, as represented in FIG. 6B.
[0062] FIG. 5 is another example of a generative model structure for post-learning for a sign-language translation application, as a more specific example of the generative model structure shown in connection with FIGs. 3A-3E. As with the methodology of FIGs. 3A-3E, the upper portion of FIG. 5 may be implemented with a particular encoder, as shown in connection with FIGs. 3B and 3D, before the latent vector data is developed at the output of the encoder. In this example and as at the left of the encoder, raw data from 6 channels (the highlighted rows in the image of FIG. 5) is shown as being used to develop the masked signal, which may be generated by decomposing data from patches with embedded circuity (including the skin-contacting electrodes). From the output of the encoder, the latent vector data may then be developed, for example, using different layered (software-based) CPU modules. As shown in this illustrated example, such different layered CPU modules may include a first layer for batching and/or normalization of the data derived from the signals carried on each channel, a second layer for data linearization to better reflect continuity and/or relationships in areas between actual electrodes and virtual (or pseudo electrodes), and a third (softmax) layer for normalization of the data into various output classes (e.g., O1, O2, O3, O4 and O5, as shown to the right of FIG. 5. This may be realized by converting each vector (channel) of numbers into a vector of probabilities, where the probabilities of each value are proportional to the relative scale applicable to each value in the vector. In implementations which use, as the logic circuitry, a neural CPU network, the softmax layer (or function) may be used to normalize the output and therein provide a probability distribution over predicted ones of such output classes; for example, for a given set of output values (e.g., O1 - O5), one for each class in the classification task, the softmax function may be used to normalize the outputs, by converting them from weighted sum values into probabilities that sum to one. Each value in the output of the softmax function is interpreted as the probability of membership for each class.
[0063] The example schematic/flow diagram of FIG. 6A corresponds to a system configured for utilizing a set of one or more latent vectors for post-training sign-language translation results. In this experimental effort, a pre-trained encoding netw ork w as adapted to reconstruct a system with a full 32-channel dataset from a 6-channel input (as the masked signal). To achieve this, the 6-channel data was input into an encoder, thereby generating a latent vector. This vector was then introduced to a linear layer for finger spelling prediction. Initially, 15 finger letters (of the American sign-language as shown in FIG. 6B) ranging from A to N were captured by the system. Using the pre-trained encoder, the model was further refined for language prediction. This subsequent training, which benefits from smaller datasets and can be expedited, can be termed as transfer learning or meta-leaming. [0064] FIG. 6C shows the results of the American sign language, in the form of a confusion matrix, as processed by the data flow and system of FIG. 6A. The confusion matrix has along its vertical axis, as the true object, each of 15 rows respectively corresponding to the 15 finger letters (e.g., as indicated by a unique motion or (near) position that corresponds to one or more of the 15 letters). The horizontal axis of the confusion matrix shows 15 rows also corresponding to the 15 finger letters but as predicted by the system of FIG. 6A. The shading map, to the right of FIG. 6C, shows the degree of accuracy associated with the prediction as a score with the darker entries showing the higher score and accuracy and, therefore, higher degree of confidence. For lower scores, the algorithm may be further trained for refinement towards higher scores (with acceptable levels of accuracy) and/or the system may generate user-feedback (e.g., visual, tactile and/or audible) on such a wearable device to prompt the user to repeat, and where appropriate, continue with signing as inputs to the system. Such feedback may be implemented using circuitry integrated into the wearable device (e.g., a vibration circuit or display/audible alert as known in many smartwatches). [0065] Also among the various applications, according to the present disclosure, are implementations of such a system for processing data for different users (aka individuals). Since a latent vector has rich information density and inherent generalization, it offers enhanced resilience to a variety of inputs. Given that biological structures exhibit consistency across individuals, a latent space trained using a select yet varied group of representatives (based on gender, weight, height) is well-positioned to generalize corresponding sets of individual data as one set of representative data that is useful for a plurality of users. Thus, by using a high-resolution map trained on this small and diverse cohort, the system can effectively generalize across different individuals.
[0066] Through experimental/more-specific examples consistent with example embodiments and related aspects of the present disclosure, and in connection with six sets of depicted data as shown respectively in FIGs. 7A-7F, results for signal reconstruction and ASL prediction are presented. For each of the six sets, a final reconstructed signal is shown from the masked information (with the masking depicted in the center of each set), with the electrode-captured raw data shown to the left, and the processed and extrapolated data shown on the right. The six sets may be different in various ways; as examples, each set may use: a different number, type and arrangement of raw electrodes; and different types and locations of masks (e.g., with some being randomly chosen and/or placed).
[0067] One or more of the above aspects may be implemented in a variety of other more specific experimental/more-specific examples. One such example involves implementation for use by different individuals (or users). Since a latent vector has rich information density and inherent generalization, it offers enhanced resilience to a variety of inputs. Given that biological structures exhibit consistency across individuals, a latent space trained using a select yet varied group of representatives (based on gender, weight, height) is well-positioned to generalize. Thus, by using a high-resolution map trained on this small and diverse cohort, the system can effectively generalize across different individuals.
[0068] Other such examples involve different types of electrophysiological sensors such as electromyography (EMG), electrocardiography (ECG), electroencephalogram (EEG), electrogastrography (EGG) and electrooculography (EOG). EMG tracks various muscle activities throughout the biological body, encompassing motions in the hands, legs, neck, and back. With the assistance of G-SNET, this can be expanded to predict a range of motions for applications like sign-language translation, gait-phase detection, and body pose detection. Key applications include rehabilitation (muscle function monitoring), sports (performance analysis), human-machine interfaces (prosthetic control, gesture recognition, silent speech interface), and health monitoring (fall detection).
[0069] An ECG, especially a multi-array ECG, offers a comprehensive view of the heart's electrical activity from multiple viewpoints. G-SNET can potentially condense the standard 12-lead electrode measurement system into a more compact channel system. Notable applications lie in health monitoring (heart rate, arrhythmia detection) and sports (performance and recovery monitoring).
[0070] Multi-array EEGs have applications in cognitive neuroscience and braincomputer interfaces. Typically requiring 256 or more channels for better spatial resolution and detailed brain activity mapping, G-SNET streamlines this by leveraging merely 10-20% of the original electrode count, ensuring enhanced user comfort. Applications span health monitoring (sleep disorders, epilepsy monitoring, mental health) to brain-computer interfaces (like prosthetic control), and also for applications involving probes (or electrodes) being inserted for uses in brain neural recording, with the brain neural recording benefiting from this for deep brain electrodes used for stimulation and accurate recording. These and other implementations according to the present disclosure may be applicable to both wearable EEG as well as implanted high-density EEG probes. [0071] EGG offers a non-invasive approach to monitor the stomach's muscle activity. With G-SNET, fewer electrodes are needed to predict a broader range of gastrointestinal activities. This technology has applications in health monitoring (detecting gastrointestinal disorders) and in research to study the correlation between gastric activities, stress, and sleep. [0072] EOG captures the comeo-retinal standing potential that exists between the front and the back of the human eye, making it possible to detect eye movements. G-SNET aims to achieve accurate readings with fewer electrodes, eliminating the need for extensive coverage around the eye. The potential applications of this technology extend to the human-computer interface (communication, wheelchair navigation, and gaze tracking in AR/VR), and healthcare (e.g., diagnosing retinal diseases and optic nerve disorders).
[0073] Other more specific experimental/more-specific examples related to one or more of the above aspects may be implemented with any of a variety of different types or combinations of sensors. Such sensor types include, among others, strain/pressure sensors, photo-diode based sensors, and ultra-sonic sensors, and such sensors need not necessarily be wearable (e.g., they may be temporarily applied or coupled to the user for temporary measurements).
[0074] Certain (e.g., wearable) strain/pressure sensors detect deformations (stretching, bending, twisting) and convert them into electrical signals. These sensors can discern changes in pressure and stretch from various sources. Traditional methods entail placing sensors on each joint and muscle. In contrast, G-SNET will discover areal strain/pressure distribution correlations, simplifying the sensory' complexity.
[0075] Applications encompass health monitoring (blood pressure, respiration rate, pulse), prosthetics (prosthetic limbs, gait analysis), sports (posture monitoring), organ movement (e.g. stomach, gut), and human-machine interfaces (gesture recognition, facial movement recognition, body movement recognition, virtual reality). In yet further applications, such methodology and corresponding structures are applicable to obtaining measurements with one or more probes (having one or more electrodes referring to or at the end of. each probe) inserted in an organ and to measure maps of electrophysiological signals both on a surface of the body as w ell as on a surface (or membrane) of internal organs, such as heart, bladder or uterus. Further, such methodology' and corresponding structures can also be applied to implantable probes used for electrophysiology measurements.
[0076] Photo-diode based sensors translate variations in the light absorption reflected by the body. When organized in a multi-array pattern on wearable devices, their sensitivity, spatial resolution, and functionality are elevated. G-SNET allows for higher sensory resolution with few channels when gauging heart rate, oxygen saturation (SpO2), glucose monitoring, and blood pressure. This can also be applied to photo-diode based imaging. [0077] With regards to use of ultra-sonic sensors, ultrasound offers a non-invasive window into the body, facilitating health monitoring and disease diagnosis. Ultrasound waves permeate the body and reflect off internal structures, generating echoes. These are captured and relayed to devices that render them into images or videos. A greater number of ultrasound sensor elements begets enhanced image resolution. However, G-SNET enables a reduction in transmitter count while preserving image quality.
[0078] It is recognized and appreciated that as specific examples, the abovecharacterized figures and discussion are provided to help illustrate certain aspects (and advantages in some instances) which may be used in the manufacture of such structures and devices. These structures and devices include the exemplary structures and devices described in connection with each of the figures as well as other devices, as each such described embodiment has one or more related aspects which may be modified and/or combined with the other such devices and examples as described hereinabove may also be found in the Appendices of the above-referenced Provisional Application.
[0079] The skilled artisan would also recognize various terminology' as used in the present disclosure. As examples, the Specification may describe and/or illustrates aspects useful for implementing the examples by way of various semiconductor materials/circuits which may be illustrated as or using terms such as layers, blocks, modules, device, system, unit, controller, and/or other circuit-type depictions. Such semiconductor and/or semiconductive materials (including portions of semiconductor structure) and circuit elements and/or related circuitry may be used together with other elements to exemplify how certain examples may be carried out in the form or structures, steps, functions, operations, activities, etc. For example, in certain of the above-discussed embodiments, one or more modules are discrete logic circuits or programmable logic circuits configured and arranged for implementing these operations/activities, as in the blocks/modules as shown in and among the figures. In certain embodiments, such a programmable circuit includes one or more computer circuits programmed to execute a set (or sets) of instructions (and/or configuration data). The instructions (and/or configuration data) can be in the form of firmware or software stored in and accessible from a memory (circuit). As an example, first and second modules include a combination of a CPU hardware-based circuit and a set of instructions in the form of firmware, in which the first module includes a first CPU hardware circuit with one set of instructions for training an algorithm, and the second module includes a second CPU hardware circuit with another set of instructions for further training of the algorithm and/or algorithm utilization. Furthermore, terms to exemplify orientation, such as upper/lower, left/right, top/bottom and above/below, may be used herein to refer to relative positions of elements as shown in the figures. Such terminology is used for notational convenience only and that in actual use the disclosed structures may be oriented different from the orientation shown in the figures.
[0080] Based upon the above discussion and illustrations, those skilled in the art will readily recognize that various modifications and changes may be made to the various embodiments without strictly following the exemplary' embodiments and applications illustrated and described herein. For example, methods as exemplified in the Figures may involve steps carried out in various orders, with one or more aspects of the embodiments herein retained, or may involve fewer or more steps. Such modifications do not depart from the true spirit and scope of various aspects of the disclosure, including aspects set forth in the claims.

Claims

What is Claimed:
1. An apparatus for use with an array of sensors to collect signals, indicative of kinematics and/or electrophysiology, corresponding to an observed region of a user, the apparatus comprising: computer processing circuitry to use the signals as collected by the array of sensors by training and/or using an algorithm to identify a pattern of physiological signals manifested by the user, and generating, via the algorithm, data knowledge about relationships between the observed region and at least one unobserved region of the user that does not include the observed region, and therein extrapolating, in response to the generated data knowledge, further physiological signals of the user or information related to the further physiological signals, corresponding to the at least one unobserved region.
2. The apparatus of claim 1, wherein the computer processing circuitry is to train the algorithm based on a set of raw signals acquired from a number of training electrodes which are associated with at least the unobserved region of the user signals, and once trained, while the array of sensors uses a number of user-coupled electrodes, coupled to the observed region, the computer processing circuitry is to use the algorithm to identify' the pattern of physiological signals by collecting the user physiological signals from the user-coupled electrodes of the array of sensors, wherein the number of training electrodes is at least twice the number of user-coupled electrodes.
3. The apparatus of claim 1, wherein the computer processing circuitry' is associated with a first software-executable module to train the algorithm by responding to raw signals from electrodes associated with the observed region of the user, and the computer processing circuitry is to use the trained algorithm, via a second software-executable module, for said extrapolating further physiological signals of the user or information related to the further physiological signals by responding to a set of user-coupled electrodes that are physically coupled to the observed region of the user without concurrently using any further electrodes coupled to the unobserved region of the user.
4. The apparatus of claim 1, wherein the computer processing circuitry- is to generate, based on the signals as collected by the array of sensors, output data indicative of the user concerning at least one of kinematics and electrophysiology- of the at least one unobserved region of the user.
5. The apparatus of claim 1, further including the array of sensors to collect physiological signals, indicative of electrophysiology, corresponding to the observed region of the user.
6. The apparatus of claim 1, further including the array of sensors to collect physiological signals, indicative of kinematics, corresponding to the observed region of the user.
7. The apparatus of claim 1, further including the array of sensors, wherein at least one of the sensors in the array is characterized by at least one of strain pressure sensors, photodiode based sensors, and ultra-sonic sensors.
8 The apparatus of claim 1, further including the array of sensors, wherein the array of sensors includes a body-adhering polymer based substrate layer that is stretchable and elastic integrated with a conductive gel that serves to lower impedance when electrodes, as part of the array of sensors, are in contact with skin of the user corresponding to the observed region.
9. The apparatus of claim 1, further including the array of sensors, wherein the array of sensors is protected by a rubber barrier layer to mitigate or prevent solvent induced swelling.
10. The apparatus of claim 1, further including the array of sensors, wherein the array of sensors includes a liquid metal electrode layer which is both stretchable and micro-patterned, encapsulated by a photo-patterned synthetic rubber which leaves only electrode areas uncovered to facilitate skin contact and signal measurement.
11. An apparatus comprising; an array of sensors to collect signals, indicative of kinematics and/or electrophysiology, corresponding to an observ ed user region of a user; and computer processing circuitry to use an algorithm, trained on a high-resolution map of a diverse cohort of biological structures, to extrapolate the physiological signals of a larger region of the body from the signals collected from the observed region by the array of sensors, via knowledge about correlations between the observed region and at least one separate unobserv ed region of the user that does not include the observed region.
12. The apparatus of claim 11. wherein the computer processing circuitry is to train the algorithm to identify a pattern of physiological signals manifested by one or more individuals.
13. The apparatus of claim 11, wherein the array of sensors is to collect signals indicative of electrophysiology linked to an aspect of the user, wherein the aspect of the user corresponds to one of the following: a portion of the user’s organ via a probe having electrodes as part of the array of sensors; surface or membrane of an internal organ.
14. The apparatus of claim 11. wherein the computer processing circuitry and the algorithm collectively correspond to a pre-trained encoding network configured based on a defined or an iteratively-refined latent vector derived from a set of raw data carried or derived from a set of electrodes, or signal channels, which are associated with the observed user region and to be used to predict a pattern of physiological signals associated with the observed user region and said at least one separate unobserved region.
15. A circuit-implemented method comprising: collecting, via an array of sensors, signals indicative of kinematics and/or electrophysiology, corresponding to an observ ed user region of a user, and using, via a computer processing circuitry and a trained computer-executable algorithm, the collected signals to identify a pattern of physiological signals manifested by the user, and by generating, via the algorithm, knowledge about relationships between the observed region and at least one unobserved region of the user that does not include the observed region, and therein extrapolating further physiological signals of the user, corresponding to the unobserved regions.
16. The circuit-implemented method of claim 15, further including training the computerexecutable algorithm to identify the pattern of physiological signals by using: a set of M electrodes to acquire signals associated with said at least one unobserved region of the user, deriving at least one latent vector from the earned signals, wherein the computerexecutable algorithm is trained to identify the pattern of physiological signals based on said at least one latent vector, and said step of generating knowledge about relationships between the observed region and at least one unobserved region uses not more than N electrodes in the array of sensors for said step of collecting signals, wherein M and N are positive integers and M is at least four times greater than N.
17. The circuit-implemented method of claim 15, further including generating a representation of physiological signals as a function of averaging-based calculations involving the physiological signals as captured by the sensors in the array, and wherein the step of using the collected signals to identify a pattern of physiological signals is in response to said generating a representation of physiological signals as a function of averaging-based calculations.
18. The circuit-implemented method of claim 15, further including processing the physiological signals as captured by the sensors in the array, via one or more calculations involving root mean square values across time, and in response generating a simplified representation of the physiological signals.
19. The circuit-implemented method of claim 15, further including training the computerexecutable algorithm to identify the pattern of physiological signals by using a learning model that operates based on an auto encoder framework and that includes an encoder circuit module and a decoder circuit module that are cooperatively arranged to process raw data, from a number of training electrodes used during training, to configure the computerexecutable algorithm to identify, after the computer-executable algorithm is trained, the pattern of physiological signals while the array of sensors for said step of collecting signals utilizes less than one half of the number of training electrodes used during training.
20. The circuit-implemented method of claim 15, wherein the physiological signals include electrophysiological signals for one of EMG (electromyography), EEG (electroencephalogram), ECG (electrocardiography), EGG (electrogastrography), and EOG (electrooculography).
PCT/US2024/056006 2023-11-15 2024-11-14 Apparatuses and methods involving sensor-based sensory extrapolation Pending WO2025106737A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363599190P 2023-11-15 2023-11-15
US63/599,190 2023-11-15

Publications (1)

Publication Number Publication Date
WO2025106737A1 true WO2025106737A1 (en) 2025-05-22

Family

ID=95743522

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2024/056006 Pending WO2025106737A1 (en) 2023-11-15 2024-11-14 Apparatuses and methods involving sensor-based sensory extrapolation

Country Status (1)

Country Link
WO (1) WO2025106737A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220142504A1 (en) * 2013-02-20 2022-05-12 Pmd Device Solutions Limited Method and device for respiratory monitoring
US20220151467A1 (en) * 2019-09-03 2022-05-19 Fujifilm Corporation Medical image processing apparatus, endoscope system, medical image processing method, and program
US20220354381A1 (en) * 2017-06-28 2022-11-10 Resonea, Inc. Diagnosis tailoring of health and disease

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220142504A1 (en) * 2013-02-20 2022-05-12 Pmd Device Solutions Limited Method and device for respiratory monitoring
US20220354381A1 (en) * 2017-06-28 2022-11-10 Resonea, Inc. Diagnosis tailoring of health and disease
US20220151467A1 (en) * 2019-09-03 2022-05-19 Fujifilm Corporation Medical image processing apparatus, endoscope system, medical image processing method, and program

Similar Documents

Publication Publication Date Title
Zheng et al. Surface electromyography as a natural human–machine interface: A review
He et al. Wrist and finger gesture recognition with single-element ultrasound signals: A comparison with single-channel surface electromyogram
Kim et al. Recent developments and future directions of wearable skin biosignal sensors
Zhang et al. Learning effective spatial–temporal features for sEMG armband-based gesture recognition
Pancholi et al. Improved classification scheme using fused wavelet packet transform based features for intelligent myoelectric prostheses
Gao et al. Use of advanced materials and artificial intelligence in electromyography signal detection and interpretation
Liu et al. Recent applications of EEG-based brain-computer-interface in the medical field
Farina et al. High-density EMG E-textile systems for the control of active prostheses
Matsushita et al. Recent use of deep learning techniques in clinical applications based on gait: A survey
Guo et al. A human-machine interface based on single channel EOG and patchable sensor
Zhou et al. Ankle foot motion recognition based on wireless wearable sEMG and acceleration sensors for smart AFO
Nsugbe Brain-machine and muscle-machine bio-sensing methods for gesture intent acquisition in upper-limb prosthesis control: A review
Zhang et al. A wearable human–machine-interface (HMI) system based on colocated EMG-pFMG sensing for hand gesture recognition
Zhang et al. ViT-LLMR: Vision Transformer-based lower limb motion recognition from fusion signals of MMG and IMU
Zhu et al. High-fidelity recording of EMG signals by multichannel on-skin electrode arrays from target muscles for effective human–machine interfaces
Xue et al. Instrumentation, measurement, and signal processing in electroencephalography-based brain-computer interfaces: situations and prospects
Tanzarella et al. Neuromorphic decoding of spinal motor neuron behaviour during natural hand movements for a new generation of wearable neural interfaces
Ahamad System architecture for brain-computer interface based on machine learning and internet of things
Gao et al. A multifeatured time–frequency neural network system for classifying sEMG
WO2025106737A1 (en) Apparatuses and methods involving sensor-based sensory extrapolation
He et al. Low-latency gesture recognition from spatial filtering of single-element ultrasound signals
Ruan et al. Long-Term Finger Force Predictions Using Motoneuron Discharge Activities
Meng et al. Robust and Lightweight Decoder for Unsupervised Multi-Finger Force Predictions Towards the Internet of Medical Things-based Applications
Epeagba et al. Ultrasound based wrist intent recognition method for robotic-assisted stroke rehabilitation
US20250352089A1 (en) Tissue spectrophotometry for human-computer and human-machine interfacing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24892267

Country of ref document: EP

Kind code of ref document: A1