[go: up one dir, main page]

WO2025217508A1 - Systèmes et procédés d'utilisation de réalité virtuelle pour simuler une tâche de travail à l'aide d'un masque professionnel et de communication bidirectionnelle entre au moins deux utilisateurs - Google Patents

Systèmes et procédés d'utilisation de réalité virtuelle pour simuler une tâche de travail à l'aide d'un masque professionnel et de communication bidirectionnelle entre au moins deux utilisateurs

Info

Publication number
WO2025217508A1
WO2025217508A1 PCT/US2025/024263 US2025024263W WO2025217508A1 WO 2025217508 A1 WO2025217508 A1 WO 2025217508A1 US 2025024263 W US2025024263 W US 2025024263W WO 2025217508 A1 WO2025217508 A1 WO 2025217508A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
vocational
mask
data
task
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/US2025/024263
Other languages
English (en)
Inventor
Arnold Kravitz
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BlueForge Alliance
Original Assignee
BlueForge Alliance
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US19/169,623 external-priority patent/US20250276398A1/en
Application filed by BlueForge Alliance filed Critical BlueForge Alliance
Publication of WO2025217508A1 publication Critical patent/WO2025217508A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B9/00—Simulators for teaching or training purposes
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B23—MACHINE TOOLS; METAL-WORKING NOT OTHERWISE PROVIDED FOR
    • B23K—SOLDERING OR UNSOLDERING; WELDING; CLADDING OR PLATING BY SOLDERING OR WELDING; CUTTING BY APPLYING HEAT LOCALLY, e.g. FLAME CUTTING; WORKING BY LASER BEAM
    • B23K9/00—Arc welding or cutting
    • B23K9/095—Monitoring or automatic control of welding parameters
    • B23K9/0953—Monitoring or automatic control of welding parameters using computing means
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00—Teaching not covered by other main groups of this subclass
    • G09B19/24—Use of tools
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/43—Querying
    • G06F16/438—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality

Definitions

  • This disclosure relates to enabling workers to perform vocations. More specifically, this disclosure relates to systems and methods for using a vocational mask and a shared virtual reality session for instruction between a master and an apprentice.
  • a welder may use a welding mask and/or a welding gun to weld an object.
  • the welder may participate in training courses prior to welding the object.
  • a master welder may lead the training courses to train the welder how to properly weld.
  • the master welder may be located at a physical location that is remote from where a student welder is physically located.
  • a first vocational mask configured to be worn by a first user includes a virtual retinal display, a memory device storing instructions and a processing device communicatively coupled to the memory device and the virtual retinal display.
  • the instructions when executed by the processing device, are configured to cause the first vocational mask to share a virtual reality session with a computing device associated with a second user.
  • the first vocational mask is configured to conduct bidirectional communications with the computing device of the second user, wherein the first user is an apprentice and the second user is a master.
  • the first vocational mask is further configured to receive and present, via the virtual retinal display to the first user, data in selected ones of a plurality of formats from the second user.
  • the data comprises directions for performing a task by the first user under direction from the second user.
  • a tangible, non-transitory computer-readable medium stores instructions that, when executed, cause a processing device to perform any operation of any method disclosed herein.
  • a system includes a memory device storing instructions and a processing device communicatively coupled to the memory device.
  • the processing device executes the instructions to perform any operation of any method disclosed herein.
  • FIG. 1 illustrates a system architecture according to certain embodiments of this disclosure
  • FIG. 2 illustrates a component diagram for a vocational mask according to certain embodiments of this disclosure
  • FIG. 3 illustrates bidirectional communication between communicatively coupled vocational masks according to certain embodiments of this disclosure
  • FIG. 4 illustrates an example of projecting an image onto a user’s retina via a virtual retinal display of a vocational mask according to certain embodiments of this disclosure
  • FIG. 5 illustrates an example of an image including instructions projected via a virtual retinal display of a vocational mask according to certain embodiments of this disclosure
  • FIG. 6 illustrates an example of an image including a warning projected via a virtual retinal display of a vocational mask according to certain embodiments of this disclosure
  • FIG. 7 illustrates an example of a method for executing an artificial intelligence agent to determine certain information projected via a vocational mask of a user according to certain embodiments of this disclosure
  • FIG. 8 illustrates an example of a method for transmitting instructions for performing a task via bidirectional communication between a vocational mask and a computing device according to certain embodiments of this disclosure
  • FIG. 9 illustrates an example of a method for implementing instructions for performing a task using a peripheral haptic device according to certain embodiments of this disclosure
  • FIG. 10 illustrates an example computer system according to embodiments of this disclosure
  • FIGS. 11A-11C illustrate an example user interface configured to display a virtual avatar or coach according to embodiments of the present disclosure
  • FIG. 12 illustrates steps of an example method of controlling the user interface and virtual coach of FIGS. 11 A-l 1C according to embodiments of the present disclosure
  • FIG. 13 illustrates steps of an example method of conducting a shared virtual reality session between an apprentice and a master according to embodiments of the present disclosure
  • FIG. 14 illustrates examples of respective user interfaces for both an apprentice and a master during training to perform a particular task according to embodiments of the present disclosure
  • FIG. 15 illustrates steps of an example of a method of conducting a shared virtual reality session between a first user and a second user operating in switchable first and second contexts according to embodiments of the present disclosure
  • FIG. 16 illustrates examples of respective user interfaces for first and second users sharing a virtual reality session with switchable contexts according to embodiments of the present disclosure. NOTATION AND NOMENCLATURE
  • first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections; however, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another region, layer or section. Terms such as “first,” “second,” and other numerical terms, when used herein, do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.
  • phrases “at least one of,” when used with a list of items, means that different combinations of one or more of the listed items may be used, and only one item in the list may be needed.
  • “at least one of: A, B, and C” includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.
  • the phrase “one or more” when used with a list of items means there may be one item or any suitable number of items exceeding one.
  • various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium.
  • application and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code.
  • program includes any type of computer code, including source code, object code, and executable code.
  • the phrase '‘computer readable medium” includes any type of medium capable of being accessed by a computer, such as read only memory’ (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), solid state drives (SSDs), flash memory, or any other type of memory.
  • ROM read only memory
  • RAM random access memory
  • CD compact disc
  • DVD digital video disc
  • SSDs solid state drives
  • flash memory or any other type of memory.
  • a “non-transitory” computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals.
  • a non-transitory’ computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.
  • FIGS. 1 through 10 discussed below, and the various embodiments used to describe the principles of this disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure.
  • the vocational tools may 7 be in the form of a vocational mask that projects work instructions using imagery’, animation, video, text, audio, and the like.
  • the vocational tools may be used by workers to enhance the efficiency and proficiency of performing professional and vocational tasks, such as but not limited to supply chain operations, manufacturing and warehousing processes, product inspection, coworker and master-apprentice bidirectional collaboration and communication with or without haptic sensory feedback, other telepresence, and the like.
  • Some of the disclosed embodiments may be used to collect data, metadata, and multiband video to aid in product acceptance, qualification, and full lifecycle product management. Further, some of the disclosed embodiments may aid a failure reporting, analysis, and corrective action system, a failure mode, effects, and criticality analysis system, other sustainment and support activities and tasks to accommodate worker dislocation and multi-decade lifecycle of some products.
  • a vocational mask employs bidirectional communication to include voice and imagery and still and audio video imagery' recording with other colleagues over a distance.
  • the vocational mask may provide virtual images of objects to a person wearing the vocational mask via a display (e.g., virtual retinal display).
  • the vocational mask may enable bidirectional communications with collaborators and/or students (e.g., in a master-apprentice relationship). Further, the vocational mask may enable bidirectional audio, visual, and haptic communication to provide a master-apprentice relationship.
  • a master-apprentice relationship may include virtually any teacher-student relationship or other type of relationship in which a wearer of the vocational mask carries out a task while receiving instructions or directions from another user via another computing device that is operatively coupled (e.g.. via a network) to the vocational mask. Examples include, but are not limited to, an apprentice carrying out a task within a skilled trade under instruction of a master, a resident carrying out a medical procedure under instruction of a licensed doctor, and so on.
  • the disclosure further contemplates the wearer of the vocational mask being a collaborator with another user via another operatively coupled computing device (which may be another vocational mask or any other suitable computing device). Accordingly, two or more users may collaborate on performing a task or a learning exercise during a virtual reality session that is shared with or by a vocational mask.
  • another operatively coupled computing device which may be another vocational mask or any other suitable computing device. Accordingly, two or more users may collaborate on performing a task or a learning exercise during a virtual reality session that is shared with or by a vocational mask.
  • the disclosure contemplates the sharing and switching of contexts between users in a shared virtual reality session, including at least one user wearing a vocational mask.
  • a user wearing a first vocational mask may share a virtual reality session with another user wearing a second vocational mask, with the first user operating in a first context and the second user operating in a second context.
  • these contexts may be switched during the virtual reality session such that, in this example, the first and second users exchange contexts.
  • a second user in the master role may send haptic data to a first user in an apprentice role to provide haptic instruction (e.g. vibration of a peripheral haptic device associated with a tool) on manipulating a tool.
  • haptic instruction e.g. vibration of a peripheral haptic device associated with a tool
  • the first user may send haptic data to the second user to enable the latter to use haptic feedback to monitor the manipulation of the tool by the first user.
  • two or more users performing a complex repair of a system may each, at times during the repair, require control of the system.
  • a first user, wearing a vocational mask may be on-site, and in a first context, may have control of the system under repair to enable certain steps to be carried out while the second user is in an observational context.
  • a context switch may be performed in which the first user transfers control of the system to be earned out remotely by the second user, with the first user assuming an observational role.
  • the vocational mask may include multiple electromagnetic spectrum and acoustic sensors/imagers.
  • the vocational mask may also provide multiband audio and video sensed imagery’ to a wearer of the vocational mask.
  • the vocational mask may be configured to provide display capabilities to project images onto one or more irises of the wearer to display alphanumeric data and graphic/animated work instructions, for example.
  • the vocational mask may also include one or more speakers to emit audio related to work instructions, such as those provided by a master trained user, supervisor, collaborator, teacher, etc.
  • the vocational mask may include an edge-based processor that executes an artificial intelligence agent.
  • the artificial intelligence agent may be implemented in computer instructions stored on one or more memory' devices and executed by one or more processing devices (e.g., edge processor or processors).
  • the artificial intelligence agent may be trained to perform one or more functions, such as but not limited to (i) perception-based object and feature identification, (ii) cognition-based scenery understanding, to identify material and assembly defects versus acceptable features, and (iii) decision making to aid the wearer and to provide relevant advice and instruction in real-time or near real-time to the wearer of the vocational mask.
  • the data that is collected may be used for inspection and future analyses of product quality, product design, and the like. Further, the collected data may be stored for instructional analyses and providing lessons, mentoring, collaboration, and the like.
  • the vocational mask may include one or more components (e.g., processing device, memory’ device, display, etc.), interfaces, and/or sensors configured to provide sensing capabilities to understand hand motions and use of a virtual user interface (e.g., keyboards) and other haptic instructions.
  • the vocational mask may include a haptic interface to allow physical bidirectional haptic sensing and stimulation via the bidirectional communications to other users and/or collaborators using a peripheral haptic device (e.g., a welding gun).
  • the vocational mask may be in the form of binocular goggles, monocular goggles, finishing process glasses (e.g., grind, chamfer, debur, sand polish, coat, etc.), or the like.
  • the vocational mask may be attached to a welding helmet.
  • the vocational mask may include an optical bench that aligns a virtual retinal display to one or more eyes of a user. Certain types of data (e.g., video and image) may be projected from the virtual retinal display into the retinal of a user.
  • the vocational mask may include a liquid crystal display welding helmet, a welding camera, an augmented reality / virtual reality headset, etc.
  • the vocational mask may augment projections by providing augmented reality cues and information to assist a worker (e.g., welder) with contextual information, which may include setup, quality control, procedures, training, and the like. Further, the vocational mask may provide a continuum of visibility 7 from visible spectrum (arc off) through high-intensity / ultraviolet (arc on). Further, some embodiments include remote feedback and recording of images and bidirectional communications to a trainer, supervisor, mentor, master user, teacher, collaborator, etc. who can provide visual, auditory, and/or haptic feedback to the wearer of the vocational mask in real-time or near real-time.
  • the vocational mask may be integrated with a welding helmet.
  • the vocational mask may be (or include) a set of augmented reality / virtual reality goggles worn under a welding helmet (e.g., with external devices, sensors, cameras, etc. appended for image / data gathering).
  • the vocational mask may be a set of binocular welding goggles or a monocular welding goggle to be worn under or in lieu of a welding helmet (e.g.. with external devices, sensors, cameras, etc. appended to the goggles for image / data gathering).
  • the vocational mask may include a mid-band or long wave context camera displayed to the user and monitor.
  • information may be superpositioned or superimposed onto a display without the user (e.g., worker, student, etc.) wearing a vocational mask.
  • the information may include work instructions in the form of text, images, alphanumeric characters, video, etc.
  • the vocational mask may function across both visible light (arc off) and high intensity ultraviolet light (arc on) conditions.
  • the vocational mask may natively or in conjunction with other personal protective equipment provide protection against welding flash.
  • the vocational mask may enable real-time or near real-time two-way communication with a remote instructor or supervisor.
  • the vocational mask may provide one or more video, audio, and data feeds to a remote instructor or supervisor.
  • the vocational mask and/or other components in a system may enable recording of all data and communications.
  • the system may provide a mechanism for replaying the data and communications, via a media player, for training purposes, quality control purposes, inspection purposes, and the like.
  • the vocational mask and/or other components in a system may provide a mechanism for visual feedback from a remote instructor or supervisor.
  • the vocational mask and/or other components in a system may provide a bidirectional mechanism for haptic feedback from a remote instructor or supervisor.
  • the system may include an artificial intelligence simulation generator that generates task simulations to be transmitted to and presented via the vocational mask.
  • the simulation of a task may be transmitted as virtual reality data to the vocational mask which includes a virtual reality headset and/or display to playback the virtual real i ty data.
  • the virtual reality data may be configured based on parameters of a physical space in which the vocational mask is located, based on parameters of an object to be worked on, based on parameters of a tool to be used, and the like.
  • Some embodiments of the system may also include an artificial intelligence agent that is implemented in instructions stored on one or more memory devices and executable on one or more processing devices of the vocational mask.
  • the artificial intelligence agent is trained such that, when executed, it may monitor one or more aspects of the virtual reality session and may additionally provide directions for performing a task to a user wearing the vocational mask.
  • the artificial intelligence agent may also monitor one or more properties of a task performed by a user wearing the vocational mask. For example, if the task is welding, the artificial intelligence agent may monitor one or more properties of a weld formed by the first user. Based on the one or more monitored properties, the artificial intelligence agent may adjust the directions provided to the user for cartying out the task.
  • the artificial intelligence agent may monitor a number of different types of tasks in addition to the example of welding given here.
  • Other tasks monitored by the artificial intelligence agent may include (but are not limited to) brazing, soldering, and other types of mechanical and/or industrial processes that may be earned out by a user, medical procedures to be carried out by a resident under instruction of a doctor (as well as procedures carried out by one doctor with the assistance of another), repair operations carried out by a technician with the assistance of an engineer or other technician, and so on.
  • FIG. 1 depicts a system architecture 10 according to some embodiments.
  • the system architecture 10 may include one or more computing devices 140, one or more vocational masks 130, one or more peripheral haptic devices 134, and/or one or more tools 136 communicatively coupled to a cloud-based computing system 116.
  • Each of the computing devices 140, vocational masks 130, peripheral haptic devices 134, tools 136, and components included in the cloud-based computing system 116 may include one or more processing devices, memory devices, and/or network interface cards.
  • the network interface cards may enable communication via a wireless protocol for transmitting data over short distances, such as Bluetooth, ZigBee, NFC, etc.
  • Network 20 may be a public network (e.g., connected to the Internet via wired (Ethernet) or wireless (WiFi)), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof.
  • Network 20 may also include a node or nodes on the Internet of Things (loT).
  • the network 20 may be a cellular network.
  • the computing devices 140 may be any suitable computing device, such as alaptop, tablet, smartphone, smartwatch, ear buds, server, or computer.
  • the computing device 140 may be a vocational mask.
  • the computing devices 140 may include a display capable of presenting a user interface 142 of an application.
  • the display may be a laptop display, smartphone display, computer display, tablet display, a virtual retinal display, etc.
  • the application may be implemented in computer instructions stored on the one or more memory devices of the computing devices 140 and executable by the one or more processing devices of the computing device 140.
  • the application may present various screens to a user.
  • the user interface 142 may present a screen that plays video received from the vocational mask 130.
  • the video may present real-time or near realtime footage of what the vocational mask 130 is viewing, and in some instances, that may include a user’s hands holding the tool 136 to perform a task (e.g., weld, sand, polish, chamfer, debur, paint, play a video game, etc.). Additional screens may be presented via the user interface 142.
  • a task e.g., weld, sand, polish, chamfer, debur, paint, play a video game, etc.
  • Additional screens may be presented via the user interface 142.
  • the application executes within another application (e.g., web browser).
  • the computing device 140 may also include instructions stored on the one or more memory devices that, when executed by the one or more processing devices of the computing devices 140 perform operations of any of the methods described herein.
  • the computing devices 140 may include an edge processor 132. 1 that performs one or more operations of any of the methods described herein.
  • the edge processor 132.1 may execute an artificial intelligence agent to perform various operations described herein.
  • the artificial intelligence agent may include one or more machine learning models that are trained via the cloud-based computing system 116 as described herein.
  • the edge processor 132.1 may represent one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the edge processor 132.1 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets.
  • the edge processor 132.1 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like.
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • DSP digital signal processor
  • the vocational mask 130 may be attached to or integrated with a welding helmet, binocular goggles, a monocular goggle, glasses, a hat, a helmet, a virtual reality headset, a headset, a facemask, or the like.
  • the vocational mask 130 may include various components as described herein, such as an edge processor 132.2.
  • the edge processor 132.2 may be located separately from the vocational mask 130 and may be included in another computing device, such as a server, laptop, desktop, tablet, smartphone, etc.
  • the edge processor 132.2 may represent one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like.
  • the edge processor 132.2 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets.
  • the edge processor 132.2 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like.
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • DSP digital signal processor
  • the edge processor 132.2 may perform one or more operations of any of the methods described herein.
  • the edge processor 132.2 may execute an artificial intelligence agent to perform various operations described herein.
  • the artificial intelligence agent may include one or more machine learning models that are trained via the cloud-based computing system 116 as described herein.
  • the cloud-based computing system 116 may train one or more machine learning models 154 via a training engine 152, and may transmit the parameters used to train the machine learning model to the edge processor 132.2 such that the edge processor 132.2 can implement the parameters in the machine learning models executing locally on the vocational mask 130 or computing device 140.
  • the machine learning models may be trained to monitor various aspects of a virtual reality 7 session between, e.g., an apprentice and a master, and may further be trained to monitor properties of tasks carried out by the apprentice (e.g., a weld, a solder joint, etc.). Based on this monitoring, an artificial intelligence agent using these machine learning models may adjust instructions/directions provided to the apprentice.
  • the edge processor 132.2 may include a data concentrator that collects data from multiple vocational masks 130 and transmits the data to the cloud-based computing system 116.
  • the data concentrator may map information to reduce bandwidth transmission costs of transmitting data.
  • a network connection may not be needed for the edge processor 132.2 to collect data from vocational masks and to perform various functions using the trained machine learning models 154.
  • the vocational mask 130 may also include a network interface card that enables bidirectional communication with any other computing device 140, such as other vocational masks 130, smartphones, laptops, desktops, servers, wearable devices, tablets, etc.
  • the vocational mask 130 may also be communicatively coupled to the cloud-based computing system 116 and may transmit and receive information and/or data to and from the cloud-based computing system 1 16.
  • the vocational mask 130 may include various sensors, such as position sensors, acoustic sensors, haptic sensors, microphones, temperature sensors, accelerometers, and the like.
  • the vocational mask 130 may include various cameras configured to capture audio and video.
  • the vocational mask 130 may include a speaker to emit audio.
  • the vocational mask 130 may include a haptic interface configured to transmit and receive haptic data to and from the peripheral haptic device 134.
  • the haptic interface may be communicatively coupled to a processing device (e.g., edge processor 132.2) of the vocational mask 130.
  • the peripheral haptic device 134 may be attached to or integrated with the tool 136. In some embodiments, the peripheral haptic device 134 may be separate from the tool 136.
  • the peripheral haptic device 134 may include one or more haptic sensors that provide force, vibration, touch, and/or motion sensations to the user, among other things.
  • the peripheral haptic device 134 may be used to enable a person remote from a user of the peripheral haptic device 134 to provide haptic instructions to perform a task (e.g., weld, shine, polish, paint, control a video game controller, grind, chamfer, debur, etc.).
  • the peripheral haptic device 134 may include one or more processing devices, memory devices, network interface cards, haptic interfaces, etc. In some embodiments, the peripheral haptic device 134 may be communicatively coupled to the vocational mask 130, the computing device 140, and/or the cloud-based computing system 116.
  • the tool 136 may be any suitable tool, such as a welding gun, a video game controller, a paint brush, a pen, a utensil, a grinder, a sander, a polisher, a gardening tool, a yard tool, a glove, medical tools, a brazing torch, a soldering iron, a mechanical tool such as a wrench or pliers, or the like.
  • the tool 136 may be handheld such that the peripheral haptic device 134 is enabled to provide haptic instructions for performing a task to the user holding the tool 136.
  • the tool 136 may be wearable by the user.
  • the tool 136 may be used to perform a task.
  • the tool 136 may be located in a physical proximity to the user in a physical space.
  • the cloud-based computing system 116 may include one or more servers 128 that form a distributed computing architecture.
  • the servers 128 may be a rackmount server, a router computer, a personal computer, a portable digital assistant, a mobile phone, a laptop computer, a tablet computer, a camera, a video camera, a netbook, a desktop computer, a media center, any other device capable of functioning as a server, or any combination of the above.
  • Each of the servers 128 may include one or more processing devices, memory devices, data storage, and/or network interface cards.
  • the servers 128 may be in communication with one another via any suitable communication protocol.
  • the servers 128 may execute an artificial intelligence (Al) engine and/or an Al agent that uses one or more machine learning models 154 to perform at least one of the embodiments disclosed herein.
  • the cloud-based computing system 116 may also include a database 129 that stores data, knowledge, and data structures used to perform various embodiments.
  • the database 129 may store multimedia data of users performing tasks using tools, communications between vocational masks 130 and/or computing devices 140, virtual reality 7 simulations, augmented reality information, recommendations, instructions, and the like.
  • the database 129 may also store user profiles including characteristics particular to each user.
  • the database 129 may be hosted on one or more of the servers 128.
  • the cloud-based computing system 116 may include a training engine 152 capable of generating the one or more machine learning models 154.
  • the machine learning models 154 may be trained to identify perception-based objects and features using training data that includes labeled inputs of images including certain objects and features mapped to labeled outputs of identities or characterizations of those objects and features.
  • the machine learning models 154 may be trained determine cognition-based scenery to identity one or more material defects, one or more assembly defects, one or more acceptable features, or some combination thereof using training data that includes labeled input of scenery images of objects including material defects, assembly defects, and/or acceptable features mapped to labeled outputs that characterize and/or identify the material defects, assembly defects, and/or acceptable features.
  • the machine learning models 154 may be trained to determine one or more recommendations, instructions, or both using training data including labeled input of images (e.g., objects, products, tools, actions, etc.) and tasks to be performed (e.g., weld, grind, chamfer, debur, sand, polish, coat, etc.) mapped to labeled outputs including recommendations, instructions, or both.
  • images e.g., objects, products, tools, actions, etc.
  • tasks to be performed e.g., weld, grind, chamfer, debur, sand, polish, coat, etc.
  • the one or more machine learning models 154 may be generated by the training engine 152 and may be implemented in computer instructions executable by one or more processing devices of the training engine 152 and/or the servers 128. To generate the one or more machine learning models 154, the training engine 152 may train the one or more machine learning models 154. The one or more machine learning models 154 may also be executed by the edge processor 132 (132.1, 132.2). The parameters used to train the one or more machine learning models 154 by the training engine 152 at the cloud-based computing system 116 may be transmitted to the edge processor 132 (132.1, 132.2) to be implemented locally at the vocational mask 130 and/or the computing device 140.
  • the training engine 152 may be a rackmount server, a router computer, a personal computer, a portable digital assistant, a smartphone, a laptop computer, a tablet computer, a netbook, a desktop computer, an Internet of Things (loT) device, any other desired computing device, or any combination of the above.
  • the training engine 152 may be cloud-based, be a real-time software platform, include privacy software or protocols, and/or include security software or protocols.
  • the training engine 152 may train the one or more machine learning models 154.
  • the one or more machine learning models 154 may refer to model artifacts created by the training engine 152 using training data that includes training inputs and corresponding target outputs.
  • the training engine 152 may find patterns in the training data wherein such patterns map the training input to the target output and generate the machine learning models 154 that capture these patterns.
  • the training engine 152 may reside on server 128.
  • the database 129, and/or the training engine 152 may reside on the computing devices 140.
  • the one or more machine learning models 154 may comprise, e.g., a single level of linear or non-linear operations (e.g., a support vector machine [SVM]) or the machine learning models 154 may be a deep network, i.e., a machine learning model comprising multiple levels of non-linear operations.
  • deep networks are neural networks, including generative adversarial networks, convolutional neural networks, recurrent neural networks with one or more hidden layers, and fully connected neural networks (e.g., each neuron may transmit its output signal to the input of the remaining neurons, as well as to itself).
  • the machine learning model may include numerous layers and/or hidden layers that perform calculations (e.g., dot products) using various neurons.
  • FIG. 2 illustrates a component diagram for a vocational mask 130 according to certain embodiments of this disclosure.
  • the edge processor 132.2 is also depicted.
  • the edge processor 132.2 may be included in a computing device separate from the vocational mask 130. and in some embodiments, the edge processor 132.2 may be included in the vocational mask 130.
  • the vocational mask 130 may include various position, navigation, and time (PNT) components, sensors, and/or devices that enable determining the geographical positon (latitude, longitude, altitude, time), pose (length (ground to sensor), elevation, time), translation (delta in latitude, delta in longitude, delta in altitude, time), the rotational rate of pose ((or, ®p, coy(northing), t)), and the like, where cor represents the roll rate, which is the angular velocity about the longitudinal axis of the vocational mask 130, cop represents the pitch rate, which is the angular velocity about the lateral axis of the vocational mask 130, coy(northing) represents the yaw rate, which is the angular velocity about the vertical axis of the vocational mask 130, references with respect to the northing direction, and t represents the time at which these rotational rates are measured.
  • PNT position, navigation, and time
  • the vocational mask 130 may include one or more sensors, such as vocation imaging band specific cameras, visual band cameras, microphones, and the like.
  • the vocational mask 130 may include an audio visual display, such as a stereo speaker, a virtual retinal display, a liquid crystal display, a virtual reality headset, and the like.
  • a virtual retinal display may be a retinal scan display or retinal projector that draws a raster display directly onto the retina of the eye.
  • the virtual retinal display may include drive electronics that transmit data to a photon generator and/or intensity modulator. These components may process the data (e.g., video, audio, haptic, etc.) and transmit the processed data to a beam scanning component that further transmits data to an optical projector that projects an image and/or video to a retina of a user.
  • the vocational mask 130 may include a network interface card that enables bidirectional communication (digital communication) with other vocational masks and/or computing device 140.
  • the vocational mask 130 may provide a user interface to the user via the display described herein.
  • the edge processor 132.2 may include a network interface card that enables digital communication with the vocational mask 130, the computing device 140, the cloud-based computing system 116, or the like.
  • FIG. 3 illustrates bidirectional communication between communicatively coupled vocational masks 130 and 302 according to certain embodiments of this disclosure.
  • a user 306 is wearing a vocational mask 130.
  • the vocational mask 130 is attached to or integrated with a welding helmet 308.
  • the user is viewing an object or workpiece 300.
  • the vocational mask 130 may include multiple electromagnetic spectrum and/or acoustic sensors / imagers 304 to enable obtaining audio, video, acoustic, etc. data while observing the object 300 and/or performing a task (e.g., welding).
  • the vocational mask 130 may be communicatively coupled to one or more other vocational masks 302 worn by other users and may communicate data in real-time or near real-time such that bidirectional audio visual and haptic communications foster a master-apprentice relationship, a teacher-student relationship, a relationship between collaborators, or other type of relationship.
  • the bidirectional communication enabled by the vocational masks 130 may enable collaboration between a teacher or collaborator and students.
  • Each of the users wearing the vocational mask 130 may be enabled to visualize the object 300 that the user is viewing in real-time or near real-time.
  • FIG. 4 illustrates an example of projecting an image onto a user’s retina 400 via a virtual retinal display of a vocational mask 130 according to certain embodiments of this disclosure.
  • the imagers and/or cameras of the vocational mask 130 receive data pertaining to the object and the vocational mask 130 processes the data and projects an image representing the object 300 using a virtual retinal display onto the user’s retina 400.
  • the bidirectional communication with other users may enable projecting the image onto their retinas if they are wearing a vocational mask, as well.
  • the image may be displayed via a computing device 140 if the other users are not wearing vocational masks.
  • FIG. 5 illustrates an example of an image including instructions projected via a virtual retinal display of a vocational mask 130 according to certain embodiments of this disclosure.
  • the example user interface 500 depicts actual things the user is looking at, such as a tool 136 and an object 300 (e.g., a workpiece), through the vocational mask 130. Further, the user interface depicts instructions 502 pertaining to performing a task.
  • the instructions 502 may be generated by one or more machine learning models 154 of the Al agent, or may be provided via a computing device 140 and/or another vocational mask being used by another user (e.g., master user, collaborator, teacher, supervisor, etc.). In the depicted example, the instructions 502 instruct the user to “1. Turn on welder; 2. Adjust wire speed and voltage’’.
  • the instructions 502 may be projected on the user’s retina via the virtual retinal display and/or presented on a display of the vocational mask 130.
  • FIG. 6 illustrates an example of an image including a warning projected via a virtual retinal display of a vocational mask according to certain embodiments of this disclosure.
  • the example user interface 600 depicts actual things the user is looking at. such as a tool 136 and an object 300, through the vocational mask 130. Further, the user interface depicts a warning 602 pertaining to performing a task.
  • the warning 602 may be generated by one or more machine learning models 154 of the Al agent, or may be provided via a computing device 140 and/or another vocational mask being used by another user (e.g., master user, collaborator, teacher, supervisor, etc.). In the depicted example, the warning 602 indicates “Caution: Material defect detected! Cease welding to avoid bum through”.
  • the warning 602 may be projected on the user’s retina via the virtual retinal display and/or presented on a display of the vocational mask 130.
  • FIG. 7 illustrates an example of a method 700 for executing an artificial intelligence agent to determine certain information projected via a vocational mask of a user according to certain embodiments of this disclosure.
  • the method 700 may be performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software, or a combination of both.
  • the method 700 and/or each of their individual functions, subroutines, or operations may be performed by one or more processing devices of a computing device (e.g., any component (server 128, training engine 152, machine learning models 154, etc.) of cloudbased computing system 116.
  • the method 700 may be implemented as computer instructions stored on a memory device and executable by the one or more processors. In certain implementations, the method 700 may be performed by a single processing thread. Alternatively, the method 700 may be performed by two or more processing threads, each thread implementing one or more individual functions, routines, subroutines, or operations of the methods.
  • the method 700 is depicted and described as a series of operations. However, operations in accordance with this disclosure can occur in various orders or concurrently, and with other operations not presented and described herein. For example, the operations depicted in the method 700 may occur in combination with any other operation of any other method disclosed herein. Furthermore, not all illustrated operations may be required to implement the method 700 in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the method 700 could alternatively be represented as a series of interrelated states via a state diagram or events.
  • one or more machine learning models may be generated and trained by the artificial intelligence engine and/or the training engine to perform one or more of the operations of the methods described herein.
  • the processing device may execute the one or more machine learning models.
  • the one or more machine learning models may be iteratively retrained to select different features capable of enabling optimization of output.
  • the features that may be modified may 7 include several nodes included in each layer of the machine learning models, an objective function executed at each node, a number of layers, various weights associated with outputs of each node, and the like.
  • a system may include the vocational mask 130, which may include one or more virtual retinal displays, memory devices, processing devices, and other components as described herein.
  • the processing devices may be communicatively coupled to the memoiy devices that store computer instructions, and the processing devices may execute the computer instructions to perform one or more of the steps of the method 700.
  • the system may include a welding helmet, and the vocational mask may be coupled to the welding helmet.
  • the vocational mask may be configured to operate across both visible light and high intensity ultraviolet light conditions.
  • the vocational mask may provide protection against welding flash.
  • the vocational mask may be integrated with goggles.
  • the vocational mask may be integrated with binoculars or a monocular.
  • the processing device may execute an artificial intelligence agent trained to perform at least one or more functions to determine certain information.
  • the functions may include (i) identifying perception-based objects and features, (ii) determining cognition-based scenery to identify one or more material defects, one or more assembly defects, one or more acceptable features, or some combination thereof, and (iii) determining one or more recommendations, instructions, or both.
  • the artificial intelligence agent may include one or more machine learning models 154 trained to perform the functions.
  • one or more machine learning models 154 may be trained to (i) identify perception-based objects and features using training data that includes labeled inputs of images including certain objects and features mapped to labeled outputs of identities or characterizations of those objects and features.
  • the machine learning models may be trained to analyze aspects of the objects and features to compare the aspects to known aspects associated with known objects and features, and the machine learning models may perceive the identity of the analyzed objects and features.
  • the one or more machine learning models 154 may be trained to (ii) determine cognition-based scenery to identify one or more material defects, one or more assembly defects, one or more acceptable features, or some combination thereof using training data that includes labeled input of scenery 7 images of objects including material defects, assembly defects, and/or acceptable features mapped to labeled outputs that characterize and/or identify the material defects, assembly defects, and/or acceptable features.
  • one scenery 7 image may include a portion of a submarine that includes parts that are welded together, and the machine learning models may be trained to cognitively analyze the scenery image to identify one or more portions of the scenery image that includes a welded part with a material welding defect, a part assembly defect, and/or acceptable welded feature.
  • the one or more machine learning models 154 may be trained to (iii) determine one or more recommendations, instructions, or both using training data including labeled input of images (e.g., objects, products, tools, actions, etc.) and tasks to be performed (e.g.. weld, grind, chamfer, debur, sand, polish, coat, etc.) mapped to labeled outputs including recommendations, instructions, or both.
  • the processing device may provide (e.g., via the virtual retinal display, a speaker, etc.) images, video, and/or audio that points out the defects and provides instructions, drawings, and/or information pertaining to how to fix the defects.
  • the output from performing one of the functions (i), (ii), and/or (iii) may be used as input to the other functions to enable the machine learning models 154 to generate a combined output.
  • the machine learning models 154 may identify a defect (a gouge) and provide welding instructions on how to fix the defect by filling the gouge properly via the vocational mask 130.
  • the machine learning models 154 may identify several potential actions that the user can perform to complete the task and may aid the user’s decision making by providing the actions in a ranked order of most preferred action to least preferred action or a ranked order of the action with the highest probability of success to the action with the lowest probability of success.
  • the machine learning models 154 may identify an acceptable feature (e.g., properly welded parts) and may output a recommendation to do nothing.
  • the artificial intelligence agent may further monitor one or more aspects of a virtual reality session. Based on the monitoring, the artificial intelligence agent may generate and provide, in real time, direction for performing a given task, during the virtual reality session, by a first user wearing the vocational mask. Alternatively, the artificial intelligence agent may generate suggested instructions by a master/teacher sharing the virtual reality session with an apprentice/student.
  • Embodiments are further contemplated in which the artificial intelligence agent may generate and provide, to various users of the shared virtual reality session, information in furtherance of completing a task where the various users are collaborators.
  • the various users of the shared virtual reality session may have different contexts, with such contexts being switchable among users.
  • a context may be control of a machine that can be carried out locally or remotely, and these contexts may be exchanged among the different users in the shared virtual reality session.
  • the artificial intelligence agent may adjust which user receives particular information based on such context switches.
  • the processing device may cause the certain information to be presented via the virtual retinal display.
  • the virtual retinal display may project an image onto at least one iris of the user to display alphanumeric data, graphic instructions, animated instructions, video instructions, or some combination thereof.
  • the vocational mask may include a stereo speaker to emit audio pertaining the information.
  • the processing device may superposition the certain information on a display (e.g., virtual retinal display).
  • the vocational mask may include a network interface configured to enable bidirectional communication with a second network interface of a second vocational mask.
  • the bidirectional communication may enable transmission of real- time or near real-time audio and video data, recorded audio and video data, or some combination thereof.
  • “Real-time”’ may refer to less than 2 seconds and “near real-time” may refer to between 2 and 20 seconds.
  • a system may include a peripheral haptic device.
  • the vocational mask may include a haptic interface, and the haptic interface may be configured to perform bidirectional haptic sensing and stimulation using the peripheral haptic device and the bidirectional communication.
  • the stimulation may include precise mimicking, vibration, and the like.
  • the stimulation may include performing mimicked gestures via the peripheral haptic device.
  • a master user may be using a peripheral haptic device to perform a task and the gestures performed by the master user using the peripheral haptic device may be mimicked by the peripheral haptic device being used by an apprentice user. In such a way, the master user may train and/or guide the apprentice user how to properly perform a task (e.g., weld) using the peripheral haptic devices.
  • the haptic interface may be communicatively coupled to the processing device.
  • the haptic interface may be configured to sense, from the peripheral haptic device, hand motions, texture, temperature, vibration, slipperiness, friction, wetness, pulsation, stiction, friction, and the like.
  • the haptic interface may detect keystrokes when a user uses a virtual keyboard presented via the vocational mask using a display (e.g., virtual retinal display).
  • the bidirectional communication provided by the vocational mask(s) and/or computing devices may enable a master user of a vocational mask and/or computing device to view and/or listen to the real-time or near real-time audio and video data, recorded audio and video data, or some combination thereof, and to provide instructions to the user via the vocational mask being worn by the user.
  • the bidirectional communication provided by the vocational mask(s) and/or computing devices may enable the user of a vocational mask and/or computing device to provide instructions to a set of students and/or apprentices via multiple vocational masks being worn by the students and/or apprentices. This technique may be beneficial for a teacher, collaborator, master user, and/or supervisor that is training the set of students.
  • the user wearing a vocational mask may communicate with one or more users who are not wearing a vocational mask.
  • a teacher and/or collaborator may be using a computing device (e.g., smartphone) to see what a student is viewing and hear what the student is hearing using the bidirectional communication provided by the vocational mask worn by the student.
  • the bidirectional communication provided by the vocational mask may enable a teacher or collaborator to receive, using a computing device, audio data, video data, haptic data, or some combination thereof, from the vocational mask being used by the user.
  • the teacher and/or collaborator may receive haptic data, via the computing device, from the vocational mask worn by the student.
  • the teacher and/or collaborator may transmit instructions (e.g., audio, video, haptic, etc.), via the computing device, to the vocational mask to guide and/or teach the student how to perform the task (e.g.. weld) in real-time or near real-time.
  • the bidirectional communication may enable a user wearing a vocational mask to provide instructions to a set of students via a set of computing devices (e.g., smartphones).
  • the user may be a teacher or collaborator and may be teaching a class or lesson on how to perform a task (e.g., weld) while wearing the vocational mask.
  • the vocational mask may include one or more sensors to provide information related to geographical position, pose of the user, rotational rate of the user, or some combination thereof.
  • a position sensor may be used to determine a location of the vocational mask, an object, a peripheral haptic device, a tool, etc. in a physical space.
  • the position sensor may determine an absolute position in relation to an established reference point.
  • the processing device may perform physical registration of the vocational mask, an object being worked on, a peripheral haptic device, a tool (e.g., welding gun, sander, grinder, etc.), etc. to map out the device in an environment (e.g., warehouse, room, underwater, etc.) in which the vocational mask, the object, the peripheral haptic device, etc. is located.
  • the vocational mask may include one or more sensors including vocation imaging band specific cameras, visual band cameras, stereo microphones, acoustic sensors, or some combination thereof.
  • the acoustic sensors may sense welding clues based on audio signatures associated with certain defects or issues, such as bum through.
  • Machine learning models 154 may be trained using inputs of labeled audio signatures, labeled images, and/or labeled videos mapped to labeled outputs of defects.
  • the artificial intelligence agent may process received sensor data, such as images, audio, video, haptics, etc., identify an issue (e.g., defect), and provide a recommendation (e.g., stop welding due to detected potential bum through) via the vocational mask.
  • the vocational mask may include an optical bench that aligns the virtual retinal display to one or more eyes of the user.
  • the processing device is configured to record the certain information, communications with other devices (e.g., vocational masks, computing devices), or both.
  • the processing device may store certain information and/or communications as data in the memory device communicatively coupled to the processing device, and/or the processing device may transmit the certain information and/or communications as data feeds to the cloud-based computing system 116 for storage.
  • FIG. 8 illustrates an example of a method 800 for transmitting instructions for performing a task via bidirectional communication between a vocational mask and a computing device according to certain embodiments of this disclosure.
  • the method 800 may be performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software, or a combination of both.
  • the method 800 and/or each of their individual functions, subroutines, or operations may be performed by one or more processing devices of a computing device (e.g., any component (server 128, training engine 152, machine learning models 154, etc.) of cloud-based computing system 116, vocational mask 130, edge processor 132 (132.1. 132.2).
  • peripheral haptic device 134, tool 136, and/or computing device 140 of FIG. 1) implementing the method 700.
  • the method 800 may be implemented as computer instructions stored on a memory device and executable by the one or more processors. In certain implementations, the method 800 may be performed by a single processing thread. Alternatively, the method 800 may be performed by two or more processing threads, each thread implementing one or more individual functions, routines, subroutines, or operations of the methods.
  • the method 800 is depicted and described as a series of operations. However, operations in accordance with this disclosure can occur in various orders or concurrently, and with other operations not presented and described herein. For example, the operations depicted in the method 800 may occur in combination with any other operation of any other method disclosed herein. Furthermore, not all illustrated operations may be required to implement the method 800 in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the method 800 could alternatively be represented as a series of interrelated states via a state diagram or events.
  • one or more machine learning models may be generated and trained by the artificial intelligence engine and/or the training engine to perform one or more of the operations of the methods described herein.
  • the processing device may execute the one or more machine learning models.
  • the one or more machine learning models may be iteratively retrained to select different features capable of enabling optimization of output.
  • the features that may be modified may include a number of nodes included in each layer of the machine learning models, an objective function executed at each node, a number of layers, various weights associated with outputs of each node, and the like.
  • the processing device may receive, at one or more processing devices of the vocational mask 130.
  • the vocational mask 130 may be attached to or integrated with a welding helmet and the task may be welding.
  • the task may be sanding, grinding, polishing, deburring, chamfering, coating, etc.
  • the vocational mask 130 may be attached to or integrated with a helmet, a hat, goggles, binoculars, a monocular, or the like.
  • the one or more first data feeds may include information related to video, images, audio, hand motions, haptics, texture, temperature, vibration, slipperiness, friction, wetness, pulsation, or some combination thereof.
  • the one or more first data feeds may include geographical position of the vocational mask 130, and the processing device may map, based on the geographical positon. the vocational mask 130 in an environment or a physical space in which the vocational mask 130 is located.
  • the processing device may transmit, via one or more network interfaces of the vocational mask 130, the one or more first data feeds to one or more processing devices of the computing device 140 of a second user.
  • the computing device 140 of the second user may include one or more vocational masks, one or more smartphones, one or more tablets, one or more laptop computers, one or more desktop computers, one or more servers, or some combination thereof.
  • the computing device 140 may be separate from the vocational mask 130, and the one or more first data feeds are at least one of presented via a display of the computing device 140, emitted by an audio device of the computing device 140. or produced or reproduced via a peripheral haptic device coupled to the computing device 140.
  • the first user may be an apprentice, student, trainee, or the like
  • the second user may be a master user, a trainer, a teacher, a collaborator, a supervisor, or the like.
  • the processing device may receive, from the computing device, one or more second data feeds pertaining to at least instructions for performing the task.
  • the one or more second data feeds are received by the one or more processing devices of the vocational mask 130. and the one or more second data feeds are at least one of presented via a virtual retinal display of the vocational mask 130, emitted by an audio device (e.g., speaker) of the vocational mask 130, or produced or reproduced via a peripheral haptic device 134 coupled to the vocational mask 130.
  • the instructions are presented, by the virtual retinal display of the vocational mask 130. via augmented reality.
  • the instructions are presented, by the virtual retinal display of the vocational mask, via virtual reality during a simulation associated with the task.
  • the processing device may cause the virtual retinal display to project an image onto at least one iris of the first user to display alphanumeric data associated with the instructions, graphics associated with the instructions, animations associated with the instructions, video associated with the instructions, or some combination thereof.
  • the processing device may store, via one or more memory devices communicatively coupled to the one or more processing devices of the vocational mask 130, the one or more first data feeds and/or the one or more second data feeds.
  • the processing device may cause the peripheral haptic device 134 to vibrate based on the instructions received from the computing device 140.
  • the processing device may execute an artificial intelligence agent trained to perform at least one or more functions to determine certain information.
  • the one or more functions may include (i) identifying perception-based objects and features, (ii) determining cognition-based scenery' to identify one or more material defects, one or more assembly defects, one or more acceptable features, or some combination thereof, and (iii) determining one or more recommendations, instructions, or both.
  • FIG. 9 illustrates an example of a method 900 for implementing instructions for performing a task using a peripheral haptic device according to certain embodiments of this disclosure.
  • the method 900 may be performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software, or a combination of both.
  • the method 900 and/or each of their individual functions, subroutines, or operations may be performed by one or more processing devices of a computing device (e.g., any component (server 128, training engine 152, machine learning models 154, etc.) of cloud-based computing system 116, vocational mask 130, edge processor 132 (132.1, 132.2), peripheral haptic device 134, tool 136, and/or computing device 140 of FIG. 1) implementing the method 900.
  • a computing device e.g., any component (server 128, training engine 152, machine learning models 154, etc.
  • vocational mask 130 e.g., edge processor 132 (132.1, 132.2), peripheral haptic device 134, tool 136
  • the method 900 may be implemented as computer instructions stored on a memory device and executable by the one or more processors. In certain implementations, the method 900 may be performed by a single processing thread. Alternatively, the method 900 may be performed by two or more processing threads, each thread implementing one or more individual functions, routines, subroutines, or operations of the methods.
  • the method 900 is depicted and described as a series of operations. However, operations in accordance with this disclosure can occur in various orders or concurrently, and with other operations not presented and described herein. For example, the operations depicted in the method 900 may occur in combination with any other operation of any other method disclosed herein. Furthermore, not all illustrated operations may be required to implement the method 900 in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the method 900 could alternatively be represented as a series of interrelated states via a state diagram or events.
  • one or more machine learning models may be generated and trained by the artificial intelligence engine and/or the training engine to perform one or more of the operations of the methods described herein.
  • the processing device may execute the one or more machine learning models.
  • the one or more machine learning models may be iteratively retrained to select different features capable of enabling optimization of output.
  • the features that may be modified may include a number of nodes included in each layer of the machine learning models, an objective function executed at each node, a number of layers, various weights associated with outputs of each node, and the like.
  • the processing device may receive, at one or more processing devices of a vocational mask 130, first data pertaining to instructions for performing a task using a tool 136.
  • the first data may be received from a computing device 140 separate from the vocational mask 130.
  • the computing device may include one or more peripheral haptic devices, one or more vocational masks, one or more smartphones, one or more tablets, one or more laptop computers, one or more desktop computers, one or more servers, or some combination thereof.
  • the task includes welding and the tool 136 is a welding gun.
  • the processing device may transmit, via a haptic interface communicatively coupled to the one or more processing devices of the vocational mask 130, the first data to one or more peripheral haptic devices 134 associated with the tool 136 to cause the one or more peripheral haptic devices 134 to implement the instructions by at least vibrating in accordance with the instructions to guide a user to perform the task using the tool 136.
  • the processing device may receive, from a haptic interface, feedback data pertaining to one or more gestures, motions, surfaces, temperatures, or some combination thereof.
  • the feedback data may be received from the one or more peripheral haptic devices 134. and the feedback data may include information pertaining to the user's compliance with the instructions for performing the task.
  • the processing device may transmit, to the computing device 140, the feedback data.
  • transmitting the feedback data may cause the computing device 140 to produce an indication of whether the user complied with the instructions for performing the task.
  • the indication may be produced or generated via a display, a speaker, a different peripheral haptic device, or some combination thereof.
  • video data in addition to the first data being received, video data maybe received at the processing device of the vocational mask 130, and the video data may include video pertaining to the instructions for performing the task using the tool 136.
  • the processing device may display, via a virtual retinal display of the vocational mask 130, the video data.
  • the video data may be displayed concurrently with the instructions being implemented by the one or more peripheral haptic devices 134.
  • audio data may be received at the processing device of the vocational mask 130, and the audio data may include audio pertaining to the instructions for performing the task using the tool 136.
  • the processing device may emit, via a speaker of the vocational mask 130, the audio data.
  • the audio data may be emitted concurrently with the instructions being by the one or more peripheral haptic devices 134 and/or with the video data being displayed by the virtual retinal display. That is, one or more of video, audio, and/or haptic data pertaining to the instructions may be used concurrently to guide or instruct a user how to perform a task.
  • virtual reality data may be received at the processing device of the vocational mask 130, and the virtual reality data may include virtual reality multimedia representing a simulation of a task.
  • the processing device may execute, via at least a display of the vocational mask 130, playback of the virtual reality multimedia.
  • an artificial intelligent simulation generator may be configured to generate a virtual reality simulation for performing a task, such as welding an object using a welding gun.
  • the virtual reality simulation may take into consideration various attributes, characteristics, parameters, and the like of the welding scenario, such as type of object being welded, ty pe of welding, current amperage, length of arc, angle, manipulation, speed, and the like.
  • the virtual reality 7 simulation may be generated as multimedia that is presented via the vocational mask to a user to enable a user to practice, visualize, and experience performing certain welding tasks without actually welding anything.
  • augmented reality data may be received at the processing device of the vocational mask 130, and the augmented reality data may include augmented reality multimedia representing at leastthe instructions (e.g., via text, graphics, images, video, animation, audio).
  • the processing device may execute, via at least a display of the vocational mask 130, playback of the augmented reality multimedia.
  • the processing device may execute an artificial intelligence agent trained to perform at least one or more functions to determine certain information.
  • the one or more functions may include (i) identify ing perception-based objects and features, (ii) determining cognition-based scenery 7 to identify one or more material defects, one or more assembly defects, one or more acceptable features, or some combination thereof, and/or (iii) determining one or more recommendations, instructions, or both.
  • the processing device may display, via a display (e.g., virtual retinal display or other display), the objects and features, the one or more material defects, the one or more assembly defects, the one or more acceptable features, the one or more recommendations, the instructions, or some combination thereof.
  • FIG. 10 illustrates an example computer system 1000, which can perform any one or more of the methods described herein.
  • computer system 1000 may include one or more components that correspond to the vocational mask 130, the computing device 140, the peripheral haptic device 134. the tool 136, one or more servers 128 of the cloudbased computing system 116, or one or more training engines 152 of the cloud-based computing system 1 16 of FIG. 1.
  • the computer system 1000 may be connected (e.g., networked) to other computer systems in a LAN, an intranet, an extranet, or the Internet.
  • the computer system 1000 may operate in the capacity of a server in a client-server network environment.
  • the computer system 1000 may be a personal computer (PC), a tablet computer, a laptop, a wearable (e.g., wristband), a set-top box (STB), a personal Digital Assistant (PDA), a smartphone, a smartwatch, a camera, a video camera, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device.
  • PC personal computer
  • PDA personal Digital Assistant
  • smartphone a smartwatch
  • camera a video camera
  • any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device.
  • the term ‘'computer” shall also be taken to include any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.
  • the computer system 1000 includes a processing device 1002, a main memory 1004 (e.g., read-only memory (ROM), solid state drive (SSD), flash memory, dynamic random access memory' (DRAM) such as synchronous DRAM (SDRAM)), a static memory 1006 (e.g., solid state drive (SSD), flash memory, static random access memory (SRAM)), and a data storage device 1008, which communicate with each other via a bus 1010.
  • main memory 1004 e.g., read-only memory (ROM), solid state drive (SSD), flash memory, dynamic random access memory' (DRAM) such as synchronous DRAM (SDRAM)
  • DRAM dynamic random access memory'
  • SDRAM synchronous DRAM
  • static memory 1006 e.g., solid state drive (SSD), flash memory, static random access memory (SRAM)
  • SRAM static random access memory
  • Processing device 1002 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processing device 1002 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processing device 1002 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 1002 is configured to execute instructions for performing any of the operations and steps of any of the methods discussed herein.
  • CISC complex instruction set computing
  • RISC reduced instruction set computing
  • VLIW very long instruction word
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • DSP digital signal processor
  • network processor or the like.
  • the computer system 1000 may further include a network interface device 1012.
  • the computer system 1000 also may include a video display 1014 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), one or more input devices 1016 (e.g., a keyboard and/or a mouse), and one or more speakers 1018 (e.g., a speaker).
  • a video display 1014 e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)
  • input devices 1016 e.g., a keyboard and/or a mouse
  • speakers 1018 e.g., a speaker
  • the video display 1014 and the input device(s) 1016 may be combined into a single component or device (e.g., an LCD touch screen).
  • the data storage device 1008 may include a computer-readable medium 1020 on which the instructions 1022 embodying any one or more of the methodologies or functions described herein are stored.
  • the instructions 1022 may also reside, completely or at least partially, within the main memory 1004 and/or within the processing device 1002 during execution thereof by the computer system 1000. As such, the main memory' 1004 and the processing device 1002 also constitute computer-readable media.
  • the instructions 1022 may further be transmitted or received over a network 20 via the network interface device 1012.
  • computer-readable storage medium 1020 is shown in the illustrative examples to be a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
  • the term “computer-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure.
  • the term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.
  • systems and methods of the present disclosure are configured to implement a virtual coach or avatar to provide instructions, encouragement, etc. to the user.
  • a user interface may be configured to display and project the virtual coach via the virtual retinal display of a vocational mask as described herein.
  • FIGS. 11A-11C show an example user interface 1100 configured to display a virtual avatar or coach 1104.
  • the user interface 1100 presents an actual environment and objects (e.g., a tool 1108, a workpiece 1112, etc.) being viewed by the user through the vocational mask (e.g., through an eye opening, window, slot, etc. of the vocational mask 130) as well as projected graphical elements including, but not limited to, the virtual coach 1104, instructions 1116 pertaining to performing a task, etc.
  • the instructions 1116 may be generated by one or more machine learning models, or may be provided via one or more computing devices and/or other vocational masks being used by other users (e.g., master user, collaborator, teacher, supervisor, etc.).
  • the instructions 1116 and/or other commands or feedback, control signals, data, etc. may be provided to the user interface 1100 (e.g., to a computing device implementing the user interface 1100 at the vocational mask, such as the computing device 140) via a cloud-based computing system 1 120.
  • the virtual coach 1104 is selected, generated, displayed, controlled, etc. based on one or more criteria as described below in more detail.
  • the virtual coach 1104 may have one or more personas (e.g.. different professions, costumes or uniforms, characters, historical figures, etc., non-human personas such as animals, etc.), demeanors, appearances, voices, or other characteristics. Characteristics of the virtual coach 1104 may be modified based on user feedback and/or user performance (e.g., in real-time, based on historical performance over time, etc.).
  • the virtual coach 1104 provides instructions via the user interface 1100 of the vocational mask as the user performs a work task (e.g., a welding task).
  • the instructions may be provided as text instructions (e.g., via a text bubble including the instructions 1116), voice instructions (e.g., audio instructions provided via speakers of the vocational mask as described herein), or combinations thereof.
  • Criteria for generating/controlling the virtual coach 1104 may include user characteristics, performance-based characteristics, task-based characteristics, or combinations thereof.
  • user characteristics may include, but are not limited to, user demographics or psychographics data, task-related physical characteristics (e.g., height, right-or-left-handedness. etc.), and so on.
  • Performance-based characteristics may include, but are not limited to, historical performance data for the user and/or other users performing the same or similar task, real-time performance data collected as the user performs the current task, predictive data (e.g., generated by an ML model based on the historical performance data, the real-time performance data, etc.) indicative of predicted performance for a remainder of the current task, etc.
  • Task-based characteristics correspond to characteristics of the specific task being performed and may include, but are not limited to, a type or category of the task (e.g., welding), an expected or average duration of the task (e.g., based on historical data), an amount of time the user has been performing the current task, an estimated amount of time remaining to perform the current task, a difficulty rating assigned to the task (e.g.. on a numerical or other scale, such as a 1-10 scale), a number of times the user performed has performed the current task (e.g., in previous sessions), etc.
  • a type or category of the task e.g., welding
  • an expected or average duration of the task e.g., based on historical data
  • an amount of time the user has been performing the current task e.g., an estimated amount of time remaining to perform the current task
  • a difficulty rating assigned to the task e.g. on a numerical or other scale, such as a 1-10 scale
  • a number of times the user performed has performed the current task
  • the virtual coach 1104 is displayed in a fixed or static region of the user interface 1100.
  • a position of the virtual coach 1104 in the user interface 1100 is variable or dynamic.
  • the virtual coach 1104 may be displayed different regions of the user interface 1100 based on the task being performed, movement of the user, movement of the tool 1108, movement of the workpiece 1112, etc.
  • the virtual coach 1104 is displayed in an unoccupied region of the user interface 1100 (e.g., a region of the user interface 1100 that does not include the tool 1108, the workpiece 1112, etc.
  • the virtual coach 1104 may move to a different, unoccupied region of the user interface 1100.
  • the computing device may be configured to determine whether a current location of the virtual coach 1 104 overlaps with other objects/graphical elements (e.g., occupies a same region as another object in an x-y coordinate system of the user interface 1100), identify an unoccupied region of the user interface 1100, and change the displayed location of the virtual coach 1104 to the unoccupied region.
  • a size of the virtual coach 1 104 may be adjusted to fit within a particular unoccupied region of the user interface 1100.
  • the workpiece 1112 substantially almost an entire field-of-view of the user interface 1100. Accordingly, in this example, the virtual coach 1104 may be omitted.
  • the workpiece 1112 may not be in the field-of- view of the user interface 1100.
  • the virtual coach 1104 is enlarged and displayed in a central region of the user interface 1100.
  • the virtual coach 1104 may be presented performing a same or similar task as the user (e.g.. performing a welding task on a workpiece similar to the workpiece 11 12, performing a welding task on a same portion of the workpiece 1112, welding along a same or similar path or welding line, etc.).
  • movement/actions of the virtual coach 1104 may be automatically controlled by the computing device (e.g., Al- or ML model-controlled) and/or controlled/responsive to commands from a trainer, teacher, or other user as described herein.
  • the virtual coach 1104 may perform preset/predetermined movements or actions (e.g.. a repeated movement, such as a looped animation, corresponding to a welding task being performed, which may demonstrate preferred motion or movement, speed, posture, etc. for the welding task).
  • preset/predetermined movements or actions e.g.. a repeated movement, such as a looped animation, corresponding to a welding task being performed, which may demonstrate preferred motion or movement, speed, posture, etc. for the welding task.
  • the virtual coach 1104 may perform one of a plurality of possible movements or actions in response to commands from the trainer.
  • the virtual coach 1104 may be controlled based on actual movements/actions of a trainer.
  • the trainer may selectively “inhabit” the virtual coach 1104 (e.g., from a remote location).
  • the trainer may be presented with, on a display, a duplicate of the user interface 1100 to observe the actions of the user while performing the task.
  • the trainer may be prompted to initiate or selectively initiate a demonstration mode in which the movement of the virtual coach 1104 is controlled by the movement of the trainer.
  • the trainer may perform a same or similar welding task on a workpiece or simply mimic preferred movements or postures for the welding task, which are then presented to the user in the user interface 1100 via movements of the virtual coach 1104.
  • movement of the trainer may be monitored using various sensors as described herein, transmitted to the computing device associated with the vocational mask (e.g., via the cloud-based computing system 1 120), and presented to the user via the virtual coach 1104.
  • the appearance of the virtual coach 1104 may change based on whether the virtual coach 1104 is being controlled by the computing system or the trainer. For example, in response to the trainer taking control or inhabiting the virtual coach 1104, the appearance of the virtual coach 1104 may change to a persona associated with the trainer.
  • one or more display-related characteristics of the virtual coach 1104 may vary based on performance of the task by the user. For example, to reduce a likelihood of distracting the user while performing a welding task, a determination of whether to display the virtual coach 1104, a size of the virtual coach 1104, movement of the virtual coach, etc.
  • the virtual coach 1104 may be triggered based on one or inputs corresponding to actions performed by the user (e.g., based on whether the user is actively welding, as indicated by a welding button or trigger 1124 on the tool 1108 being actuated). For example, in response to the trigger 1124 being actuated, the virtual coach 1104 may be removed from the user interface 1100, a size of the virtual coach 1104 may be decreased, movement of the virtual coach 1104 may be stopped, volume of audio instructions may be decreased (or, in some examples, increased), etc.
  • a graphical element representing the user such as a virtual avatar or user 1128
  • the virtual user 1128 may mirror/imitate a posture, movements, etc. of the user.
  • the virtual coach 1104 may overlay the virtual user 1128 (e.g., the virtual coach 1104 may be at least partially transparent such that the virtual user 1128 is visible through the virtual coach 1104).
  • the virtual coach 1104 may be controlled to demonstrate a target or desired posture or movement.
  • the virtual coach 1104 may be represented as a looped or repeated animation demonstrating the task being performed by the user.
  • a state or frame of animation of the virtual user 1128 may correspond to a current position or state of the actual user performing the welding task. According, the user can view a comparison between the user’s movements/posture and the target or desired movements/posture.
  • welding tasks are typically associated with challenges including, but are not limited to, user coordination (e.g., coordination of wire feeding, hand position, tool alignment, straightness of a weld, etc.), heat control (e.g., sufficient heat to perform the weld but maintaining heat below an amount that could damage the workpiece 1112), travel speed (e.g., travel speed of the tool 1108), and various safety considerations (heat, presence of fumes/gases, etc.).
  • user coordination e.g., coordination of wire feeding, hand position, tool alignment, straightness of a weld, etc.
  • heat control e.g., sufficient heat to perform the weld but maintaining heat below an amount that could damage the workpiece 1112
  • travel speed e.g., travel speed of the tool 1108
  • various safety considerations heat, presence of fumes/gases, etc.
  • characteristics of the virtual coach 1104 may change based on performance data indicative of one or more of user coordination, heat control, travel speed, and safe considerations.
  • Other performance data may include, but is not limited to, data indicative of an amount of welding material used, thickness of a weld, porosity of a weld, penetration of a weld, and so on.
  • a displayed characteristic such as a color of clothing or other element, persona, expression or demeanor, size, position on the user interface 1100, etc. of the virtual coach 1104 may change based on the performance data.
  • a color, persona, expression, etc. of the virtual coach 1104 may have two or more levels or tiers, such as a one level indicating satisfactory performance (e.g. measured indicator below a threshold) and another level indicating unsatisfactory' performance (e.g., a measured indicator greater than or equal to the threshold).
  • the displayed characteristic may include a first level indicating that the measured indicator is at least a first amount less than the threshold, a second level indicating that the measured indicator is approaching the threshold (e.g., within a certain range of the threshold), and a third level indicating that the measured indicator is greater than or equal to the threshold.
  • the various levels may be represented by discrete states of the virtual coach 1104 and/or as a continuum or gradient.
  • the measured indicator may correspond to a distance or amount of variation of an actual yveld line from a target weld line and the threshold may correspond to a maximum allowable distance from the target yveld line.
  • the measured indicator may correspond to a measured, sensed, estimated, or modeled heat value (e.g., based on inputs from sensors in the vocational mask, the tool 1108, in a vicinity of the workpiece 11 12, etc.) and the threshold may include a first threshold corresponding to a minimum alloyvable heat and a second threshold corresponding to a maximum alloyvable heat.
  • the measured indicator may correspond to a measured, sensed, estimated, or modeled speed (e.g., based on inputs from a sensor in the tool 1108 and/or attached to the user, a calculation based on observed movement of the tool 1108 within the user interface 1100, etc.) and the threshold may include a first threshold corresponding to a minimum alloyvable travel speed and a second threshold corresponding to a maximum allowable travel speed.
  • the measured indicator may correspond to measured, sensed, estimated, or modeled values of environmental indicators such as heat, gases, etc. and the threshold may correspond to a maximum measured amount or value.
  • a color of the virtual coach 1104 may cycle through three possible colors (e.g., green, yellow, and red) in accordance with the first, second, and third levels as described above.
  • an expression or demeanor of the virtual coach 1104 may cycle through three possible expressions, such as a happy or smiling expression, a neutral or concerned expression, and an angry or fearful expression.
  • a posture of the virtual coach 1104 may cycle through three possible postures, such as a positive posture (e.g., a “thumbs up” gesture), a neutral or concerned posture (e.g., a wagging finger), and a negative posture (e.g.. a “thumbs down” gesture).
  • a positive posture e.g., a “thumbs up” gesture
  • a neutral or concerned posture e.g., a wagging finger
  • a negative posture e.g.. a “thumbs down” gesture
  • the changing or cycling may be the same for each display characteristic (e.g., the same possible colors for each level of heat control, travel speed, etc.) or may be different for each display characteristic.
  • the cycling between levels may be accompanied by one or more icons 1132 (e.g., a fire extinguisher for heat control) or other graphical elements indicating which performance data is being indicated by the cycling display characteristic.
  • each display characteristic may be associated with a particular type of performance data. For example, cycling expression may indicate different coordination levels, cycling color may indicate different heat control levels, cycling posture may indicate different safety consideration levels, etc.
  • display characteristics of the virtual coach 1104 may dynamically change to provide different types of information to the user corresponding to different performance or other types of indicators.
  • display characteristics may refer to audio characteristics, such as characteristics of a voice or vocal instructions associated with the virtual coach 1104, such as volume, tone, etc. of the voice.
  • Display characteristics of the virtual coach 1104 may change based on progress (e.g., a percentage or other measurement of a portion of the task that has been completed) and/or performance (e.g., one or more measures of performance, such as a calculated performance score or grade). For example, performance may be measured based on a comparison between performance of the current task and a baseline, reference, or target performance level of the task, a comparison betw een the performance of the current task and previous performances of the task by the same user or other users, etc. In an example, performance may be calculated (e.g., as a performance score) based on one or more performance values, which may correspond to measured or sensed values, calculated values, modeled or estimated values, or combinations thereof.
  • a performance score may be based on a calculation of an overall amount of variation of performance values from one or more target values (e.g., a running average of performance values), an amount of times respective performance values varied (e.g., by at least a predetermined amount) from one or more target values, etc.
  • target values e.g., a running average of performance values
  • an amount of times respective performance values varied e.g., by at least a predetermined amount
  • the performance values may be obtained using one or more techniques, including, but not limited to, imaging techniques (e.g., image processing of images displayed via the user interface 1100 to monitor and measure characteristics of the weld and work environment), acoustic sensing techniques (e.g., using one or more acoustic sensors as described herein), modeling techniques (e.g., techniques responsive to inputs from various sensors in the work environment, vocational mask, on or around the user, etc ), Al and ML techniques, and so on.
  • imaging techniques e.g., image processing of images displayed via the user interface 1100 to monitor and measure characteristics of the weld and work environment
  • acoustic sensing techniques e.g., using one or more acoustic sensors as described herein
  • modeling techniques e.g., techniques responsive to inputs from various sensors in the work environment, vocational mask, on or around the user, etc
  • Al and ML techniques and so on.
  • the performance score is calculated (e g., using one or more of the ML models 154) based on a combination of performance values each corresponding to one or more of a plurality of measured, sensed, calculated, estimated, etc. values.
  • the performance score may be a numerical value, such as an integer or non-integer value (e.g. , a value on a scale from 1 to 10, from 1 to 100, etc.) or a percentage.
  • Each individual performance value may be calculated in accordance with a variation from a target value, where a maximum performance value (e.g., 100) is assigned for measurements within a minimum threshold of a target or baseline value and a minimum performance value (e.g., 0) is assigned for measurements greater than or equal to a maximum threshold of the target or baseline value.
  • the target or baseline value may be a predetermined value (e.g., a predetermined target temperature), a dynamic value based on historical performance data (e.g., an average or other combination of temperatures measured during previous performances of the current task by the same or different users), or combinations thereof.
  • the performance score is an average or weighted average of the individual performance values.
  • one or more of the ML models 154 may be trained to obtain weights for each of the performance values based on assessments of overall weld quality (i.e., for a completed weld by the user and/or other users) and the various performance values calculated during performance of the respective welds.
  • an ML model 154 may be trained to identify which performance values have the greatest effect on overall weld quality and assign weights accordingly.
  • the weights may correspond to percentages or decimal values between 0 and 1, a ranking of the performance values (e.g., from 1 to n or from n to 1 , where n is an integer greater than one), etc.
  • the performance score may be continuously calculated and updated as the user performs the welding task and the virtual coach 1104 is generated and controlled based on the performance score (e.g., display characteristics such as colors, persona, demeanor, etc. of the virtual coach 1104 are changed based on the performance score).
  • the display characteristics may be changed between discrete values or levels as described herein, changed in a gradient manner, etc.
  • the virtual coach 1104 may be controlled based in part on improvement or a rate of improvement (over time for the current task, a plurality of tasks, etc.) for the user and/or other users.
  • improvement may be measured based on improvement in the overall performance score and/or individual performance values for the current task, improvement from task-to-task, over time, for a plurality of tasks, etc.
  • various rates of improvement may be indexed to or correlated with (e.g., via an ML model) respective display characteristics and levels as described herein such that responsiveness to various display characteristics, for a given user or group of users, can be calculated.
  • different users may show increased or decreased rates of improvement in response to different display characteristics of the virtual coach 1104 during specific stages or actions of a task.
  • some users or types of users may be more responsive to higher levels (e.g., the third level described herein) and demonstrate greater rates of improvement while other users or ty pes of users may be less responsive to higher levels and demonstrate lower rates of improvement or degraded performance.
  • data indicative of improvement and/or rate of improvement may be referred to as “improvement data.’'
  • the ML models according to the present disclosure may be trained to cycle through the various levels, omit or skip certain levels, use or not use certain colors or expressions, and change the display based on observed increases or decreases in performance, rates of improvement, etc. of various users over time.
  • training data may include labeled inputs (e.g.. increased or decreased performance scores, rates of improvement, etc. of users over time) mapped to labeled outputs (e.g., various levels, colors or expressions, etc. to modify a display).
  • the training data may be used to train one or more of the ML models.
  • FIG. 12 illustrates steps of an example method 1200 of controlling a user interface and virtual coach according to embodiments of the present disclosure.
  • One or more computing systems, computing devices, processors or processing devices. AI/ML models, etc. as described herein may be configured to perform various steps of the method 1200. While the method 1200 may be implemented for performing different types of tasks, the steps of the method 1200 are described with respect to a welding task for illustration purposes only.
  • a user begins performing a welding task (e.g.. on a workpiece.). Beginning to perform the welding task may include, but is not limited to, putting on a vocational mask as described herein and powering up and initializing welding equipment.
  • the user may log in or otherwise provide identification data (e.g.. to a computing system) that includes or provides an indication of various user characteristics, previous performance data (e.g., historical data), user settings or preferences, etc.
  • beginning to perform the welding task may include providing an indication of the welding task, such as a type of welding task, a type of workpiece, etc.
  • the method 1200 includes, at step 1204, obtaining data associated with the user and/or the welding task (e.g., from the database 129).
  • the data may include, is not limited to, data identifying one or more characteristics of the user, historical data associated with performance of the welding task or other task by the user or other users, operating characteristics (e.g., target measurement or performance values) associated with the welding task or workpiece, etc.
  • the method 1200 includes generating a virtual coach (and/or, in some examples, a virtual user) and displaying the virtual coach on a user interface of the vocational mask.
  • the virtual coach may be a default (i.e., predetermined) virtual coach for the user or group of users, a virtual coach selected based on characteristics of the user, a virtual coach selected based on previous performances (e.g., welding sessions) of the user. etc.
  • the method 1200 includes obtaining performance data while the user performs the welding task.
  • the performance data may include measured, sensed, calculated, estimated, and/or modeled values as described herein.
  • the method 1200 includes obtaining performance values based on the performance data. The performance values may be calculated based on comparisons between the performance values and one or more respective reference, baseline, or target values as described herein.
  • the method 1200 includes generating a performance score based on the performance data and/or performance values.
  • generating the performance score may include providing the performance values and/or performance data, historical performance data (e.g., for the user and/or other users), user characteristics (e.g., for the user and/or other users), or any other data described herein to one or more ML models trained to generate the performance score.
  • the performance score is a weighted average of the performance values, and the one or more ML models are configured to calculate and assign weights to the performance values as described herein.
  • the method 1200 includes controlling the virtual coach (e.g., controlling one or more display characteristics of the virtual coach) based on the performance score and, in some examples, further based on historical performance scores, values, and/or data for the user and/or other users.
  • the display characteristics may simply be controlled based on a comparison between the performance score and various ranges of performance scores (e.g., a determination of whether the performance score falls within a first range, a second range, a third range, etc.).
  • a first range may be defined as a range of performance scores between 0 and m
  • a second range may be defined as a range between m and n
  • a third range may be defined as a range between n and 100.
  • the ranges may be same or different sizes, the ranges may be variable, and more or fewer ranges may be implemented.
  • the display characteristics may be controlled based on comparisons between respective performance values and various ranges of performance values (e.g., comparisons between temperature values and ranges, travel speed values and ranges, etc.). In other words, in various examples, the display characteristics may be controlled based on an overall performance score (e.g., a performance score based on a combination of a plurality of performance values) and/or based on a plurality of different performance values corresponding to different measured (or calculated, modeled, etc.) values. [0159] In other examples, controlling the virtual coach includes controlling the display characteristics further based on improvement data as described herein. For example, the method 1200 may include providing the performance score, performance data, etc.
  • the ML model may be configured to select/adjust various display characteristics further based on the improvement data.
  • the performance ranges may be adjusted based on the improvement data (e g., one or more performance ranges may be increased or decreased based on improvement data indicating that the user or other users or more likely or less likely to improve in response to specific display characteristics).
  • one or more performance ranges may be eliminated.
  • various display characteristics may be omitted (e.g., one or more colors, expressions, personas, etc.
  • audio characteristics may be adjusted (e.g., volume may be increased or decreased, voice or audio tones may be omitted or favored, etc.).
  • the ML model is configured to generate control outputs based on improvement data indicative of which display characteristics are most effective for improving performance of a user or group of users.
  • FIG. 13 illustrates steps of an example method 1300 of conducting a shared virtual reality session between an apprentice and a master.
  • method 1300 includes wearing, by a first user, a first vocational mask.
  • the first vocational mask may include a virtual retinal display, a memory device storing instructions, and a processing device communicatively coupled to the memory and the virtual retinal display.
  • the first vocational mask may correspond to any of the various embodiments of the same discussed above.
  • the method includes executing, using the processing device, the instructions stored in the memory device, while at step 1313, the method includes sharing, in response to execution of the instructions, a virtual reality session with a computing device associated with a second user.
  • the sharing of the virtual reality session comprises the first user conducting bidirectional communications, via the first vocational mask, with the computing device associated with the second user, wherein the first user is an apprentice and the second user is a master.
  • a master may conduct training of the apprentice in the performance of a specific task, such as welding, brazing, soldering, various mechanical tasks, and so on.
  • a network such as network 20 of Fig. 1, this training may be conducted even when the master and the apprentice are at geographically disparate locations with respect to one another.
  • method 1300 includes presenting, to the first user using the first vocational mask, incoming data received from the computing device associated with the second user.
  • the incoming data is received in a plurality of formats, and wherein the data includes directions for performing a task by the first user under direction from the second user.
  • method 1300 may include projecting the incoming data, using the first vocational mask, to a retina of the first user, wherein the incoming data is presented in one or more of the plurality of formats.
  • Sharing the virtual reality session with the computing device associated with a second user may include, in one non-limiting example, sharing the virtual reality session with a second vocational mask worn by the second user.
  • the disclosure further contemplates, as the computing device associated with the second user, tablets, smartphones, laptop computers, desktop computers, and any other computing device suitable for sharing a virtual reality session with the first user via the first vocational mask.
  • the plurality of data formats may include one or more of the following image data, video data, audio data, and haptic feedback.
  • a master may convey video data indicating a proper procedure for carrying out a particular task, still image data indicating an appearance of an end result of the task (e.g., a properly formed weld), and audio data to supplement the other formats.
  • data may be conveyed in the form of haptic feedback in order to instruct an apprentice of the proper feel for carrying out the particular task for which instruction is being provided.
  • the method may, in various embodiments, include sharing the virtual reality session when the first user and the second user being located remotely with respect to one another.
  • Such remote locales may, in various examples, include different locations within the same facility wherein direct verbal communications may be impractical or impossible, in different cities, or in different geographic regions for which the communications are conducted using long-distance communication mechanisms (e.g., via the internet).
  • Method 1300 is not limited to a master instructing a single apprentice. Accordingly method 1300 may include, in various embodiments, sharing the virtual reality session among a plurality of computing devices associated with respective ones of a plurality of users, wherein the plurality of computing devices includes the first vocational mask and one or more additional vocational masks worn by respective ones of a subset of a plurality of users, wherein the subset of the plurality of users includes the first user and one or more additional users, wherein the one or more additional users are apprentices.
  • at least one apprentice may be remotely located with respect to at least one other apprentice, and may also be located remotely with respect to the master.
  • a master in a first geographic location may, in one embodiment of the method, concurrently instruct a first apprentice in a second geographic location, a second apprentice in a third geographic location, and a third apprentice in fourth geographic location, wherein the first, second, third, and fourth geographic locations are all remote with respect to one another.
  • executing the instructions stored in the memory device comprises executing an artificial intelligence agent trained to monitor one or more aspects of the virtual reality session and further trained to provide, in real time (e.g., less than 2 seconds), directions for performing the task to the first user during the virtual reality session.
  • the task is welding, and wherein the method further comprises monitoring, using the artificial intelligence agent, one or more properties of a weld formed by the first user and adjusting, using the artificial intelligence agent, one or more properties of the weld based on the monitoring.
  • the use of the artificial intelligence agent may be similarly applied to other tasks as well.
  • FIG. 14 illustrates examples of respective user interfaces for both an apprentice and a master during training to perform a particular task.
  • the illustrated example includes an apprentice user interface (UI) 1401 , and a master UI 1402.
  • UI apprentice user interface
  • master UI 1402. Through the apprentice UI, a work piece 1410 comprising a cylindrical object and a flat object are shown, along with a welding gun 1411.
  • the work piece 1410 along with the welding gun 1411 can be seen, allowing the master to monitor the performance of the task by the apprentice.
  • the work piece 1410 and the welding gun 1411 may be virtual graphical objects presented concurrently by the UI 1401 and the master UI 1402.
  • instructions for carrying out the task are displayed.
  • the instructions include adjusting a wire speed and voltage, as well as the positioning of the welder at 45° to weld a bevel around the top of the cylinder.
  • the performance of the task is monitored by a master through the master UI 1402.
  • an Al agent may be executed on the vocational mask of the apprentice.
  • the Al agent may provide monitoring in addition to that provided by the master through the master UI.
  • the Al agent may monitor the weld joint between the cylinder and the flat object for quality and proper form.
  • the Al agent may be trained, using training data, to identify images, screens, objects, and the like in the virtual reality’ session, determine contextual meaning corresponding to the identified images, screens, objects, etc., and determine, based on the contextual meaning, recommended directions and/or information to present to the apprentice.
  • the training data may include labeled inputs (e.g., images, video, audio, haptic data, etc.) mapped to labeled outputs (e.g., directions, information, recommendations).
  • the master UI 1402 displays information that the weld joint is too thin and that the apprentice needs to increase the travel speed of the weld gun.
  • the apprentice UI 1401 displays an additional instruction that the apprentice is to increase the travel speed of the weld gun, along with the reasoning behind this instruction.
  • the disclosure contemplates a system comprising a first vocational mask configured to be worn by a first user, wherein the first vocational mask comprises: a virtual retinal display; a memory device storing instructions; and a processing device communicatively coupled to the memory device and the virtual retinal display, wherein the instructions, when executed by the processing device, are configured to cause the first vocational mask to: share a virtual reality session with a computing device associated with a second user, wherein the first vocational mask is configured to conduct bidirectional communications with the computing device of the second user, wherein the first user is an apprentice and the second user is a master; and receive and present, via the virtual retinal display to the first user, data in selected ones of a plurality of formats from the second user, the data comprising directions for performing a task by the first user under direction from the second user.
  • the first vocational mask is configured to receive the data from a second vocational mask worn by the second user.
  • the plurality of formats (by which data may be received at the first includes audio data.
  • the plurality of formats includes video data.
  • the plurality of formats includes image data.
  • the first vocational mask includes one or more peripheral haptic devices, and wherein the instructions, when executed by the processing device, are configured to cause the first vocational mask to present, as one of the plurality of formats, haptic feedback to the first user.
  • the first vocational mask in various embodiments is configured to project data in at least one of the plurality of formats to a retina of the first user via the virtual retinal display. [0178] The first vocational mask is configured to convey feedback data in one or more of the plurality of formats to the computing device of the second user.
  • the first user is located remotely with respect to the second user, and thus the first vocational mask is located remotely with respect processing device used by the second user.
  • the system further comprises a plurality of vocational masks including the first vocational mask, wherein each of the plurality of vocational masks is configured to conduct bidirectional communications with the computing device associated with the second user.
  • at least a subset of the plurality of vocational masks may be configured to receive and present data in a plurality of formats from the computing device associated with the second user, wherein respective users of the subset are apprentices receiving the directions for performing the task.
  • the instructions when executed by the processing device, are configured to execute an artificial intelligence agent trained to monitor one or more aspects of the virtual reality session and further trained to provide, in real time, directions for performing the task to the first user during the virtual reality session.
  • the task is welding, and wherein the artificial intelligence agent is trained to monitor one or more properties of a weld formed by the first user, and further trained to adjust directions to the first user based on the one or more properties of the weld.
  • the disclosure further contemplates a non-transitory computer-readable medium storing instructions that, when executed by a processing device of a first computing system, cause the first computing system to generate, for a first user wearing a first vocational mask that comprises the first computing system, a virtual reality session, wherein the first vocational mask includes a virtual retinal display; share the virtual reality session with a second computing system associated with a second user, wherein sharing the virtual reality session comprises the first user conducting bidirectional communications, via the first vocational mask, with a second computing system associated with the second user, wherein the first user is an apprentice and wherein the second user is a master; and present, to the first user, using the first vocational mask, data received from the computing device associated with the second user, wherein the data is received in a plurality of formats, and wherein the data comprises directions for performing a task by the first user under direction from the second user.
  • the first vocational mask includes a virtual retinal display, and wherein the instructions are executable to cause data to be displayed in at least a subset of the plurality of formats as visual data displayed to the first user via the virtual retinal display.
  • the subset of the plurality of formats includes video data and image data.
  • the first vocational mask includes at least one haptic peripheral device
  • the instructions are executable to cause haptic feedback, as one of the plurality of formats, to be presented to the first user via the at least one haptic peripheral device.
  • the instructions when executed by the processing device of the first computing system, cause the first computing system to execute an artificial intelligence agent trained to monitor one or more aspects of the virtual reality session and further trained to provide, in real time, directions for performing the task to the first user during the virtual reality session.
  • the disclosure further contemplates that the instructions, when executed by the processing device of the first computing system, cause the first computing system to share the virtual reality session among a plurality of computing devices associated with respective ones of a plurality of users, wherein the plurality of computing devices includes the first vocational mask and one or more additional vocational masks worn by respective ones of a subset of a plurality of users, wherein the subset of the plurality of users includes the first user and one or more additional users, wherein the one or more additional users are apprentices.
  • the disclosure further contemplates that the instructions, when executed by the processing device of the first computing system, cause the first computing system to send feedback data to the second computing system.
  • the disclosure additionally contemplates a vocational mask configured to be worn by a first user, wherein the vocational mask comprises: a virtual retinal display; a memory device storing instructions; one or more peripheral haptic devices; a processing device communicatively coupled to the memory' device and the virtual retinal display, wherein the processing device executes the instructions to: share a virtual reality session with a computing device associated with a second user, wherein the vocational mask is configured to conduct bidirectional communications with the computing device of the second user, wherein the first user is an apprentice and the second user is a master; receive and present, via the virtual retinal display to the first user, data in selected ones of a plurality of formats from the second user, the data comprising directions for performing a task by the first user under direction from the second user; cause the vocational mask to present, via at least one of the one or more peripheral haptic devices, haptic feedback to the first user; and execute an artificial intelligence agent trained to monitor one or more aspects of the virtual reality session and further trained to provide, in real time, directions
  • FIG. 15 illustrates steps of an example of a method of conducting a shared virtual reality session between a first user and a second user operating in switchable first and second contexts.
  • Method 1500 includes, at step 1504, wearing, by a first user, a first vocational mask, the first vocational mask having a virtual retinal display, a memory' device storing instructions, and a processing device communicatively coupled to the memory and the virtual retinal display.
  • the method includes executing, using the processing device, the instructions stored in the memory device.
  • the method includes sharing, in response to execution of the instructions, a virtual reality session with a computing device associated with a second user, wherein sharing the virtual reality session comprises conducting bidirectional communications with the computing device of the second user, wherein the first user is operating in a first context and wherein the second user is operating in a second context.
  • the method includes presenting, via the virtual retinal display to the first user, data in selected ones of a plurality of formats received from the computing device associated with the second user, the data comprising information regarding a task to be performed, at least in part, by the first user.
  • the method includes executing instructions to cause the first user to switch to the second context concurrent with the second user switching to the first context.
  • the first user is a teacher and the second user is a student.
  • the first and second users are collaborators on the task.
  • Various embodiments of the method include comprising the first vocational mask receiving the data from a second vocational mask worn by the second user.
  • the plurality of formats includes video data and still image data.
  • the plurality of formats includes audio data.
  • Some embodiments of the method may include executing instructions to cause the first vocational mask to present, as one of the plurality of formats, haptic feedback to the first user via a haptic peripheral device coupled to the first vocational mask.
  • the first and second users are collaborators on the task
  • the method further comprises the artificial intelligence agent providing a first description indicating a first portion of the task to be performed by the first user and a second portion of the task to be performed by the second user.
  • FIG. 16 illustrates examples of respective user interfaces for first and second users sharing a virtual reality session with switchable contexts.
  • a first user UI (user interface) 1601 and a second user UI 1602 are shown.
  • the first user UI 1601 may be a display from a vocational mask as discussed above.
  • the second user UI 1602 may be that of a computing device used by the second user.
  • the computing device associated with the second user may be another instance of a vocational mask in one implementation.
  • the disclosure contemplates that the computing device associated with the second user may be a laptop computer, a desktop computer, a smart phone, a tablet computer, or any other suitable computing device.
  • the first and second users in the example shown may be collaborators working to perform a task together, even though they may otherwise be separated in distance.
  • the task carried out may be a repair on a complex piece of equipment that can be controlled locally in some instances and remotely in others.
  • the repair may at various times utilize local control of the equipment, while utilizing remote control of the equipment at other times.
  • the repair operation may include the first user controlling the equipment locally, while the second user remotely performs software updates remotely.
  • the operation involving the first and second users may thus include separate contexts.
  • one of the users may perform an active role during the repair operation, while the other user may perform an observational role.
  • these contexts may be switched between the users during the performance of the tasks.
  • the first user U1 1601 is operating in a first context (Context #1) and the second user UI 1602 is operating in a second context (Context #2).
  • the first context in this example is an operational context, and initially includes instructions for the first user to perform various steps of the task, while the second context is an observational context.
  • the context may be switched, with the first (operational) context being transferred to second user UI 1602 while the second (observational) contexts being transferred to first user UI 1601.
  • the second user may perform portions of the task as outlined in the instruction displayed on second user UI 1602, while the first user may assume an observational role.
  • the example discussed herein is one in which the first and second users are collaborators in the performance of a task.
  • the disclosure is not limited to such examples.
  • the first and second users may have a master-apprentice or teacher-student relationship as well.
  • the switching of contexts may enable a master/teacher to demonstrate a task with the apprentice/student observing, before switching contexts to allow the apprentice/student to perform the previously demonstrated task while the master/teacher observes.
  • the disclosure further contemplates the use of the systems and methods described here in an augmented environment. For example, a surgeon may carry out a complex surgery' on a patient with augmentation of another user in an observational context through which advice and instruction may be relayed.
  • doctor-resident relationship is also contemplated within the contexts discussed herein.
  • the disclosure also contemplates examples in which multiple users operate in a first context with one or more additional users operating in a second context. Examples with multiple users and more than two contexts are also possible and contemplated.
  • the disclosure contemplates a system comprising a first vocational mask configured to be worn by a first user, wherein the first vocational mask comprises: a virtual retinal display; a memory’ device storing instructions; and a processing device communicatively coupled to the memory device and the virtual retinal display, wherein the instructions, when executed by the processing device, are configured to cause the first vocational mask to: share a virtual reality session with a computing device associated with a second user, wherein the first vocational mask is configured to conduct bidirectional communications with the computing device of the second user, wherein the first user is operating in a first context and wherein the second user is operating in a second context; receive and present, via the virtual retinal display to the first user, data in selected ones of a plurality' of formats received from the computing device associated with the second user, the data comprising information regarding a task to be performed, at least in part, by the first user.
  • the processing device is further configured to execute instructions to cause the first user to switch to the second context concurrent with the second user switching to the first context.
  • the first user is a teacher and the second user is a student.
  • the first and second users are collaborators on the task.
  • the first vocational mask is configured to receive the data from a second vocational mask worn by the second user.
  • the plurality of formats includes video data and still image data.
  • the plurality' of formats includes audio data.
  • the first vocational mask includes one or more peripheral haptic devices, and wherein the instructions, when executed by the processing device, are configured to cause the first vocational mask to present, as one of the plurality of formats, haptic feedback to the first user.
  • the first vocational mask is configured to project data in at least one of the plurality of formats to a retina of the first user via the virtual retinal display.
  • the first user is located remotely with respect to the second user.
  • the instructions when executed by the processing device, are configured to execute an artificial intelligence agent trained to monitor one or more aspects of the virtual reality session and further trained to provide, in real time, information regarding the task to the first user during the virtual reality session.
  • the information regarding the task may comprise directions for performing the task.
  • the information may comprise a first description indicating a first portion of the task to be performed by the first user, wherein the artificial intelligence agent is further configured to generate a second description indicating a second portion of the task to be performed by the second user.
  • a non-transitory computer-readable medium storing instructions that, when executed by a processing device of a first computing system, cause the first computing system to: generate, for a first user wearing a first vocational mask that comprises the first computing system, a virtual reality session, wherein the first vocational mask includes a virtual retinal display: share, in response to execution of the instructions, a virtual reality session with a computing device associated with a second user, wherein sharing the virtual reality session comprises the first user conducting bidirectional communications, via the first vocational mask, with a second computing system associated with the second user; operate, by the first user using the first vocational mask, in a first context concurrent with the second user operating in a second context using the second computing system; and present, via the virtual retinal display to the first user, data in selected ones of a plurality of formats received from the computing device associated with the second user, the data comprising information regarding a task to be performed, at least in part, by the first user.
  • the instructions are further executable to cause the first user to switch to the second context concurrent with the second user switching to the first context.
  • the instructions are further executable to operate an artificial intelligence agent trained to monitor one or more aspects of the virtual reality session and further trained to provide, in real time, information regarding the task to the first user during the virtual reality session.
  • the information regarding the task may comprise directions for performing the task.
  • the first and second users may be collaborators on the task, and wherein the information may comprise a first description indicating a first portion of the task to be performed by the first user, wherein the instructions are further executable to cause the artificial intelligence agent to generate a second description indicating a second portion of the task to be performed by the second user.
  • the first user is a teacher and the second user is a student.
  • the first and second users are collaborators on the task.
  • the disclosure further contemplates a vocational mask configured to be worn by a first user, wherein the vocational mask comprises: a virtual retinal display; a memory device storing instructions; one or more peripheral haptic devices; a processing device communicatively coupled to the memory device and the virtual retinal display, wherein the processing device executes the instructions to: generate, for a first user wearing a first vocational mask that comprises the first computing system, a virtual reality session, wherein the first vocational mask includes a virtual retinal display; share, in response to execution of the instructions, a virtual reality session with a computing device associated with a second user, wherein sharing the virtual reality 7 session comprises the first user conducting bidirectional communications, via the first vocational mask, with a second computing system associated with the second user; operate, by the first user using the first vocational mask, in a first context concurrent with the second user operating in a second context using the second computing system; and present, via the virtual retinal display to the first user, data in selected ones of a plurality of formats received from the computing device associated with the
  • a system comprising:
  • a first vocational mask configured to be worn by a first user, wherein the first vocational mask comprises:
  • a processing device communicatively coupled to the memory device and the virtual retinal display, wherein the instructions, when executed by the processing device, are configured to cause the first vocational mask to:
  • [0222] share a virtual reality session with a computing device associated with a second user, wherein the first vocational mask is configured to conduct bidirectional communications with the computing device of the second user, wherein the first user is an apprentice and the second user is a master;
  • [0223] receive and present data, via the virtual retinal display to the first user, wherein the data is received in selected ones of a plurality of formats from the second user and comprises directions for performing a task by the first user under direction from the second user.
  • the first vocational mask includes one or more peripheral haptic devices, and wherein the instructions, when executed by the processing device, are configured to cause the first vocational mask to present, as one of the plurality' of formats, haptic feedback to the first user.
  • the task is welding
  • the artificial intelligence agent is trained to monitor one or more properties of a weld formed by the first user, and further trained to adjust directions to the first user based on the one or more properties of the weld.
  • a method comprising:
  • sharing in response to execution of the instructions, a virtual reality session with a computing device associated with a second user, wherein sharing the virtual reality session comprises the first user conducting bidirectional communications, via the first vocational mask, with the computing device associated with the second user, wherein the first user is an apprentice and the second user is a master;
  • [0240] presenting, to the first user using the first vocational mask, data received from the computing device associated with the second user, wherein the data is received in a plurality of formats, and wherein the data comprises directions for performing a task by the first user under direction from the second user.
  • sharing the virtual reality session with the computing device associated with a second user comprises sharing the virtual reality session with a second vocational mask worn by the second user.
  • executing the instructions stored in the memory device comprises executing an artificial intelligence agent trained to monitor one or more aspects of the virtual reality session and further trained to provide, in real time, directions for performing the task to the first user during the virtual reality session.
  • a non-transitory computer-readable medium storing instructions that, when executed by a processing device of a first computing system, cause the first computing system to:
  • [0256] generate, for a first user wearing a first vocational mask that comprises the first computing system, a virtual reality session, wherein the first vocational mask includes a virtual retinal display;
  • [0257] share the virtual reality session with a second computing system associated with a second user, wherein sharing the virtual reality session comprises the first user conducting bidirectional communications, via the first vocational mask, with a second computing system associated with the second user, wherein the first user is an apprentice and wherein the second user is a master; and [0258] present, to the first user, using the first vocational mask, data received from the second computing system associated with the second user, wherein the data is received in a plurality of formats, and wherein the data comprises directions for performing a task by the first user under direction from the second user.
  • a vocational mask configured to be worn by a first user, wherein the vocational mask comprises:
  • a memory device storing instructions; [0268] one or more peripheral haptic devices; and
  • a processing device communicatively coupled to the memory device and the virtual retinal display, wherein the processing device executes the instructions to:
  • [0271] receive and present, via the virtual retinal display to the first user, data in selected ones of a plurality of formats from the second user, the data comprising directions for performing a task by the first user under direction from the second user;
  • [0273] execute an artificial intelligence agent trained to monitor one or more aspects of the virtual reality session and further trained to provide, in real time, directions for performing the task to the first user during the virtual reality session.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Educational Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Educational Administration (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Plasma & Fusion (AREA)
  • Mechanical Engineering (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

L'invention concerne des systèmes et des procédés permettant d'utiliser la réalité virtuelle pour simuler une tâche de travail à l'aide d'un masque professionnel et de communication bidirectionnelle entre au moins deux utilisateurs. Un masque professionnel conçu pour être porté par un premier utilisateur comprend un dispositif d'affichage rétinien virtuel, un dispositif de mémoire qui stocke des instructions et un dispositif de traitement qui est couplé au dispositif de mémoire et au dispositif d'affichage rétinien virtuel. L'exécution des instructions amène le masque professionnel à partager une session de réalité virtuelle avec un dispositif informatique associé à un second utilisateur. Le masque professionnel peut établir des communications bidirectionnelles avec le dispositif informatique du second utilisateur, le premier utilisateur étant un apprenti et le second utilisateur étant un maître. Le masque professionnel peut recevoir et présenter dans divers formats des données provenant du second utilisateur. Les données comprennent des instructions destinées à ce que le premier utilisateur effectue une tâche sous la direction du second utilisateur.
PCT/US2025/024263 2024-04-12 2025-04-11 Systèmes et procédés d'utilisation de réalité virtuelle pour simuler une tâche de travail à l'aide d'un masque professionnel et de communication bidirectionnelle entre au moins deux utilisateurs Pending WO2025217508A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202463633166P 2024-04-12 2024-04-12
US63/633,166 2024-04-12
US19/169,623 2025-04-03
US19/169,623 US20250276398A1 (en) 2023-12-07 2025-04-03 Systems and methods for using virtual reality to simulate a work task using a vocational mask and bidirectional communication between at least two users

Publications (1)

Publication Number Publication Date
WO2025217508A1 true WO2025217508A1 (fr) 2025-10-16

Family

ID=97350396

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2025/024263 Pending WO2025217508A1 (fr) 2024-04-12 2025-04-11 Systèmes et procédés d'utilisation de réalité virtuelle pour simuler une tâche de travail à l'aide d'un masque professionnel et de communication bidirectionnelle entre au moins deux utilisateurs

Country Status (1)

Country Link
WO (1) WO2025217508A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150375327A1 (en) * 2014-06-27 2015-12-31 Illinois Tool Works Inc. System and method for remote welding training
US20160163221A1 (en) * 2014-12-05 2016-06-09 Illinois Tool Works Inc. Augmented and mediated reality welding helmet systems
US20200273365A1 (en) * 2017-09-14 2020-08-27 Vrsim, Inc. Simulator for skill-oriented training
US20200337407A1 (en) * 2015-03-06 2020-10-29 Illinois Tool Works Inc. Sensor assisted head mounted displays for welding
US20230129708A1 (en) * 2021-10-23 2023-04-27 Simulated Inanimate Models, LLC Procedure guidance and training apparatus, methods and systems

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150375327A1 (en) * 2014-06-27 2015-12-31 Illinois Tool Works Inc. System and method for remote welding training
US20160163221A1 (en) * 2014-12-05 2016-06-09 Illinois Tool Works Inc. Augmented and mediated reality welding helmet systems
US20200337407A1 (en) * 2015-03-06 2020-10-29 Illinois Tool Works Inc. Sensor assisted head mounted displays for welding
US20200273365A1 (en) * 2017-09-14 2020-08-27 Vrsim, Inc. Simulator for skill-oriented training
US20230129708A1 (en) * 2021-10-23 2023-04-27 Simulated Inanimate Models, LLC Procedure guidance and training apparatus, methods and systems

Similar Documents

Publication Publication Date Title
JP6710261B2 (ja) コンピュータ化されたアイウェアデバイス
Burova et al. Utilizing VR and gaze tracking to develop AR solutions for industrial maintenance
US11373383B2 (en) Immersive ecosystem
US9381426B1 (en) Semi-automated digital puppetry control
KR100721713B1 (ko) 몰입형 활선작업 교육시스템 및 그 방법
US20190130788A1 (en) Virtual Reality Microsimulation Platform
CA2554498C (fr) Systeme et procede servant a entrainer et ameliorer la mobilite corporelle
White et al. Low-cost simulated MIG welding for advancement in technical training
Fast et al. Virtual welding—a low cost virtual reality welder training system phase II
US20130252214A1 (en) Body motion training and qualification system and method
CN107656505A (zh) 使用增强现实设备控制人机协作的方法、装置和系统
JP7125872B2 (ja) 作業支援装置、および、作業支援方法
JP2019200257A (ja) 学習支援システム及びプログラム
JP2023073415A (ja) 溶接を支援するためのコンピュータ化アイウェア装置を提供するシステムおよび方法
US20250276398A1 (en) Systems and methods for using virtual reality to simulate a work task using a vocational mask and bidirectional communication between at least two users
Yang et al. Virtual reality based welding training simulator with 3D multimodal interaction
US20250252865A1 (en) Systems and methods for using artificial intelligence and machine learning to generate a virtual coach
WO2025217508A1 (fr) Systèmes et procédés d'utilisation de réalité virtuelle pour simuler une tâche de travail à l'aide d'un masque professionnel et de communication bidirectionnelle entre au moins deux utilisateurs
US20250252385A1 (en) Systems and methods for using artificial intelligence and machine learning to generate optimized operating parameters for a work tool based on track haptic feedback and quality of job performed
US12512009B2 (en) Systems and methods for using a vocational mask with a hyper-enabled worker
EP4567770A1 (fr) Systèmes et procédés d'utilisation d'un masque covatoire avec un travailleur hyper-activé
EP4567772A1 (fr) Systèmes et procédés d'utilisation d'un masque covatoire avec un travailleur hyper-activé
EP4567771A1 (fr) Systèmes et procédés d'utilisation d'un masque covatoire avec un travailleur hyper-activé
US20250259120A1 (en) Systems and methods for using artificial intelligence and machine learning to monitor work tasks
US20250336308A1 (en) Systems and methods for using artificial intelligence and machine learning with a wearable mask to identify a work setting and to control operation of a tool

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25787309

Country of ref document: EP

Kind code of ref document: A1