RU2725782C2 - System for communication of users without using muscular movements and speech - Google Patents

Abstract

FIELD: information technology.SUBSTANCE: invention relates to a system for communication of users without using speech and movement. Said result is achieved by a system for communication of users without using speech and movements, comprising a user electroencephalogram registration means, a network interface, a server for forming a social network, a data processing unit configured to generate on the display screen varying light stimuli having different time variation parameters, wherein each of the stimuli is matched with a predetermined action, the data processing unit is configured to determine, in real time, stimuli directly selected by the user, and performing actions corresponding to user-selected stimuli by tracking selection of objects from a plurality of objects, device screen is configured to display results of performing actions based on results of stimuli presentation to multiple users on multiple screens, transmission of information on actions on results of presentation of stimuli between data processing units of various users is carried out by means of a remote server.EFFECT: technical result is creating a universal means for communication without using speech and movements.17 cl, 7 dwg

Description

FIELD OF TECHNOLOGY

The invention relates to the field of computer technology and communications, and is intended for both remote and direct communication of users who do not have the ability to confidently use speech and limb movements to control communication tools.

BACKGROUND

A known system (see US20180255361, published 06.09.2018), moreover, the system contains a data warehouse and interaction management platform, which is associated with a data warehouse and entertainment platform, and the interaction management platform is configured to: generate the first interface available to the first user on the first computing device receiving from the first computing device the first "live" (live) transmission in which the first user aloud says a question that the second user must answer; generating a second interface available to the second user on the second computing device; receiving from the second computing device a second "live" transmission in which the user aloud answers the question of the first user; converting the first and second live broadcasts to MPEG-4 format and for redirecting the first and second live broadcasts to an entertainment platform for broadcasting through a broadcast television network.

Known interfaces for network communication of users suggest that the physical capabilities of the user correspond to the average level. For example, to implement the solution shown above requires the use of speech and user images for communication. At the same time, communication aids are made in the form of touch screens, which require the confident use of muscles for control.

The brain-computer interface system is also known (see US20150309572, published October 29, 2015), where the brain-computer interface system includes: a device for acquiring an electroencephalogram (EEG) connected to a user, the device for acquiring EEGs configured to collect EEG signals generated by user; a display configured to display images to a user using a display panel, the images corresponding to a user interface environment; a processor configured to divide the images into several categories, for presenting to the user at least one image from each category from a plurality of categories on the display panel and for blinking images, and the processor is configured to determine a user selected image based on the collected EEG signals after presenting by the blinking unit at least one image to the user, and to perform an action based on the image selected by the user.

The disadvantage of this solution is the impossibility of creating a system of user interaction with each other, including the creation of social networks for users to communicate with each other.

The objective of the present invention is to overcome these drawbacks, and to enable communication with the outside world and control service devices for patients without using the movements and functions of speech, as well as simplifying the use of communication tools.

ESSENCE

The technical result achieved by the implementation of the invention is the creation of a universal tool for communication without the use of speech and movements, in particular, users with limited motor and speech functions, moreover, this tool improves the security of use when reaching a speed of exchange of information between users, comparable to the average speed sharing information on social networks of general use. At the same time, it is possible to use the invention as a means to accelerate socialization and increase the effectiveness of rehabilitation measures.

According to one implementation option, a system is proposed for communication of users without using speech and movement, comprising means for recording an electroencephalogram of a user, the information outputs of which are connected to a data processing unit, the video output of which is connected to the display, and the network interface is designed to connect to a remote server designed for the formation of a social network configured to communicate with several users in real time, while the electroencephalogram registration tool is configured to register an electroencephalogram by determining the activity of the brain by measuring the electrical potentials on the surface of the user's scalp in real time, extracting parameters from the electroencephalogram at least one component of evoked potentials (VP) with predetermined basic characteristics and the formation of the output signal in the form of a description of the parameters of at least one component of the evoked potentials in digital form, the data processing unit is configured to produce varying light stimuli on the display screen having various parameters of change in time such that, while maintaining user attention on different stimuli, at least one component of the evoked potentials has different parameters, at the same time, each of the stimuli is assigned a predetermined action, the data processing unit determines, in real time, the stimuli selected by the user and performs the actions corresponding to the stimuli selected by the user, while the stimuli selected by the user are considered stimuli, reaction parameters in response to which maximally correspond to the parameters of at least one component of the evoked potentials that arise during the formation of stimuli on the device screen, while the device screen is configured to display the results of actions based on the results of incentive phenomena to several users on several screens, the selection of the parameters of at least one component of the evoked potentials from the electroencephalogram is performed by the data processing unit, and information about actions based on the results of the presentation of incentives between the data processing units of various users is transmitted via a remote server ..

In one particular embodiment, the separation of the parameters of at least one component of the evoked potentials from the electroencephalogram is performed by a data processing unit using machine learning that takes into account the nature of the user's limited motor and speech functions.

In one of the private implementation options, machine learning is performed for each user separately.

In one of the private embodiments, the means of recording the electroencephalogram is made containing electrodes intended for connection with the scalp of the user, installed in the mount with the ability to move.

In one of the private embodiments, the fastening means is configured to install and remove electrodes.

In one particular embodiment, the component of the evoked potential with predetermined characteristics is wave P300.

In one of the private embodiments, the proposed system contains a block for setting stimulus parameters.

In one of the private options for implementation, one of the parameters of the stimulus is the time moment of the beginning of the display of the stimulus.

In one particular embodiment, at least one of the other parameters of the stimulus is the moment the stimulus is extinguished, the speed of the stimulus moving along the display, the brightness of the stimulus, the frequency of the stimulus flicker, the geometric dimensions of the stimulus, the outline of the stimulus, and the information depicted on the stimulus.

In one particular embodiment, the geometric dimensions of the stimulus are set in accordance with the angular dimensions of the displayed stimulus relative to the position of the patient, and these sizes are adjusted individually for each user to provide the best visual distinction between the stimulus and the most effective detection of the stimulus-induced response.

In one particular embodiment, the instants of the incentive display start time are set by various settings to ensure that the incentives are displayed on the screen in cycles.

In one particular embodiment, the frequency of the stimulus display cycles is selected such that the potential negative consequences for the user from observing the stimuli on the display screen are minimized.

In one particular embodiment, the sizes of the stimuli provide a distinguishability of the stimuli on the screen and the information related to the stimuli.

In one of the private embodiments, the proposed system is implemented with the display on one display of a message set module with a virtual keyboard, a module for displaying dialed messages, and a correspondence display module.

In one particular embodiment, the correspondence display module is configured to display the history of correspondence with at least one other user.

In one particular embodiment, the correspondence display module displays messages generated by another user in real time.

In one particular embodiment, the means of recording the electroencephalogram is made with the possibility of extracting by the data processing unit from the electroencephalogram the parameters of more than one component of the evoked potentials with predetermined basic characteristics and generating an output signal in the form of a description of the parameters of the components of the evoked potentials in digital form, and the stimuli selected by the user are considered , reaction parameters in response to which correspond most closely to the parameters of the EP components arising in the process of stimulus formation on the device screen.

BRIEF DESCRIPTION OF GRAPHIC MATERIALS

Additional objectives, features and advantages of the present invention will be apparent from reading the following description of an embodiment of the invention with reference to the accompanying drawings, in which:

FIG. 1 illustrates an exemplary embodiment of a system implementing the present invention;

FIG. 2 illustrates an exemplary embodiment of averaging EEG eras;

FIG. 3 illustrates a block diagram of a classification algorithm according to the present invention;

FIG. 4 illustrates an exemplary embodiment of a neuro headset placed on a user's head;

FIG. 5 illustrates part of the described system for a single user;

FIG. 6 illustrates an example embodiment of a display screen with displayed objects for user interaction.

FIG. 7 illustrates an example of a general purpose computer system.

DETAILED DESCRIPTION

The objects and features of the present invention, methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below, it can be embodied in various forms. The essence described in the description is nothing more than the specific details provided to assist the specialist in the field of technology in a comprehensive understanding of the invention, and the present invention is defined only in the scope of the attached claims.

Used in the present description of the invention, the terms “component”, “element”, “system”, “module”, “part”, in particular, “component”, “block” and the like are used to refer to computer entities (for example, objects related with a computer, computing entities), which can be hardware, in particular equipment (for example, a device, tool, apparatus, equipment, part of a device, in particular a processor, microprocessor, printed circuit board, etc.), software (for example, executable program code, a compiled application, software module, a piece of software and / or code, etc.) or firmware (firmware / firmware). So, for example, a component can be a process running / executing on a processor, a processor, an object, an executable file, a program, a function, a method, a library, a subprogram and / or a computing device (for example, a microcomputer or a computer) or a combination of software or hardware. As an illustration: as an application running on a server, in particular on a central server, can be a component or module, so the server can be a component or module. At least one component may be located within the process. A component may reside on a single computing device (eg, a computer) and / or may be distributed between two or more computing devices. So, for example, in a particular case, an application (component) can be represented by a server component (server part) and a client component (client part). In the particular case, the client component is installed on at least one computing device, and the server component is installed on the second computing device, from which, in the particular case, control and / or configuration of the first computing device (and / or its component components / In a particular case, a module (etc.) can be implemented (be) one or a set of files, including an executable file (s), which in turn can be associated with at least one program library , for example, implemented as a dll file, which is a compiled form of a dynamic link library (from the English Dynamic Link Library), and also with at least one file, for example, containing application service data, application metadata, data necessary for functioning of the application, and / or a service (local and / or remote, for example, a web service / web service), including applications and services built on the server c-oriented architecture service-oriented architecture / SOA), including, but not limited to, REST technologies (from Representative State Transfer - “transfer of presentation state”), remote procedure calls (from Remote Procedure Call, RPC) and others.

In FIG. 1 shows an exemplary embodiment of a system implementing the present invention.

Shown in FIG. 1, the system described in the framework of the present invention includes an electronic computing device, for example, a computer (130A, 130B), an electroencephalogram recording (recording) means (120A, 120B), and a display (110A, connected to a computer (130A, 130B) 110B), in the particular case, providing updating of frames with a frequency of at least 40 and not more than 150 Hz and a response time of not more than 7 milliseconds (ms). In the particular case, an electronic computing device, for example, a computer (130A, 130B) and a display (110A, 110B) can be implemented as a single device, for example, a candy bar, tablet, smartphone, etc.

As described in the framework of the present invention, the means of recording the electroencephalogram (120A, 120B) is configured to extract the data processing unit from the electroencephalogram parameters of more than one component of the evoked potentials with predetermined basic characteristics and generating an output signal in the form of a description of the parameters of the components of the evoked potentials in digital form, and user-selected stimuli are considered to be stimuli, the reaction parameters in response to which correspond to the parameters of the EP components that arise during the formation of stimuli on the device screen.

The means for recording an electroencephalogram, that is, the means for recording the electrical activity of the brain 120A is used to register the electroencephalogram (EEG) of the first user (patient) 105A, and the means for recording the electrical activity of the brain) 120B is used for recording the electroencephalogram of the second user 105B. Displays (110A, 110B) are designed to display objects. for example, incentives presented in the form of digital or alphabetic characters. icons or menu items with angular dimensions, relative to the eyes of users (105A, 105B), allowing you to switch attention from one presented object to another. In the particular case of implementation, the number of elements, as well as their location on the display on the display (110A, 110B), is set by the stimulus generation module (513, FIG. 5) in accordance with the angular dimensions, which make it possible to distinguish objects for the user (105A, 105B). In the particular case of implementation, the number of screen objects and / or their angular dimensions are set taking into account the individual characteristics of the user (105A, 105B). Among the individual features of the user can be attributed the properties of the user's visual perception of objects and the user's ability to keep objects in the field of attention. If it is impossible for the user (105A, 105B) to hold visual attention on a screen object with small angular dimensions due to blurring of the field of view or trembling of the point of fixation, it is possible, with various nosologies, the angular dimensions of the screen object can be increased, the distance between objects can be increased and the number of objects simultaneously present on the screen. In the particular case, the visual visibility of objects is tested, for example, when configuring the system to work with a new user, or when changing the distance from the user to the display, as well as when changing other parameters. At the initial configuration and reconfiguration, the geometric dimensions of the display objects and other characteristics of the display objects displayed on the screen (display) are set. In the particular case, the user's visual acuity is taken into account. If the used parameters of the objects are not satisfactory, for example, the user cannot select the desired stimulus on the screen due to insufficient visual acuity, then increasing the size of objects with their contrast provides the user with the opportunity to select objects, if changing the geometric characteristics does not give a positive result, you can use changes to other parameters, for example, the frequency of flickering of objects, the sequence of their occurrence on the screen can be changed. Interface settings can be made in an automated mode, as well as in the mode of manual control by a user or operator. When manually adjusting the interface, the display screen displays interface elements whose dimensions and position can be changed, as well as parameter values that significantly affect the perception of elements (incentives) by the user. In the particular case, you can change the set of stimuli, grouped within one screen.

After making changes to the settings, the distinguishability of the stimuli is tested, and, if necessary, an additional adjustment is made. In the particular case, when grouping stimuli across several screens, the parameters adopted for one of the screens apply to all screens. and if changing the settings has led to a change in the number of screens, for example, due to a significant increase or decrease in the size of the elements, additional redistribution of incentives across the screens is performed.

As display objects or stimuli, various graphic elements are used, for example, symbols in the form of letters and numbers, so-called emoticons, other pictograms, as well as representations of controls. Symbols can refer to elements of transmitted messages, as well as be system control elements, for example, such elements can be buttons for switching to another screen, buttons for turning on and off the system. In the particular case of the invention, the service buttons of various screens are located in different places on the display, thus significantly reducing the likelihood of a false command to go to another screen when changing screens.

For menu system call objects and menu system objects, the same arrangement of menu call buttons on different screens is excluded. Incentives designed to activate (turn on) and deactivate (turn off) the system can be located outside the boundaries of the display and can be made in the form of separate light sources. In the particular case of the invention, the objects on and off of the system can be dedicated displays, for example, flickering LED lamps.

The system can also use additional types of objects designed to speed up typing. Such special objects can be made in the form of buttons of a larger size than buttons with individual letters, and words and short phrases can be displayed on them, formed, for example, by implementing the function of predictive typing.

The characters displayed on such buttons dynamically change depending on the combination of characters already typed by the user. The predictive typing subsystem can use a dynamically changing dictionary generated, in a particular case, as a result of analysis of the previous user interaction (105A, 105B) with the described system. For example, a computer (130A, 130B) or a central server 140 can store the most frequently used words, phrases, sentences, and other grammatical forms in a specialized database.

Special buttons can be located on the same screen as the buttons on the alphabetic keyboard, but are located at a distance from any buttons on the keyboard, significantly greater than the distance between adjacent buttons on the keyboard. Such a spatial separation allows the user to confidently switch attention to special buttons, as a result of which the accuracy of object recognition is increased.

Additionally, the processes of turning on and / or turning off the system can be started with the predetermined behavior of the user or system. For example, if the user is inactive for a long time, the system may be turned off, the system may be turned off or restarted if the user does not use the system interface for a long time. In the event that the user stops using the interface without completing the input of a word or phrase, the system can additionally activate the emergency call facilities. Additionally, an “invitation to turn off”, or “invitation to turn on” can be activated at predetermined hours of user activity.

The user equipment 101 of the system shown in FIG. 1 is shown in more detail in FIG. 5 with the corresponding description.

Shown in FIG. 1, the system contains and uses electrodes (115A, 115B), configured to provide contact with the skin of the user's head to transfer the corresponding potentials to the analog data processing unit. The electrodes can be fixed in a means of fixing the electrodes on the head of the user. The electrodes can be located in close proximity and connected, through the conductors, to the analog data processing unit that converts the analog data into digital data according to a predetermined algorithm that digitizes the analog signals at predetermined time intervals. The digital data of the analog signal processing unit can be transmitted to a computer (130A, 130B), via a wired interface or using wireless means of communication, in particular, such as a standardized Wi-Fi connection. In general, the system uses equipment that is usually used to implement the P300 brain-computer interface (IMC-P300, or the English “P300 BCI” - brain-computer interface). In the particular case of implementation, the number of electrodes mounted on the user's head, as well as their location, is selected taking into account the individual characteristics of the user. For example, according to the results of surgical operations for brain lesions, or as a result of head injuries, scalp sites may be absent or cannot be used to install electrodes. In this case, an electrode fastener is used, in which the number of electrodes, as well as their installation location, are set taking into account the identified features. In the particular case, additional electrodes can be used, installed as close as possible to the site of injury. In the event of a significant change in the pattern of potential distribution on the scalp of the user, additional training of the system is used to ensure the functioning of the system and the allocation of the "P300" wave. In other cases, for example, with organic damage to the cerebral cortex, an increase in the number of electrodes is used to provide the ability to obtain the required parameters of brain activity from replacement data.

An electronic computing device implemented, for example, in the form of a portable or stationary computer (130A, 130B), provides an analysis of the activity of the brain according to the measurement results. The activity analysis is carried out, for example, using a data processing and analysis module (523, FIG. 5). For the analysis, electroencephalograms (EEGs) are used, as well as data relating to the parameters of the stimuli formed on the display. At the same time, the main parameter of the stimulus used in the analysis is the moment of its occurrence on the screen (display). The formation of visual stimuli is carried out by the stimulus generation module (513, FIG. 5), which generates the data necessary to display the stimulus on the display at predetermined moments, ensuring that the stimuli are displayed on the display (screen) (110A, 110B). In the particular case, the described system provides such a sequence or sequence of illumination and suppression of stimuli on the display that the simultaneous presence on the screen of two objects or stimuli, the places of formation of which are located next to each other. In the particular case, it is possible to simultaneously display objects located on one row or in one column. In this case, the definition of the object of attention is carried out by sequentially determining the column and row at the intersection of which the object of attention is located. Moments of occurrence (times of occurrence), extinction moments (times of disappearance), and other characteristics or parameters of visual stimuli created can be determined automatically by the stimulus generation module (513, FIG. 5). Incentive parameters can be set as a result of machine learning, including the possibility of using artificial intelligence. Also, the time of creation, as well as the time of disappearance, and the characteristics of the generated visual stimuli can be set by the user, the administrator of the central server 140, the observer or attending physician of a user who is a patient of a medical institution. The creation time, as well as the time of disappearance or extinction, as well as other characteristics of the generated visual stimuli can be set using the stimulus generation module (513, FIG. 5), for example, using the graphical user interface displayed on the screen.

In the event that the registered electroencephalogram, or in other words, the registered parameters of the brain’s electrical activity, measured by the electroencephalogram meter (120A, 120B) and transferred to the computer for processing, equally corresponds to several different stimuli, the user may be asked to select one of the objects. corresponding to stimuli either by forming an additional stimulus - a hint, or by completely extinguishing all stimuli except potential attention objects and displaying the remaining stimuli in this mode to provide a more accurate definition of the object of interest.

In the particular case, the central server can be implemented (or be) a web server and / or specialized computing device (in particular, a computer) and / or specialized equipment for performing service software on it (including services of certain tasks) .

In the particular case, at least one user is offered tips related to the possibility of communication within a given circle of contacts, for example, a given circle of friends, a medical hospital or department.

The characteristics or parameters of the stimuli (objects) displayed on the display can be the brightness of the object, the frequency of appearing on the display, belonging to a group of objects displayed on one group screen, the frequency of flickering during the display, the length of time the display has disappeared, the coordinates of the location of the object on the display, the size and shape of the object, the speed of movement of the speaker changes in size and / or shape, etc.

The characteristics of visual stimuli can be changed or selected in the process of using the system by the user so as to achieve the maximum speed for choosing the stimuli by the user, or to reduce the user's fatigue, or minimize the number of system errors per unit time, or achieve the optimal combination of parameters. In addition, during the initial setting of stimulus parameters, the physiological characteristics of the user can be taken into account, in particular, visual acuity, the rate of information transfer within various structures of the brain, the presence of brain injuries, the presence of specific diseases of the nervous system and brain that affect perception processes can be taken into account. In addition, user experience when interacting with the system interface can be taken into account. In particular, more trained users can use lower brightness and faster blinking speed, which allows them to get tired less and dial faster (sequential selection of stimuli for generating messages or other alphanumeric and / or visual structures).

In addition to visual stimuli, the user may be presented with sound or tactile stimuli, however, in order to disclose the essence of the invention, stimuli, unless otherwise indicated, are understood to mean visual or visual stimuli, that is, objects formed on the screen or display that can be perceived visually.

The tool for recording an electroencephalogram (120A, 120B), that is, a tool for recording electrical activity of the brain (EEG), is equipped with a means of attaching sensors of electric potentials on the user's head and allows you to analyze the EEG. In this case, it is desirable to use tools that record electrical potentials with a frequency of at least 50 times per second. which makes it possible to determine, using the data processing and analysis module (523, FIG. 5) and / or the data processing module, which may be part of the encephalograph, (120A, 120B), specific components of changes in the EEG when the user focuses on the objects for a short time displayed on the display.

In the context of the description of the invention, a graphic display or “display” is used as an example of the implementation of a visual stimulus environment, which can be performed in various ways, for example, using separate elements representing symbols and having individual illumination means, LED arrays, using mechanical or moving mechanical elements, for example, using curtains to hide characters, etc.

Controls or incentives can be grouped, and the user, in the process of using the system interface, can be presented with different groups of elements to enable reliable determination of the selected element of the group. When using several groups of elements, each of the groups contains at least one stimulus, when selected by the user, it is provided either switching to another group of elements or displaying a menu using which the transition to another group is provided.

To select one command by the user (105A, 105B), cycles of presenting objects to the user are used. To make a choice of one or another displayed (screen) object, the user (105A, 105B) focuses on the said object, which is associated with the function of interest, for example, a command, a text symbol (element), graphic element, etc. may contain an image that can be unambiguously correlated with the result of selecting an object, for example, it may contain a graphic element, a text element, a command designation (for example, sending a message, adding an image to a message, adding another user as a friend, connecting to the Internet system , launching the application, interface, etc.). In the implementation of stimulation or, in other words, providing the possibility of selecting an object by the user, the user monitors the behavior of the object, and the appearance of the object on the display causes the formation of a specific picture of the dynamic distribution of electric potentials on the surface of the user's head. In general, a specific impulse is generated in the P300 wave (or P300 wave pattern). Temporal parameters of the pattern, for example, the moment of occurrence and duration, depend on the temporal parameters of the stimulus, to which the user's attention is drawn. Within the scope of the present invention, the generated pattern is recognized by the electronic computing device, and the pattern parameters are related to the stimulus parameters presented to the user, for example, to the moment of formation, type and / or nature of the generated on-screen (visual) stimuli. The object to which the pattern corresponds to the maximum extent is identified as the object to which the user mentally reacts, after which the system performs actions corresponding to the object, for example, supplements the text line with the selected symbol or starts displaying the symbols (stimuli) of another screen.

Specific components may be the deviation of the dynamics of the data of the electroencephalogram averaged over the target stimuli, in particular, during a certain predefined time interval, for example, within 600-1000 milliseconds, after the target stimulus is presented from the same averaging upon presentation of a non-target stimulus, and when individual functions are lost, sections of the cerebral cortex, an additional refinement is used, namely, that in a particular case, the data obtained from the changed, reduced or increased number of electrodes are analyzed, and the analysis of the dynamics of the potential distribution is carried out for all electrodes in conjunction with the use of machine learning methods or with the use of artificial intelligence . At the same time, at the training stage, information about the object selected by the user is obtained without using computational tools and transmitted to the system additionally, or the sequence of stimuli that the user should focus on is set and this sequence is used as a learning one when comparing patterns and stimuli.

That is, the reactions represented by the specific components disclosed above are subjected to real-time automated analysis, for example, using machine learning tools or artificial intelligence. In the particular case, carry out preliminary special settings shown in FIG. 1 system with user participation (105A, 105B), or the system is configured or tuned in operating mode. The main requirement for the configuration mode is the quick response of the user to system responses interpreting his commands in the "true - false" mode.

Using the processing and analysis elements integrated in the EEG tool (120A, 120B) and / or in the computer (130A, 130B) or the central server 140, for example, which implement artificial intelligence structures, it is possible to interpret user reactions as commands for controlling the reception functions -transmission of messages, including text messages, as well as service device control modules, for example, an alarm device, a message service to a mobile phone or electric drives of a wheelchair. Mentioned elements of processing and analysis, including. artificial intelligence tools can be part of a data processing and analysis module (523, FIG. 5) or can be a computer module (130A, 130B) associated with an electroencephalogram recording tool (120A, 120B) or another module, for example, a central server module 140 and / or computer (130A, 130B), including a data processing and analysis module (523, FIG. 5) a data processing and analysis module (523, FIG. 5) or at least one part thereof, may be part of a central server 140 and / or electroencephalogram recording means (120A, 120B) and / or central server 140.

The control commands mentioned may be control commands for web browsers, instant messaging systems (instant messaging systems, from the English Instant messaging, IM), short message services (from the English Short Message Service) and other software and / or hardware means (systems, etc.) of interaction, in particular, communication, users with each other, communities, user groups, including those united by various parameters, for example, hobbies, etc. In the particular case, the mentioned processing and analysis elements, in the particular case of artificial intelligence, convert the mentioned reactions into commands for controlling the functions of the Internet browser (web browser), etc.

The described system, under the code name "NeuroChat", uses an algorithm that uses the property of the user's brain (human, patient, medical staff, etc.) to change the electrical parameters of brain activity, if there is uncertainty in the behavior of an object, in particular, a screen object, to which the user's attention is directed, and such changes in the electrical parameters of the user's brain activity are recorded by means of an electroencephalogram recording tool (120A, 120B). The described NeuroChat system works on any (fairly “powerful”) computer (130A, 130B) of the user (105A, 105B) and on any modern display (110A, 110B), in particular, a monitor of sufficient size, in particular on systems combining a computer and a display (in particular, a monitor) in one device, in a particular case, on monoblocks, tablets and smartphones. A standard (commonly used) refresh rate of 60 Hz is sufficient, as well as standard response times for such monitors.

Shown in FIG. 1, the system provides the display on the display (110A, 110B) of objects, each of which is characterized by a unique set of properties that affect the electrical activity of the user's brain (105A, 105B) in a unique way. Shown in FIG. 1, the system recognizes the object that the user's attention is directed to (105A, 105B) by the electrical activity of the brain of such a user corresponding to the specified set of properties. In the particular case of said recognition, or at least part of such recognition, of an object can be carried out by at least one computer module (130A, 130B) and / or an electroencephalogram recording tool (120A, 120B), and / or central server 140.

The main properties of objects that affect the activity of the brain, in a particular case, are the time parameters for the appearance of objects (images of objects) on the display (110A, 110B) and their removal from the display (display screen). In a particular case, the frequency of blinking (flickering, in particular, presentations, etc.) of images of objects is used as such a property. In the particular case, the aforementioned blinking is a change in the brightness and / or color and / or transparency of the (on-screen) object, as well as its image (appearing and disappearing) on the display (110A, 110B), which, in the particular case, is selected (set, set) ) not lower than, for example, 1 Hertz (Hz) and, for example, not higher than 10 Hz, while the images of objects, in a particular case, flash at the same frequency, but appear on the display (110A, 110B) alternately.

In the particular case of implementation, the behavior of objects can change over time, increasing the level of uncertainty in the behavior of the object for the user.

NeuroChat (in particular, the NeuroChat system) is algorithmically based on the principle of neurocontrol using the so-called technology “brain-computer interface on the P300 wave” (IMC-P300, or the English “P300 BCI” - brain-computer interface). The control in this technology is based on the analysis of evoked potentials (VP) of the human brain, or potentials associated with events (MSS). These potentials represent a specific reaction of the brain in response to a rare and / or significant for the user (105A, 105B) stimulus that can be obtained after the processing (and analysis) of the recorded (recorded) electroencephalogram. In most cases, in communication facilities based on IMC-P300, visual modality is used for stimuli (for example, the appearance of images or symbols on the screen, or their illumination - i.e., an increase in brightness).

A system, and, in a particular case, a means of neurocontrol (user interactions with each other, with various devices, modules, systems, with each other using intermediate and auxiliary devices, tools, modules, etc.), in particular, based on IMC-P300, organized as follows. In general, the user (105A, 105B) sees a display (110A, 110B) in front of him with objects (screen objects, visual objects), in the particular case being symbols (symbol elements) representing various control commands (including letters, for example, if the interface is intended for typing, interacting with other users, systems, including operating systems, etc.) organized in the form of a matrix, in particular, a matrix table. Thus, the user (105A, 105B) on the display (110A, 110B) displays incentives in the form of objects, in particular, images of objects (screen objects).

Individual objects, in particular, symbol elements of the matrix are alternately “highlighted” (flicker, blink, etc., or are highlighted in any other way and means) in a random order. In the particular case, the mentioned objects can be allocated by the system, in particular, by the stimulus generation module (513, FIG. 5) in a strictly defined order, for example, a certain sequence with various pauses between such selection, with different selection intensity, display time, highlighting and other ways of distinguishing, in particular, presenting, objects to at least one user, etc.

The user (105A, 105B) focuses on the stimulus (command symbol, stimulus command, object, target stimulus, target symbol, target) that he wants to choose, and whenever this object, in particular, the stimulus ( symbol, etc.) is highlighted (“lights up”, displayed, highlighted, starts blinking, blinking, etc.), must mentally respond to this event (for example, calculate the total number of highlights of this symbol). In the process of user interaction (105A, 105B) with an interface, in particular, a graphical interface that performs at least displaying stimuli on the display, the user (105A, 105B) registers an EEG using an electroencephalogram recording tool (120A, 120B).

At the training stage, the described system uses information that unambiguously indicates a stimulus, in particular, a symbol that is the object of attention of the user (105A, 105B), and the object of attention of the user is a (screen) object displayed on the display (110A, 110B). So, for example, the system notifies the user (105A, 105B), in particular, through the data processing and analysis module (523, FIG. 5), for example, using the stimulus generation module (513, FIG. 5), about the stimulus, in particular , symbol, command, etc., on which the user should focus, and for this stimulus, brain activity is recorded (EEG data is recorded using an electroencephalogram recording tool (120A, 120B)). The obtained EEG data relating to both the flickering of the symbol and the location of the symbol on the screen are recorded and used, in particular, by at least one module, for example, a computer module (130A, 130B), central server 140, electroencephalogram recording means (120A , 120B), etc., for building an expert subsystem. The mentioned expert subsystem is created to identify individual differences between users in response to target and non-target stimuli, namely, differences in the data of the electroencephalogram, and when using the system to determine at least one stimulus that the user selects, it occurs using (based on) identified differences. Obtained, in particular, recorded or registered brain activity data can be processed (in whole or in part) as a computer (110A, 110B), in particular, at least one of its modules, for example, and / or data processing and analysis module (523, FIG. 5), the central server 140, for example, at least one of its module, and / or means for recording an electroencephalogram (120A, 120B), or at least one of its module, if such a means (module, device, etc.) of recording an electroencephalogram (120A, 120B) allows for such processing, for example, using at least one processor, microprocessor, as well as a controller, microcontroller, etc.

The training of the expert subsystem can continue at the stage of using the system, for example, taking into account the number of errors or, according to the correct results, obtained under conditions of insufficient definiteness of the correspondence of the EEG and any of the symbols displayed on the display (110A, 110B).

Reactions to highlighting the target command symbol are distinguished by at least one module (means, device, etc.), for example, a data processing and analysis module (523, FIG. 5), among reactions to other stimuli, in particular symbols that are inappropriate stimuli and to which the user (105A, 105B) should not pay attention. In such target reactions (or VP), potential fluctuations of a significantly larger amplitude are present than in non-target VPs. The mentioned potential fluctuations can be processed by a data processing and analysis module (523, FIG. 5) and / or a central server 140, and / or an electroencephalogram recording tool (120A, 120B), or at least one of their modules. One of the largest and most universal components of these oscillations is a specific pattern - the so-called P300 wave (therefore, the technology has received the corresponding name).

The components of the reaction to external stimuli (EP) are usually not identified on the background EEG, since these reactions have an amplitude of several microvolts, while the background vibrations of the EEG usually have amplitudes one to two orders of magnitude higher. Therefore, to extract the VP, a technique is used, in particular, implemented by an algorithm (implemented by a software and / or hardware component, in particular, a module), of synchronous averaging of individual EEG segments. One such segment of the EEG is called the era (EEG era). Its duration may vary; in the described system, it, in the particular case, can be about 1 second. Since spontaneous EEG oscillations are usually not synchronized with the instants of the stimulus (external stimulus), when averaging the EEG epochs relative to the moment of this stimulus, the time-synchronized signal of the potentials associated with the events accumulates, certain peaks (reaction components) are released, and non-synchronous EEG vibrations (“noise”) are attenuated (their amplitude drops sharply). It should be noted that the stimulus is the presentation of the object (or, in a particular case, the presented object) to the user, in particular, the selection of the object (or group of objects), and the stimulus in the described system is external to the user (for example, in contrast to other types of IMC , where the user can consciously initiate any state without (various, any) incentives). When a set of stimuli is presented sequentially (only one of which is currently targeted for the user), epochs are identified (determined, calculated) from the EEG that are associated with the moment (time) of the appearance (highlighting, or any other selection) of each individual stimulus, as shown in FIG. 2, i.e., in the particular case, the time of the onset of one era coincides with the time of appearance, in particular, of the presentation of the stimulus.

The mentioned selection of at least one era is carried out by a computer, in particular, a data processing and analysis module (523, FIG. 5), and / or an electroencephalogram recording tool (120A, 120B), a central server 140 and, and / or a module (modules) of said means, central server 140, computer, etc.

In FIG. 2 shows an exemplary embodiment of averaging EEG eras.

As noted above, for reliable recognition of target EP reactions, each of the stimuli is presented to the user (105A, 105B) repeatedly, after which the responses to each stimulus are averaged separately. Averaging the reaction separately for two groups, the first of which is the group with target stimuli, and the second is the group with non-target stimuli, allows you to highlight a number of characteristics (for example, frequency-amplitude characteristics, which are a positive deviation with latency, in the particular case, about 300 ms and above after the start (presentation) of the stimulus), according to which the target reactions are distinguished, in particular, by the data processing and analysis module (523, FIG. 5). The P300 wave arises (or sharply increases in amplitude) on the target stimulus in the P300 wave - a positive deviation with a latency of 300 ms or higher after the stimulus starts. In order for this component to appear, the user (105A, 105B) is given the task of focusing attention on the target stimulus and, in the particular case, counting the moments of its appearance among non-target stimuli. The mentioned task can be given to the user (105A, 105B) by an operator (computer, server, in particular, central server 140, etc.), for example, by medical personnel, a doctor, etc. The task may be voice and / or visual, i.e. presented to the user (105A, 105B) in the form of images (drawings, graphic elements, photographs, etc.), text, etc. The task may, in particular, be presented to the user (105A, 105B) using a computer (130A, 130B), a central server 140, etc., in particular, at least one computer module or module, for example, using task module, tuning module, etc., including using a display (110A, 110B), sound reproduction means connected to a computer, etc. When presenting an audio (audio) task to the user (105A, 105B), such an audio task can be previously stored in the computer (130A, 130B) and / or central server 140, for example, as an audio file, or the audio task can be presented to the user using a microphone associated with a computer (130A, 130B) or central server 140. In general, the amplitude of the P300 reflects the likelihood of a stimulus (positively correlates with its unexpectedness). For the emergence of P300, it is important that the target stimulus should not only be rare, but should also relate to the task, i.e. the subject, in particular the user (105A, 105B), should be actively included in the task of recognizing the target stimulus. In various variations of such a task, in addition to the P300 wave, it is carried out, for example, by a computer (130A, 130B), in particular, by a data processing and analysis module (523, FIG. 5), amplification and a number of other components (not shown in FIG. 1), the most the most significant of which for this IMC are the following: N1 (or N170) is a negative peak with a latency of 170-200 ms, P200 is a positive peak with a latency of 200 to 300 ms, N400 is a negative peak with a latency of 400-600 ms. All these components, together with the P300, are a reflection of the user's attention (105A, 105B) to the target stimulus and are used by the IMC algorithms to detect the target command. Since the duration of each epoch is about 1000 ms (1 second), which are determined (set) based on the most probable time during which significant amplitude features can appear, then (all) these components are a priori analyzed, in particular, by the data processing and analysis module (523, FIG. 5). Responses to stimuli may contain other EP components. In addition, it is not necessary to distinguish between target and non-target eras (moreover, at least P300 is mandatory, but, in the particular case, an analysis of a longer time interval is carried out, as was said above) the presence of all components, and their presence and severity is an individual feature of each user (subject) or at least one user. In the particular case, the reactions (of the user) to the target and non-target stimuli may differ in the time range of the P300 wave. Also, in some cases, the maximum (in particular, the main) component that distinguishes between target and non-target incentives may be P200. So, for example, for some users it (component) can “functionally” replace the P300, in particular, have all its properties, but have earlier latency. Also, for some users, the largest component may be negative N170, which in this case can make the main contribution to the recognition of the target stimulus.

The process of dividing the specific features of the EP signal into two classes — target and non-target stimuli — is called classification (a block diagram of an example algorithm of which is shown in FIG. 3), which is the first stage of the system’s operation, that is, “classifier training” occurs. The separation (or at least one part of such separation) of the specific characteristics of the VP signal into two classes is carried out by the data processing and analysis module (523, FIG. 5) and / or the central server 140. To classify the reactions of attention of the IMC user to the target symbol, the team converts the VP into final attributes, i.e. data of a universal form, which are further used in the classification algorithm executed by the computer (130A, 130B) and / or the central server 140, in particular, by at least one module of the said computer and / or server. As such (final) signs, the amplitude values of the VP signal are used, subjected to procedures (in particular, operations, methods implemented, for example, by software algorithms) processed by a computer (130A, 130B) and / or central server 140, and / or means electroencephalogram (120A, 120B), in particular, at least one of their modules. Such procedures (in particular, processing methods) may include filtering the original signal (EEG signal), cleaning such a signal of noise (removing noise from the signal) and artifacts, as well as “downsampling”, i.e. lowering the dimension of the EEG data. The mentioned artifacts may be amplitude (in particular, significant) changes in the EEG data that are not related to the bioelectrical activity of the brain, can be caused by oculomotor or muscle actions. The mentioned downsampling, as a rule, is used to increase the speed and reliability of the VP extraction algorithm and allows to reduce the sensitivity (makes it less sensitive) to some natural changes in the EEG signal (for example, rhythmic changes in the EEG of each user or group, in particular, type users), allowing the described system to work effectively with some variations of this signal, and such variations can be due to both natural changes in the EEG signal and artifacts, i.e. any type, or at least one of them, of “side” activity that complicates the isolation of the target signal. In the filtering procedure (process) of said signal, various band-pass frequency filters can be used (for example, a Butterworth filter). The boundaries of the filtering can vary, but, as a rule, range from 1 Hz to 5-10 Hz, in the particular case, which are an effective filtering band for highlighting a useful signal for the purpose of the described system, and the above-mentioned borders allow to remove the “noise” as much as possible in the form of random oscillations (for example, not related to the response to the stimulus), but at the same time they still do not distort the target signal strongly (EP components). In particular, automatic adjustment of the filtration boundaries can be used, in particular, carried out by a computer (130A, 130B) and / or central server 140, and / or an electroencephalogram recording tool (120A, 120B). After that, data, in particular, obtained after processing the original the signal (after the above procedures) from an array of homogeneous erases are divided into eras (in particular, two arrays with target and non-target eras are formed) relative to the moment of the start of each stimulus, by a computer (130A, 130B) and / or central server 140, and / or means EEG recording (120A, 120B). For the mentioned automatic adjustment of the filtration boundaries, an algorithm can be used that changes the boundaries of the filtration band, for example, if the performed filtering on a standard filtration band does not allow us to identify signals useful for the operation of the described system. The resulting data array (represented as eras) is divided into two new arrays, each consisting of only target and only non-target eras, in particular, one array contains target eras, and the second array contains non-target eras. Then, the aforementioned downsampling (“thinning” of the data) is carried out, during which the dimensionality of the data is reduced by selecting (selecting) each N-th sample from the total array of filtered data, so that the sampling frequency of the said signal decreases. The choice of the degree of downsampling can be set manually, for example, by an operator, a user, etc., or adjusted automatically (in particular, using an algorithm that enumerates several different downsampling parameters and selects the option that gives the most meaningful information for further analysis), for example, by a computer (130A, 130B) and / or a central server 140, and / or an electroencephalogram recording means (120A, 120B). After carrying out all the preparatory procedures (the processing methods described above, such as downsampling, filtering, removing artifacts) by a computer (130A, 130B) and / or a central server 140, and / or an electroencephalogram recording tool (120A, 120B), signs are generated: data the arrays with target and non-target epochs after all preparatory operations are regrouped by a computer (130A, 130B) and / or central server 140, and / or an electroencephalogram recording tool (120A, 120B) so that two two-dimensional matrices are obtained. Such said feature matrices are input to said classifier. In the particular case, both linear and non-linear classifiers can be used. In a particular case, at the output, the classifier produces a weight vector, which is used in the next stage to determine at least one target command.

The next stage (stage) is, on the basis of the existing classifier, the need to determine, by comparing the EP reactions to all stimuli (command stimuli) in the matrix, which stimulus is the target, i.e. what stimulus the user's attention is directed to (105A, 105B). Thus, the user selects (105A, 105B) commands in the IMC-P300 by focusing the user (105A, 105B) on highlighting (highlighting, blinking, etc.) the target stimulus command and ignoring the highlighting of the other stimulus commands. That is, it becomes possible to select any command from a whole set of command characters presented on the display (110A, 110B). To determine the target team, the algorithm uses the previously trained classifier to receive new EEG data. To do this, the same preprocessing operations described above (preparatory procedures) are performed with the original signal. Further, the existing mentioned classifier weights are applied to the obtained attributes, in particular, to the obtained attributes after carrying out the described preparatory procedures, the said classifier is applied to the received data arrays. Based on the application of the classifier weights to the data obtained from the EEG, the system, for example, using a computer (130A, 130B) and / or central server 140, selects among all presented stimuli, in particular, symbols, one target stimulus, in particular, a symbol.

That is, during operation, the described system implements the formation of a menu of stimuli, in particular, of at least two, of symbols, graphic, text, etc. presenting such stimuli on the display (110A, 110B), registers the EEG using an electroencephalogram recording tool (120A, 120B) and compares the received brain reactions of the user (105A, 105B) using a computer (130A, 130B) and / or central server 140 reactions that could be formed in response to the behavior (in particular, the presentation and possible types of such presentation) of each of the stimuli (in particular, symbols) present on the display (110A, 110B). When matching is found, the system executes the command that is assigned to the corresponding stimulus.

In FIG. 3 shows a block diagram of a classification algorithm.

In step 314, using the display (110A, 110B), target and non-target stimuli are presented to the user (105A, 105B) within the framework of the IMC-P300 paradigm, in particular, sequential highlighting, in particular, highlighting of stimulus groups, in particular, IMC symbols.

In step 324, using the EEG tool (120A, 120B), EEG is recorded at positions with the expected largest differences in reactions in response to target and non-target stimuli (where the largest differences in reactions in response to target and non-target stimuli are expected).

In step 334, the computer (130A, 130B) and / or the central server 140 performs the preprocessing (preprocessing procedure) of the signal, for example, filtering, etc., as described above.

In step 344, the computer (130A, 130B) and / or the central server 140 selects segments (epochs) of a given duration, tied to the time of filing, in particular, the presentation, of each stimulus.

In step 354, a computer (130A, 130B) and / or a central server 140 performs the formation of two feature vectors (distinguished epochs for each stimulus in each EEG channel) - for target and non-target stimuli.

In step 364, the computer (130A, 130B) and / or the central server 140 performs a classification, in particular, weights are calculated for each feature that determine its significance for distinguishing between the class of target and non-target stimuli.

In FIG. 4 shows an exemplary embodiment of a neuro headset placed on a user's head.

A special case of an electroencephalogram recording tool (120A, 120B) is a multi-electrode electroencephalograph. In a particular case, for patients (users) with organic damage to brain regions, as well as for patients who have not lost functionality, the system or at least one part of it, in particular, the encephalograph (electroencephalograph), is set up, which consists in that test objects are presented to the user (105A, 105B) and post-processing of the EEG signals (data) is provided in such a way that the maximum distinguishability of the waves used to determine the object (symbol or other object) on which the user's attention is focused (105A, 105B) is ensured. At the same time, the presentation, in particular, of the flicker frequency, stimuli (symbols or other displayed objects) can be adjusted to provide maximum distinguishability, for example, P300 waves, or frequencies that have an individual negative effect, for example, causing a pre-epileptic state, can be excluded. In the particular case, additionally the disappearance of the stimulus (symbol or other object) from the display (display screen) can be carried out by gradually reducing the brightness of such a stimulus. In a particular case, system tuning involves training the algorithms described in this invention.

The tool for recording the electroencephalogram (120A, 120B), in particular, the electroencephalograph is an integral part of the neuro-headset, which provides fastening on the user's head (105A, 105B) of the electrodes (115A, 115B) in the necessary (certain) positions. In the particular case, during the aforementioned system setup, the system (s) are mounted, in particular, the location of the electrodes (115A, 115B) is adjusted so that all the electrodes fit snugly against the surface of the user's head (105A, 105B) to ensure reliable EEG signal. The electrodes are located in positions where the recorded EEG signal is most informative for detecting differences between reactions in response to target and non-target stimuli. Preferred areas are the parietal and lateral occipital areas of the cortex, as shown in FIG. 4.

Taking into account the principle of operation of the system described above, the distinguishability of reactions to target and non-target objects of attention is provided (blinking sequentially at different times and also differing in position within the display). Accordingly, the system is tuned, in particular, tuning of parameters such as the temporal characteristics of stimuli (duration of the backlight, the interval between the two closest backlights, the total number of backlights, etc.). These parameters are selected so that the overall frequency of stimulation is comfortable (for the user) and does not create side effects, including in the form of epilepsy. The frequency characteristics of the stimuli are determined by the duration of the “backlight + pause” interval, which, in the particular case, is about 200 ms (the exposure time of the stimulus itself along with the time when no symbol is active). Depending on the individual capabilities (in particular, features) and user preferences (105A, 105B), this interval can be changed (for example, reduced to increase the overall speed of work (in particular, a set (input) of characters), or increased if the user does not manages to respond to character highlighting).

In FIG. 5 shows a part of the described system for one user, in particular, user equipment 101 is shown.

Shown in FIG. 5, user equipment 101 relates to one user, in particular to user 105A, computer 130A, electroencephalogram recording means 120A, associated electrodes 115A, display 110A, although it is understood that the user equipment or the local part of the system can relate to any user for example, to a user 105B, a computer 130B, an EEG tool 120B, electrodes 115B associated with it, a display 110B and / or at least one other user. The number of users in the described system, as well as the computers, means, modules, electrodes, displays, etc. mentioned in the particular case is not limited, and may depend on the connection speed of the computer, or any other part of the described system (for example, if their composition, or related computing devices, tools, modules, etc. with the possibility of connecting between such devices, tools, modules, etc.), with the Internet network (system), data processing speeds of the described devices, tools , modules, etc. and their other characteristics.

In the particular case of implementation, the EEG reactions to the presentation of, in particular, the backlight, command screen stimuli, such as, for example, symbols associated with the communication module 533, a particular case of which is the Internet communication module, and at least one control module, including at least one external device, in a particular case, which is the smart home system (control), a verticalizer, a wheelchair, etc.

In the particular case, the pace of presenting stimuli, in particular, highlights, is determined individually individually comfortable and safe for a particular user, for example, reducing the risk of an epileptic seizure.

In the particular case of implementation, the described system contains means for determining the vital signs of the user and provides shutdown (in particular, shutdown) of the communication module 533 and / or display (110A, 110B), and / or the stimulus generation module 513, and / or other modules (means , devices, etc.), for example, when there is a threat to health and / or life, and / or indicators recorded by the mentioned modules (means, devices, etc.) of changes, in particular, deterioration of the user's condition (105A, 105B ), for example, with an increase in blood pressure or an increase in the user's heart rate (105A, 105B), their parts, devices, etc., registered by the corresponding modules, for example, tonometers, heart rate monitors, etc., associated with at least one part of the described system, in particular, at least one module, and such recording modules can be located both on the user (105A, 105B), and can be remote, in particular, allowing carry out the mentioned registration at a distance from the user.

When communicating between users, the statuses of communication participants (users) can be set by users or by an automated decision-making system, in a particular case, which is part of a central server 140, computer, etc. User statuses may differ from the statuses set by the users themselves. For example, if the means of tracking the parameters of the vital functions of users (means of determining the vital indicators of the user), in a particular case, integrated into the system or, in the particular case, a subsystem of the described system, tracking the parameters of the vital functions of users, detect, in particular, registering that the user sleeps, then his status is automatically switched by the described system to the “do not disturb” mode. In the particular case, if it is registered by the mentioned means that the user 105A is in an excited state, then the described system or at least one of its modules checks the history of previous communications, in particular, connections, contacts, etc., with other users , and displays the statuses of other users (participants). The system determines the connections of the user 105A and the statuses of other users, which can worsen the state of the user 105A, and translates the status of the user 105A into the status "inactive", even if other users are ready for communication. In addition, the system can be equipped with a software “emergency button” used to send alarm messages to users, medical personnel, etc.

An additional feature of the organization of the interface of the described system, in the particular case, is the simultaneous location on one screen of the display 110A of the message set module (software, in particular virtual, keyboard and window with input text), and correspondence, in particular, the history of correspondence, at least , with one user, in particular, with the selected user, including messages appearing in real time from this user. In the particular case, part of the screen or the entire screen can be used to enter information, in particular characters, with or without other information (associated with at least one user, user messages, user communities, user comments, attached images , notifications, etc.), and, in the particular case, to view the sent and received messages, the user can exit the information (data) input mode, in particular, type (enter) text, or remain in the information input mode.

In a particular case, the described system may be a social network or part of such a social network, or another means of user interaction with each other (including may be a computer game system) using various devices (modules, etc.), which may include at least one processor and / or at least one microcontroller, means for recording user actions, for example, used to interact, including communicate, users with each other using electronic computing devices, for example, equipped with (or related) means of displaying information, such as displays, monitors, etc., and registering user actions, and terminal devices (for example, computer (s), means (means) for recording an electroencephalogram (120A, 120B) etc.) support, including continuous, communication with a central server 140 providing a call to active and / or inactive current users and forwarding (sending) messages.

In the particular case, the stimulus generation module 513 creates, in particular, the generation of stimuli displayed on the display 110A to the user 105A. The creation of stimuli by the stimulus generation module 513 is carried out using the characteristics of the stimuli. Characteristics of stimuli and display of stimuli, such as, for example, the number of presented stimuli, (geometric) sizes of stimuli, the rate of change (increase and decrease) of the sizes of stimuli, the start and end times of presentation of stimuli, the frequency of flickering of stimuli, the rate of change of brightness of stimuli, the location of stimuli, rate of change of location of stimuli, speed of movement of stimuli, relative position of stimuli relative to other stimuli, screen boundaries, program window (interface container), etc. are set by the stimulus generation module 513. In the particular case, the mentioned stimulus characteristics can be entered into the stimulus generation module 513 by an operator, in particular, by medical personnel, for example, using the stimulus characteristic setting module, which can be part of a central server 140, computer 130A, a submodule stimulus generation module 513 or may be associated with them in any known manner. The stimulus generation module 513 may also transmit data to the data processing and analysis module 523 for further processing or transmission to the central server 140. Such transmitted data may be characteristics (parameters) of the generated (and presented) stimuli, for example, the time of creation (appearance) of stimuli , the disappearance time of the stimuli, the number of stimuli, etc. that can be used by the data processing and analysis module 523, as described in the framework of the present invention.

In the particular case, the means for recording (recording) the electroencephalogram 120A using the connected electrodes 115A registers the electroencephalogram (EEG), in particular, to register the brain activity (brain activity) of the user 105A to the stimuli displayed to the user 105A, in particular, to the ones selected by the user 105A stimuli displayed on the display 110A, the response to which is recorded by the electroencephalogram recording means 120A. The registered electroencephalogram is transmitted to the data processing and analysis module 523 of the computer 130A using wired and / or wireless means of data transmission. In the particular case, the recorded electroencephalogram can (also) be transmitted to the central server 140. In the particular case, before being transmitted to the data processing and analysis module 523, the registered electroencephalogram can be pre-processed by means of the recording of the electroencephalogram 120A or at least one part thereof, for example , pre-processing module. Said pre-processing (preprocessing) may include, for example, splitting one signal recorded by the EEG meter 120A into several signals, extracting one signal from the remaining signals, averaging the epochs, removing (eliminating) the noise of at least one signal, extracting the wave P-300 etc. In the particular case, such preprocessing can be performed by a central server 140 or a computer, for example, a data processing and analysis module 523 and / or another module, for example, a data preprocessing module, which is part of the central server 140 and / or computer 130A. In the particular case, the data processing and data analysis module 523 or at least one part thereof, in particular, a submodule (submodule) may be part of a central server 140.

The data transmitted to the data processing and analysis module 523, in particular, transmitted by the electroencephalogram recording means 120A, are analyzed (and / or processed) by such a module and / or transmitted to the central server 140, for example, if at least all or part processing and / or analysis of such data is carried out on a central server 140.

In the particular case, the data processing and analysis module 523 correlates (in particular, comparing) the registered brain activity of the user 105A (according to the timeline for presenting stimuli) with the creation time and nature, in particular, parameters (characteristics, etc.), stimuli transmitted by the means of recording the electroencephalogram 120A and / or the stimulus generation module 513, including, in accordance with (depending on) the disease (s) of the user 105A, stimulus parameters, etc. In the particular case, the characteristics of the stimuli (and the parameters, in particular, the features, their presentation) are determined, in particular, calculated by the data processing and analysis module 523 with the use of machine learning, artificial intelligence, etc., and the characteristics can be transferred to the stimulus generation module 513 and can be combined with the characteristics of the stimuli defined (and / or stored) in the stimulus generation module 513.

Characteristics of stimuli and / or data after analysis and processing, as described in the framework of the present invention, and / or data recorded (in real time) or recorded by means of electroencephalogram recording 120A are transmitted to communication module 533, in particular, for transmission and processing by server 140 and / or another computer, for example, computer 130B. In the communication module 533, stimulus characteristics from the stimulus generation module 513 can also be transmitted by computer 130A.

By selecting stimuli, in particular a response to stimuli recorded by the electroencephalogram recording means 120A and processed by the computer 130A, as described in the framework of the present invention, the user 105A can communicate with other users, including, for example, entering text messages, managing such messages for example, send messages, select the user to whom the message will be sent, switch the layout of the (software) keyboard, select the register, size, color, etc. text, select and add an image to the message, reply to messages of other users, add comments to messages, etc.

Data from the data processing and analysis module 523 is transmitted to the communication module 533, which transmits such data to the central server 140. The central server 140 receives data from the communication module 533, and such received data can be transmitted in message format, encoded, etc. d. In the particular case, on the server 140 (in the database of the server 140 on the information storage device, in particular, the data storage), the settings described in the framework of the present invention, for example, for users (110A, 110B) transmitted from the computer (130A, 130B) can be stored with certain diseases, damage to certain parts of the brain, etc. The central server 140 transmits data, in particular, messages received from one computer 130A, to another computer, for example, to a computer 130B, in particular, to the communication module of such a computer 130B with their display on the display of such a computer, for example, on the display 110B . The mentioned database can be hierarchical, object, object-oriented, document-oriented, object-relational, relational, network and / or functional database, each of which can be centralized, concentrated, distributed, heterogeneous, homogeneous, fragmented (partitioned) replicated, spatial, temporal, spatio-temporal, cyclic, super-large database, etc., moreover, various database management systems (DBMS) can be used to manage, create and use databases. In a particular case, the data storage may be a temporary storage device (for example, Random Access Memory (RAM)), a permanent data storage, for example, (Programmable) Permanent Storage Device (ROM or ROM), including, at least, implemented , a single chip or chipset, etc. Also, the data in the data warehouse can be stored in at least one file, in particular, in the form of a text file, or the data can be stored in any at least one other currently known format for storing data (information) or in a data format invented later.

In FIG. 6 shows an exemplary embodiment of a display screen with displayed objects for user interaction. In the particular case of at least one element depicted in FIG. 6 is an incentive. As described in the framework of the present invention, displayed objects, in particular grouped (or modules, in particular program modules) on the display screen 110A, may include a message set module 656, with which the user enters messages, for example, at least one other user. Displayed objects may also include the history of correspondence 646, for example, with at least one user, in the form of a display of previous and current (in particular, user-dialed in real time) messages.

Also, the displayed objects may include a window, in particular, a text box for entering messages 616, in which, in a particular case, the user enters messages and / or in which the user displays a message 626. In the particular case, the message box 616 is also a stimulus, when selected, the user can start or continue entering a message, the program cursor 636, which can also be a stimulus (user-selectable), can be controlled by the user, for example, can be moved by the user.

In the particular case, the objects presented to the user can be grouped into at least one group, for example, into a group of objects that is a module of message set 656, etc. (which are shown by a dashed line in FIG. 6), wherein the user can switch between groups as described in the framework of the present invention, for example, the group can be presented to the user as an object to which the user is responding, or objects displayed on the screen can be used by the user for switching a user between said groups, where said switching objects can be located within at least one such group, located outside such a group, and at least one additional object display layer can be used to display such switching objects, at least on one screen or in a program window.

In FIG. 7 shows an example of a general-purpose computer system that includes a multipurpose computing device in the form of a computer 20 or a server, or a mobile (computing) device, or a module of the system described in the present invention, which, in a particular case, can be terminal (computing) devices (for example, a user, an operator, etc.) including a processor 21, a system memory 22, and a system bus 23 that couples various system components, including the system memory to the processor 21.

The system bus 23 may be any of various types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. System memory 22 includes read-only memory (ROM) 24 and random access memory (RAM) 25. The ROM 24 stores the basic input / output system 26 (BIOS), consisting of basic routines that help exchange information between elements within the computer 20, for example at the time of launch.

Computer 20 may also include a hard disk drive 27 for reading from and writing to a hard disk (not shown), a magnetic disk drive 28 for reading from or writing to a removable magnetic disk 29, and an optical disk drive 30 for reading from or recording to a removable optical disc 31 such as a CD, a digital video disc, and other optical means. The hard disk drive 27, the magnetic disk drive 28, and the optical disk drive 30 are connected to the system bus 23 by means of the hard disk drive interface 32, the magnetic disk drive interface 33, and the optical drive interface 34, respectively. Storage devices and their respective computer-readable means provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for computer 20.

Although the typical configuration described here uses a hard disk, a removable magnetic disk 29, and a removable optical disk 31, one skilled in the art will appreciate that other types of computer-readable means that can store data that are accessible by a computer may also be used in a typical operating environment. such as magnetic cassettes, flash memory cards, digital video discs, Bernoulli cartridges, random access memory (RAM), read-only memory (ROM), etc.

Various software modules, including operating system 35, may be stored on a hard disk, magnetic disk 29, optical disk 31, ROM 24, or RAM 25. Computer 20 includes a file system 36 associated with or included with the operating system 35 or more software application (s) 37, other program modules 38, and program data 39. A user may enter commands and information into computer 20 using input devices such as a keyboard 40 and pointing device 42. Other input devices (not shown) may include a microphone, joystick, gamepad, satellite dish, scanner, or any other.

These and other input devices are connected to the processor 21 often through a serial port interface 46, which is connected to the system bus, but can be connected via other interfaces, such as a parallel port, game port, or universal serial bus (USB). A monitor 47 or other type of visual display device is also connected to the system bus 23 via an interface, for example, a video adapter 48. In addition to the monitor 47, personal computers typically include other peripheral output devices (not shown), such as speakers and printers.

Computer 20 may operate in a networked environment through logical connections to one or more remote computers 49. The remote computer (or computers) 49 may be another computer, server, router, network PC, peer to peer device, or other node on a single network, and typically also includes includes most or all of the elements described above with respect to computer 20, although only an information storage device 50 is shown. Logical connections include a local area network (LAN) 51 and a global computer network (GSC) 52. Such network environments typically common in institutions, corporate computer networks, the Internet.

Computer 20 used in a LAN network environment is connected to a local area network 51 via a network interface or adapter 53. Computer 20 used in a GCS network environment typically uses a modem 54 or other means to establish communication with a global computer network 52, such as the Internet.

The modem 54, which may be internal or external, is connected to the system bus 23 via the serial port interface 46. In a networked environment, program modules or parts thereof described with reference to computer 20 may be stored on a remote information storage device. It should be noted that the network connections shown are typical, and other means may be used to establish communication communication between computers.

In conclusion, it should be noted that the information provided in the description are examples that do not limit the scope of the present invention defined by the claims. One skilled in the art will recognize that there may be other embodiments of the present invention consistent with the spirit and scope of the present invention.

Claims (17)

1. A system for communication of users without the use of speech and movement, comprising means for recording an electroencephalogram of a user, the information outputs of which are connected to a data processing unit, the video output of which is connected to a display, and the network interface is designed to connect to a remote server designed to form a social network made with the ability to communicate with multiple users in real time, while the EEG recording tool is configured to record the EEG by determining the activity of the brain by measuring the electrical potentials on the surface of the user's scalp in real time, extracting from the EEG parameters of at least one component caused by potentials (VP) with predetermined basic characteristics and the formation of the output signal in the form of a description of the parameters of at least one component caused by the potential In digital form, the data processing unit is configured to generate varying light stimuli on the display screen that have different parameters of change over time such that, while maintaining user attention on different stimuli, at least one component of the evoked potentials has different parameters, with each of the stimuli given in accordance with a predetermined action, the data processing unit is responsible for determining, in real time, the stimuli directly selected by the user and performing actions corresponding to the selected stimuli by the user by monitoring the selection of objects from a variety of objects, while the selected stimuli are the stimuli and reaction parameters in the answer to which maximally corresponds to the parameters of at least one component of the evoked potentials that arise during the formation of stimuli on the device screen, while the device screen is configured to display the results of actions According to the results of presenting incentives to several users on several screens, the separation of the parameters of at least one component of the evoked potentials from the electroencephalogram is performed by the data processing unit, and information about actions based on the results of presenting incentives between the data processing units of different users is transmitted via a remote server. 2. The system according to claim 1, characterized in that the separation of the parameters of at least one component of the evoked potentials from the electroencephalogram is performed by the data processing unit using machine learning that takes into account the nature of the limited motor and speech functions of the user. 3. The system according to claim 1, characterized in that machine learning is performed for each user separately. 4. The system according to p. 1, characterized in that the means of recording the electroencephalogram is made containing electrodes intended for connection with the scalp of the user, installed in the mount with the ability to move. 5. The system according to p. 4, characterized in that the fastening means is configured to install and remove electrodes. 6. The system according to claim 1, characterized in that the P300 wave is a component of the evoked potential with predetermined characteristics. 7. The system according to claim 1, characterized in that it contains a block for setting stimulus parameters. 8. The system according to claim 7, characterized in that one of the parameters of the stimulus is the time moment of the beginning of the display of the stimulus. 9. The system according to p. 8, characterized in that at least one of the other parameters of the stimulus is the moment of quenching of the stimulus, the speed of movement of the stimulus on the display, the brightness of the stimulus, the frequency of the flicker of the stimulus, the geometric dimensions of the stimulus, the outline of the stimulus, the information shown in stimulus. 10. The system according to p. 9, characterized in that the geometric dimensions of the stimulus are set in accordance with the angular dimensions of the stimulus displayed on the display, determined relative to the position of the patient, and these sizes are adjusted individually for each user to provide the best visual distinction between them for the most effective detection of stimulus-induced VP reactions. 11. The system according to p. 8, characterized in that the time moments of the beginning of the display of stimuli are set by various settings, providing a cyclic display of stimuli on the screen. 12. The system according to p. 11, characterized in that the frequency of the stimulus display cycles is selected so that potential negative consequences for the user from observing the stimuli on the display screen are minimized by eliminating the frequencies causing the pre-epileptic state when selecting the cycle frequency. 13. The system according to p. 10, characterized in that the dimensions of the stimuli provide a distinguishability of the stimuli on the screen and the information related to the stimuli. 14. The system according to claim 1, characterized in that it is configured to display on one display a message set module with a virtual keyboard, a typed message display module, and a correspondence display module. 15. The system of claim 14, wherein the correspondence display module is configured to display the history of correspondence with at least one other user. 16. The system of clause 15, wherein the correspondence display module displays messages generated by another user in real time. 17. The system according to any one of the preceding paragraphs, characterized in that the electroencephalogram recording means is configured to extract from the electroencephalogram parameters of more than one component of the evoked potentials with predetermined basic characteristics and generating an output signal in the form of a description of the parameters of the components of the evoked potentials in digital form , and the stimuli selected by the user are considered to be stimuli, the reaction parameters in response to which correspond to the parameters of the EP components that arise during the formation of stimuli on the device screen.

Images