RU2776924C1 - Method for generating the signature of an unwanted electronic message - Google Patents

Abstract

FIELD: information security.

SUBSTANCE: method for generating the signature of an unwanted electronic message includes the stages of: determining the attributes of at least two consecutively intercepted electronic messages; classifying the first electronic message as unwanted using a trained model for classifying electronic messages; determining that the first and second electronic messages belong to the same cluster of electronic messages, based on certain attributes; generating the signature of the unwanted electronic message based on the determined cluster of electronic messages.

EFFECT: ensured high detection rate and reduced probability of type I and II errors occurring when detecting unwanted electronic messages due to the generation of the signature of an unwanted electronic message using machine learning methods.

7 cl, 4 dwg

Description

Technical field

The invention relates to the field of information security.

State of the art

At the moment, almost any owner of an electronic device (for example, a computer, mobile phone, smartphone, tablet, etc.) encounters unwanted electronic messages: SMS messages and emails from banks, medical centers, insurance companies, beauty salons and others. companies that use telemarketing in their work. Also, emails are often used by fraudsters to obtain confidential user information (for example, credit card information, social security numbers, etc.).

For example, unsolicited emails such as spam emails are a serious problem as they already account for 70-90% of total email traffic. This amount of spam e-mails causes inconvenience for e-mail users and disruption of mail servers. In particular, it leads to a deterioration in throughput, a waste of resources, and an increase in the processing time of emails by postal services.

There are two main approaches for detecting spam emails: signature-based and using machine learning methods.

With the signature approach, the mass distribution of electronic messages is disassembled into parts, from subsets of which signatures are formed. Signatures allow unambiguous detection of unwanted electronic messages in electronic messaging systems (for example, in mail services, instant messaging service). The advantage of this approach is a high detection rate and an almost zero probability of a Type I error, i.e. false positive. A false positive in this case is when a legitimate email message is determined to be spam. The disadvantage of this approach is the omission of unwanted e-mails (ie, the occurrence of a Type II error).

For example, patent publication US20190281000A1 describes a technology for detecting unwanted electronic messages using a signature. The signature is a hash calculated from the body text of the spam e-mail message.

Machine learning methods use a collection of spam emails and a collection of legitimate emails. Signs are extracted from the messages, which are used to train the classifier. The advantage of this approach is a high generalization ability, which allows to achieve a minimum number of skips of unwanted electronic messages. At the same time, the disadvantage of this approach is the high probability of false positives, as well as the low detection rate compared to the signature approach.

The present invention allows to eliminate the noted shortcomings and solve the problem of detecting unwanted electronic messages by generalizing the signature verdict using machine learning methods.

Disclosure of invention

The invention is intended to ensure information security.

The technical result of the present invention is to generate a signature of an unsolicited electronic message.

These results are achieved by using a spam email signature generation method, the method comprising: determining attributes of at least two successively intercepted email messages; classifying the first email message as spam using the trained email classification model; determine that the first and second electronic messages belong to the same cluster of electronic messages based on certain attributes; generating an unsolicited electronic message signature based on the determined cluster of electronic messages.

In a particular case of implementing the method, an electronic message is intercepted during transmission through at least: e-mail; messenger; instant messaging service.

In another particular case, the implementation of the method is classified as an unsolicited electronic message transmitted at least for: fraud; unauthorized receipt of confidential information; imposition of goods and services.

In another particular case of the implementation of the method, at least one of the following classification algorithms acts as a trained model for classifying electronic messages: Bayesian classifiers; logistic regression; MRF classifier; support vector machine; nearest neighbor methods; decision tree; recurrent neural network.

In a particular case of the implementation of the method as an attribute of the electronic message is at least: a sequence of words identified from the text of the electronic message; fuzzy hash calculated based on the sequence of words of the text of the electronic message; a vector characterizing the text of an electronic message.

In another particular case of implementing the method, an unwanted electronic message signature is generated based on one of the following criteria: the largest common subsequence of words in the text of electronic messages that are contained in a certain cluster of electronic messages; the largest common subsequence of characters of fuzzy hashes calculated from the text of electronic messages contained in a particular cluster of electronic messages.

In a particular case of the implementation of the method, the signature of an unwanted electronic message is formed on the basis of a redefined cluster of electronic messages in such a way that more unwanted electronic messages are determined using the generated signature than with the current one.

Brief description of the drawings

Fig. one illustrates one of the options for the architecture of the electronic messaging system.

Fig. 2 illustrates one of the system architecture options that implements a method for generating a spam email signature .

Fig. 3 illustrates a method for generating a spam email signature.

Fig. 4 represents an example of a general purpose computer system, a personal computer or a server.

Although the invention may have various modifications and alternative forms, the characteristic features shown by way of example in the drawings will be described in detail. It should be understood, however, that the purpose of the description is not to limit the invention to a particular embodiment thereof. On the contrary, the purpose of the description is to cover all changes, modifications, included in the scope of this invention, as defined by the attached claims.

Description of embodiments of the invention

The objects and features of the present invention, methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below, but may be embodied in various forms. The gist of the description is nothing but the specific details necessary to assist a person skilled in the art in a thorough understanding of the invention, and the present invention is defined within the scope of the appended claims.

On FIG. 1 shows one of the options for the architecture of the electronic messaging system.

This system includes an electronic messaging system 100 , a user device 110 , an electronic message 111 , an agent 120 , a storage device 130 , classification attributes 140 , a classification model 150 .

The electronic messaging system 100 is a system of physical and logical communication channels that implements a protocol for transmitting an electronic message 111 between user devices 110 .

Examples of electronic messaging systems 100 are at least:

• Post service;

• instant messaging service;

• messenger.

The electronic message 111 has a certain structure. In addition to the body (English body), the electronic message 111 contains headers (English header) - this is service information, including information about the route of the electronic message 111 . The headers provide information about when, from where, and by what route the e-mail came 111 , as well as information added to the e-mail 111 by various utilities. Each header is defined by its name and value. The header value is information presented in a predetermined form. For example, for a header that contains information about the sender of email 111 , the name would be "from" and the value would be the sender's email address, such as username@domain.com.

The user device 110 generates an electronic message 111 and transmits it to the electronic messaging system 100 and also receives an electronic message 111 from other devices 110 .

Agent 120 intercepts email 111 at least:

• tracking the traffic received and transmitted via the protocol of transmission of an electronic message (POP3, SMTP, IMAP, NNTP);

• keeping track of files in the storage of the electronic messaging system 100 ;

• keeping track of files in storage on the user's device 110 .

For example, agent 120 is a module (software component) of a security application of JSC Kaspersky Lab (for example, Kaspersky Internet Security).

The agent 120 determines the classification attributes 140 contained in the intercepted email message 111 and transmits them to the storage device 130.

For example, the attributes of the classification 140 of an e-mail are the values of the headers related to information about the route of passage , as well as service information generated by mail clients :

• Message_ID – a unique e-mail identifier assigned by the first mail server it encounters on the way;

• X-mailer (mailer_name) – value of the header field in which the mail client or service with which the email was created identifies itself;

• sequence of e-mail header values.

In another implementation, the Agent 120 determines the classification attributes 140 of the electronic message 111 based on an event associated with the indicated electronic message. For example, the instant messaging email classification 140 attributes are:

• flag of viewing the electronic message 111 by the user;

• a flag for interacting with the user's device 110 while viewing the electronic message 111 (eg, scrolling through the message);

• a flag for deleting an electronic message 111 by the user;

The data storage device 130 is designed to store and process classification attributes 140 . For example, the classification attributes 140 are used to train the classification model 150.

In the preferred embodiment, the data storage device 130 is a cloud storage device that processes the classification attributes 140 in the so-called “cloud”, where “cloud” is a storage model that stores data on the Internet using a cloud computing resource provider that exposes the data warehouse as a service and manages it.

For example, the data storage device 130 can be a tool containing the Kaspersky Security Network (KSN) system of Kaspersky Lab JSC.

On FIG. 2 shows one of the options for the architecture of the system that implements the method for generating the signature of an unsolicited electronic message .

Block diagram of a system that implements a method for generating an unwanted electronic message signature contains an electronic message111(including111a,111b,111v), agent120, storage device130, classification attributes140, clustering attributes210(including210a,210b,210v),classification model150,means of classification220, similarity tool230, signature generator240,signature250,spam cluster260.

In one embodiment, said classifier220, similarity tool230 and signature generator240are integral parts of the data storage device130.

In another embodiment, the classifier220, similarity tool230 and signature generator240may be separate, independent, but related, technical means that can be implemented using separate computer systems.

Agent120designed to intercept email111(including111a,111b,111v),classification attribute definitions140,clustering attributes210(including210a,210b,210v),as well as passing classification attributes140, clustering attributes210aand210bstorage device130.

The attributes of clustering 210 are at least:

• a sequence of words identified from the text of an electronic message 111 ;

• fuzzy hash calculated based on the sequence of words in the text of the electronic message 111 ;

• a vector characterizing the text of an electronic message 111 .

The storage device 130 is used to classify the electronic message 111a , determine that the electronic messages 111a and 111b belong to the same cluster , and generate a signature 250 .

Classifier220, in the preferred implementation being part of the data storage device130, designed to classify an electronic message111a based on classification attributes140using a classification model150. Model150classifies the email111a at least like:

• unwanted;

• legitimate.

For example, the spam email is an email 111 sent for at least:

• implementation of fraud;

• unauthorized receipt of confidential information;

• the imposition of goods and services.

Classification model150in advance teach with classification attributes140, transferred to the storage device130, so that the classification model150 determines, based on the specified attributes, the signs by which it classifies the electronic message111a as unwanted with some probability.

The classification algorithm is at least one of the following algorithms (or a combination thereof):

• Bayesian classifiers (English naive bayesian classifier);

• logistic regression;

• MRF classifier (eng. MRF classifier);

• support vector machine (SVM);

• nearest neighbor methods (eng. k-nearest neighbor);

• decision tree;

• Recurrent neural network (RNN).

For example, the classification model 150 may be based on deep learning methods. Specifically, classification attributes 140 are represented as a matrix, where each character of classification attribute 140 is encoded as a vector of fixed length numbers, and subjected to a neural network transformation that calculates how similar these attributes are to poison message attributes. The attributes of the classification 140 transformed by the layer of the neural network act as features.

Classifier220defines electronic message111aas unwanted,if email similarity111a with spam e-mails above a predetermined value (eg 0.7).

Similarity tool230, in the preferred implementation being part of the data storage device130, is intended to determine that electronic messages111aand111bbelong to the same cluster based on clustering attributes210a and210b.

For example, an agent120transfers to the storage device130certain clustering attributes210a and210brelevant electronic messages111aand111b,where as clustering attributes210use a fuzzy hash calculated based on the sequence of words in the text of electronic messages111aand111b.Similarity tool230calculates the Levenshtein distance, defined as the minimum number of single-character operations (inserts, deletes, substitutions) required to turn one sequence of characters into another, and determines that email messages111aand111bbelong to the same cluster if the specified metric does not exceed a predetermined value.

The data storage device 130 determines that the emails 111a and 111b belong to the spam email cluster 260 if the email 111a was classified by the classification engine 220 as spam and the similarity engine 230 determined that the emails 111a and 111b belong to the same cluster.

Signature generator240, in the preferred implementation being part of the data storage device130, designed to form a signature250spam e-mail based on a cluster of spam e-mails260.

The signature 250 of the spam cluster 260 is generated based on at least one of the following criteria:

• the largest common subsequence of words in the body of email messages 111a and 111b that are contained in the spam email cluster 260 ;

• the largest common character subsequence of fuzzy hashes calculated from the text of the email messages 111a and 111b that are contained in the spam email cluster 260 .

In one embodiment, the spam email signature is generated based on the re-determined cluster of spam emails such that more spam emails are detected with the generated signature than with the current signature 250 .

Additionally, the signature generator 240 passes the generated signature 250 to the agent 120 , which determines whether the clustering attributes 210b of the email message 111b contain the signature 250 . and ensures information security.

Ensuring information security includes at least:

• blocking unwanted e-mail 111v ;

• informing the recipient about the unwanted nature of the electronic message 111v ;

• transferring the classification attributes 140 and the clustering attributes 210 of the electronic message 111 to the data storage means 130 .

On FIG. 3 shows a method for generating an unsolicited email signature.

The block diagram of the method for generating a spam email signature includes step 310 , which determines the attributes of at least two sequentially intercepted electronic messages, step 320 , which classifies the first electronic message as spam, step 330 , which determines that the first and second electronic messages belong to the same cluster, step 340 , in which the spam email signature is generated.

At step 310 , the attributes 210 of the two sequentially intercepted email messages 111 are determined by the agent 120 .

At 320 , the classification model 150 classifies the first email 111 as spam.

At step 330 , it is determined by the similarity determiner 230 that the first and second email messages 111 belong to the same cluster 260 .

At step 340 , a signature 250 of an unsolicited e-mail message is generated by the signature generator 240 .

Fig. 4 shows an example of a general purpose computer system, a personal computer or a server 20 ', comprising a central processing unit 21 ', system memory 22 ', and a system bus 23 ', which contains various system components including memory associated with the central processing unit 21 '. The system bus 23 is implemented as any bus structure known in the art, comprising in turn a bus memory or bus memory controller, a peripheral bus, and a local bus capable of interfacing with any other bus architecture. The system memory contains read-only memory (ROM) 24 , random access memory (RAM) 25 . The main input/output system (BIOS) 26 contains the main procedures that ensure the transfer of information between the elements of a personal computer 20 , for example, at the time of booting the operating system using ROM 24 .

The personal computer 20 , in turn, contains a hard disk 27 for reading and writing data, a magnetic disk drive 28 for reading and writing to removable magnetic disks 29 and an optical drive 30 for reading and writing to removable optical disks 31 , such as CD-ROM, DVD -ROM and other optical storage media. The hard disk 27 , the magnetic disk drive 28 , the optical drive 30 are connected to the system bus 23 via the hard disk interface 32 , the magnetic disk interface 33 , and the optical drive interface 34 , respectively. Drives and related computer storage media are non-volatile means of storing computer instructions, data structures, program modules, and other personal computer data 20 .

The present description discloses an implementation of a system that uses a hard disk 27 , a removable magnetic disk 29 and a removable optical disk 31 , but it should be understood that other types of computer storage media 56 that are capable of storing data in a computer-readable form (solid-state drives, flash memory cards, digital disks, random access memory (RAM), etc.), which are connected to the system bus 23 through the controller 55 .

The computer 20 has a file system 36 that stores the recorded operating system 35 as well as additional software applications 37 , other program modules 38 and program data 39 . The user has the ability to enter commands and information into the personal computer 20 through input devices (keyboard 40 , mouse 42 ). Other input devices (not shown) may be used: microphone, joystick, game console, scanner, etc. Such input devices are normally connected to the computer system 20 through a serial port 46 , which in turn is connected to the system bus, but may be connected in other ways, such as a parallel port, game port, or universal serial bus (USB). A monitor 47 or other type of display device is also connected to the system bus 23 via an interface such as a video adapter 48 '. In addition to the monitor 47 , the personal computer may be equipped with other peripheral output devices (not shown), such as speakers, a printer, etc.

The personal computer 20 is capable of operating in a networked environment using a network connection to another or more remote computers 49 . The remote computer (or computers) 49 are the same personal computers or servers that have most or all of the elements mentioned earlier in the description of the nature of the personal computer 20 shown in FIG. 4 . Other devices may also be present in the computer network, such as routers, network stations, peer-to-peer devices, or other network nodes.

The network connections may form a local area network (LAN) 50 and a wide area network (WAN). Such networks are used in corporate computer networks, internal networks of companies and, as a rule, have access to the Internet. In LAN or WAN networks, the personal computer 20 is connected to the local network 50 via a network adapter or network interface 51 . When using networks, personal computer 20 may use a modem 54 or other means to communicate with a wide area network, such as the Internet. Modem 54 , which is an internal or external device, is connected to system bus 23 via serial port 46 . It should be clarified that network connections are only indicative and are not required to represent the exact network configuration, i.e. in fact, there are other ways to establish a connection by technical means of communication from one computer to another.

In conclusion, it should be noted that the information given in the description are examples that do not limit the scope of the present invention defined by the formula.

Claims (29)

1. A method for generating an unsolicited electronic message signature, comprising the steps of: a) determine the attributes of at least two sequentially intercepted electronic messages; b) classifying the first email message as spam using the trained email classification model; c) determining that the first and second electronic messages belong to the same electronic message cluster based on certain attributes; d) generating an unsolicited electronic message signature based on a certain cluster of electronic messages. 2. The method of claim 1, wherein the electronic message is intercepted in transit through at least: • email; • messenger; • instant messaging service. 3. The method according to claim 1, which classifies as unsolicited electronic communication transmitted at least for: • implementation of fraud; • unauthorized receipt of confidential information; • the imposition of goods and services. 4. The method of claim 1, wherein the trained electronic message classification model is at least one of the following classification algorithms: • Bayesian classifiers; • logistic regression; • MRF classifier; • support vector machine; • nearest neighbor methods; • decision tree; • recurrent neural network. 5. The method according to claim 1, wherein at least one of the attributes of the electronic message is: • the sequence of words revealed from the text of the electronic message; • fuzzy hash calculated on the basis of the sequence of words in the text of the electronic message; • a vector characterizing the text of an electronic message. 6. The method of claim 1, which generates an unsolicited email signature based on one of the following criteria: • the largest common subsequence of words in the text of electronic messages that are contained in a certain cluster of electronic messages; • the largest common subsequence of characters of fuzzy hashes calculated from the text of electronic messages contained in a certain cluster of electronic messages. 7. The method of claim 1, wherein the spam email signature is generated based on the re-determined cluster of email messages such that more spam emails are detected with the generated signature than with the current one.

Images