RU2787308C1 - Spam disposal system - Google Patents

Abstract

FIELD: computer protection.

SUBSTANCE: invention relates to the field of computer protection; it can be used for extraction of information useful to an end user from an array of digital data, including spam intended for deletion. The method contains stages of: reception of messages from spam folders; storage of unprocessed messages to at least one database of unprocessed messages; analysis, by means of at least one spam processing module, of text and service information of each spam message in the database of unprocessed messages in accordance with a given criterion for clustering spam according to the given criterion; extraction of useful information from the formed cluster; antivirus check for prevention of infection by means of an antivirus protection unit and a database with information about infected messages; storage of processed information obtained as a result of the above-mentioned processing in a database of processed information, where the database of processed information is associated with at least one mentioned spam processing module; and provision of a user with the results of processing through the module for provision of processed information associated with at least one mentioned database of processed information.

EFFECT: provision of protection of a user’s computing device from spam messages, while simultaneously extracting data from an array of digital data classified as spam.

1 cl, 3 dwg

Description

The invention relates to solutions for extracting information useful to the end user from an array of digital data, including spam, intended to be deleted.

Consider the definition of spam.

Wikipedia gives the following definition of "spam": "Spam (English spam) - mass mailing of advertising correspondence to persons who did not express a desire to receive it, as well as mass mailing."

According to Kaspersky Lab's definition, "Spam is anonymous mass unsolicited mail."

As you can see, none of the definitions says that spam is something that does not deserve attention. But the current practice of working with spam is arranged in such a way that all spam passes by the attention of both the end user and the possible persons responsible for the work of the spam recipient (heads of the corporate user). Namely, the current practice involves storing spam in a separate repository that is outside the main attention of the end user, storing spam in this repository for some short time (for example, "Google Gmail" - 30 days, "Yandex Mail" - 10 days) , and deleting spam without the possibility of subsequent recovery.

At the same time, statistics show that spam makes up the vast majority of the entire information flow at the present time. For example, according to Kaspersky Lab data, at the beginning of 2011, the share of spam in the total volume of email traffic was almost 80% [https://web.archive.org/web/20120724031934/http:/www.computery.ru /news/news2010.php?nid=8302]. The latest data we have shows that spam still accounts for more than half of total traffic as of Q2 2019 [https://securelist.ru/spam-and-phishing-in-q2-2019/94529/].

As a result, it can be stated that more than half of the information flow of messages (with all the costs of forwarding, storing, processing, etc.) is deleted without bringing any benefit to anyone.

The proposal (the essence of the spam disposal system) is to collect spam from recipients with the consent of them, or persons with the right to give such consent for the recipient (guidance in the case of corporate recipients) and extract information from these messages that may be useful to end users service.

The relevance of spam.

The urgency of the problem of spam today no one doubts. Suffice it to cite the figure that more than half of all e-mails entering corporate networks are spam in one form or another. Spam losses incurred by corporate users are significant and, according to some estimates, amount to tens of thousands of dollars due to the loss of working time and the use of network resources.

The message sent during spam, depending on the goals and objectives of the sender (spammer), may contain commercial information, or may not be related to commercial activities. According to the content of the message, "commercial" spam is distinguished - unsolicited commercial e-mail (common abbreviation - UCE) and "non-commercial" - unsolicited bulk e-mail (UBE).

From the point of view of organizing the text of an e-mail, the latter may contain in the Subject field information about what it is (for example, that it contains advertising information), and in the body of the letter - an indication of the reason why the sender considered it necessary to contact the recipient. As a rule, it contains information about what actions the recipient must take in order not to receive messages from the sender in the future.

These signs indicate that the sender understands that the information he offers is being imposed on the recipients, and is conscientiously trying to reduce the possible negative impact. However, very often the spammer is not only unwilling to reduce the discomfort of his recipients, but also refuses responsibility for his actions by falsifying the sender's address, using the address of a third party, and falsifying message headers. All this is done in order to make it as difficult as possible to identify the sender and take appropriate measures.

Spam problem.

Spam should be distinguished from legitimate mailing lists, which, although they share many of the features of spam, are requested by the user and should be delivered to him.

The perniciousness of such mailings (spam) lies in the fact that it costs the spammer practically nothing, but they are expensive for everyone else, for example, the spam recipient or its provider. A large amount of promotional mail can lead to excessive load on the channels and mail servers of the provider, which is why regular mail, which recipients may be waiting for, will be much slower. The recipient of the spam pays for everything, paying their ISP for time on the Web to receive unsolicited mail from the mail server.

In addition to direct costs, separating unsolicited information from the necessary and deleting it requires time, which is also very inconvenient, especially for active people - it often takes them a lot of time. In addition, spam is often used in conjunction with various virus technologies.

The reason for spam is that it is a cost-effective marketing tool, despite its extremely low response rate. After all, its effectiveness is calculated not by the absolute number of responses, but by the ratio of results to costs. And the cost of mass mailings is minimal. Therefore, spamming is effective even at a very low score.

The spam business is becoming more and more profitable and attracting more and more customers. The latter are captivated by the opportunity to spread their advertising to millions of addresses for little money. According to information from open sources, on average, mailing costs from $20-50 (100 thousand addresses) to $500 (several million addresses. Several hundred thousand messages can be sent out in one night. According to spammers, the response to such advertising is at least one out of a thousand recipients, which is a lot for mailings of millions, and the costs are minimal.

In addition, a spammer starting his business does not need to invest a lot of money - he can get by with the several hundred dollars needed to pay for broadband Internet access. True, serious spammers take the matter much more seriously. They contain a staff of qualified programmers, linguistic specialists, etc.

The incredible survivability of spam is due to the almost zero cost of sending it. After all, to conduct spam mailings, all you need is a computer and Internet access. One person is quite capable of sending out up to several million letters daily. All advertising costs are ultimately borne by its recipients. This is the main difference between spam and regular email advertising.

However, mail servers of providers are practically not protected from spam attacks (unless, of course, they have a special filter installed). If the corporate network can be strictly configured to allow incoming information, then the provider should accept all messages by default.

Ways to fight spam.

For the user, spam is the time spent on sorting out mail garbage. For the provider - also lost money. In particular, they are forced to pay for additional traffic generated by spammers, as well as anti-spam filters and the work of staff to set up filters and resolve user complaints.

The Internet community is trying to fight spam.

Recently, various methods of combating spam have been constantly proposed. These methods can be divided into the following categories:

- Legal methods (legislative framework).

- Technical methods (spam filtering on the mail server or client, or other technological methods).

Currently, several technological methods are used to combat spam, but not all of them are sufficiently effective.

First of all, these are "black lists" and automatic filters.

"Blacklists" in addition to the fact that recently they have practically lost their relevance due to all sorts of tricks of spammers to substitute addresses and capture user machines, they are often quite aggressive and may contain addresses of providers whose network was used to send letters or even host sites, advertised by mass mailing.

Automatic filters that search for specified terms (words and phrases), in combination with other methods, are able to suppress up to 95% of unsolicited correspondence. However, spammers routinely get around these barriers by inserting spelling errors and other minor changes to email subject lines and body text. So, very often letters that come to us from domestic spammers contain Latin letters that replace Russian ones similar to them, in particular - "I" and "U", "B", "T", etc. All this is done in order to fool the filters.

To protect addresses in HTML pages, it is recommended to use address encoding. Indeed, in modern browsers there are many possibilities for this method. This and the use of HTML encoding, when the letter "A", for example, is replaced by its code (ie A). There are many special programs for encoding.

Just like a few years ago, many mail servers have installed anti-virus filters, recently large portals (yandex.ru, mail.ru) are starting to use anti-spam filters.

Now new schemes are being developed, more effective methods of protection that use intellectual analysis of text and graphics. But they are quite expensive - according to some reports, the cost of such software is usually from $0.25 to $2 for one mailing address from the provider. Thus, for a medium-sized Internet company, protection will cost several thousand dollars.

Western providers also use harsh measures against spammers. For example, in November last year, the Scandinavian operator TeliaSonera began disconnecting spam users from the network. Recently, a major US cable Internet provider, Comcast, threatened its users with shutdowns if they didn't clean their computers of spamming Trojans that had landed on them. At the same time, the American provider has already tried to deal with such a scourge, but attempts to install a spam filter led to a massive blocking of letters sent to Russia, and the practice had to be canceled.

Legislative measures

It should be noted that in the West the fight against spam has reached the legislative level only recently. The European Union has adopted a corresponding law, which must be adapted to the legislation of its member countries - it is clear that this process is quite lengthy.

In the United States, on January 1, 2004, the federal "anti-spam" law came into force, as well as relevant state laws. Interestingly, each state has a different degree of liability for spam. The state of Maryland passed a law providing for penalties of up to five years in prison for sending spam using fraudulent methods.

There are also shifts in Russia. There are already several associations that aim to legally counter spam. For example, in 2003, the "National Coalition against Spam" was organized, founded by the Russian representative office of Microsoft, Rambler, Mail.ru companies. Golden Telecom, Subscribe.ru, Kaspersky Lab and Ashmanov & Partners (now the Coalition includes 15 companies).

The creators of the coalition intend to develop common technological tools and standards for filtering unsolicited letters, develop standards for legitimate advertising mailings and prepare a legislative framework to combat this phenomenon.

However, the struggle between spammers and anti-spam filters is becoming more and more like a confrontation between viruses and antiviruses, which will probably never stop. Spammers come up with more and more tricks. Even in the seemingly reliable system of filtering by the content of the letter, spammers have found many loopholes. In Runet, it all started with the use of similar letters. If you do not see the difference between the Latin letter T and the Russian T, then for spam filters these are two completely different characters. Many loopholes leave HTML letters. You can, for example, insert letters of the same color as the background between the letters of the text. You will see only a small gap in this place, and the filter program does not recognize this word.

According to network security companies, email users themselves support spammers: about a third of email owners have followed links in spam messages, despite the fact that clicking on such a link is a direct and, therefore, the shortest way to infect a computer with possible viruses and trojans that steal personal information.

In addition, the answer to a spam letter is a guarantee that the next time the box will be "bombed" a hundred times more actively. About 10% of people bought products advertised in such a dishonorable way. And this means that spammers are not working in vain. Anti-spam filters are only a partial solution to the problem, experts say. The main problem is in the users themselves.

Conclusions and tasks to be solved:

- The problem of spam today is serious for the Internet community and is associated with significant costs to eliminate it. Considering that all measures taken to combat spam are met with increasingly sophisticated opposition from spammers aimed at bypassing filters and other methods of combating spam, the problem of spam will be relevant for a long time to come.

- The proposed method for processing (utilizing or processing) spam does not involve fighting spam, which often resembles a battle with windmills, but extracting useful information for the end user from an array of digital data, including spam, intended for deletion.

- The entire volume of spam can be systematized and useful information can be extracted from there, in accordance with the desire of the consumer, in a variety of ways: by the subject of information; by the amount of information; by possible interested audience; by the time of arrival of spam; by addressees, etc.

- The conclusions that follow from such an analysis may be of interest to both the consumer of spam (both the management of the organization receiving spam and individual individuals) and the sender of spam (for example, in comparing the topics of messages sent by other senders).

- The information may be useful to corporate users, Internet service providers and other participants in the digital market. The consequence of this will be savings on the costs of destroying spam, and in the future - with the further development of the "Spam Disposal" method - exceeding these costs and making a profit by the user.

- Also, spam can be conditionally divided into benign and malignant. Benign - i.e. not bearing subsequently harmful effects on the consumer. Malignant - having as its possible goal the infliction of any damage to the consumer (impact on the consumer's information resources, on the psychology of users, on the secrecy of commercial and technical information, on the moral and political views of the subscriber, etc.).

- The conclusions obtained as a result of spam analysis can later be used as a means of influencing personnel (both corporate and third-party) or for other purposes at the request of the beneficiary of the received conclusions.

Analysis of existing patents and patent applications on the topic of spam handling.

1. US Application No. US20100082749A1.

Patent name: RETROSPECTIVE SPAM FILTERING (RETROSPECTIVE SPAM FILTERING).

Document #US20100082749A1 talks about reverting mislabeled "negative" information to user-relevant "positive" information (block 0004). The point of reprocessing in the US patent is to avoid being misplaced in spam, or to "recover" information from spam when "re-processing".

Those. there is a spam rule. There is a variant when information gets into spam by mistake. The spam rules can then be changed, or the spam-not-spam scoring criteria can be changed. For this, re-processing of information is provided in order to return information to “positive”, i.e. up to date for the user.

This process is simply related to the classification of information "spam" - "not spam", and is in no way connected with the proposed system.

In the proposed technical solution, any information (both “positive” and “negative”) is “useful” information.

The essence of the spam processing system is that after any amount of processing and elimination of the “spam - not spam” type (this is “negative” information in the US patent) DO NOT DELETE the information in spam, but USE it further, extracting useful information from all spam.

In the future, useful information extracted from spam may be in demand on the "spam market".

2. US Application No. US2016028673A1

Patent name: MULTI-TIERED ANTI-SPAMMING SYSTEMS AND METHODS

This source talks about reducing the amount of spam for the user.

Also, the words “recycling” or “utilization” are never found in the text of the patent.

3. RF patent No. RU2710739C1.

This solution is aimed at improving the accuracy of spam detection.

Also, the words “recycling” or “disposal” never occur in the text of the patent.

Thus, all of the listed patent documents contain a description of technical solutions aimed at improving the quality of the spam detection process, and not at working with information that is finally defined as spam.

Accordingly, no analogues of the proposed technical solution, that is, technical solutions of the same purpose, have been identified in the prior art.

At the same time, the technical result of the claimed solution is to protect the user's computing device from spam messages while extracting data from an array of digital data classified as spam.

The essence of the proposed solution.

In the spam disposal system, it is proposed to consider "spam clusters" (or simply clusters).

It should be noted that all considered patent documents provide for the operation with one message.

The spam disposal system uses the entire set of messages of a particular user, i.e. it's a job with multiple messages. Each message will be processed, but as an end result, multiple message clusters are of interest.

The result of this work will be a grouping (cluster, union) of messages, on the basis of which statistics and not only statistics will be built.

Perhaps these groupings (or associations) are simply sold, or used in other ways of "recycling".

Those. if we take all messages from one domain (at a specific address), we get a new object, new information - a “spam cluster” or just a “cluster”.

"Clusters" can be formed according to any characteristics, parameters: by time, by topics, addresses, territories, etc. (the same “specificity” of spam information processing modules mentioned above).

In this case, the message itself is not interesting. It's interesting to get a "cluster" which is new information, adding a list of messages (if needed).

Thus, the "cluster" is a new material object, which will be represented as a new set of database records stored on the system server.

"Clusters" can intersect with each other.

The material part (system) for implementing the method consists of a long-term information storage, presented in the form of a standard one (or more) server, containing at least two databases: a database of raw information, and a database of processed results, presented in the form of "spam clusters ".

The system also includes an information processing server with a set of spam information processing modules. Each module has access to the database of raw information, receives data from the corresponding tables of this database, and, based on the algorithm specific to this module, creates a “spam cluster”, which is stored by software in the database of processing results as a “product” in the form new records in the corresponding tables of this database.

Another part of the system is the server providing processing results to clients. This server is in the form of an application server for obtaining a "product" by customers through requests sent over the Internet, or as an email server for providing customers with a "product" by sending e-mail messages.

Implementation of a spam disposal system.

The general principle of the implementation of the method is shown in Fig. 1.

Incoming spam 1 (after all processing, marked for deletion) is perceived not as spam, not as information, but as an object with potential commercial value.

This object can be purchased from users, users can transfer it by agreement. And this object, processed by the proposed system (recycling process - 2), acquires new consumer properties (clusters 3).

According to the applicability of the method of extracting information from spam, the main hypotheses are as follows:

- grouping by time - statistics for corporate users;

- grouping by territories - marketing information for firms conducting marketing research, possibly for manufacturers of certain groups of goods;

- grouping by all possible signs - spammers and companies engaged in the fight against spam;

- reducing the costs of corporate users who spend resources to destroy spam (buy spam from them or negotiate the transmission of spam, eventually giving out the information the user needs, maybe even by sections, based on their user requests).

- when receiving spam from numerous different clients, it is possible to form “clusters” for several clients. As a result, analytics (statistics) can be formed, which may be of interest not only for a particular client. Such statistics, for example, can be grouped by types of customers (manufacturing, service companies, educational institutions, etc.)

Figures 2 and 3 show a method for extracting information from spam using email spam as an example. A similar scheme can be applied to spam of all types of messages, where spam is pre-filtered and stored separately from other messages.

Figure 2 shows the first part of the system (and the first stage of the implementation of the method, respectively), which consists in receiving spam messages from customers and storing these messages in raw (unchanged) form in the database. Storing raw messages serves at least two purposes:

- the ability to restore one or more messages at the request of the user (which can be considered as one of the final options for using the service);

- the ability to process messages with new algorithms that may appear after receiving a message to the service.

The system includes a module (system) for connecting to email boxes and receiving messages from a spam folder (1.1) associated with one or more email service providers (1.2) that have access to spam storages of end customers (1.3), an anti-virus protection unit (1.4), a database with information about infected messages (1.5 and 2.1), a database of unprocessed email messages (1.6 and 2.2), message processing modules (2.3), a database of processed information (2.4), a module (system) for providing processed information (2.5).

The system for connecting to clients' e-mail boxes and receiving messages from a spam folder (1.1) with a specified frequency (for example, once every 24 hours) connects to one or several e-mail service providers (1.2). Through the provider functionality, the system receives all e-mail messages from the spam storage of end customers (1.3). After that, all received messages go through the stage of anti-virus scanning (1.4) to prevent infection of the service. The antivirus check functionality is a third party product. As a result of an anti-virus scan, messages identified as infected with a virus are not saved anywhere. The maximum possible safely received amount of information about infected messages is stored in a separate service database (1.5). Copies of messages defined as safe after anti-virus scanning are stored in the service database (1.6) (without destroying messages from client mailboxes).

At the second stage (Fig. 3), one or more message processing modules (2.3) are obtained at specified intervals from the Database with information about infected messages (2.1) (represented in Fig. 2 under the number (1.5)) and the Database of unprocessed messages (2.2) (represented in Fig. 2 under the number (1.6)) previously unprocessed by this Module messages, and process each of the messages according to the algorithm embedded in the Module in order to obtain information specific to each Module. During processing, both the text of the message and any service information of the message can be processed. New information obtained as a result of processing is stored in the Processed Information Database (2.4). New Modules with new information processing algorithms may eventually be added to the service as it develops.

The system for providing summary information (2.5) serves to provide the end users of the service with the results of processing in the required form (graphs, pivot tables, reports, etc.).

The block diagram described above can be illustrated by the following example.

Spam message data is received by the service (spam disposal system) via an external program interface in the form of a typed object representing a message with a set of all message properties. The functionality of information delivery to the service is a program (or various programs) that are not part of the system (service). The condition for the operation of this program (programs) is compatibility with the service access interface. Such program(s) may be created by third party developers.

The message passing interface to the service implies that the data is transmitted in a standard format, described, in particular, in the "Properties" document.

Further operation of the system can be represented by the following examples.

There is a company with the email domain "company.com".

An employee of the company who is assigned a corporate e-mail address "engineer1@company.com" receives a message from the domain "good_rest.rest" with the heading "Summer holiday offer at European resorts".

Accordingly, the object that delivers the data of such a message to the system will have the following properties:

From: mail_sender@ good_rest.rest

Subject: Summer vacation offer in European resorts

To: engineer1@company.com

... other properties not relevant to this example.

This object by means of the service responsible for saving data in the raw information database (1.6 (2.2)) will be saved as a record in the table of this database in such a way that the value of each of the object's properties (From, Subject, To, etc.) will be saved in a separate field (column) of the table.

Further, the service installed on the server (for the Windows system, represented as a Windows Service) is an application that is automatically (if configured) executed by the system when the Windows operating system starts and runs regardless of the user's status. It has common features with the concept of demons in Unix) runs a set of programs (processing modules) installed on this server according to a schedule.

Suppose that there is only one processing module 2.3, focused on processing the data of the sender, recipient and checking the message header for violation of the law.

This module, after starting, receives from the database of raw information 1.6 (2.2) records with information of messages that have not yet been processed by this module.

Module 2.3 extracts the values of the From, To, and Subject properties from the message object. The module checks the message header (the value of the Subject property) for the presence of illegal words in it.

The result of this check will be represented by a boolean value: either "yes" (violating) or "no" (not violating). In this example, the result will be "no".

Next, the module checks whether the database contains cluster processing results for the sender “mail_sender@ good_rest.rest”, the recipient “engineer1@company.com” and the cluster of messages that do not violate the law.

If at least one of these clusters does not exist in the processing results database, then the module creates a corresponding cluster in the processing results database 2.4.

Next, module 2.3 adds to each cluster information from the processed message specific to this module.

The module then records in the raw information database that the given message was successfully processed by the given module.

If exactly the same message was sent from the same sender to an employee of the company with the e-mail address controller2@company.com, then the processing module 2.3 described in the example should check the data of this message and add the data of this message to the sender's cluster (since we consider that such the cluster was created when processing the previous example), the legal cluster (because we assume that such a cluster was created when processing the previous example), and create a cluster for the recipient “controller2@company.com” (if such a cluster was not created earlier).

Claims (8)

A method for extracting information from spam, comprising the steps of: receive messages from spam folders; storing unprocessed messages in at least one database of unprocessed messages; by means of at least one spam processing module, parsing the text and service information of each spam message in the database of raw messages in accordance with a given spam clustering criterion by a given criterion; extracting useful information from the formed cluster; performing an anti-virus check to prevent infection by means of an anti-virus protection unit and a database with information about infected messages; storing the processed information obtained as a result of the above processing in a processed information database, where the processed information database is associated with at least one of said spam processing module; and provide the user with the processing results by means of a processed information providing module associated with at least one said database of processed information.

Images