CN1240964A - Publish and subscribe data processing apparatus, method and computer program product - Google Patents

Abstract

A publish/subscribe data processing proxy device having: means for receiving from a publishing application a data message published by a publishing application in a stream defining a topic; and for distributing the received published data message to a subscribing application means for which the subscribing application has requested to receive messages in a stream in which a publish message has been published, the means for distributing comprises a plurality of connected distributing proxy data processing units; one of the streams is distributing proxy data for publishing The messages published by the processing unit are reserved and ready to be received by the subscription distribution agent data processing unit located adjacent to the publish distribution agent data processing unit.

Description

Publish/subscribe data processing apparatus, method and computer program product

The present invention relates to the field of data processing, and more particularly to data processing for distributing messages from data message suppliers (hereinafter "publishers") to consumers of such messages (hereinafter "subscribers").

Publish/subscribe data processing systems have grown in popularity in recent years as a means of distributing data messages from a publishing computer to subscriber computers. The growing popularity of the Internet, which connects all kinds of computers around the world, has helped to make this type of publish/subscribe system even more popular. The Internet's web browser application (the term "application" or "process" refers to a software program, or a portion thereof, that runs on a computer) can be used with publishers or subscribers to graphically display messages. Such systems are particularly useful when the data provided by publishers changes frequently and a large number of subscribers require rapid updates with the latest data. Perhaps the best example of its usefulness is the distribution of stock transaction data.

In this type of system, the data message publishing application does not have to know the identity or location of the subscribing application to receive the message. Publishers need only connect to a pub/sub distribution agent process, included in a set of such processes used to make up a broker system (hereinafter referred to as brokers), which sends messages to the distribution agent process and specifies the sending The subject of the message to the dispatch agent process. The distribution broker process then distributes the published messages to subscribing applications that have previously indicated to the broker that they wish to receive data messages of a defined topic. Therefore, the subscriber also does not have to know the identity and location of the publisher. Subscribers need only connect to the distribution agent process.

One such publish/subscribe system in use today is shown in Figure 1. Publishers 11 and 12 are connected to publish/subscribe broker 2 and send published messages to broker 2 which distributes the messages to subscribers 31,32,33,34. Publishers 11 and 12, which are data processing applications for outgoing data messages, connect to broker 2 using a well-known inter-application data connection protocol known as Remote Procedure Call (or RPC). Each publishing application can run on a separate machine, or alternatively, a single machine can execute multiple publishing applications. The agent program 2 consists of a plurality of distribution agents (21 to 27) connected in a hierarchical manner which will be referred to below as a "tree structure". These distribution agents, which may each run on a separate machine, are data processing applications that distribute data messages from publishers to subscribers through the agent program 2 . Subscribing applications 31, 32, 33 and 34 are connected to broker 2 via RPC to receive published messages.

Publishers 11 and 12 are first connected directly by RPC to a root distribution agent 21, which in turn is connected by RPC to second-level distribution agents 22 and 23, which in turn are connected by RPC to third-level distribution agents 24, 25, 26 and 27 (also known as "leaf allocation agents" because they are the last allocation agents in the tree structure). Each allocation agent can run on its own machine, or alternatively, an array of allocation agents can run on one machine. The leaf distribution agent connects via RPC to the subscribing applications 31 to 34, each of which can run on its own machine.

In order for Broker 2 to determine which Publish message should go to which Subscriber, the Publisher provides the Root Distribution Broker 21 with the distribution stream name for each Publish message. A distribution stream (hereinafter "stream") is an ordered sequence of messages with a name (eg "stock" for a stock trade quote stream name) that is used to distinguish one stream from the other. Similarly, subscribers provide the leaf distribution agents 31 to 34 with the names of the streams they wish to subscribe to. In this way, Broker 2 keeps track of which subscribers are interested in which streams, so when publishers publish messages to these streams, these messages can be distributed to the corresponding subscribers. Subscribers are also allowed to provide filter representations to brokers to limit messages received in specific streams (for example, a subscriber 31 interested only in IBM stock quotes can subscribe to the "stock" stream by issuing an RPC call to the leaf distribution agent 24 and Included is a filter indicating that only IBM stock-related messages in the "stock" stream are required to be sent to the subscriber 31).

It is often desirable for a dispatch agent to notify its "children" (ie, those dispatch agents directly beneath it) of certain events so that the children can take appropriate action. For example, if the wake-up agent 22 loses its connection to its parent (the root distribution agent 21), the distribution agent 22 should notify its child distribution agents 24 and 25 so that these children can take appropriate actions that have been pre-planned by the hypervisor (For example distribution agents 24 and 25 may either try to connect with other parents, such as distribution agent 23, or simply stay with distribution agent 22, waiting for distribution agent 22 to regain its connection to root distribution agent 21).

In prior art publish/subscribe brokers, this type of communication between a parent distribution broker and its children requires that the parent should send a special command outside of the normal publish/subscribe message flow. This increases the variety of communications passed between distribution agents. Furthermore, a child may receive messages in a different order than one of its siblings, since the father must send different commands to each child. Thus, the prior art in this field results in an inefficient use of available resources and a generally unsatisfactory architectural approach.

According to one aspect, the present invention provides a publish/subscribe data processing proxy device having: a means for receiving from the publishing application a data message in a stream of defined topics published by the publishing application; A means for distributing received published data messages to a subscribing application that has requested to receive messages in a stream that has published published messages, the means for distributing comprises a plurality of connected distribution agent data processing units; one of the streams is Reserved for messages published by a Publishing Distribution Agent data processing unit and intended to be received by a Subscribing Distribution Agent data processing unit located adjacent to a Publishing Distribution Agent data processing unit, or to be received by a Subscribing application located adjacent to a Publishing Distribution Agent data processing unit Program received.

According to a second aspect, the present invention provides a publish/subscribe data processing agent device having: a means for receiving from the publishing application a data message in a stream of defined topics published by the publishing application; A means for distributing a published data message to a subscribing application that has requested to receive a message in the stream that has published the published message, the means for distributing includes a plurality of The distribution agent data processing unit of the ordered hierarchy including the top level closest to the publishing application and the bottom level closest to the subscribing application; one of the streams is reserved for messages published by the publishing distribution agent data processing unit located one level up in the hierarchy and is intended to be received by a Subscription Distribution Agent data processing unit located on an adjacent layer in the hierarchy, or by a subscribing application when a Published Distribution Agent data processing unit is located on a lower layer in the hierarchy.

Upon receipt of a published message in the retained stream, the subscription distribution agent data processing unit or subscription application preferably takes a predetermined action based on the content of the received published message. In addition, at least one of the publishing application or the subscribing application preferably utilizes an Internet web browser to enable the sending or receiving of published messages over the Internet.

The present invention reuses the existing streaming architecture to enable communication between adjacent distribution agents (and between distribution agents and their directly connected subscribers). It is therefore not necessary to use another communication type for this type of communication, thus resulting in an efficient use of available system resources.

Furthermore, data messages from e.g. a parent allocating agent to its children are always delivered in order in the stream, thereby guaranteeing that each such child receives all such messages in exactly the same order, thus serving to provide high stability and predictability.

According to third and fourth aspects, the present invention provides a data processing method having method steps corresponding to each component of the data processing apparatus of the first and second aspects of the present invention, respectively.

According to the fifth and sixth aspects, the present invention provides a computer-readable storage medium having stored thereon a computer program which, when run on a computer, performs the data processing of the third and fourth aspects of the present invention, respectively. method function.

The invention will be better understood by reference to the detailed description of the preferred embodiment described in conjunction with the accompanying drawings, in which:

Fig. 1 shows the architecture of the publish/subscribe system related to the preferred embodiment of the present invention.

Figure 2 is a flowchart showing the steps taken by a parent distribution agent according to a preferred embodiment of the present invention;

Figure 3 is a flowchart showing steps taken by a sub-allocation agent according to a preferred embodiment of the present invention; and

FIG. 4 shows an alternative version of the broker architecture of FIG. 1 after performing the steps in the flowcharts of FIGS. 2 and 3. FIG.

The publishing application 11 running on the computer in FIG. 1 is, for example, a provider of live stock trading data quotes. That is, the publishing application 11 provides minute-by-minute news reporting the current value of stock prices. In this example, the publishing application 11 is publishing messages in a stream called "stock" that has already been configured in the broker 2. As is well known, when a publisher 11 wishes to publish a stock quote message to the "stock" stream, the publisher 11 issues an RPC call to the root dispatch agent 21 at the top of the agent tree structure. In this example, the subscription application 32 running on another computer has sent a subscription request via RPC call to the leaf distribution agent 24 at the bottom of the tree structure, indicating that the subscriber 32 wishes to subscribe to the "stock" stream.

Thus, anytime a publisher 11 publishes a data message to the "stock" stream, the distribution tree structure of the broker 2 sends the message down through the root distribution agent 21, through any intermediate distribution agents (such as 22 in the example of Figure 1) And to the subscriber 32 through the leaf distribution agent 24. This involves a series of RPC calls (ie 11 to 21, 21 to 22, 22 to 24 and 24 to 32) that occur between each successive circle connecting the publisher 11 and the subscriber 32 in Figure 1 .

The operation of a prior art publish/subscribe broker has been described so far. Preferred embodiments of the present invention are now described.

If the distribution agent 22 loses its connection with its parent (i.e. the root distribution agent 21), then this means that the distribution agent 24, which is a child of the distribution agent 22, before the distribution agent 22 regains its connection with the root distribution agent 21 and 25 will not be able to receive any further data messages. Of course, subscribers 31 and 32 connected to distribution agent 24 (and any subscribers connected to distribution agent 25) will also not receive any data messages. The distribution agent 22 must notify its children 24 and 25 of the lost connection so that these children can take appropriate action (eg connect to a sibling of the distribution agent 22, such as the distribution agent 23). The details of what appropriate action a child should take should be pre-specified by the hypervisor as a pre-configured policy.

2 is a flowchart illustrating the operational steps taken by distribution agent 22 in accordance with the preferred embodiment of the present invention.

In step 201, the distribution agent 22 detects that it has lost its connection to its parent (the root distribution agent 21). In step 202, the distribution agent 22 publishes a message in a stream called the "message stream" reserved for the transfer of information between the parent distribution agent and its children. Only Allocation Agents can be used as Publishers in this stream, a security check is made to determine if a Publisher is an Allocation Agent before allowing a Publisher to publish in this stream. Whenever a new child (whether it is a distribution agent or a subscriber) connects to a distribution agent (which is the child's parent), the child subscribes to the "feed" stream so that the child can easily distribute from its parent The agent receives the message. The distribution agent 22 notifies the children 24 and 25 of the agent 22 by issuing a message in step 202, telling them that the agent 22 has lost its connection with the agent 21.

3 is a flowchart illustrating the operational steps taken by child distribution agents 24 and 25 upon receipt of the message issued by parent distribution agent 22 in step 202 in accordance with the preferred embodiment of the present invention.

At step 301, the child distribution agents 24 and 25 receive messages published by the parent distribution agent 22. This is done by the parent distribution agent 22 initiating an RPC call to each of the child distribution agents 24 and 25 . When child Allocation Agents first join the tree as children of Allocation Agent 22, they are already subscribed to the "Information Stream" stream. In step 302, sub-allocation agents 24 and 25 follow pre-configured policies set by the hypervisor to notify agents 24 and 25 that when they receive a message from their father in the "info" stream telling them that the father has What they should do when they lose connection to their parent. In this example, the pre-configured policy is that in this case the sub-allocation agent should connect to another alternate parent 23 which is a brother of the original parent 22 which has lost connection to its parent 21 . Thus, agents 24 and 25 issue RPC calls to agent 23 (which in turn issues an RPC call to root distribution agent 21), requesting that publish messages that would have been sent to agents 24 and 25 via agent 22 should now be sent via agent 23. Figure 4 shows a modified version of the tree structure of the Publish/Sub Broker 2 illustrating this change.

It should be noted that another generation policy followed by the child after step 301 is to simply wait until the parent broker 22 regains its connection to the root broker 21. However, a consequence of this policy is that subscribers 31 and 32 experience a delay in receiving messages when the connection is lost. When the connection is re-established, the messages published during the lost connection can be retrieved from persistent storage (maintained by the root broker 21) and sent to the subscribers 31 and 32 via the brokers 22 and 24.

In the example above where a parent communicates with its children using the "message flow" flow, a parent sends a message to each of its children. As with other flows in the distribution tree, the parent can also choose which of its children to send a particular message to in the "Information Flow" flow, based on the nature of the message being sent. For example, another use of "Information Stream" is when a parent distribution agent wishes to notify Subscriber 31 (which is a child of Agent 24) that Subscriber 31 has lost its authorization to receive messages in a stream (such as the "Stock" stream). For example, this could be a company that has been restructured and has changed the subscribers who can view certain data. Subscriber 32, on the other hand, has not lost its authorization to receive messages in this stream. Therefore, the father 24 publishes a message via RPC only to the subscriber 31 in the "Information Stream" stream, the content of which is to inform the subscriber 31 that the subscriber 31 is no longer authorized to receive messages in the stream (eg "Stock" stream). In this case, the subscriber need not take any special response to this message. Subscriber 31 is however clearly informed that it should not expect any more news in the "stock" stream.

Another use of the "information flow" flow is to load balance the situation, assuming that the leaf distribution agent 24 is busy sending messages to one of its children (subscriber 31), ie the message load to the subscriber 31 is heavy. The parent distribution agent 24 may publish a message in the "info" stream telling its other children (subscribers 32) to temporarily connect the distribution agent 25 to the subscriber 32 during periods of heavy communication. Subscriber 32 then connects to proxy 25 by issuing an RPC call to proxy 25 . At a later time, if the proxy 25 becomes overloaded with too much load, the proxy 25 can publish a message in the "info" stream to reconnect to its original parent 24 via the subscriber 32 . In response to this publish message, the subscriber 32 then reconnects to its original parent 24 through an RPC call to the original parent 24 .

Although a hierarchical tree structure of distribution agent processes is described in the preferred embodiment for purposes of illustration, many other architectures are also within the scope of the present invention. The invention is also applicable to a set of cooperating processes spanning different types of networks (including local and wide area networks) and different types of machines and using different inter-process communication protocols (such as RPC, sockets, TCP/IP or any other well-known inter-process communication protocol) group environment. For example, the invention may also be used in the context of the Internet, which involves many different types of networks connecting different types of machines. In embodiments that do not use a tree structure, "message flow" is used not only to pass information between parent and child processes, but more generally between adjacent processes (that is, processes that are next to each other, and using the well-known Interprocess communication protocols communicate directly with each other without going through intermediate processes).

Claims (8)

1. a publish/subscribe data processing agent equipment comprises:

Be used for from issuing the device that application program receives the data-message of being issued at the stream that limits theme by the issue application program; And

The distributing data distribution of messages that is used for receiving is to the device of subscribing to application program, and this subscription application program once asked to receive the message of once issuing therein in the stream that gives out information, and is used for the assignment agent data processing unit that assigned unit comprises a plurality of connections;

It is that the serve as reasons message of issue assignment agent data processing unit issue keeps that a stream is wherein arranged, and prepare to receive by the subscription assignment agent data processing unit that is positioned at the contiguous place of issue assignment agent data processing unit, perhaps prepare to receive by the subscription application program that is positioned at the contiguous place of issue assignment agent data processing unit.

2. a publish/subscribe data processing agent equipment comprises:

Be used for from issuing the device that application program receives the data-message of being issued at the stream that limits theme by the issue application program; And

The distributing data distribution of messages that is used for receiving is to the device of subscribing to application program, this subscription application program once asked to receive the message of once issuing therein in the stream that gives out information, this is used for assigned unit and comprises a plurality of assignment agent data processing units that are arranged in the hierarchical structure that comprises a plurality of orderly levels, and this orderly level comprises the top layer of the most close issue application program and the bottom of the most close subscription application program;

One of them stream is to keep for the message that is arranged in the issue assignment agent data processing unit issue on level one deck, and prepare to receive by the subscription assignment agent data processing unit that is arranged on the level adjacent layer, maybe when being arranged on the level bottom, prepares issue assignment agent data processing unit to receive by subscribing to application program.

3. the equipment of claim 1 wherein after the message of issuing in receiving the stream of reservation, is subscribed to the assignment agent data processing unit or is subscribed to application program and take predetermined action according to the content of receiving that gives out information.

4. the equipment of claim 1, wherein at least one issue application program or subscribe to the application program utilization and sending on the Internet or receiving so that will give out information because of special World Wide Web browser.

5. a publish/subscribe data processing Proxy Method may further comprise the steps:

Limiting the data-message of issuing in the stream of theme from the reception of issue application program by the issue application program; And

To subscribing to application program, this subscription application program once asked to receive the message of once issuing therein in the stream that gives out information, and was used for the assignment agent data processing unit that assigned unit comprises a plurality of connections with the distributing data distribution of messages received;

It is that the serve as reasons message of issue assignment agent data processing unit issue keeps that a stream is wherein arranged, and prepare to receive, or prepare to receive by the subscription application program that is positioned at the contiguous place of issue assignment agent data processing unit by the subscription assignment agent data processing unit that is positioned at the contiguous place of issue assignment agent data processing unit.

6. a publish/subscribe data processing Proxy Method may further comprise the steps:

Limiting the data-message of issuing in the stream of theme from the reception of issue application program by the issue application program; And

With the distributing data distribution of messages received to subscribing to application program, this subscription application program once asked to receive the message of once issuing therein in the stream that gives out information, be used for assigned unit and comprise a plurality of assignment agent data processing units that are arranged in the hierarchical structure that comprises a plurality of orderly levels, this orderly level comprises the top layer of the most close issue application program and the bottom of the most close subscription application program;

One of them stream is to keep for the message that is arranged in the issue assignment agent data processing unit issue on level one deck, and prepare to receive by the subscription assignment agent data processing unit that is arranged on the level adjacent layer, maybe when being arranged on the level bottom, prepares issue assignment agent data processing unit to receive by subscribing to application program.

7. computer program that is stored on the computer-readable storage medium, when moving on computers, this computer program is carried out the publish/subscribe data processing Proxy Method that may further comprise the steps:

Limiting the data-message of issuing in the stream of theme from the reception of issue application program by the issue application program; And

To subscribing to application program, this subscription application program once asked to receive the message of once issuing therein in the stream that gives out information, and was used for the assignment agent data processing unit that assigned unit comprises a plurality of connections with the distributing data distribution of messages received;

One of them stream is that the serve as reasons message of issue assignment agent data processing unit issue keeps, and prepare to receive, or prepare to receive by the subscription application program that is positioned at the contiguous place of issue assignment agent data processing unit by the subscription assignment agent data processing unit that is positioned at the contiguous place of issue assignment agent data processing unit.

8. computer program that is stored on the computer-readable storage medium, when moving on computers, this computer program is carried out the publish/subscribe data processing Proxy Method that may further comprise the steps:

Limiting the data-message of issuing in the stream of theme from the reception of issue application program by the issue application program; And

With the distributing data distribution of messages received to subscribing to application program, this subscription application program once asked to receive the message of once issuing therein in the stream that gives out information, be used for assigned unit and comprise a plurality of assignment agent data processing units that are arranged in the hierarchical structure that comprises a plurality of orderly levels, this orderly level comprises the top layer of the most close issue application program and the bottom of the most close subscription application program;

One of them stream is to keep for the message that is arranged in the issue assignment agent data processing unit issue on level one deck, and prepare to receive by the subscription assignment agent data processing unit that is arranged on the level adjacent layer, maybe when being arranged on the level bottom, prepares issue assignment agent data processing unit to receive by subscribing to application program.

Images