JP2018148259A - Network system, data processor, image processor, image processing method, and packet data structure - Google Patents

Abstract

Image processing is performed with less memory resources.
As a packet data structure of a packet used when acquiring and transferring image data used for image processing from a target image frame, n × m (n × m (n), which is a unit of image processing, arranged on the image frame , M is an integer of 2 or more) Among the blocks composed of pixels, a data part for storing pixel data relating to one or a plurality of blocks is provided, and the data part includes pixel data for one row or one column constituting the block. Are sequentially stored in units of blocks.
[Selection] Figure 1

Description

  The present invention relates to an image processing technique used when image processing is performed on data included in a received packet.

  Conventionally, functions related to network services provided to users have been configured with dedicated hardware such as routers and gateways. However, in recent years, NFV (Network Functions) that operates as software on a general-purpose server for the purpose of reducing equipment costs and the like. Virtualization) technology is attracting attention.

As a mechanism for flexibly providing a network service by freely combining multiple NFV applications scattered on the network, service data is transferred in packets in an appropriate order according to the functions included in the network service. SFC (Service Function Chaining) technology is being studied (see, for example, Patent Document 1).
In general, a system using SFC technology has a data processing function (SF: Service Function) that performs service processing based on an arbitrary algorithm, and a packet transfer function (SFF) that transfers a packet including service data based on header information. : Service Function Forwarder) and a control function for controlling the entire network system.

  SFC technology has been developed mainly for the purpose of virtualizing functions related to network services and enabling flexible service operation, but similar effects can be expected when applied to functions other than networks. . For example, consider virtualizing an image processing function and classifying images on a network. Originally, the user needs to classify images in which the target object is captured after acquiring all the images of interest. However, by classifying images on the network, the user can receive only necessary images, and the processing at the terminal can be reduced.

JP 2016-46736 A

"FPGA Utilization Circuit & Sample Description Collection (3)-Video Signal Processing Circuit", [online], March 25, 2009, CQ Publishing, [Search on December 21, 2016], Internet <URL: http: //www.kumikomi.net/archives/2009/03/11circ3.php?page=7>

  Since such an image processing function generally has a large amount of calculation and is not suitable for software processing, hardware such as an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or a field-programmable gate array (FPGA) is used. It is desirable to calculate. Therefore, an image processing apparatus that performs image processing needs to have a hardware processing function in addition to a software processing function. In addition, since a plurality of functions that operate independently are mounted on the image processing apparatus, hardware that has limited computational resources is required to have a design that achieves both resource saving and high performance.

  In an image processing circuit used in an image processing apparatus, a frame buffer or a line buffer may be used to control data transfer between modules. For example, in Non-Patent Document 1, a line buffer is used for a preceding circuit that supplies data to a 3 × 3 filter circuit. In order to realize such a circuit, four line buffers that hold image data for one row are required. For example, assuming an image of VGA (= 640 pixels × 480 pixels) size as input data, the required line buffer size is 7680 bytes (= 640 pixels × 4 lines × 3 bytes), which is equivalent to 9 pixels at most. In order to send data, it is necessary to hold data of 2560 pixels.

  As described above, the image processing circuit often requires abundant memory resources. For this reason, there is a problem that image processing cannot be performed with a small amount of memory resources, which is not suitable for hardware implementation in which memory resource restrictions are severe.

  The present invention has been made to solve such problems, and an object thereof is to provide an image processing technique capable of performing image processing with a small amount of memory resources.

  In order to achieve such an object, a network system according to the present invention includes a data processing device for transmitting a packet storing image data, and image data from the packet received from the data processing device via a communication link. An image processing device that acquires and performs predetermined image processing, and the data processing device is an n × m (n and m are units of image processing) arranged on an image frame that is a target of image processing. Data that sequentially acquires pixel data for one row or one column constituting the block and sequentially stores it in the packet for one or a plurality of blocks among a plurality of blocks made up of an integer of 2 or more pixels. A processing unit, and the image processing apparatus includes: a buffer unit that holds pixel data for one block; Generates the pixel data for one block by sequentially acquiring the pixel data for one column and sequentially storing it in the buffer unit, and executes the image processing in units of the obtained pixel data for one block. And an image processing unit.

  In one configuration example of the network system according to the present invention, the buffer unit includes m (or n) buffers that hold pixel data for n (or m) pixels.

A data processing apparatus according to the present invention includes a data processing apparatus used in the network system.
An image processing apparatus according to the present invention includes an image processing apparatus used in the network system.

  In addition, an image processing method according to the present invention includes a data processing device that transmits a packet storing image data, and obtains image data from the packet received from the data processing device via a communication link, and performs predetermined image processing. An image processing method used in a network system including an image processing apparatus that performs n × m (unit of image processing arranged by the data processing apparatus on an image frame targeted for image processing) n and m are integers greater than or equal to 2) Among a plurality of blocks made up of pixels, pixel data for one row or one column constituting the block is sequentially obtained for each one or a plurality of blocks, and the packet is obtained. A data processing step of sequentially storing, wherein the image processing device receives pixel data for one row or one column from the received packet; Data is sequentially obtained and stored in a buffer unit that holds pixel data for one block to generate pixel data for one block, and the image processing is performed in units of the obtained pixel data for one block. And an image processing step to be executed.

  In one configuration example of the image processing method according to the present invention, the buffer unit includes m (or n) buffers that hold pixel data for n (or m) pixels.

  The packet data structure according to the present invention is a packet data structure of a packet used when acquiring and transferring image data used for image processing from a target image frame, and is arranged on the image frame. A data portion for storing pixel data relating to one or a plurality of blocks among blocks composed of n × m (n and m are integers of 2 or more) pixels, which are units of image processing; The pixel data for one row or one column that constitutes is sequentially stored in block units.

  According to the present invention, in an image processing apparatus that has received a packet, pixel data for one row or one column is sequentially obtained from the data portion of the received packet and sequentially stored in the buffer portion, thereby being equivalent to one block. Pixel data is generated, and image processing is executed using the obtained pixel data for one block as a unit. Therefore, it is sufficient that the buffer unit has memory resources for one block, and image processing can be performed with extremely small memory resources, ie, pixel data for at least one block. For this reason, image processing can be realized by hardware mounting even under severe restrictions on memory resources, and it is possible to achieve both resource saving and high performance in the image processing apparatus and the network system.

It is a block diagram which shows the structure of a network system. It is a block diagram which shows the structure of an image processing apparatus. It is explanatory drawing which shows the relationship between an image frame and a block. It is explanatory drawing which shows the packet format used with a network system. It is an explanatory view example showing the data structure of a packet. It is explanatory drawing which shows a response | compatibility with a packet data structure and image data. It is explanatory drawing which shows the image data acquisition in an image processing apparatus.

Next, an embodiment of the present invention will be described with reference to the drawings.
[Network system]
First, a network system 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of a network system.
This network system 1 is a function that provides various network services to users by linking a plurality of data processing functions distributed and arranged on a network based on SFC (Service Function Chaining) technology. have.

  The network system 1 includes a plurality of data processing devices 10 that are connected to each other via a communication link L and execute data processing constituting a network service used by a user, and SFC header information that adds SFC header information to a packet. A control device that provides a network service to a user by controlling the data processing device 10 to cooperate with each other in a predetermined order based on a service pattern indicating cooperation between the granting device 22 and data processing constituting the network service 20.

  In the data processing apparatus 10, as a main functional unit, a data processing unit (SF) 11 that executes data processing constituting a network service used by a user, and rearranges the processing order based on the header information of each packet. A packet sequence control unit (SQ) 12 and a packet transfer unit (SFF) 13 for transferring service data processed by the data processing unit 11 are provided.

  In the network system 1, SFC header information is given to the packet from the external communication network NW by the SFC header information giving device 22. In the network system 1, routing is performed by the packet transfer unit 13 of the data processing device 10 based on this SFC header information, and the data processing unit 11 of the own device is passed through the packet order control unit (SQ) 12. Alternatively, the packet is transferred to the packet transfer unit 13 or transferred to the packet transfer unit 13 of another data processing apparatus 10. The packet is calculated in one or more data processing units 11 and sent to the user terminal 21. With this mechanism, processing required by the user is performed on the network, so processing at the user terminal 21 is reduced.

  One of these data processing apparatuses 10 is an image processing apparatus 10A that performs image processing. The image processing apparatus 10A includes a data processing unit that performs predetermined image processing as the data processing unit (SF) 11, acquires image data from the received packet, performs image processing by the data processing unit 11, and obtains the obtained image. It has a function of storing data in a packet and transferring it. Therefore, the configuration of the image processing apparatus 10 </ b> A has almost the same configuration as the other data processing apparatus 10 except that the data processing unit (SF) 11 includes a data processing unit that performs predetermined image processing.

[Image processing device]
Next, an image processing apparatus 10A according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of the image processing apparatus.
The image processing apparatus 10A is a software processing unit 10S realized mainly by software executed by a CPU, a hardware processing unit 10H mainly realized by hardware, and one that performs data communication via a communication link L. Or it is comprised from the several input-output part 15, for example, is implement | achieved by combining a general purpose server and FPGA (Field-Programmable Gate Array).

  The software processing unit 10S includes, as main functional units, one or more data processing units (SF) 11S that collectively process one or more packets including service data, and an external device via the input / output unit 15. A packet transfer unit (SFF) 13S for transferring a packet received from the data processing unit 11S or another data processing device 10, and a corresponding data processing unit 11S and packet transfer unit 13S provided for each data processing unit 11S. A packet order control unit (SQ) 12S for rearranging the processing order based on header information of each packet exchanged between the packet and a transfer control unit for obtaining packet transfer destination information from outside the apparatus via the input / output unit 15 14S.

  The hardware processing unit 10H includes, as main functional units, one or more data processing units (SF) 11H that collectively process one or more packets including service data, and an input / output unit 15 A packet transfer unit (SFF) 13H that transfers a packet received from the outside to the data processing unit 11H or another data processing apparatus 10, and data of a packet input from the packet transfer unit 13H, provided for each data processing unit 11H Is transferred to the corresponding data processing unit 11H, a packet sequence control unit (SQ) 12H (packet sequence control circuit) that sequentially selects and transfers data according to the sequence of packet IDs included in the header information of these packets, and an input There is provided a transfer control unit 14H that acquires packet transfer destination information from the outside of the apparatus via the output unit 15.

In the configuration example of FIG. 2, an example in which two data processing units 11 (11S and 11H) are provided in each of the software processing unit 10S and the hardware processing unit 10H is shown. The number is not limited to this.
The transfer control unit 14 (14S, 14H) has a function of exchanging packet transfer destination information with each other. The input / output unit 15 has a function of exchanging packets with each other.

  The packet received by the input / output unit 15 is transferred to the packet transfer unit 13 (13S, 13H). The packet transfer unit 13 transfers the input packet to the input / output unit 15, the packet transfer unit 13, or the data processing unit 11 based on the SFC header information of the input packet. The transfer destination information regarding the packet is managed by the transfer control unit 14.

  When data processing is performed using a packet, the packet is transferred from the packet transfer unit 13 to the data processing unit 11 via the packet order control unit 12 (12S, 12H). At this time, the packet order control unit 12 performs control regarding the packet processing order and packet parsing.

The data calculated by the data processing unit 11 is shaped into a packet by the packet order control unit 12 and then transferred to the packet transfer unit 13.
A packet for which necessary processing has been completed in the image processing apparatus 10A is transferred to another data processing apparatus 10 via the input / output unit 15. By repeating this in each data processing device 10, processing required by the user is executed in order, and the final calculation result reaches the user terminal 21.

[Image processing]
Typical image processing executed by the image processing apparatus 10A includes area calculation processing such as spatial filtering processing and feature amount calculation processing. In the spatial filtering process, not only pixel data (pixel value) of a pixel of interest (pixel) but also pixel data of a pixel located in a certain region including surrounding pixels is used to obtain new pixel data of the pixel of interest. Arithmetic processing. The feature amount calculation process is a calculation process for obtaining a feature amount indicating a gradient or distribution related to pixel data of a pixel located in a certain region. As a feature amount calculation method, there is a known method such as a method for specifying the position of an edge or a corner of an object included in an image.

  In such area calculation processing, a pixel area to be calculated is extracted from the image frame and calculation processing is performed, and such a pixel area is generally called a block. FIG. 3 is an explanatory diagram showing the relationship between image frames and blocks.

  This image frame 30 has A blocks in the horizontal direction, B blocks in the vertical direction, and A × B blocks (blocks # 1, # 2,..., #A, # A + 1,. A × B) 31. The block 31 includes C pixels in the horizontal direction, D pixels in the vertical direction, and C × D pixels (pix # 1, # 2,..., #C, # C + 1,. C × D) 32.

  As described above, in the area calculation process, the vicinity of the pixel of interest, that is, the block 31 is in the calculation process range, and pixels away from the pixel of interest are not required for the calculation process. On the other hand, conventionally, image data used for image processing is divided and stored into a plurality of ordered packets in the order of rows or columns in a row direction or a column direction, and reaches the image processing apparatus 10A.

For this reason, when pixel data acquired from an image frame in units of one row or one column is input to the image processing circuit, it is necessary to hold pixel data of a plurality of rows / columns in order to align the pixel data necessary for the arithmetic processing. Will occur and the buffer size will increase.
For example, in the example of FIG. 3, when pixel data is acquired in units of one row from an image frame, in order to prepare pixel data belonging to one block 31, at least A × C pixel data is held for D rows. There is a need to.

  The present invention pays attention to the difference between block-unit image data required for such area calculation processing and image data stored in a packet, and a data structure for storing image data in units of such blocks. The image data is transferred using a packet having.

FIG. 4 is an explanatory diagram showing a packet format used in the network system.
The packet 40 used in the present invention uses a general Ethernet (registered trademark) II frame format, and its data portion 41 includes IP header information 42, UDP header information 43, user-defined header information 44, and a data portion. 45.

  In an OSI (Open Systems Interconnection) reference model, IP (Internet Protocol) is a main protocol in the network layer, and UDP (User Datagram Protocol) is a protocol in the transport layer. As another protocol in the transport layer, there is (Transmission Control Protocol) or the like, but UDP is often used for communication in an image / video that often requires real-time property.

  The IP header information 42 and the UDP header information 43 are header information defined by the standard, but the user-defined header information 44 is unique header information used in the application layer. In the user-defined header 43, various unique information relating to the application such as SFC header information 45 given by the SFC header assigning device 12 and a packet ID indicating the order concerning processing data such as image data stored in the packet are stored. Hold.

FIG. 5 is an explanatory diagram showing a data structure of a packet. The data part 45 of the packet 40 has a data structure as shown in FIG.
Here, an example in which the pixel data is RGB data having values of R (Red), G (Green), and B (Blue) is shown, and the data unit 45 continuously arranges pixels in the block. Has a specified data structure. Further, image data relating to a plurality of blocks can be stored in one packet as long as the data length allows, and the remaining area of the data portion 45 that is less than the block size may be filled with 0 or the like. .

  In the example of the data structure of FIG. 5, when one block is composed of 8 × 8 pixel regions and there is no overlapping region between blocks in the image frame, the pixel (0, 0) to pixel ( 7, 7) are continuously arranged, and then pixel (8, 0) to pixel (15, 7) constituting block # 2 are continuously arranged. By repeating this, pixels (72,0) to pixels (79,7) constituting the block # 10 are arranged, and the remaining area that is insufficient to arrange the next block # 11 is padded with zeros.

  On the other hand, if there are overlapping areas in the blocks in the image frame, the same pixel data is required in a plurality of blocks, so the data transfer amount may increase. This problem can be alleviated by defining the block to a size larger than the local area required for one calculation. That is, a trade-off can be generated between the calculation accuracy and the data transfer amount, and the design parameter can be obtained.

  FIG. 6 is an explanatory diagram showing the correspondence between the packet data structure and the image data. In general, pixel data for one row or one column is sequentially obtained from an image frame and stored in the data portion of the packet. In the present invention, pixel data is sequentially assigned to the block from the image frame. The data is acquired and stored in the data portion of the packet, and the pixel data for one row or one column is sequentially acquired in the block. As a result, the construction of the data part of the packet in the data processing apparatus 10 on the transmission side is somewhat more complicated than in the past, but in the image processing apparatus 10A on the reception side, the image data required for image processing is very easily obtained in units of blocks. Can be acquired.

[Operation of this embodiment]
Next, the operation of the network system 1 according to the present embodiment will be described.
Here, in FIG. 1, an arbitrary data processing device 10 transmits a packet storing image data from an image frame, and the image processing device 10A receives this packet to acquire image data, and performs image processing for each block. An example of execution will be described.

  First, as shown in FIG. 3, the data processing unit 11 of the data processing device 10 divides an image frame to be processed into a predetermined size, for example, 8 × 8 pixels, and provides a block. For each block selected in the order of processing, eight pixels for one row are acquired in order and stored in the data portion 45 of the bucket. When the remaining area of the data part 45 is less than one block, the remaining area is zero-padded and the packet is transmitted to the image processing apparatus 10A. This is repeatedly executed for all the blocks in the image frame, and if there is a next image frame, the same processing as described above is repeatedly executed.

  On the other hand, the hardware processing unit 10H of the image processing apparatus 10A receives the packet from the data processing apparatus 10 by the packet transfer unit 13H via the input / output unit 15, and transfers the packet to the packet order control unit 12H. The packet order control unit 12H controls the processing order of packets according to, for example, the order of packet IDs included in the packet from the packet transfer unit 13H, and outputs the image data stored in the packet data unit 45 to the data processing unit 11H. To do.

FIG. 7 is an explanatory diagram showing image data acquisition in the image processing apparatus.
The data processing unit 11H has a buffer 70 for holding pixel data for one row or one column for each row or column constituting the block in an image processing circuit that performs image processing.
For example, when the block is configured by a pixel area of 8 × 8 pixels, the buffer 70 holds 8 pixel data from the buffer # 0 that holds 8 pixel data of the 1st row. Up to buffer # 7, a total of 8 buffers are configured.

  When the image data 71 stored in the data part 45 of a new packet is input from the packet order control unit 12H, the data processing unit 11H has eight pixel data corresponding to one line from the top of the image data 71. Are stored in order in buffer # 0 to buffer # 7. As a result, image data for one block (block # 1) is prepared, and the data processing unit 11H executes image processing such as region calculation processing for each block.

  Therefore, when the image frame size is W × H pixels and the block size is 8 × 8 pixels, according to the prior art, a buffer for at least W × 8 pixels or H × 8 pixels is required. According to the invention, a buffer of 8 × 8 pixels corresponding to the block size may be used, and image processing can be performed with extremely small memory resources. Further, since this buffer size is not affected by the image frame size, it can correspond to various image frame sizes.

[Effects of the present embodiment]
As described above, this embodiment is a unit of image processing arranged on an image frame as a packet data structure of a packet used when acquiring and transferring image data used for image processing from a target image frame. A data portion that stores pixel data related to one or a plurality of blocks among blocks of n × m (n and m are integers greater than or equal to 2) pixels, and the data portion includes one row constituting the block or One column of pixel data is sequentially stored in units of blocks.

  Thus, in the image processing apparatus 10A that has received the packet, the pixel data for one row or one column is sequentially obtained from the data portion 45 of the received packet and sequentially stored in the buffer portion 70 to thereby store one block. Pixel data is generated, and image processing is executed using the obtained pixel data for one block as a unit.

  Therefore, the buffer unit 70 may be provided with memory resources for one block, and image processing can be performed with extremely small memory resources such as pixel data for at least one block. For this reason, image processing can be realized by hardware mounting even under severe restrictions on memory resources, and the image processing apparatus 10A and the network system 1 can achieve both resource saving and high performance.

[Extended embodiment]
The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  DESCRIPTION OF SYMBOLS 1 ... Network system, 10 ... Data processing apparatus, 10A ... Image processing apparatus, 10S ... Software processing part, 10H ... Hardware processing part, 11, 11S, 11H ... Data processing part (SF), 12, 12S, 12H ... Packet Sequence control unit (SQ), 13, 13S, 13H ... packet transfer unit (SFF), 14, 14S, 14H ... transfer control unit, 15 ... input / output unit, 20 ... control device, 21 ... user terminal, 22 ... SFC header Information adding device (CL), 30 ... image frame, 31 ... block, 32 ... pixel, 40 ... packet, 45 ... data part, 70 ... buffer part, 71 ... image data, L ... communication link.

Claims (7)

A data processing device for transmitting a packet storing image data;
An image processing device that obtains image data from the packet received from the data processing device via a communication link and performs predetermined image processing;
The data processing device includes:
For each one or a plurality of blocks among a plurality of blocks composed of n × m pixels (n and m are integers of 2 or more), which are units of the image processing, arranged on an image frame to be subjected to image processing A data processing unit for sequentially acquiring pixel data for one row or one column constituting the block and sequentially storing the pixel data in the packet;
The image processing apparatus includes:
A buffer unit for holding pixel data for one block;
Pixel data for one block is generated by sequentially obtaining pixel data for one row or one column from the received packet and sequentially storing the pixel data in the buffer unit, and the obtained pixel data for one block An image processing unit that executes the image processing in units of The network system according to claim 1,
The buffer system comprises m (or n) buffers that hold pixel data for n (or m) pixels.   A data processing apparatus used in the network system according to claim 1.   An image processing apparatus used in the network system according to claim 1. A network system comprising: a data processing device that transmits a packet storing image data; and an image processing device that obtains image data from the packet received via a communication link from the data processing device and performs predetermined image processing An image processing method used,
One of a plurality of blocks of n × m (n and m are integers of 2 or more) pixels, which are units of the image processing, arranged on the image frame to be subjected to image processing by the data processing device. Alternatively, for each of a plurality of blocks, a data processing step of sequentially obtaining pixel data for one row or one column constituting the block and sequentially storing the pixel data in the packet is provided.
The image processing apparatus sequentially acquires pixel data for one row or one column from the received packet, and sequentially stores the pixel data for one block in a buffer unit that holds pixel data for one block. An image processing method comprising: generating image data; and executing the image processing in units of the obtained pixel data for one block. The image processing method according to claim 5,
The image processing method according to claim 1, wherein the buffer unit includes m (or n) buffers that hold pixel data for n (or m) pixels. A packet data structure of a packet used when acquiring and transferring image data used for image processing from a target image frame,
A data portion for storing pixel data related to one or a plurality of blocks among blocks of n × m (n and m are integers of 2 or more) pixels, which are units of the image processing, arranged on the image frame. Prepared,
The data portion stores pixel data for one row or one column constituting the block in order in units of blocks.

Images