WO2018193359A1 - Automatic positioning method for mounting probes for scanning and optical spectroscopy in situ and device - Google Patents

Abstract

Method and device capable of automatically fixing a microscopic plasmon structure to a macroscopic support structure. The proposed method and device allow the construction of probes that can be used for scanning probe microscopy (SPM) procedures, including the coupling thereof to optical systems, generating scanning probe optical microscopy (SPOM) techniques such as scanning near-field optical microscopy (SNOM) or tip enhanced Raman spectroscopy (TERS). The proposed method and device are based on closed loop positioning control and computer viewing means, using visual feedback.

Description

 AUTOMATIC POSITIONING METHOD FOR ASSEMBLING SCAN AND SPECTROSCOPY PROBES

IN SITU OPTICAL AND DEVICE _.

[01] This patent application describes a method and device capable of automatically attaching a microscopic plasmid structure to a macroscopic support structure. The proposed method and device allow the construction of probes that can be used for Scanning Probe Microscopy (SPM) procedures, including their coupling with optical systems, generating optical scanning microscopy techniques. Scanning Probe Optical Microscopy (SPOM), such as Scanning Near-field Optical Microscopy (SNOM) or Probe Effect Raman Spectroscopy (TERS). The proposed method and device are based on computational vision and closed-loop positioning control capabilities using visual feedback.

[02] SPOM procedures, such as SNOM and TERS, are useful in precision, microelectronics and biomedical manufacturing industries, among other areas of science and technology, where the information obtained through these techniques results in improved quality and product reliability. Optical procedures using these techniques are capable of exceeding the light diffraction limit and producing atomic resolution images, as well as extracting information on morphology, electrical conductivity, hardness, and magnetic and optical properties of the materials under study. But this is possible only when the probes used have specific plasmon properties, as well as a small enough apex diameter. For this reason, probes used in TERS experiments performed in a laboratory environment, for For example, they are difficult to make with the necessary quality. In addition, the success of experiments using these probes is associated with the quality of their manufacture (Johnson, TW, Lapin, ZJ, Beams, R., Lindquist, C, Rodrigo, SG, Novotny, L, & Oh, SH (2012) Highly reproducible near-field optical imaging with sub-20 nm resolution based on template-stripped gold pyramids (ACS Nano, 6 (10), 9168-9174.).

 [03] A SPOM probe is composed of a nanometric or micrometer sized plasmid structure, which must be attached to a macroscopic support structure, which has the function of connecting the plasmid structure to the control system. This macroscopic structure can be, for example, a thin wire.

 [04] To attach a plasmonic structure to the macroscopic structure, the operator must move the macroscopic structure containing adhesive material at its end and position it in the correct orientation over the plasmonic structure located on a substrate, and keep it in this position for as long as necessary for the two structures to adhere. To guide this process, the operator uses the image provided by an optical magnifier attached to a camera.

[05] The article ^' Highly Reproducible Near-Field Optical Imaging with Sub-20-nm Resolution Based on Template-Stripped Gold Pyramids _. (JOHNSON ET AL, 2012), proposes a method of producing gold pyramidal tips for making probes without, however, presenting an automatic method for positioning and fixing the pyramids in the macroscopic structure (which is what forms the probe). .

[06] The article entitled "Computer vision for nanoscale imaging" deals with a review of the techniques employed in general purpose nanoscale computational vision, not directly related to The presented technology utilizes part of such methods and applies them for the manufacture of scanning probes and in situ optical spectroscopy (Eraldo Ribeiro and Mubarak Shah. 2006. Computer Vision for Nanoscale Imaging. Mach. Vision Appl. 17, 3 (July 2006 ), 147-162.).

[07] US patent BR10201501 12335 METHOD AND EQUIPMENT FOR AUTOMATIC POSITIONING FOR scanning probe microscopy OPTICS AND SPECTROSCOPY IN SITU of 15/05/2015 _^, using a similar methodology proposed in the present invention, but with computer vision techniques, procedures and distinct applications, aimed at positioning probes already mounted on the laser focus, to perform SPOM procedures such as SNOM and TERS.

 [08] These procedures require the presence of a specialist technician, requiring time, dexterity and training, being subject to human error, variations in the quality and duration of the procedure. These difficulties constitute limitations to the use and dissemination of SPOM techniques, and therefore barriers not only against the advancement of the techniques themselves, but also of all the innovation that may come from their use.

[09] In view of these difficulties, an automatic system was then developed for the step of fixing the plasmonic structure to the macroscopic support structure that together constitute the probe. The device of the present invention adds greater repeatability, reliability, speed and robustness to the manufacture of probes for SPOM experiments such as SNOM and TERS.

[010] The technology proposed here can be used, for example, to assemble probes with the plasmid structure proposed in the THighly Reproducible Near-Field Optical Imaging with Sub-20-nm Resolution Based on Template-Stripped Gold Pyramids _. (JOHNSON ET AL, 2012), which proposes a method of producing gold pyramidal tips for the probe making. The manufacture of this type of probe will be used to illustrate the function of the method and device proposed herein. The present application has a different scope from the article cited above, since the methods and devices described herein do not concern the production of tips themselves, but the automation of the adhesion of the prepared plasmid structure to the other components that constitute the probe.

BRIEF DESCRIPTION OF THE FIGURES

[011] FIGURE 1 ^" It presents a simplified flowchart formed by the steps related to the manipulation of the images obtained by camera, aiming to obtain the contours of the elements of interest, specifically the contours of the plasmonic structures in the substrate, the macroscopic support structure and its reflection on the substrate.

[012] FIGURE 2 - Flowchart indicating the contour manipulation steps, obtained from the actions indicated in the flowchart shown in Figure 1, of the plasmonic structures, the macroscopic support structure and its reflection. The manipulation of these contours is intended to provide their positions in image coordinates.

[013] FIGURE 3 - Flowchart with the steps of location of plasmonic structures, macroscopic structure and its reflection, aiming to provide the current positions of the elements to the software.

[014] FIGURE 4 - Flowchart showing the positioning steps of the macroscopic structure, which are performed iteratively until the end of the positioning.

[015] FIGURE 5 ^" Flowchart of actions describing the completion of one mounting procedure and the beginning of the next.

[016] FIGURE 6 ^" Shows the optical image of the desired situation at the end of positioning. The macroscopic structure (1) and its reflection (2) are close enough and in contact with the plasmonic structure (3) contained in the substrate. plasmonic structures (4). After contact between (1) and (3), it is necessary to wait for the adhesive substance to dry to remove the tip of the matrix that composes the substrate (4).

[017] FIGURE 7 ^" Displays the optical image of an example mounted probe. The elements of the probe present in the image are: (3) plasmid structure (gold probe), (5) adhesive substance (epoxy glue) and (1 to ) tungsten wire composing the macroscopic structure (1), without limitation.

 [018] FIGURE 8 - Detail of the macroscopic structure automatic exchange device (1). Formed by the elements: (6) representing the movement system in the x, y and z axes; (7) the motor responsible for the rotation of the shaft (8) containing the drum (9). At (9) a plurality of macroscopic structures (1) to be used in the process will be fixed along its perimeter. The assembly forming the macroscopic structure (1) comprises a tuning fork [1 (b)] and a tungsten wire [(1 (a)]

FIGURE 9 ^" Top view of a substrate containing plasmid structures (3) that are still in the matrix and the empty cavity [3 (a)] containing a plasmid structure that was successfully removed.

[020] FIGURE 10 ^" Presents a simplified block diagram showing the main hardware components of the system and the information flow between the equipment. (10) represents the process computer that executes the control and vision algorithms. ( 20) represents the controller that receives motion requests from the process computer and forwards them appropriately to the actuators, in addition to sending feedback of the movements actually performed. (30) represents the set of actuators responsible for moving the macroscopic structure (40) in The scene is captured by a camera connected to a long-range microscope (50) that passes the image to the process computer (10). DETAILED DESCRIPTION OF TECHNOLOGY

 [021] The present invention proposes a method and device capable of automatically attaching a microscopic plasmid structure to a macroscopic support structure. The proposed method and device allow the construction of probes that can be used for probe scanning microscopy (SPM) procedures, including their coupling with optical systems, generating probe scanning optical microscopy (SPOM) techniques such as near field optical microscopy (SNOM) or probe effect Raman spectroscopy (TERS). The proposed method and device are based on computational vision and closed-loop positioning control capabilities using visual feedback.

[022] The method is divided into three subsystems: a vision subsystem, a positioning subsystem, and a intelligence and control subsystem. As shown in the block diagram of FIG. 10, the vision module is comprised of a camera and an optical magnifier (50); The positioning subsystem consists of actuators (30) capable of three-dimensionally moving the macroscopic support (40). These actuators must communicate with drivers, which in turn communicate with an intelligence and control algorithm that resides in a process computer (10). Finally, the intelligence and control subsystem (the software) consists of computer vision, closed-loop positioning control algorithms, and modules for communicating with the sensors (camera) and actuators used in the proposed solution.

[023] The method used to automatically make a probe used for SPOM procedures is performed by the intelligence and control subsystem. The method comprises the following steps:

a) acquire images by means of a camera controlled by an image acquisition system capable of controlling cadence and resetting it during the execution of the method, time the camera activities and request the acquisition of images, in addition to counting the images;

b) recognize the substrate in an acquired image by applying a filter, scanning the image and comparing the substrate plasmonic structures found in the scan with the standard substrate plasmid structure, locating the regions in which the plasmonic structures found are more similar to the pattern, verify the color and brightness of the region previously found with the established thresholds, generate a binary image and obtain the contours of such regions, represent the contours as a corresponding geometric shape and define the structures. of interest (the bases of the plasmonic structures) of the substrate from the size, the geometric shape of the contour representation (for example the base rectangle of the pyramid mentioned in Example 1) and the location of such shapes, which will represent the representation of the structures. substrate plasmonics;

c) recognize in an image a macroscopic structure and its reflection and determine the positions of both, applying a filter, binarizing the image through Otsu's automatic method, extracting its contours and identifying contour pairs of relevant areas, verifying horizontal alignment and vertical spacing between such contours and defining the pair corresponding to the macroscopic structure and its reflection, defining the contour of the macroscopic structure containing the adhesive material, validating its position by comparing it with the expected position for the macroscopic structure relative to its reflection, use the validated position as a coordinate of the macroscopic structure and, similarly, repeat the above operations to locate the coordinates of the macroscopic structure reflex;

d) verify and update data concerning the positions of the substrate and the macroscopic structure and its reflection, submitting the images to a low-pass filter to provide greater stability to the procedure and prevent the sending of commands to actuators based on erroneous measurements. , stability is achieved by analyzing the coherence of image variations by verifying that changes are physically possible and expected before updating them;

e) control the movements of the macroscopic structure based on the calculation of the relative, horizontal (x) and vertical (Y) distances between the elements of interest (substrate and the set formed by the macroscopic structure and its reflection) that are calculated through the images obtained after step cl_; define based on the images obtained after step cl_ the directions and directions of the movements of approximation of the macroscopic structure to the region of interest of the substrate to promote the collage, define the type of movement based on the calculated distances, and the movement of the probe is It is performed by motors and can be subdivided into three scales: larger, moderate and step movements, in accordance with the increasing order of intrinsic positioning accuracy; send the resulting commands to the actuators containing all movement information and perform the steps iteratively until the necessary conditions for positioning are met (contact between macroscopic structure and substrate for bonding); f) After the end of positioning, keep the macroscopic structure in contact with the region of interest of the substrate until the adhesive material dries;

 g) move the assembly formed by the macroscopic structure adhered to the plasmonic structure (mounted probe), present on the substrate at the moment prior to the contact caused by positioning, to a region destined to the conditioning of structures whose adhesion step is finished, detach the mounted probe in such a region and position a new macroscopic structure to be fixed.

 Alternatively in steps e_ and c_, it is possible to use filters such as the mean and median filter, preferably Gaussian.

 [025] In steps ¾_ and c_, you can use segmentation methods such as pattern matching for pattern recognition or the Otsu method for image binarization, followed by contour extraction, combined with later manipulation of the patterns. same for the correct identification of the macroscopic structure and its reflex.

[026] Step do_ of the method described above can be implemented using techniques such as pattern matching (Template Matching), searching for a template in a larger image. For this, the reference model traverses the image (bidirectional cross-correlation between the image and the model), and is compared with the elements of the image. At the end of this scan, you will find the position that is most compatible with the model used. For applications where there are multiple objects to be recognized, such as in the recognition of a matrix of plasmonic structures for the production of multiple probes, some thresholding technique is used to select an array. plurality of objects that presented greater compatibility to a predefined model.

 [027] The choice of template to be used in step ¾_ can be made clinically ^ in the sense that the technician selects on screen a plasmonic structure to be used as template before the procedure starts. This gives greater flexibility as to the relative position between the optical magnifier and the substrate, the illumination of the scene and the type of plasmonic structure to be used. Another possibility is the use of a database containing several images of plasmid structures on substrates to be used as a model.

[028] The process of positioning the macroscopic support begins by detecting its positions and the plasmonic structure located in the lower left corner of the substrate by measuring the relative distances of both in the X and Y directions of the coordinate system used. (in this case, the coordinate system of the image itself). Then, an algorithm determines the procedures adopted to approach the macroscopic support to the plasmonic structure, which represents the reference of the control coordinate system.

[029] In step ID ^ if the detected position is changed abruptly, it is a probable measurement error and not a real movement. When moved by the actuators, the coordinate variation is gradual.

[030] The steps of image acquisition, detection of macroscopic support and substrate plasmon structures, emission of control signals for assembly movement occur iteratively until the process of pasting a set of macroscopic structures to a set is completed. of plasmonic structures.

Based on the updates considered in the previous step, the relative, horizontal (X axis), and vertical (Y axis) distances between the elements of interest (target substrate structures, macroscopic support and its reflection) in coordinates in the image are calculated. Based on these distances, the direction and type of movement are determined. After establishing the directions and distances of movement, the commands are sent to the actuators. Finally, it is checked whether the necessary conditions for the end of the procedure have been met.

[032] For each new image acquired by the camera, the positions of the elements of interest are updated. However, sudden changes in lighting intensity and other disturbances, such as focus change, can lead to detection and position errors. Thus, the detection algorithm could sometimes give erroneous results for subsequent modules. To avoid such a problem, in step cL a signal conditioning module, consisting of an anti-spike filter and a low pass filter, was implemented to provide greater procedural stability and to prevent commands being sent to actuators based on erroneous measurements. .

 For production of a large number of probes, the repetition of steps a through g is applied to bond a plurality of macroscopic structures to a plurality of plasmonic structures.

 An additional step after step g_ can be used to perform substrate inspection to verify that all plasmonic structures have indeed been removed for this purpose by using a reflection optical microscopy image of the substrate as seen from below. above so that it is possible to see clearly whether or not there are remaining plasmonic structures; If so, the structures corresponding to the positions at which the plasmonic structures were not successfully removed are discarded.

[035] The invention also proposes a device comprising at least one macroscopic structure (1) with at least one tuning fork [1 (b)] and at least one tungsten wire [(1 (a)], a tuning fork bracket (9), an axle (8), an engine (7), x-axis, y and z-axis (6) as shown in Figure In this embodiment, the macroscopic structure (1) closest to the substrate is detected by the positioning system for mounting the complete probe as described above.

 [036] Mounting the probe, the system changes so that another macroscopic structure (1) becomes closest to the substrate, and this procedure is repeated until all macroscopic structures (1) of the automatic exchange system have a plasmonic structure (3) adhered to its extremity, or according to other stopping criteria established by the operator. Once this procedure is completed, the substrate can be taken for inspection and sets corresponding to unsuccessful adhesion attempts are discarded.

[037] A plurality of macroscopic structures [1 (b)] (tuning fork) and [(1 (a)] (tungsten wire) attached to the support (9) can be changed throughout the process. illustrated in figure 7, a novel macroscopic structure (1) represented by step g1 can be manual or involve an automatic exchange system.

Thus there is an automatic positioning device for mounting probes for scanning and optical spectroscopy in situ, characterized in that it comprises at least one macroscopic structure (1) formed by at least one tuning fork [1 (b)], at least one tungsten wire [(1 (a)], a tuning fork bracket (9) an axle (8), a motor (7), x-axis, y and z-axis (6); 7) are motors, preferably step motors or piezoelectric systems, and the tuning fork holder (9) is preferably cylindrical in shape.

[039] The invention may be better understood by way of the non-limiting example below. Example 1 ^" Experimental Prototype Results

 [040] The prototype constructed of the scanning probe mounting system provided with the present automatic visual feedback positioning technology has the following constituent elements: HP Compaq DC 5800 Small Form Factor computer corresponding to component (10) in Figure 10; two controllers composed of two Arduino Motor Shields L293D coupled to two Arduinos Uno (20); three stepper motors SM1 .8 - A1734CMN (30); motorized long-distance microscope KC VideoMaxu Long Distance Microscope (50); Invent Vision V200e camera (50) and a ThorLabs model PT3 / M XYZ translation stage (30). In addition to the devices that correspond to the main hardware components, mechanical structures were also used to provide a favorable configuration of components such as the pyramidal point matrix (which is the incorporation of the plasmonic structures), support for the long distance microscope with the camera and a connector for connecting the macroscopic support structure to the XYZ translation stage.

 [041] Software containing the automatic visual feedback positioning method described in this application is operated through a parallel graphical user interface.

[042] The procedure was initiated by gluing (using epoxy) a piece of tungsten wire cut from a spool and fixed to a quartz tuning fork [(1) in Figure 6] that fulfills the role of macroscopic support in this experiment. . The tuning fork-wire set was then fixed to the translational stage XYZ [(30) in Figure 10]. Through the developed software, the operator is able to position the tip of the wire on a surface where epoxy has been applied. In this step, the glue was applied to the tip of the thread that will be fixed later to the gold tip (3). [043] After applying epoxy to the end of the wire, the assembly was placed in the initial position for alignment of the assembly with one end in the die. When performing initial positioning, the command was triggered by the software that initiates automatic alignment following the method documented in the detailed description of this invention, taking into consideration the following specificities: the steps ¾_ and c_ of the method were implemented using as a central technique the matching of standards, the base standard being obtained from previous manufacturing procedures; The filtering method implemented for step cl_ consists of a filter that rejects sudden changes in position that are outside a certain tolerance limit (anti-spike filtering); To control the actuator movements in step e_, two PID controllers (Integral and Derivative Proportional) were used, one for X and one for Y, and decouplers were also used to decrease the influences of one control loop over the other.

[044] Figure 6 shows the moment when the wire support assembly reaches the desired position on the tip matrix. Figure 7 shows the final result after bonding the gold tip to the end of the tungsten wire. Using the device shown in Figure 8, the macroscopic structure is replaced with one not yet containing the probe and the procedure is restarted. At the end of mounting all probes attached to the holder, the shape is taken for inspection using a microscopy image as shown in Figure 9. In this inspection, which plasmonic structures were successfully removed (3a) from the substrate are identified. and which were not ((3) in Figure 9). Properly assembled probes are stored and those corresponding to unremoved plasmonic structures are discarded.

Claims

1. Automatic positioning method for mounting probes for scanning and optical spectroscopy in situ, comprising the following steps:  a) acquire images by means of a camera controlled by an image acquisition system capable of controlling the cadence and resetting it during the execution of the method, timing the camera activities and requesting the acquisition of images, as well as counting the images; b) recognize the substrate in an acquired image by applying a filter, scanning the image, and comparing the substrate plasma structures found in the scan with the standard substrate plasmid structure, locating the regions where the plasmon structures found are more similar to the pattern, check the color and brightness of the region previously found with the established thresholds, generate a binary image and obtain the contours of these regions, represent the contours as a corresponding geometric shape and define the structures. of interest (the bases of the plasmonic structures) of the substrate from the dimension, the geometric shape of the contour representation and the location of such shapes, which will represent the representation of the substrate plasma structures; c) recognize detect in an image a macroscopic structure and its reflection and determine the positions of both, applying a filter, binarizing the image, extracting its contours and identifying pairs of contours of relevant areas, verify horizontal alignment and vertical spacing between such. contours and define the pair corresponding to the macroscopic structure and its reflection, define the contour of the macroscopic structure containing the adhesive material, validate its position by comparing it with the expected position of the macroscopic structure in relation to its reflex, use the validated position as the coordinate of the macroscopic structure and, similarly, repeat the above operations to locate the coordinates of the macroscopic structure reflex; d) verify and update data concerning the positions of the substrate and the macroscopic structure and its reflection, submitting the images to a low-pass filter to provide greater stability to the procedure and prevent the sending of commands to actuators based on erroneous measurements. , stability is achieved by analyzing the coherence of image variations by verifying that changes are physically possible and expected before updating them; e) control the movements of the macroscopic structure based on the calculation of the relative, horizontal (x) and vertical (Y) distances between the elements of interest (substrate and the set formed by the macroscopic structure and its reflection) that are calculated through the images obtained after step cl_; define based on the images obtained after step cl_ the directions and directions of the movements of approximation of the macroscopic structure to the region of interest of the substrate to promote the collage, define the type of movement based on the calculated distances, and the movement of the probe is It is performed by motors and can be subdivided into three scales: larger, moderate and step movements, in accordance with the increasing order of intrinsic positioning accuracy; send the resulting commands to the actuators containing all movement information and perform the steps iteratively until the conditions required for the positioning are achieved (contact between macroscopic structure and substrate for bonding);  f) After the end of positioning, keep the macroscopic structure in contact with the region of interest of the substrate until the adhesive material dries;  g) move the assembly formed by the macroscopic structure adhered to the plasmonic structure (mounted probe), present on the substrate at the moment prior to the contact caused by positioning, to a region destined to the conditioning of structures whose adhesion step is finished, detach the mounted probe in such a region and position a new macroscopic structure to be fixed. Method according to claim 1, steps b and c, characterized by the use of filters, such as the mean and median filter, preferably the Gaussian.  Method according to claim 1, step b_, characterized in that it uses for the recognition of an array of plasmonic structures for the production of multiple probes, a thresholding technique to select a plurality of objects that showed greater compatibility with a pre-model. -defined.  Method according to claim 1, steps b_ and c_, characterized by using segmentation methods such as pattern matching for pattern recognition or the Otsu method for image binarization.  Method according to claim 1, step d, characterized in that it uses an antispike filter and a low-pass filter. Method according to claim 1, characterized in that the steps a through g are repeated to glue a plurality of macroscopic structures to a plurality of plasmonic structures, proceeding with the acquisition of images, detection of the macroscopic support and substrate plasmon structures, emission of control signals for iterative movement of the set until the completion of the bonding process of a set of macroscopic structures to a set of plasmonic structures.  Method according to claim 1, characterized by the presence of an additional step after step g_ comprising the substrate inspection to verify that all the plasmonic structures have indeed been removed for this purpose using a microscopic image. reflection optics of the substrate seen from above, so that it is possible to see clearly whether or not there are still remaining plasmonic structures; If so, discard the structures corresponding to the positions at which the plasmonic structures were not successfully removed.  Automatic positioning device for mounting probes for scanning and optical spectroscopy in situ, comprising at least one macroscopic structure (1) formed by at least one tuning fork [1 (b)], at least one tungsten wire [1 (a)], a tuning fork bracket (9) an axle (8), a motor (7), x-axis, y and z-axis motion system (6).  Device according to Claim 5, characterized in that the element (7) is motors, preferably stepping motors or piezoelectric systems.  Device according to Claim 5, characterized in that the element (9) is preferably cylindrical in shape.