Contact
Sounkalo DEMBÉLÉ
Objectives
The framework of this research is instrumentation at the small scales, more particularly characterization of small objects by means of robotics and scanning electron microscopy (SEM). Indeed, SEM, particularly the one of ROBOTEX platform (Carl Zeiss Auriga) provides clean environment of manipulation and high resolution images from which accurate informations can be retrieved, notably motion and structure. These informations are required for many applications including robot calibration and control, and object characterization. But SEM exhibits some singularities, particularly at high magnification, that make it use challenging: charging of samples, drift of images, depth ambiguity (i.e. invariance to translation along Z axis), bas-relief ambiguity (i.e. invariance to fronto-parallel rotation by depth product), Necker ambiguity (i.e. invariance to reflexion with respect to image plane).
Our research aims to tackle the singularities of SEM and to provide robust and accurate solutions to the topics of modelling, motion, structure and control.
Modelling of SEM imaging
Geometrical modelling is an important issue of any imaging system, and particularly for SEM imaging. SEM is modelled by an affine camera because the ratio field-of-view over the working distance is very low, particularly at high magnification. This property leads to some singularities described above and makes it use challenging. Many developments have been achieved in this topic of modelling, notably autofocus and autocalibration.
Accurate and robust methods of autofocus has been developed: static and dynamic autofocus based on robust gradient descent.
Dynamic autofocus enables focused images along with the 3D position (Z coordinate) of object with respect to the objective lens of the microscope (Ultramicroscopy, PhD thesis). The following figures show two scenes with and without autofocus.
Autocalibration is a technique allowing to compute camera motion and model. In contrast with manual calibration which requires the use of a calibration object, and then, is tedious and rigid, autocalibration is performed directly on the images of working object. An autocalibration approach based on non-linear optimization with a global search approach has been developed. From a high number of SEM images acquired, the method allows full estimation of camera matrices with high accuracy for all views in the sequence (PhD thesis).
Structure
Object structure, i.e. object 3D model, recovery from images is important for object characterization, recognition and manipulation. Indeed it enables accurate and robust 3D geometry measurement and 3D motion measurement.
An accurate and practical 3D reconstruction method has been developed: from a set of SEM images the autocalibration method stated above is performed to recovery the SEM model and motion, a linear rectification and a linear triangulation are implemented to deliver a dense 3D cloud of object. The following figures show respectively the 3D model of a cutting tool obtained from 4 images and a pollen grain obtained from 15 images (PhD thesis).
Motion metrology
Motion measurement is an important issue for robot calibration and control, and path planning. Many accurate motion measurement approaches from SEM images have been developed: 2D motion using Fourier transform and template matching, 3D motion using dynamic autofocus and autocalibration (Ultramicroscopy).
ICP (Iterative Closest Point) implementation of PCL (Point Cloud Library) has also been used to recovery 3D motion between two 3D point clouds.
Dimensional metrology
Dimensional metrology is an important topics in for industry.
Cutting edge radius is a major parameter of micromachining tool, it affects the whole machining process and workpiece quality. It is then important to be able to estimate accurately its value. The method developed is based on robust 3D curve (plane and cylinder)
fitting along with a practical method for obtaining edge point cloud from auto-calibration. Both methods are implemented in C++ with open source libraries OpenCV and PCL for 2D and 3D image processing, respectively.
Control
Accurate and robust control is very important for task at small scales. Many developments have been achieved to achieve this objective.
Autofocus and 2D visual servo has been combined to control a robot and to perform its kinematic calibration accurately. This results has lead to the 3D positioning of a 100 µm pollen grain and 10 µm microsphere.
Visual servoing based on motion measurement using Fourier transform enabled the 2D positioning with an accuracy of 80 nm at x1000 magnification.
Software
A software, Pollen 3D, has been developed that combines ours results described above. Based on OpenCV, NLOpt, C++, Qt, it interactively enables 3D reconstruction of objects from a high number of SEM images.
People involved
Nadine PIAT, Valerian GUELPA
Selection of publications
Kudryavtsev Andrey, Dembele Sounkalo and Piat Nadine, "Autofocus on Moving Object in Scanning Electron Microscope", Ultramicroscopy, vol. 182, pp. 216 - 225, jan. 2017
Marturi Naresh, Tamadazte Brahim, Dembele Sounkalo and Piat Nadine, "Image-Guided Nanopositioning Scheme for SEM.", IEEE Transactions on Automation Science and Engineering, vol. 99, pp. 1-12, jun. 2016
Dembele Sounkalo, Lehmann Olivier, Medjaher Kamal, Marturi Naresh and Piat Nadine, "Combining Gradient Ascent Search and Support Vector Machines for Effective Autofocus of a Field Emission – Scanning Electron Microscope.", Journal of Microscopy, vol. 264, 1, pp. 79 - 87, oct. 2016
Kudryavtsev Andrey, Dembele Sounkalo and Piat Nadine, "Stereo-image Rectification for Dense 3D Reconstruction in Scanning Electron Microscope", MARSS , Montréal - Canada, pp. 42 - 47, jul. 2017
Kudryavtsev Andrey, Dembele Sounkalo and Piat Nadine, "Full 3D Rotation Estimation in Scanning Electron Microscope", IEEE/RSJ IROS, Vancouver Canada, pp. 1 - 6, sep. 2017