SEM-based vision

Contact

Sounkalo DEMBÉLÉ

Objectives

The framework of this research is instrumentation at the small scales, more particularly characterization of small objects by means of robotics and scanning electron microscopy (SEM). Indeed, SEM, particularly the one of ROBOTEX platform (Carl Zeiss Auriga) provides clean environment of manipulation and high resolution images from which accurate informations can be retrieved, notably motion and structure. These informations are required for many applications including robot calibration and control, and object characterization. But SEM exhibits some singularities, particularly at high magnification, that make it use challenging: charging of samples, drift of images, depth ambiguity (i.e. invariance to translation along Z axis), bas-relief ambiguity (i.e. invariance to fronto-parallel rotation by depth product), Necker ambiguity (i.e. invariance to reflexion with respect to image plane).

Our research aims to tackle the singularities of SEM and to provide robust and accurate solutions to the topics of modelling, motion, structure and control.

Modelling of SEM imaging

Geometrical modelling is an important issue of any imaging system, and particularly for SEM imaging. SEM is modelled by an affine camera because the ratio field-of-view over the working distance is very low, particularly at high magnification. This property leads to some singularities described above and makes it use challenging. Many developments have been achieved in this topic of modelling, notably autofocus and autocalibration.

Accurate and robust methods of autofocus has been developed: static and dynamic autofocus based on robust gradient descent.
Dynamic autofocus enables focused images along with the 3D position (Z coordinate) of object with respect to the objective lens of the microscope (Ultramicroscopy, PhD thesis). The following figures show two scenes with and without autofocus.

Illustration of autofocus

Autocalibration is a technique allowing to compute camera motion and model. In contrast with manual calibration which requires the use of a calibration object, and then, is tedious and rigid, autocalibration is performed directly on the images of working object. An autocalibration approach based on non-linear optimization with a global search approach has been developed. From a high number of SEM images acquired, the method allows full estimation of camera matrices with high accuracy for all views in the sequence (PhD thesis).

Motion

Motion measurement is an important issue for robot calibration and control, and path planning. Many accurate motion measurement approaches from SEM images have been developed: 2D motion using Fourier transform and template matching, 3D motion using dynamic autofocus and autocalibration (Ultramicroscopy).

3D motion by means of dynamic autofocus

ICP (Iterative Closest Point) implementation of PCL (Point Cloud Library) has also been used to recovery 3D motion between two 3D point clouds.

Structure

Object structure, i.e. object 3D model, recovery from images is important for object characterization, recognition and manipulation. Indeed it enables accurate and robust 3D geometry measurement and 3D motion measurement.

An accurate and practical 3D reconstruction method has been developed: from a set of SEM images the autocalibration method stated above is performed to recovery the SEM model and motion, a linear rectification and a linear triangulation are implemented to deliver a dense 3D cloud of object. The following figures show respectively the 3D model of a cutting tool obtained from 4 images and a pollen grain obtained from 15 images (PhD thesis).

3D model of cutting tool    3D model of Potamo pollen grain

Control

Accurate and robust control is very important for task at small scales. Many developments have been achieved to achieve this objective.

Autofocus and 2D visual servo has been combined to control a robot and to perform its kinematic calibration accurately. This results has lead to the 3D positioning of a 100 µm pollen grain and 10 µm microsphere.

Visual servoing based on motion measurement using Fourier transform enabled the 2D positioning with an accuracy of 80 nm at x1000 magnification.

Principe of visual servoing using Fourier transform         Positioning with an accuracy of 80 nm

Software

A software, Pollen 3D, has been developed that combines ours results described above. Based on OpenCV, NLOpt, C++, Qt, it interactively enables 3D reconstruction of objects from a high number of SEM images.

People involved

Nadine PIAT, Valerian GUELPA, Rahima DJAHEL

Selection of publications

Kudryavtsev Andrey, Dembele Sounkalo and Piat Nadine, "Autofocus on Moving Object in Scanning Electron Microscope", Ultramicroscopy, vol. 182, pp. 216 - 225, jan. 2017

Marturi Naresh, Tamadazte Brahim, Dembele Sounkalo and Piat Nadine, "Image-Guided Nanopositioning Scheme for SEM.", IEEE Transactions on Automation Science and Engineering, vol. 99, pp. 1-12, jun. 2016

Dembele Sounkalo, Lehmann Olivier, Medjaher Kamal, Marturi Naresh and Piat Nadine, "Combining Gradient Ascent Search and Support Vector Machines for Effective Autofocus of a Field Emission – Scanning Electron Microscope.", Journal of Microscopy, vol. 264, 1, pp. 79 - 87, oct. 2016

Kudryavtsev Andrey, Dembele Sounkalo and Piat Nadine, "Stereo-image Rectification for Dense 3D Reconstruction in Scanning Electron Microscope", MARSS ,  Montréal - Canada, pp. 42 - 47, jul. 2017

Kudryavtsev Andrey, Dembele Sounkalo and Piat Nadine, "Full 3D Rotation Estimation in Scanning Electron Microscope", IEEE/RSJ IROS, Vancouver Canada, pp. 1 - 6, sep. 2017