We report the use of ensemble learning to achieve significant improvements in the performance of diffractive optical classifiers on CIFAR-10 image dataset. We initially created a pool of 1252 diversely-trained diffractive network models; using a novel iterative pruning algorithm, we trimmed this down to an ensemble size of 14 diffractive networks to achieve a blind testing accuracy of 61.14% on CIFAR-10 image classification, which performs >16% higher in its inference accuracy compared to the average performance of the individual diffractive networks within the ensemble. These results signify a major advancement in all-optical inference and image classification capabilities of diffractive networks.

We develop a novel high‐profile application of machine learning techniques by elevating digital holography and sensing in robotics to a new level. The extraction of unknown metrics such as focusing distance and in plane positioning without full image restoration from digital holograms is performed by pre‐processing approach in space‐domain and/or in Fourier‐domain, including real‐time constraints. Measuring a single hologram, we successfully determine the axial distance of a complex object to the 10x microscope objective over a range of 100 µm with an accuracy of 1.25 µm. We apply a machine learning technique to the hologram to speed up tracking in the plane of the pseudo-periodic target position up to several tens of frames per second (fps). Such high frame rates enable real-time processing in many different application scenarios.

We present a weakly-supervised deep learning framework for human breast cancer-related optical biomarker discovery based on label-free autofluorescence multiharmonic (SLAM) microscopy. This framework consists of three stages: self-supervised consistency training for image representation learning at multiple scales; cancer region identification by weakly-supervised Multiple Instance Learning (MIL); optical biomarker discovery based on channel-wise attribution maps. Currently, the model has achieved an average AUC of 0.86 on the breast cancer global detection task. The attribution maps on different scales highlight distinct structures in SLAM which facilitate new insights into tumor micro-environment and field cancerization.

We report a virtual image refocusing framework for fluorescence microscopy, which extends the imaging depth-of-field by ~20-fold and provides improved lateral resolution. This method utilizes point-spread function (PSF) engineering and a cascaded convolutional neural network model, which we termed as W-Net. We tested this W-Net architecture by imaging 50 nm fluorescent nanobeads at various defocus distances using a double-helix PSF, demonstrating ~20-fold improvement in image depth-of-field over conventional wide-field microscopy. W-Net architecture can be used to develop deep-learning-based image reconstruction and computational microscopy techniques that utilize engineered PSFs and can significantly improve the spatial resolution and throughput of fluorescence microscopy.

We present a thin camera which encodes 3D scenes into the 2D image sensor through a single piece of thin custom-fabricated microlens array and reconstructs the 3D scenes through a deep learning framework. The microlens array is designed to have a balanced frequency support among different spatial frequency. The deep learning framework is assisted with an adversarial learning model, and has a high speed in reconstruction. We validate the system in both simulations and experiments. Our thin 3D camera demonstrates the great potential of combining custom-designed micro-optics and deep learning algorithms in computational imaging.

Imec is a leading R&D center for nanotechnology and advanced prototyping. Photonic technology based on wafer-scale CMOS lines is receiving more attention due to its compactness, high performance, scalability, and cost reduction. These advantages have already affected the communication market, where Si photonics-based technology is the leading platform. In this conference, we will cover other emerging market segments where integrated photonics provide clear advantages, such as AR/VR, quantum computing, and AI applications.

The development of metasurfaces has enabled unprecedented portability and functionality in flat optical devices. Spaceplates — devices that can mimic free propagation to replace space in an imaging system — have recently been introduced as a complementary element to reduce the space between individual metalenses. Spaceplates require an angle-dependent optical phase response which depends on the transverse spatial frequency component of a light field. Therefore, it is challenging both to design them and to assess their ultimate performance and potential. Here, we employ inverse-design techniques to explore the behaviour of general thin-film-based spaceplates.

Scalability is essential for computing, yet classical 2D integration of neural networks faces fundamental challenges in this regard. Using 3D printing via two photon polymerization-based direct laser writing, we overcome this challenge and create low loss waveguides and demonstrate dense as well as convolutional network topologies that scale linear in size. Air-clad high-confinement waveguides allow for high-density multimode photonic integration. Leveraging the writing laser’s power as a degree of freedom in a (3+1)D printing technique, we also achieve precise control over refractive index contrast, which enables single mode propagation and low-loss evanescent couplers for next generation 3D integrated photonic circuits.

An optical computing framework based on spatiotemporal nonlinear effects of multimode fibers is presented. Experimentally, a powerful computation engine can be realized using linear and nonlinear interactions of spatial fiber modes. With the present optical scheme, we demonstrated excellent performance on a variety of classification and regression tasks. Our studies showed that spatiotemporal fiber nonlinearities are as effective as digital neural network structures in challenging computational tasks. Featuring better energy efficiency and easy scalability, our method provides a new approach to optical computation.

Optical computing has been proposed as a replacement for electrical computing to reduce energy use of math intensive programmable applications like machine learning. Objective energy use comparison requires that data transfer is separated from computing and made constant, with only computing variable. This shows that energy use is dominated by data transfer, and computing energy use is a small fraction of the total. Switching to optical from electrical programmable computing does not reduce energy use. This has been the case for years in optical communications, where optical computing has been entirely replaced by CMOS DSP.

Coherent Ising machines (CIMs) are an experimentally promising class of physics-based computational architectures that embed hard combinatorial optimization problems into systems of coupled nonlinear optical oscillators. The solution-finding mechanisms employed by CIMs feature complicated dynamical bifurcations occurring on a network scale, posing significant challenges to the development of theory and models for their underlying principles of operation. These difficulties are especially pronounced in the ultra-low-power or quantum regimes where the benefits in computational efficiency over conventional optimization algorithms are expected to be largest. We discuss some of our recent approaches and results at this intersection of dynamical systems theory and quantum model reduction, which have highlighted some potentially useful architectures and applications on the horizon for CIMs.

We will discuss diffractive optical networks that can all-optically implement various complex functions, as the input light diffracts through spatially-engineered surfaces. These diffractive processors will find applications in all-optical image analysis, feature detection and object classification, also enabling task-specific camera designs and new optical components.

With Moore’s law and Dennard scaling now being limited by fundamental physics, the trend in processor heterogeneity suggests the possibility for special-purpose photonic processors such as neural networks or RF-signal & image filtering. Here unique opportunities exist, for example, given by algorithmic parallelism of analog and distributed non-van Neuman architectures enabling non-iterative O(1) processors with ps-short delay towards real-time decision making. Here, I will share our latest work on photonic information processors to include a photonic tensor core including multistate photonic nonvolatile random-access memory [Appl. Phys. Rev.], and a massively parallel Fourier-optics convolutional processor [Optica]. In summary, photonics connects the worlds of electronics and optics, thus enabling new concepts of efficient intelligence information processing via algorithm-hardware homomorphism empowered by the distinctive properties of light.

In this paper, we present the design and fabrication of a newly developed high speed two-dimensional photodetector array device, and its application for optical signal processing in advanced fiber communications and advanced optical wireless communications. The PD array device configured with 6 x 6 square shape alignment arrangement was designed, which gave us multi-parallel output configuration at over 10 GHz, as well as a large photodetective area. In application for optical signal processing, a phase retrieval coherent receiver, space diversity optical wireless receiver, and its related technology will be introduced.

Here we show two PIC-based prototypes of a photonic convolution layer. System 1) is a Fourier-optics based 4F system integrated into a PIC. Unliked our earlier demonstration of a massively-parallel optical DMD-based CNN layer (Miscuglio, Sorger et al. OPTICA 2020), which processes 1000x1000 pixel matrices in a single time-step at 20KHz update rates (8x faster than SOW GPUs), this first-ever PIC-based 4F processor processes only 10’s of pixels, but at GHz rates (10^6 times faster than DMD, and 10^8 times faster than SLM). System 2) is a PIC-based joint-transform correlator where both the data and the convolution kernel are fed front-end and auto-convolve in the Fourier domain (autocorrelation). Note, the rapid 10GHz update rate of the kernel using foundry PIC components allows to perform online training on the system as well. Rapid and low SWaP ASICs are powerful tools for network edge processing and enable ns-short latency for rapid target tracking, for example.

This study reports a new diffractive optical network design strategy that incorporates object scaling, translation, and rotation as part of the forward training model using uniformly-distributed random variables to provide immunity and resilience against such variations at the input object plane. By guiding the evolution of the diffractive layers towards a scale-, shift- and rotation-invariant network solution, this training strategy provides >30-70% improvement in the all-optical blind inference accuracies achieved under various unknown object transformations. This training method constitutes a promising approach to bring the advantages of all-optical diffractive inference with low-latency, power-efficiency, and parallelization to various machine vision applications.

We report an all-optical computational imager to restore diffuser-distorted images at the speed of light, without a computer. For seeing through random/unknown diffusers, we trained diffractive networks consisting of successive transmissive layers. After its training, the resulting diffractive layers are fabricated, forming a passive optical network, placed behind random, new diffusers to perform all-optical reconstruction of unknown images entirely covered by unknown diffusers. All-optical diffractive reconstructions are completed at the speed of light propagation from the input to the output, do not require power except for illumination, and might find applications in e.g., atmospheric sciences, biomedical imaging, defense/security, among others.

We present both data-free and data-driven methods for the all-optical synthesis of an arbitrary complex-valued linear transformation using diffractive surfaces. Our analyses reveal that if the total number (N) of spatially-engineered diffractive features/neurons is larger than a threshold, dictated by the multiplication of the number of pixels at the input (I) and output (O) fields-of-views, i.e., N>IxO, both methods succeed in all-optical implementation of the target transformation. However, compared to data-free designs, deep learning-based diffractive designs with multiple diffractive layers are found to achieve significantly larger diffraction efficiencies and their all-optical transformations are much more accurate when N< IxO.

A vision system is a critical element to develop the driver assistance systems for improved mobility, semi-automated or fully automated driving functions, and enhancing safety. However, vision systems in the developing autonomous vehicles have been plagued by poor visibility conditions caused by glare. We demonstrate a miniaturized integrative electrical polarizer system allowing for rapid correction of multimodal glare to reveal hidden objects. We show our method enables classification of glare, fast response time and adaptability to multimodal glare.

This talk will present a science-as-a-service application that will provide public access to a new family of computational imaging algorithms that are inspired by optical physics. These algorithms are emerging as the best-in-class tools for certain digital image processing functions such as edge and texture detection, and more. A cloud application developed in collaboration with and hosted by AWS, the application features various tools, sample data, and workflows. The cloud approach provides data security, a crucial issue in the biomedical industry and exposes the broader biomedical community to the ongoing innovations in algorithms.

A fundamental challenge in the design of nanophotonic devices is the optimization of subwavelength structures to achieve tailored and high-performance electromagnetic responses. To this end, topology or shape optimizers based on the adjoint variables method have been widely adopted to push the performance limits of electromagnetic systems. However, the understanding of such freeform structures remain obscure, and such gradient-based optimizers can get trapped in low-performance local minima. Accordingly, to elucidate the relationships between device performance and nanoscale structuring, while mitigating the effects of local minima trapping, we present an inverse design framework that combines adjoint optimization, AutoML, and explainable AI.

We show that spectral mapping of data onto femtosecond optical pulses and a projection into an implicit, higher dimensional space using nonlinear optical transformation of data reduces the latency in data classification by several orders of magnitude. The approach is validated by the classification of various datasets, including brain intracranial pressure, cancer cell imaging, spoken digit recognition, and the classic Exclusive OR (XOR) benchmark for nonlinear classification. Single-shot operation is demonstrated using time stretch data acquisition. Due to the modest degrees of freedom in the optical domain, the classification accuracy is data-dependent.

Implementing frequency-encoded photonic linear transformations can be of significant interest not only for quantum information processing and machine learning hardware accelerators, but also for optical signal processing, communications, and spectrotemporal shaping of light. We present a flexible, reconfigurable architecture to implement such arbitrary linear transformations for photons using the synthetic frequency dimension of dynamically modulated microring resonators. Inverse design of the coupling between the frequency modes enables arbitrary scattering matrices to be scalably implemented with high fidelity, allowing for nonreciprocal frequency translation, unitary and nonunitary transformations. Our results introduce new functionalities for linear transformations beyond those possible with real-space architectures that are typically time-invariant.

The generation of an optical supercontinuum in nonlinear fibers exhibits highly complex nonlinear dynamics. Here, we show that one can train a neural network to learn the complex propagation dynamics for supercontinuum generation solely based on the input pulse parameters for a variety of scenarios ranging from higher-order soliton compression to broadband octave-spanning supercontinuum. The speed of our approach exceeds that of the direct integration of the generalized nonlinear Schrödinger equation by several orders of magnitude, allowing for “real-time” optimization or analysis of optical systems.

We use machine learning methods to control the spectral broadening experienced by femtosecond pulses in a highly nonlinear fiber. Combining a programmable spectral filter with a genetic algorithm or neural network allows us to optimize the nonlinear propagation dynamics to generate an on-demand target spectrum. Our approach is generic and can be adapted to a wide range of optical fibers and pump pulses. We expect our results to provide significant advances for adaptative control and tailored light sources.

The race to heuristically solve non-deterministic polynomial-time (NP) problems through efficient methods is ongoing. Recently, optics was demonstrated as a promising tool to find the ground state of a spin-glass Ising Hamiltonian, which represents an archetypal NP problem. However, achieving completely programmable spin couplings in these large-scale optical Ising simulators remains an open challenge. Here, by exploiting the knowledge of the transmission matrix of a random medium, we experimentally demonstrate the possibility of controlling the couplings of a fully connected Ising spin system. By further tailoring the input wavefront we showcase the possibility of modifying the Ising Hamiltonian both by accounting for an external magnetic field and by controlling the number of degenerate ground states and their properties and probabilities. Our results represent a relevant step toward the realisation of fully-programmable Ising machines on thin optical platforms, capable of solving complex spin-glass Hamiltonians on a large scale.