Computing power is the cornerstone and driving engine of the intelligent era, while vision serves as the most critical way for humans and machines to perceive the world. As a fundamental building block for artificial intelligence, in-sensor computing chips empower compact, portable terminals such as smartphones and robots with powerful perception and computing capabilities.
The research team led by Professor Kaiyu Cui from the Department of Electronic Engineering, Tsinghua University, has developed the world’s first Matter Spectral Chip — also known as the 2nd-generation spectral imaging chip, formally named the Spectral Convolutional Neural Network (SCNN) Chip. Designed for complex visual tasks, it is the first optical computing chip that directly takes natural light carrying material spectral information as input. This breakthrough addresses the major bottleneck that has prevented most existing optical neural networks from practical deployment, enabling real-world complex visual computing. Dr. Cai Xusheng, Dr. Huang Zhilei and Dr. Wang Yu, co-founders of Seetrum, participated in the chip’s R&D.
The research findings were published online in Nature Communications under the title Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light.
Against the backdrop that demand for computing power driven by big data and large AI models is growing far beyond Moore’s Law, conventional electronic computing platforms suffer from high energy consumption and limited operating speed, hindering the deployment of AI models on edge devices. Optical neural networks feature remarkable advantages including ultra-fast computing, high parallelism and low power consumption, making them one of the most promising next-generation parallel computing solutions. Nevertheless, restricted by on-chip integration limitations and reliance on coherent light sources, most existing optical neural networks can only perform simple tasks such as image edge detection and handwritten digit recognition, and are difficult to apply in real-world scenarios.
Inspired by biological vision, Convolutional Neural Networks (CNN) can extract high-dimensional image features and drastically reduce parameter volume for image processing, and have been widely adopted in image recognition, segmentation, detection and other machine vision tasks. The research team proposed an innovative in-sensor computing solution: the Spectral Convolutional Neural Network (SCNN). By massively integrating spectral modulation structures on a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor (CIS), large-scale parallel vector inner product computation can be realized in the spectral domain. The image sensor integrated with spectral modulation structures acts as both the input layer and the first convolutional layer. Combined with subsequent small-scale electronic convolutional layers, it forms a hybrid optoelectronic neural network.
Two versions of the SCNN-based Matter Spectral Chip were fabricated using optical metasurfaces and pigment-based spectral modulation structures respectively, verifying the feasibility of the SCNN framework. The metasurface-based chip delivers superior spectral modulation performance and holds potential for full optical field perception including polarization and phase detection as well as incident angle sensing. The pigment-based chip has achieved mass production on 12-inch wafers, featuring higher integration density and lower manufacturing costs.
Technical Advantages
1. The optically computed convolutional layer built on commercial image sensors supports direct perception of incoherent natural light, which covers two spatial dimensions and one broadband spectral dimension. It requires no coherent light sources.
2. The architecture realizes true in-sensor computing: image acquisition and computation are completed simultaneously by the sensor. It enables high-dimensional spectral image capture and processing on edge and mobile terminals with limited computing resources. This realizes Matter Meta-imaging and breaks reliance on GPUs, allowing spectral imaging technology to be widely deployed on various end devices.
3. The hybrid optoelectronic computing framework combines the high speed, parallelism and low power consumption of optical computing with the flexibility of electronic computing. Leveraging high-density image sensor arrays, it enables per-pixel computation for cameras with millions or even hundreds of millions of pixels.
Application Prospects
To validate the effectiveness and flexibility of the Matter Spectral Chip (3×3.5 mm²), the team deployed it for two distinct real-world complex tasks: pathological diagnosis and face anti-spoofing.
Face Anti-spoofing: Spectral characteristics reflect inherent material compositions and are extremely difficult to forge. Powered by SCNN-based spectral perception and computation, the chip achieves pixel-level living body detection with an accuracy of 96.23%, and nearly 100% accuracy for image-level face anti-spoofing.
Tissue Section Diagnosis: The chip can distinguish normal thyroid tissues from four types of diseased tissues (including cancerous lesions) without using a microscope, demonstrating great potential for real-time intraoperative pathological diagnosis.
In summary, the Matter Spectral Chip (Spectral Convolutional Neural Network Chip) can directly process natural images. It performs highly parallel inner product computation across spatial dimensions with millions to hundreds of millions of pixels, and captures material information via continuous spectral perception. The chip dynamically identifies material compositions and maps them into feature spaces, pioneering a new paradigm of in-sensor edge computing for Matter Meta-imaging (MMI). It equips machine vision and edge terminals with brand-new material imaging capabilities, opening a new era for neural network chips that deliver vision beyond human eyes.
