Sensors group INI student projects
(Link to edit this source document) (link to draft projects file) Scan QR code for this document →
The Sensors group (sensors.ini.uzh.ch) at the Inst. of Neuroinformatics has interesting, exciting projects for semester-projects and bachelor- or master-theses; and projects for talented high school students doing a Maturaarbeit. The projects span a broad range from device physics to event-driven deep networks to neuromorphic vision and audition with event sensors, including projects in machine learning, artificial intelligence, robotics, and control.
Looking for an internship with a company? Check out our spinoffs inivation, inilabs, and insightness.
Interested? Contact the group directors Shih-Chii Liu (shih@ini.uzh.ch ) and Tobi Delbruck (tobi@ini.uzh.ch ) along with the responsible person.
Overview of projects
Machine learning, hardware | NEW USB camera interface for Deep Network neuroprosthesis Implement camera interface on FPGA to a vision neural network accelerator. | Assistants for visual impaired with cortical prosthesis Design event-driven deep network architectures and test them on embedded systems acting as assistants for visually impaired. | NEW Fast video semantic edge segmentations on edge devices Design event-driven deep network architectures for video semantic edge segmentation and deploy the model on an edge device. | NEW Virtual Reality Demonstration of Neuroprosthesis Vision Develop a demonstration of what a neuroprosthetic vision would allow a blind person to see. |
Audio algorithms and systems | Multi-channel speech separation algorithms using deep learning Implement real-time multi-channel speech separation algorithms | Audio multi-channel beamforming Implement speech enhancement and beamforming algorithms on WHISPER | Speech recognition using recurrent neural networks on embedded systems Design deep neural network hardware for embedded ASR edge applications. Study tradeoffs in energy, latency, throughput. | Deep networks using temporal information in spikes Develop new spike-based algorithms and networks that use timing information |
Hardware/ Real world AI, robotics, control | NEW Combine AI and Control Theory to control a dynamical system in real-time. See expanded description below and our SIROP post “Join us to develop brain-inspired control algorithms and deploy them to real robots!” | Explore computer vision, control and reinforcement learning with the Brio Labyrinth | Use end to end and reinforcement learning technology to autonomously sail Argo, our robot boat | |
Machine learning, theory, practice | NEW Using DNN models to understand neuroscience data for adaptive visual neuroprosthetics (Image: Chen & Roelfsema, KNAW) Develop a DNN to extract information from a multi-channel recording array. | NEW 🔥 Efficient closed-loop visual neuroprosthesis simulation for video Develop deep learning algorithm for biologically plausible smooth phosphenes. | Night vision CNN denoisingTeach a 3D CNN to denoise night video, using self-supervised learning. | Neuromorphic multisensor fusion for real-time beverage tasting Develop a neural network fusion system for predicting beverage classes. |
Event sensors | Event-based lipreading: audio-visual deep network speech recognition with spikes Learn to train deep networks for a real time A-V recognition system using event-based sensors | NEW Novel strategies for DVS noise filtering Explore new ideas to extract signal information from noise DVS output | NEW Understanding and modeling DVS noise as a random process Develop a mathematical theory for DVS noise | Event sensor processing and learning We have various other projects involving event-driven processing and learning using event-based sensors, the DAVIS, DVS, and the COCHLP driving deep networks |
ASIC circuit design/measurement | VLSI design of spike-based DN classifier using cochlea spikes Design low-power audio VLSI circuits for next generation audio TinyML devices. | NEW Exploring new design ideas for the DVS pixel Analog/mixed-signal design of new ideas to improve the DVS pixel. | NEW Characterization and benchmarking of DVS cameras Lab measurement, characterization and benchmarking of DVS cameras. | Neuromorphic circuit design/measurement projects Interested in mixed-signal design. We have neuromorphic circuit design projects. |
Starting a project:
- Verbal and written English are a requirement for all projects.
- Projects are initiated with a discussion with the project supervisor(s).
- After a written project description is accepted (see here for Sensors student project proposal template), and the project is registered by the student in myStudies, the project can start.
- Students are required to sign an IPR rights assignment for some projects that assigns IP ownership to UZH, and that also ensures inventors will benefit.
- Projects are graded based on a set of standard criteria.
Open projects
Audio processing (algorithms and systems)
Multi-channel speech separation networks using deep learning | |
Acronym | MC-SP |
Status | Open |
Type | masters (short/long) |
Contact | Prof. Shih-Chii Liu (shihatini.uzh.ch ) |
Last update date | 01.11.21 |
Project Description | Together with an industry partner, we will investigate multi-channel speech separation algorithms that can run in real-time on embedded platforms. Algorithms will be based on deep network solutions. The student will investigate ways of implementing multi-channel speech enhancement and source separation for reverberant environments. |
Requirements | General knowledge of Signal Processing, Machine Learning and Neural Networks; Knowledge of Matlab, Python and git. |
Background material |
|
Results | Optional information/links to project results |
Audio multi-channel beamforming systemLearn about beamforming algorithms and their real-time implementation on WHISPER | |
Acronym | AUDSYS |
Status | Open |
Type | bachelors/semester/masters (short/long) |
Contact | Ilya Kiselev (kiselevatini.uzh.ch) and Prof. Shih-Chii Liu (shihatini.uzh.ch ) |
Last update date | 01.05.21 |
Project Description | This project will look at the beamforming capabilities of our WHISPER ad-hoc multi-microphone platform when microphones are placed in arbitrary different spatial locations. The student will have an opportunity to learn about beamforming algorithms and their real-time implementation. The student will look at how the beamforming results change with different microphone configurations when using this platform. The WHISPER platform can also be mounted on a mobile robot for tracking experiments. |
Requirements | Skills needed in one of the one following areas:
software programming in Java, Matlab or python or working with Raspberry-PI |
Background material | Kiselev, I. et al, “WHISPER: Wirelessly Synchronized Distributed Audio Sensor Platform” , 2017 https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8110202 |
Results | Optional information/links to project results |
Speech recognition using recurrent neural network on FPGA | |
Acronym | SRDRNN |
Status | Open |
Type | masters (short/long) |
Contact | Chang Gao (chang@ini.uzh.ch), Prof. Shih-Chii Liu (shih@ini.uzh.ch), Prof. Tobi Delbruck (tobi@ini.uzh.ch 044 635 3038) |
Last update date | 28.02.2021 |
Project Description | In this project, you will design a deep neural network based on recurrent neural networks (RNN) and probably convolutional neural networks as a front-end feature extractor to realize an end-to-end automatic speech recognition (ASR) system. You will also have the opportunity to build a real-world demo running on embedded systems like NVIDIA’s Jetson Nano/TX2 platform and our DeltaRNN accelerator. Potential tasks include:
|
Requirements | If you want a short project to do only tasks 1:
If you want a long project to do everything and hope to have a paper:
|
Background material |
|
Results | Optional information/links to project results |
Deep networks using temporal information in cochlea spikesDevelop new event-driven deep networks that extract feature-level information based on temporal information from spiking cochleas | |
Acronym | SPIKEDNCOCH |
Status | Open |
Type | bachelors/semester/masters (short/long) |
Contact | Prof. Shih-Chii Liu (shihatini.uzh.ch ) |
Last update date | 01.11.21 |
Project Description | The student will investigate deep network architectures that extract feature-level information based on temporal information in the cochlea spikes and spectrogram samples for solving a task. The student will also study different encoding schemes of the cochlea for audio tasks. We have state-of-art hardware implementations of binaural cochlea chips that prod.uce asynchronous spike outputs (AEREAR2 and COCHLP) that can be tested with the networks. |
Requirements | Strong interest in deep learning, signal processing and neural models. SKnowledge of Python; previous experience with standard deep learning libraries |
Background material |
|
Results | Optional information/links to project results |
Circuit design/measurement
VLSI design of spike-based deep network classifier that uses a cochlea front endDevelop circuits for a silicon cochlea + deep network classifier. | |
Acronym | COCHDESIGN |
Status | Open |
Type | bachelors/masters (short/long) |
Contact | Prof. Shih-Chii Liu (shihatini.uzh.ch) |
Last update date | 01.11.21 |
Project Description | Interested in building event-driven neuromorphic circuits? We have an open circuit project for a student to design one block of a low power acoustic sensor with a deep network backend. |
Requirements | The student should have taken an analog/digital VLSI design class or NE1. |
Background material | |
Results | Optional information/links to project results |
Exploring new design ideas for the DVS pixel | |
Acronym | |
Status | Open |
Type | masters (short/long) |
Contact | Rui Graca (rpgraca@ini.uzh.ch), Tobi Delbruck (tobi@ini.uzh.ch) |
Last update date | 06.12.22 |
Project Description | Some ideas to improve DVS performance have been suggested but not sufficiently explored. These include:
In this project, some of these possible improvements will be explored and implemented in DVS pixel design. |
Requirements |
|
Background material | [1] P. Lichtsteiner, C. Posch and T. Delbruck, "A 128×128 120 dB 15μs Latency Asynchronous Temporal Contrast Vision Sensor," in IEEE Journal of Solid-State Circuits, vol. 43, no. 2, pp. 566-576, Feb. 2008, doi: 10.1109/JSSC.2007.914337. [2] T. Delbruck, C. Li, R. Graca, and B. Mcreynolds, ‘Utility and Feasibility of a Center Surround Event Camera’. arXiv, 2022. [3] T. Finateu et al., "5.10 A 1280×720 Back-Illuminated Stacked Temporal Contrast Event-Based Vision Sensor with 4.86µm Pixels, 1.066GEPS Readout, Programmable Event-Rate Controller and Compressive Data-Formatting Pipeline," 2020 IEEE International Solid- State Circuits Conference - (ISSCC), 2020, pp. 112-114, doi: 10.1109/ISSCC19947.2020.9063149. [4] Y. Suh et al., "A 1280×960 Dynamic Vision Sensor with a 4.95-μm Pixel Pitch and Motion Artifact Minimization," 2020 IEEE International Symposium on Circuits and Systems (ISCAS), 2020, pp. 1-5, doi: 10.1109/ISCAS45731.2020.9180436. [5] M. Yang, S. -C. Liu and T. Delbruck, "A Dynamic Vision Sensor With 1% Temporal Contrast Sensitivity and In-Pixel Asynchronous Delta Modulator for Event Encoding," in IEEE Journal of Solid-State Circuits, vol. 50, no. 9, pp. 2149-2160, Sept. 2015, doi: 10.1109/JSSC.2015.2425886. |
Results | Characterization through circuit simulation of new ideas for improvements on the DVS pixel |
Characterization and benchmarking of DVS cameras | |
Acronym | DVSmeas |
Status | Open |
Type | masters (short/long) |
Contact | Rui Graca (rpgraca@ini.uzh.ch), Tobi Delbruck (tobi@ini.uzh.ch) |
Last update date | 06.12.22 |
Project Description | DVS operation is complex and affected by many non-idealities, often not properly modeled by simulation. Other than that, DVS behavior depends on several biases and illumination in a way that is not always intuitive. This is a hands-on project that aims to measure and characterize DVS cameras, improving current modeling and benchmarking. |
Requirements |
|
Background material | [1] P. Lichtsteiner, C. Posch and T. Delbruck, "A 128×128 120 dB 15μs Latency Asynchronous Temporal Contrast Vision Sensor," in IEEE Journal of Solid-State Circuits, vol. 43, no. 2, pp. 566-576, Feb. 2008, doi: 10.1109/JSSC.2007.914337. [2] R. Graça, B. McReynolds and T. Delbruck, "Shining light on the DVS pixel: A tutorial and discussion about biasing and optimization," 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vancouver, BC, Canada, 2023, pp. 4045-4053, doi: 10.1109/CVPRW59228.2023.00423. [3] R. Graca and T. Delbruck, ‘Unraveling the paradox of intensity-dependent DVS pixel noise’. arXiv, 2021. https://arxiv.org/abs/2109.08640 [4] B. McReynolds, R. Graca, T. Delbruck, "Experimental methods to predict dynamic vision sensor event camera performance," Opt. Eng. 61(7) 074103 (25 July 2022) https://doi.org/10.1117/1.OE.61.7.074103 |
Results | Improved understanding about DVS cameras. New knowledge to improve models and benchmarks. |
Neuromorphic circuit design projects | |
Acronym | CIRDES |
Status | Open |
Type | bachelors/masters (short/long) |
Contact | Prof. Shih-Chii Liu (shihatini.uzh.ch) and Prof. Tobi Delbruck (tobiatini.uzh.ch) |
Last update date | 01.11.21 |
Project Description | Interested in building neuromorphic circuits? We have circuit projects for students to design novel mixed signal circuits. |
Requirements | NE1 or analog integrated circuits course |
Background material | See our web pages for papers about IC designs |
Results | Optional information/links to project results |
Control/Robotics/Machine learning
Learning to control | |
Acronym | L2C |
Status | Open |
Type | bachelors/semester/masters (short/long) |
Contact | Marcin Paluch (marcin.p.paluch@gmail.com ), Tobi Delbruck (tobi@ini.uzh.ch ) See also our SIROP post “Join us to develop brain-inspired control algorithms and deploy them to real robots!” |
Last update date | 6.12.2022 |
Project Description | Have you ever seen a bat catching an insect or an octopus maneuvering its eight arms? Help us to close the gap between the control we observe in nature and robotic applications! As the Neural Control team at the Sensors Group (Institute of Neuroinformatic, a joint initiative of ETH and UZH) we use machine learning tools to find the optimal control fast and with limited power budget. We are interested in learning system models and studying how the uncertainty can be included to provide more robust control strategies. The algorithms are tested on multiple systems in simulation and on physical cartpole. We prepare tests on a powerful f1tenth car (in collaboration with PBL at ETH) and a quadruped (in collaboration with Amber Lab at Caltech). We aim in proving the usefulness of our ideas in the competitive setting of F1TENTH international autonomous race. Possible questions, which you could research on as your project, include:
More topics are possible! You can even come up with your own idea and if it fits our research direction we will help you realize it. Please contact us and we will willingly search together for a project which best fits your interests, prerequisites and the workload of your thesis. Write us something about your interests, send us your CV and transcript and let us have a zoom meeting. We are happy to get to know you! We can directly supervise students from UZH, ETH D-ITET and ETH D-PHYS. We also willingly accept students from other ETH departments, in which case supervision is formally provided by one of our collaborators. We are open to semester projects, bachelor and master thesis. |
Prerequisites | Strongly dependent on a particular project. Background in control theory and AI as well as basic Python skills are helpful. |
Control the Labyrinth robotExplore computer vision, control and reinforcement learning with the Brio Labyrinth | |
Acronym | LABYRINTH |
Status | CURRENTLY OPEN |
Type | Any (bachelors/semester/masters (short/long)) |
Contact | Prof. Tobi Delbruck (tobi@ini.uzh.ch, 0446353038) |
Last update date | 1.6.19 |
Project Description | See https://youtu.be/VbHLn3vcv34 for existing status of LABYRINTH. The Brio Labyrinth is a game that offers a rich environment for exploring computer vision, control and reinforcement learning. The goal is to navigate a ball through the maze from while avoiding falling into the holes. The image above shows the current implementation of the robot version of labyrinth. Two hobby servos have been attached to the knobs controlling the tilt of the table and we built the first implementation of a silicon retina ball tracker and PID controller. See https://youtu.be/VbHLn3vcv34 to get the best idea of the current state of this robot. To solve the full problem requires innovative development of several components
|
Requirements | Background in one or more of the following fields is needed for work on aspects of the project. Mechanical engineering: For reinforcement learning, it will be essential to have a mechanism to return the balls to a starting point. The balls are precision stainless steel bearings that are ferromagnetic, so an electromagnet could grip them. Or some kind of rolling tape arrangement would allow endless repetition and enable the massive number of trials that needed for reinforcement learning. Machine learning and computer vision: To solve the ball tracking problem, one possible approach is to use a convolutional neural network (CNN) to solve the what/where problem: We need to find if the ball is in the scene and where it is in the scene. A CNN can be trained by collecting and labeling data while a human controls the ball around the table. Then a CNN can be trained to solve the problem. Finally, this CNN can be run very quickly on our NullHop FPGA CNN accelerator to achieve the low latencies needed for control. Many interesting aspects arise from this problem including attention, prior biasing, tracking, etc. Low level control and reinforcement learning: The youtube video shows a basic PID controller that can control the ball on a blank table. However, this controller assumes linearity and the PID parameters are hand-tuned for optimal performance. There is a great opportunity here to explore RL learning techniques to optimize this PID controller for more precise control, faster response, and better stability. The most recent techniques for continuous-value deep reinforcement learning can also be explored here to develop an adaptive controller. The real labyrinth game requires nonlinear control. The control of the table is via strings that stretch and slip and the tilt of the table is resonant; it overshoots a target. Also, as soon as the ball hits a wall of the maze, the response of the ball depends very nonlinearly on the control input, since the wall prevents restricts motion along one dimension for at least a range of control inputs. Strategic planning and reinforcement learning: A human can learn how to play labyrinth quite well with a few days practice. Typically a human solves it by going between safe corner points. Could a RL algorithm learn this same approach? The existing code already implements a map of the maze including the holes, walls and path. It also includes a simple planner that can move a ball along a specified set of path points. But a lot more interesting work needs to be done to develop methods that might be capable of learning a true strategy to solve this game. Can it actually learn the human approach of going between safe corners? |
Background material |
|
Results | Optional information/links to project results |
Argo: Learning to sailUse modern machine end-to-end and reinforcement learning technology to autonomously sail Argo, our robot sailboat (see Argo photo album) | |
Acronym | ARGO |
Status | Open |
Type | bachelors/semester/masters (short/long) |
Contact | Prof. Tobi Delbruck (tobi@ini.uzh.ch, 044 635 3038) |
Last update date | 8.5.19 |
Project Description | RC sailing is an enthusiast activity where people enjoy sailing remote controlled sailboats around ponds, for example Irchel pond. The aim of ARGO is to use modern machine learning technology to autonomously sail such a boat, for example, with the objective of a normal sailboat race or to pass a chosen waypoint. This project has various levels. For example, one hardware subproject is to research and acquire the sailboat, outfit it with knotmeter, wind speed and direction meter, high quality GPS, and IMU, along with a very low power embedded linux computer to control the two servos that determine rudder and sail position. (This part is complete) For machine / reinforcement learning, the data can be acquired by manually sailing the boat under many conditions to collect an end-to-end (E2E) training dataset. Another aspect of the project would develop an interface to a sailboat simulator to collect training data, to explore transfer learning and reinforcement learning approaches in a simpler model-based environment. The model would then need to be validated on a real sailboat to credibly demonstrate it works. Finally, the learned model can be tested on recorded data and then on the actual sailboat. UPDATE: Argo, our robot sailboat based on DragonForce 65 is now built and running smoothly to collect E2E training data (see Argo photo album). We have routinely been sailing Argo on the Irchel pond. We are starting to collect data, but no analysis of this data except to assess quality of the sensors has been performed. |
Requirements | Various, depending on project component. For setup, experience with sensor integration and embedded linux is important. For machine learning, an introductory course in machine learning is essential. |
Background material | No academic papers found on using reinforcement or deep learning methods for robot sailing.
|
Results | D-ITET group project students Imre Kertesz and Andre Mock built boat with T Delbruck fall of 2018. Argo is sailing now in 2019. |
Machine Learning / Theory & Practice
Assistants for visual impaired with cortical prosthesis | |
Acronym | VisModeViper |
Status | OPEN |
Type | bachelors/masters (short/long) |
Contact | Prof. Shih-Chii Liu (shih@ini.uzh.ch) |
Last update date | 12.04.2021 |
Project Description | In this project we will develop networks, embedded systems, and integrated circuit versions of the network to help the visually impaired person interact with and navigate their surroundings (www.neuraviper.eu). We will combine areas of Deep Learning such as video parsing and speech processing. Projects include
|
Requirements |
|
Background material | Review of video scene parsing using deep learning: https://www.sciencedirect.com/science/article/abs/pii/S1077314220301120 Possible starting point: |
Results | Deep network that uses multiple sensors. Transfer of network to embedded systems suitable for visual prosthesis. Real-time demonstrator that combines multiple modalities. |
Fast video semantic edge segmentation on edge devices | |
Acronym | ViSESeg |
Status | OPEN |
Type | bachelors/semester/masters (short/long) |
Contact | Zuowen Wang (wangzu@ethz.ch), Prof. Shih-Chii Liu (shih@ini.uzh.ch) |
Last update date | 08.12. 2022 |
Project Description | Semantic edge detection combines both edge detection and semantic classification associating edge pixels with one or more object categories. For every pixel on an image the task solves whether the pixel lies on an edge and which class(es) it belongs to. In this project we intend to address the issue of low inference speed of currently existing works while maintaining good prediction performance. Potential tasks consist of:
|
Requirements | Basic machine learning/ computer vision courses. Experience with deep learning frameworks, including pytorch. Tensorflow 1 is a plus since we might have to compare with existing (older) works. FPGA experience is a plus. |
Background material | |
Results |
|
Night vision CNN denoising￫ Teach a convolutional recurrent CNN to denoise night video, using self-supervised learning. | |
Acronym | NIGHTVIS |
Status | OPEN |
Type | bachelors/semester/masters (short/long) |
Contact | |
Last update date | 05.3.20 |
Project Description | The book chapter below shows amazing results from a collaboration between Eric Warrant, a Swedish professor who studies night vision in flying insects, and two Swedish mathematicians, on hand crafted methods that do an amazing job on improving the quality of video that is extremely shot noise limited. This project aims to replace the hand crafted feature engineering with a 3D (space time) CNN that is trained using self-supervised methods. Eventually such CNN could run on a hardware CNN accelerator such as the ones developed by the Sensors group in real time and low power. |
Requirements | Machine Learning, e.g. pyTorch or TensorFlow experience |
Background material |
|
Results | |
Efficient closed-loop visual neuroprosthesis simulation for videoLeft: original video, Right: simulated phosphenes, scale up for best view for the phosphenes | |
Acronym | vidNVPsim |
Status | Open |
Type | bachelors/semester/masters (short/long) |
Contact | Zuowen Wang(wangzu@ethz.ch), Prof. Shih-Chii Liu (shih@ini.uzh.ch) |
Last update date | 06.Feb.2023 |
Project Description | In this project carried out under NeuraViPeR , an EU project aiming at providing rudimentary vision for the visually impaired, the student will extend an existing phosphene simulator to create biologically plausible and smooth phosphene temporal patterns. They will explore efficient deep learning algorithms for video inputs and optimize these networks for a closed-loop system. The student will also implement the phosphene network on the Jetson Nano for a portable setup. Extensions of this work, for e.g. for a closed-loop online learning pipeline, or with an event camera is also possible. |
Requirements | Basic machine learning/ deep learning courses and computer vision knowledge. Experience with python and pytorch. Neuroscience knowledge especially on the visual cortex is a plus but not required. |
Background material |
|
Results | Previous student projects in our group under the NeuraViPeR project:
|
Using DNN models to understand neuroscience data for adaptive visual neuroprosthetics(Image: Chen & Roelfsema, KNAW) | |
Acronym | DNNNeuro |
Status | OPEN |
Type | bachelors/masters (short/long) |
Contact | Zuowen Wang (wangzu@ethz.ch), Prof. Shih-Chii Liu (shihatini.uzh.ch) |
Last update date | 01.02.2023 |
Project Description | The student will apply deep learning methods to determine the amount of dynamic information in the responses of recordings from a multi-channel electrode system implanted in the visual cortex and study the feasibility to use the information as feedback control for a neuroprothesis. This work is part of the NeuraViPeR project. |
Requirements | Knowledge of Python; previous experience with standard deep learning libraries; strong interest in deep learning and neuroscience. |
Background material |
neuroprosthesis in monkey visual cortex
|
Results |
|
Neuromorphic multisensor fusion for real-time beverage tasting | |
Acronym | REALTASTE2 |
Status | Open |
Type | semester/masters (short/long) |
Contact | Dr. Josep M. Margarit (josep@ini.uzh.ch), Prof. Shih-Chii Liu (shih@ini.uzh.ch) |
Last update date | 01.12.2022 |
Project Description | This project is a collaboration between the Institute of Neuroinformatics (INI) of ETH/UZH and the Institute of Microelectronics of Barcelona (IMB-CNM(CSIC)) of the Spanish National Research Council. This project extends on previous work to develop a neuromorphic network implementation of electrochemical tasting on Intel’s Loihi research platform. The system includes an array of amperometric, impedimetric and potentiometric microsensors and its respective readout electronic equipment. The student will develop a fusion network to fuse dynamic multimodal sensor data so as to discriminate between different types of commercial beverages. The student will develop a CNN architecture to improve model performance on neuromorphic hardware such as Intel’s Loihi; explore the online learning of new classes/sensors; and run the network on the edge using the Loihi Kapoho Bay USB stick. They will benchmark the system against other baseline regression/classification results in a Python GUI for a live demonstration. |
Requirements |
|
Background material | [1] J.M. Margarit-Taulé, P. Giménez-Gómez, R. Escudé-Pujol, M. Gutiérrez-Capitán, C. Jiménez-Jorquera, and S.C. Liu, “Live Demonstration: A Portable Microsensor Fusion System with Real-Time Measurement for On-Site Beverage Tasting”, 2019 IEEE ISCAS, Jun 2019; https://doi.org/10.1109/ISCAS.2019.8702184 [2] P. Giménez-Gómez, R. Escudé-Pujol, F. Capdevila, A. Puig-Pujol, C. Jiménez-Jorquera, and M. Gutiérrez-Capitán, “Portable electronic tongue based on microsensors for the analysis of cava wines,” Sensors, vol. 16, no. 11, p. 1796, 2016. https://doi.org/10.3390/s16111796 [3] M. Davies et al., "Loihi: A Neuromorphic Manycore Processor with On-Chip Learning," IEEE Micro, vol. 38, no. 1, pp. 82-99, 2018. https://doi.org/10.1109/MM.2018.112130359 |
Results | |
Virtual Reality Demonstration of Neuroprosthesis Vision | |
Acronym | VRneuro |
Status | OPEN |
Type | bachelors/masters (short/long) |
Contact | Pehuen Moure (pmoure@ini.ethz.ch), Prof. Shih-Chii Liu (shih@ini.uzh.ch) |
Last update date | 02.12.2022 |
Project Description | We are interested in building a VR based tool for demonstrating the visual neuroprosthesis output as part of the NeuraViPeR project. As part of demonstration the VR Headset will simulate a limited phosphene based output for a patient for the non-visually impaired participant. These outputs will be generated by a deep neural network or a spiking neural network that will be used for patients. The participant will be asked to take part in a basic task using the simulated output, an example of a task vs simulated view can be seen below. We are interested in developing such a system that functions on a standard VR headset (Oculus or homemade) and running some basic user studies. Projects include
Sample Camera Input Sample Neuroprosthetic Output |
Requirements |
|
Background material | [1] Optimization of Neuroprosthetic Vision via End-to-end Deep Reinforcement Learning: https://www.biorxiv.org/content/10.1101/2022.02.25.482017v1.full.pdf [2] Chen et al, Science 2020: Shape perception via a high-channel-count neuroprosthesis in monkey visual cortex |
Results | Demonstration of VR system with real-time execution of artificial neural network. |
Hardware deep neural network accelerators (CNNs and RNNs)
USB camera interface for Deep Network neuroprosthesis | |
Acronym | CBNVP |
Status | Open |
Type | Masters |
Contact | Dr. Qinyu Chen (qinyu.chen@uzh.ch), Prof. Shih-Chii Liu (shih@ini.uzh.ch) |
Last update date | 20.11.2022 |
Project Description | In this project, the student will develop a USB interface to stream camera data to a downstream convolutional neural network accelerator used to create stimulation patterns for a visual neuroprosthesis in the NeuraViPeR project (www.neuraviper.eu). Your tasks:
|
Requirements |
|
Reference (Background Material) |
neuroprosthesis in monkey visual cortex |
Novel applications of event sensor processing and learning
Event-based lipreading: audio-visual deep network speech recognition with spikes | |
Acronym | EVLIPREAD |
Status | Open |
Type | masters (short/long) |
Contact | Shih-Chii Liu (shihatini.uzh.ch ) and Shu Wang (shuatini.uzh.ch ) |
Last update date | 01.11.21 |
Project Description | The student will design a neuromorphic audio-visual fusion system for speech recognition using the Dynamic Vision Sensor (DVS) and Dynamic Audio Sensor (DAS) to extract events as input features. They will train a deep convolutional neural network (CNN) on audio and video spikes for predicting words in a speech dataset. With the SNN toolbox we can convert the CNN to spiking networks for the final implementation on the Intel Loihi neuromorphic platform. |
Requirements |
|
Background material |
|
Results | |
Novel strategies for DVS noise filtering | |
Acronym | DVSnoiseFilt |
Status | Open |
Type | masters (short/long) |
Contact | Rui Graca (rpgraca@ini.uzh.ch), Tobi Delbruck (tobi@ini.uzh.ch) |
Last update date | 06.12.22 |
Project Description | In this project, the student will explore new ideas for extracting information from DVS output, separating useful signal from parasitic noise. Depending on the interest of the student, the noise filtering algorithms can be implemented either in software or in hardware (FPGA). |
Requirements | Depending on the direction chosen for this project, some of the following skills are important:
|
Background material | [1] S. Guo and T. Delbruck, "Low Cost and Latency Event Camera Background Activity Denoising," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 785-795, 1 Jan. 2023, doi: 10.1109/TPAMI.2022.3152999. http://dx.doi.org/10.1109/TPAMI.2022.3152999 [2] R. Graca and T. Delbruck, ‘Unraveling the paradox of intensity-dependent DVS pixel noise’. arXiv, 2021. https://arxiv.org/abs/2109.08640 [3] Y. Hu, S-C. Liu, and T. Delbruck. v2e: From Video Frames to Realistic DVS Events. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), URL: https://arxiv.org/abs/2006.07722, 2021 |
Results | The expected result of this project is a new strategy for DVS noise filtering, implemented either in software or hardware, and a comparison between this new strategy and currently existing algorithms. |
Understanding and modeling DVS noise as a random process | |
Acronym | DVSnoiseMod |
Status | Open |
Type | masters (short/long) |
Contact | Rui Graca (rpgraca@ini.uzh.ch), Tobi Delbruck (tobi@ini.uzh.ch) |
Last update date | 06.12.22 |
Project Description | This project focuses on developing a mathematical theory relating DVS noise event rate to DVS cutoff frequency, event threshold and voltage noise, using stochastic signal theory. The theory developed will be confronted with simulations of a simplified functional model of a DVS pixel, as well as with real data already acquired with a DVS camera. Depending on the interest of the student, this project can go in a more theoretical oriented direction, in a modeling oriented direction, or even in a more hands-on approach with lab measurements of a DVS camera. |
Requirements |
|
Background material | [1] R. Graca and T. Delbruck, ‘Unraveling the paradox of intensity-dependent DVS pixel noise’. arXiv, 2021. https://arxiv.org/abs/2109.08640 [2] S. Guo and T. Delbruck, "Low Cost and Latency Event Camera Background Activity Denoising," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 785-795, 1 Jan. 2023, doi: 10.1109/TPAMI.2022.3152999. http://dx.doi.org/10.1109/TPAMI.2022.3152999 [3] Y. Hu, S-C. Liu, and T. Delbruck. v2e: From Video Frames to Realistic DVS Events. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), URL: https://arxiv.org/abs/2006.07722, 2021 |
Results | A good understanding of statistics of DVS noise events should result from this project. This can be then applied in several directions, depending on the interest of the student. Some of the possible directions are:
|
Event sensor processing and learningWe have various other projects involving event-based processing and learning using deep networks and our event-based sensors, the DAVIS, DVS, and the COCHLP | |
Acronym | various |
Status | Open |
Type | bachelors/semester/masters (short/long) |
Contact | Shih-Chii Liu (shihatini.uzh.ch ) and Tobi Delbruck (tobiatini.uzh.ch ) |
Last update date | 10.05.19 |
Project Description | We have other projects involving event-based processing and learning using deep networks and our event-based sensors, the DAVIS, DVS and the COCHLP. |
Requirements | Various, but signal processing and machine learning backgrounds are important for most projects, along with coding ability. |
Background material | See Sensors group Research and Publications pages |
Results | Optional information/links to project results |