PhD student, Department of Computer Science, ETH ZĂŒrich
I am a PhD student at the Sensing, Interaction & Perception Lab at ETH ZĂŒrich with Christian Holz.
My ambition is to foster productivity in VR over prolonged periods of time. For this, I develop novel interaction techniques and learning-based perception algorithms by combining my experiences in machine learning with my background in electrical engineering.
Before starting my PhD, I completed a masterâs degree in Electrical & Electronic Engineering at Imperial College London and worked as an AI research intern at TikTok.
Please reach out via email if you want to know more about my work.
Todayâs Mixed Reality head-mounted displays track the userâs head pose in world space as well as the userâs hands for interaction in both Augmented Reality and Virtual Reality scenarios. While this is adequate to support user input, it unfortunately limits usersâ virtual representations to just their upper bodies. Current systems thus resort to floating avatars, whose limitation is particularly evident in collaborative settings. To estimate full-body poses from the sparse input sources, prior work has incorporated additional trackers and sensors at the pelvis or lower body, which increases setup complexity and limits practical application in mobile settings. In this paper, we present AvatarPoser, the first learning-based method that predicts full-body poses in world coordinates using only motion input from the userâs head and hands. Our method builds on a Transformer encoder to extract deep features from the input signals and decouples global motion from the learned local joint orientations to guide pose estimation. To obtain accurate full-body motions that resemble motion capture animations, we refine the arm jointsâ positions using an optimization routine with inverse kinematics to match the original tracking input. In our evaluation, AvatarPoser achieved new state-of-the-art results in evaluations on large motion capture datasets (AMASS). At the same time, our methodâs inference speed supports real-time operation, providing a practical interface to support holistic avatar control and representation for Metaverse applications.
@inproceedings{eccv2022-avatarposer,author={Jiang, Jiaxi and Streli, Paul and Qiu, Huajian and Fender, Andreas and Laich, Larissa and Snape, Patrick and Holz, Christian},title={AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing},year={2022},booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}}
TapType: Ten-finger text entry on everyday surfaces via Bayesian inference
Paul Streli,
Jiaxi Jiang,
Andreas Fender,
Manuel Meier,
Hugo Romat,
and Christian Holz
In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.
2022.
Despite the advent of touchscreens, typing on physical keyboards remains most efficient for entering text, because users can leverage all fingers across a full-size keyboard for convenient typing. As users increasingly type on the go, text input on mobile and wearable devices has had to compromise on full-size typing. In this paper, we present TapType, a mobile text entry system for full-size typing on passive surfacesâwithout an actual keyboard. From the inertial sensors inside a band on either wrist, TapType decodes and relates surface taps to a traditional QWERTY keyboard layout. The key novelty of our method is to predict the most likely character sequences by fusing the finger probabilities from our Bayesian neural network classifier with the characters' prior probabilities from an n-gram language model. In our online evaluation, participants on average typed 19 words per minute with a character error rate of 0.6% after 30 minutes of training. Expert typists thereby consistently achieved more than 25Â WPM at a similar error rate. We demonstrate applications of TapType in mobile use around smartphones and tablets, as a complement to interaction in situated Mixed Reality outside visual control, and as an eyes-free mobile text input method using an audio feedback-only interface.
@inproceedings{chi2022-taptype,author={Streli, Paul and Jiang, Jiaxi and Fender, Andreas and Meier, Manuel and Romat, Hugo and Holz, Christian},title={TapType: Ten-finger text entry on everyday surfaces via Bayesian inference},year={2022},isbn={9781450391573},publisher={Association for Computing Machinery},address={New York, NY, USA},doi={https://doi.org/10.1145/3491102.3501878},booktitle={Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems},numpages={1--16}}
TouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images
Karan Ahuja,
Paul Streli,
and Christian Holz
In Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology.
2021.
Today's touchscreen devices commonly detect the coordinates of user input using capacitive sensing. Yet, these coordinates are the mere 2D manifestations of the more complex 3D configuration of the whole handâa sensation that touchscreen devices so far remain oblivious to In this work, we introduce the problem of reconstructing a 3D hand skeleton from capacitive images, which encode the sparse observations captured by touch sensors. These low-resolution images represent intensity mappings that are proportional to the distance to the user's fingers and hands. We present the first dataset of capacitive images with corresponding depth maps and 3D hand pose coordinates, comprising 65,374 aligned records from 10 participants. We introduce our supervised method TouchPose, which learns a 3D hand model and a corresponding depth map using a cross-modal trained embedding from capacitive images in our dataset. We quantitatively evaluate TouchPose's accuracy in touch contact classification, depth estimation, and 3D joint reconstruction, showing that our model generalizes to hand poses it has never seen during training and that it can infer joints that lie outside the touch sensor's volume. Enabled by TouchPose, we demonstrate a series of interactive apps and novel interactions on multitouch devices. These applications show TouchPose's versatile capability to serve as a general-purpose model, operating independent of use-case, and establishing 3D hand pose as an integral part of the input dictionary for application designers and developers. We also release our dataset, code, and model to enable future work in this domain.
@inproceedings{uist2021-touchpose,author={Ahuja, Karan and Streli, Paul and Holz, Christian},title={TouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images},year={2021},isbn={9781450386357},publisher={Association for Computing Machinery},address={New York, NY, USA},doi={https://doi.org/10.1145/3472749.3474801},booktitle={Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology},pages={997â1009},numpages={13}}
CapContact: Super-Resolution Contact Areas from Capacitive Touchscreens
Paul Streli,
and Christian Holz
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.
2021.
(best paper award).
Touch input is dominantly detected using mutual-capacitance sensing, which measures the proximity of close-by objects that change the electric field between the sensor lines. The exponential drop-off in intensities with growing distance enables software to detect touch events, but does not reveal true contact areas. In this paper, we introduce CapContact, a novel method to precisely infer the contact area between the user's finger and the surface from a single capacitive image. At 8x super-resolution, our convolutional neural network generates refined touch masks from 16-bit capacitive images as input, which can even discriminate adjacent touches that are not distinguishable with existing methods. We trained and evaluated our method using supervised learning on data from 10 participants who performed touch gestures. Our capture apparatus integrates optical touch sensing to obtain ground-truth contact through high-resolution frustrated total internal reflection. We compare our method with a baseline using bicubic upsampling as well as the ground truth from FTIR images. We separately evaluate our method's performance in discriminating adjacent touches. CapContact successfully separated closely adjacent touch contacts in 494 of 570 cases (87%) compared to the baseline's 43 of 570 cases (8%). Importantly, we demonstrate that our method accurately performs even at half of the sensing resolution at twice the grid-line pitch across the same surface area, challenging the current industry-wide standard of a âŒ4mm sensing pitch. We conclude this paper with implications for capacitive touch sensing in general and for touch-input accuracy in particular.
@inproceedings{chi2021-capcontact,author={Streli, Paul and Holz, Christian},title={CapContact: Super-Resolution Contact Areas from Capacitive Touchscreens},year={2021},isbn={9781450380966},publisher={Association for Computing Machinery},address={New York, NY, USA},doi={https://doi.org/10.1145/3411764.3445621},booktitle={Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems},articleno={289},numpages={14}}
TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing
Manuel Meier,
Paul Streli,
Andreas Fender,
and Christian Holz
In 2021 IEEE Virtual Reality and 3D User Interfaces (VR).
2021.
(Accompanying demo received best demonstration award (juried)).
Current Virtual Reality systems typically use cameras to capture user input from controllers or free-hand mid-air interaction. In this paper, we argue that this is a key impediment to productivity scenarios in VR, which require continued interaction over prolonged periods of time-a requirement that controller or free-hand input in mid-air does not satisfy. To address this challenge, we bring rapid touch interaction on surfaces to Virtual Reality-the input modality that users have grown used to on phones and tablets for continued use. We present TapID, a wrist-based inertial sensing system that complements headset-tracked hand poses to trigger input in VR. TapID embeds a pair of inertial sensors in a flexible strap, one at either side of the wrist; from the combination of registered signals, TapID reliably detects surface touch events and, more importantly, identifies the finger used for touch. We evaluated TapID in a series of user studies on event-detection accuracy (F1 = 0.997) and hand-agnostic finger-identification accuracy (within-user: F1 = 0.93; across users: F1 = 0.91 after 10 refinement taps and F1 = 0.87 without refinement) in a seated table scenario. We conclude with a series of applications that complement hand tracking with touch input and that are uniquely enabled by TapID, including UI control, rapid keyboard typing and piano playing, as well as surface gestures.
@inproceedings{vr2021-TapID,author={Meier, Manuel and Streli, Paul and Fender, Andreas and Holz, Christian},booktitle={2021 IEEE Virtual Reality and 3D User Interfaces (VR)},title={TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing},year={2021},volume={},number={},pages={519-528},doi={https://doi.org/10.1109/VR50410.2021.00076}}
TapID: Rapid Touch Interaction in Virtual Reality using Wearable SensingIn 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 2021. (Accompanying demo received best demonstration award (juried)).