File: OpenXR.m

package info (click to toggle)
psychtoolbox-3 3.0.19.14.dfsg1-1
  • links: PTS, VCS
  • area: main
  • in suites: forky, sid, trixie
  • size: 86,796 kB
  • sloc: ansic: 176,245; cpp: 20,103; objc: 5,393; sh: 2,753; python: 1,397; php: 384; makefile: 193; java: 113
file content (203 lines) | stat: -rw-r--r-- 11,266 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
% OpenXR - How to set up and use Psychtoolbox OpenXR support.
%
% Since Psychtoolbox version 3.0.19, Psychtoolbox supports the modern
% Khronos open, cross-platform, cross-vendor, cross-device OpenXR api for
% implementing (V)irtual (R)eality VR, (A)ugmented (R)ealtiy AR, and/or
% (M)ixed (R)eality MR applications, summarized under the umbrella term
% e(X)tended (R)eality XR, hence the name OpenXR. See:
% https://www.khronos.org/openxr
%
% Unless otherwise specified by a users script, the PsychVRHMD('AutoSetupHMD')
% function from now on will try to use a system installed OpenXR runtime to
% run VR applications, with fallbacks to the older legacy drivers like
% PsychOculusVR1 for recent Oculus devices on MS-Windows, PsychOculusVR for
% Oculus Rift DK-1/DK-2 on Linux/X11 and MS-Windows, or to PsychOpenHMDVR
% on Linux/X11. The driver is designed to be reasonably backwards compatible,
% so most scripts should continue to work unmodified "plug & play".
%
% In the current release, we support the XR subset of VR virtual reality
% applications by use of any VR Head mounted display (VR-HMD) and for VR
% input devices which are supported by a OpenXR 1.0 compliant runtime that
% provides the following minimum set of OpenXR 1.0 extensions:
%
% - All: XR_KHR_opengl_enable, XR_EXT_debug_utils, and XR_KHR_composition_layer_depth.
% - Additionally on MS-Windows: XR_KHR_WIN32_convert_performance_counter_time.
% - Additionally on Linux/Unix: XR_KHR_convert_timespec_time.
% - Optional, but not tested without it: XR_FB_display_refresh_rate.
% - Optional, for improved input controller support: XR_EXT_hp_mixed_reality_controller,
%   XR_HTC_vive_cosmos_controller_interaction, XR_HTC_vive_focus3_controller_interaction,
%   XR_KHR_binding_modification, XR_EXT_dpad_binding.
% - Optional, for basic eye gaze tracking: XR_EXT_eye_gaze_interaction.
%
% On VR HMDs with a builtin eye gaze tracker, our OpenXR driver is also
% capable of using such hardware to track and report eye gaze. Basic eye
% gaze tracking should work with any HMD and OpenXR runtime that supports
% the optional OpenXR XR_EXT_eye_gaze_interaction extension. This allows
% reporting of a "cyclops eye" gaze vector and 2D gaze position, possibly
% synthesized from separate eye gaze measurements on a binocular eye
% tracker, or the single measurement from a monocular eye tracker.
% Additionally, on MS-Windows only, with certain HTC HMDs only, more
% advanced eye gaze tracking is supported via HTC's SRAnipal eye tracker
% runtime on suitable HMDs, e.g., the "HTC Vive Pro Eye". This allows
% additionally for binocular eye gaze reporting, reporting of eye opening,
% and estimated eye pupil diameter. Some demos have support and demonstration
% of eye gaze tracking. E.g., GazeContingentDemo.m for gaze contingent
% manipulation of 2D monoscopic stimuli, VRHMDDemo.m for 2D tracking and
% visualization of gaze wrt. monoscopic or stereoscopic stimuli.
% VRInputStuffTest.m shows gaze tracking wrt. 3D VR stereoscopic scenes,
% both showing 2D gaze position and 3D gaze rays. VREyetrackingTest.m
% allows to test and assess various aspects of 2D gaze tracking of 2D or 3D
% stimuli presented monoscopically or stereoscopically.
%
% So far successfully tested with the PTB 3.0.19.5/3.0.19.6 release are:
%
% - The open-source Monado(XR) runtime version 21.0.0 for Linux/X11, as shipping
%   with Ubuntu 22.04-LTS and later, or as a 3rd party ppa for Ubuntu 20.04-LTS,
%   as well as part of Debian GNU/Linux 12/unstable/testing. See the following link
%   for more information about Monado:
%
%   https://monado.freedesktop.org
%
%   This has been tested on Ubuntu 20.04.6-LTS and 22.04-LTS with AMD and NVidia
%   gpu's so far.
%
% - The proprietary Valve SteamVR runtime version 1.24.7 and 1.25.7 on Linux
%   (Ubuntu 20.04.6-LTS and 22.04.3-LTS), and SteamVR 1.25.7 and 2.0 on Microsoft
%   Windows 10 21H2 - 22H2.
%
% - The proprietary OculusVR runtime version 1.81.0 on Microsoft Windows 10
%   21H2 - 22H2.
%
% Testing so far occured with a OculusVR Oculus Rift CV-1 HMD with 2 Oculus
% tracking cameras and 2 Oculus touch controllers, as well as a Oculus
% Remote control, and a Microsoft XBox 360 gamepad controller. Additionally
% testing was performed under Monado and SteamVR with a HTC Vive Pro Eye
% and its two "Vive Wand" controllers, tracked by a Valve Lighthouse
% version 2 system with two Vive lighthouse emitter stations.
%
% Eye gaze tracking was tested under Windows 10 22H2 under SteamVR 1.25.6
% and and SteamVR 2.0 and HTC SRAnipal SDK 1.3.6.8, under Octave 7.3 and
% Matlab R2022b and R2023b, using a HTC Vive Pro Eye VR headset. Both OpenXR
% "cyclops" gaze tracking, and also binocular gaze tracking with eye opening
% and pupil diameter reporting worked well.
%
% A limitation of the current OpenXR spec is that it doesn't provide any
% means for reliable, robust, trustworthy, accurate and precise visual
% stimulus onset timestamping. We are investigating a future solution for
% reliable and trustworthy timestamping for the open-source MonadoXR
% runtime on Linux and hope to find a solution there in the foreseeable
% future, stay tuned. For now, as of Psychoolbox 3.0.19.6 we have a hack
% that only works on a modified version of Monado + a modified version of
% Mesa, on AMD or Intel gpu's under Linux. The hack has various restrictions
% and it impacts performance. It is also not quite plug and play to set up.
% See 'help PsychOpenXR', the section about the Monado metrics hack, for
% instructions.
%
% Testing also showed that all tested proprietary OpenXR runtimes, ie.
% both OculusVR and SteamVR, violate the OpenXR specs stimulus timing
% requirements, as of December 2023. The only exception was the open-source
% Monado(XR) runtime for Linux.
%
% The same limitations are true for the old OculusVR runtimes on
% MS-Windows. To get at least approximately correct timestamps, the driver
% therefore will switch to a multi-threaded mode of operation if it detects
% the need for timestamping or timing, or if that need is specified with
% new 'basicRequirements' keywords to PsychVRHMD('AutoSetupHMD') or to
% PsychVRHMD('SetupRenderingParameters'), e.g., 'TimestampingSupport' for
% timestamps, or 'TimingSupport' for onset timing. The switch to
% multi-threading will cost some performance and possibly introduce extra
% latency. In the case of SteamVR on MS-Windows it may even cause bugs and
% hangs in some cases. Therefore additional keywords like
% 'NoTimestampingSupport', 'NoTimingSupport', or 'ForbidMultiThreading'
% allow your script to specify also if it doesn't need precise timing or
% timestamping or does not want multi-threading to be used.
%
% Testing showed that MonadoXR was the most reliable and bug-free runtime,
% whereas both OculusVR and SteamVR exposed various other serious bugs. Our
% driver tries to work around such known bugs on those runtimes, sometimes
% by use of multi-threading, which impacts performance. Therefore various new
% keywords beyond the ones mentioned above exist to control these
% quality/reliability vs. performance tradeoffs for your specific script
% and paradigm.
%
% 'help PsychVRHMD' lists those new keywords in the section for
% 'AutoSetupHMD', and the 'help PsychOpenXR' sometimes gives more detailed
% infos.
%
% If you need precise timing at all costs, potentially to the detriment of
% most other functionality, performance or quality, there is also the
% keyword 'TimingPrecisionIsCritical' to specify, in addition to the other
% timing/timestamping keywords. This keyword will force the selection of
% the driver with the highest possible timing precision/reliability. At the
% moment this means to probe for the PsychOculusVR driver for the old
% Oculus v0.5 runtime for Linux/X11 and MS-Windows, only usable for the
% original Oculus Rift developer kits DK-1 and DK-2. Then a fallback to a
% potentially timing enhanced MonadoXR implementation, once such a thing
% exists. Then back to standard OpenXR as a last resort measure, in which
% case timestamps will not be reliable or trustworthy at all!
%
%
% Basic Setup:
% ============
%
% MS-Windows:
% -----------
%
% - Oculus: If you bought and set up a Oculus HMD, then the OculusVR-1
%   OpenXR runtime will have been installed and setup already and should
%   just work(tm) with modern Oculus devices like Rift DK1/DK2/CV1/S and
%   the new Meta Quest devices and associated controllers.
%
% - SteamVR: The same should be true for SteamVR supported HMDs if you
%   followed the setup instructions, e.g., for the Valve Index HMDs or
%   early HTC Vive HMDs, including the Vive Pro and Vive Pro Eye.If you
%   chose a (W)indows(M)ixed(R)eality WMR-HMD, you need to install SteamVR
%   and set it up as OpenXR runtime for those HMDs, as the Microsoft
%   Windows built-in WMR OpenXR runtime does not support OpenGL interop, so
%   SteamVR is needed as a middle-man and translator between Psychtoolbox
%   OpenGL rendering and WMR's Direct3D only rendering.
%
%   If you want to use binocular eye tracking via SRAnipal on HTC HMDs with
%   eye tracker, the following URLs may provide useful setup and getting
%   started instructions:
%
%   - HTC official guide: https://developer.vive.com/resources/openxr/openxr-pcvr/tutorials/how-use-sr_runtime
%   - Much more detailed webpage, very well done by some motivated volunteer:
%     https://github.com/corycorvus/Eye-Tracking-Wiki
%
% - Other OpenXR runtimes exist from HTC for their latest devices, or from
%   other vendors like Varjo for their devices.
%
% Linux/X11:
% ----------
%
% - MonadoXR is provided via 'sudo apt install monado' on Ubuntu 22.04-LTS
%   and later, Debian GNU/Linux 11 and later, and probably other distros.
%   Also as a 3rd party ppa for Ubuntu 20.04-LTS, but we now recommend
%   using at least Ubuntu 22.04-LTS. Note that for some HMDs, e.g., the
%   Oculus Rift CV-1, you also need OpenHMD, and potentially build Monado
%   from source against OpenHMD. See the "supported hardware" section on
%   Monado's website, for natively supported devices, and for devices that
%   additionally need OpenHMD, and potentially building Monado from source
%   code against an installed libOpenHMD. The HTC Vive devices do have
%   basic 3 degree of freedom orientation tracking support in Monado, but
%   for full 6 degrees of freedom tracking you will need to install
%   libsurvive and compile Monado from source against libsurvive, as an
%   external more capable tracking library at the moment.
%
% - SteamVR can be installed to use SteamVR supported HMD's, e.g., HTC
%   Vive, Valve Index, Oculus Rift. Follow setup instructions after
%   installing SteamVR. Use of Oculus devices needs MonadoXR as a SteamVR
%   driver plugin on Linux (see setup instructions under
%   https://monado.freedesktop.org/steamvr.html). When displaying in
%   'Monoscopic' or 'Stereoscopic' 2D mode, it has been shown beneficial at
%   least on Linux with Oculus Rift, to disable asynchronous reprojection,
%   as this reduces jitter and tracking noise. HTC Vive devices do not need
%   extra external Monado plugins for use with SteamVR.
%
% macOS:
% ------
%
% No OpenXR (or other Psychtoolbox supported) virtual reality runtime
% exists on Apples iToys operating system.
%