Author: eilemann@gmail.com
State: asynchronous readbacks implemented in 1.4 beta
Overview
Equalizer 1.2 implements synchronous pixel download and upload from the render threads using an external plugin API. This documents describes the extension of the plugin API and Equalizer to support asynchronous download and uploads, as per issue 64 and other issues referencing it.
Implementation
Asynchronous downloads are started from the render thread and completed in another thread (the current transmit thread), freeing the render thread to start with the next frame earlier. Asynchronous uploads are started on reception of the FrameData, and completed by the render thread(s), freeing the render thread from the actual upload time.
Each asynchronous operations causes a referencing of the rendering frame, which will be unreferenced after the operation has been completed. The last unreferencing will cause the frame to be released on the server.
When synchronous downloads are used, the asynchronous readback thread is not used. Setting ready of the frame and scheduling of the transmit operations is either effected by the render pipe thread when only synchronous readbacks are used, or by the readback thread after the last readback operation has been finished. The sequence diagram on the right shows a high-level overview of the operations.
Download
The download implementation is relatively
straight-forward: eq::Channel::frameReadback starts the download
operation using EqCompressorStartDownload, schedules a command on
the pipe download thread, which will finish the operation
using EqCompressorFinishDownload and set the ready
flag on the frame data. The download thread is created lazy on first use.
Most of the asynchronous download plugins will need a shared OpenGL context with the main render window. Other implementations may use a similar concept, e.g., a separate CUDA context for the finish operation. The download and upload threads will manage auxilary SystemWindows which share the context with the main window.
Upload
Asynchronous uploads will always need to go to a temporary buffer instead of directly to the framebuffer, since the upload needs to be started before the window is ready to receive the data.
TBD: Add destination pipe/window/channel information to FrameData
TBD: Impact on Image and Compositor code
Programming Interface
SystemWindow* eq::WindowSystem::createWindow( const WindowSettings& settings ); //? bool eq::WindowSystem::releaseWindow( SystemWindow* window );
Issues
1. How does the receiver node know which pipes need the received data?
The FrameData is received by the receiver thread and uncompressed by the command thread. The upload should then be started immediately, most likely by the command thread. Therefore it needs to know which GPUs will use the FrameData.
2. Who is responsible for the shared OpenGL context in the auxilary threads?
Resolution: The download and upload threads (cf. Issue 3) manage an auxilary eq::SystemWindow sharing with the main window.
The read finish operation in the transmit thread and the start upload
operation in the command thread optionally need a shared OpenGL context with
the corresponding render thread. The most consistent way is to have one
additional SystemWindow instance for each auxilary thread and
window, sharing with the main window.
3. Do we keep the model of having one transmit thread per node?
Resolution: Yes, but lazily create additional download and upload threads for each pipe thread.
Using one compression and decompression thread per node seems like the natural model to optimize CPU resources. Having one upload and download thread per render thread seems the natural model to optimize uploads and downloads.
4. How is the lifetime of the auxilary SystemWindows managed?
Resolution: Introduce create/destroy methods (see API changes above)
The 'owner' of the SystemWindows is the corresponding eq::Window. Consequently, it is responsible for their lifetime. The main system window is managed using configInit/ExitSystemWindow(). Due to historical reasons this is inconsistent with the NodeFactory create/release methods. New create/destroy methods will be used to create all system windows of a given eq::Window. The shareWindow parameter may be 0.
