Application Data Transmission

Author: eilemann@gmail.com
State: Design/Abandoned

Overview

The data transmission API is a simple interface to send data from the application to the render clients. It sends unstructured data directly to running nodes. The main purpose is to efficiently transmit the model database during initialization.

The implementation has been deferred, since most of its use can be implemented using distributed objects.

Design

The API has to provide an efficient way to send data. The data is not frame-synchronized, that is, latency is not accounted for. For frame-specific data the usage of versioned eq::net::Objects is highly recommended. Synchronization of data to config frames can also be implemented by the application, using the current frame number.

One approach is not to provide a new API, but to use the command packet API. This requires getting a list of nodes from the 'client' API in order to know where to send it to. The proposed API is easier to use and can implement optimizations later on.

The local (app) node also receives the broadcasted data. This allows for the same code to be deployed on the app node. Furthermore, when initializing the model, the application node might want to initialize a copy of the model as well. If not, it can always discard the received data.

The received data is enqueued by the receiver thread in a separate CommandQueue. The CommandQueue transfers the received packet, thus avoiding a data copy. To access the data from the enqueued Commands, a wrapper API around the CommandQueue is provided. Handling in the receiver thread without queueing can be implemented by overwriting the command handler (advanced usage).

Potential overlap with the to-be-defined frame data API (a.k.a. cull queues) exists. The frame data API will most likely require objects to provide additional information for frustum and range culling. It also requires frame synchronization and therefore queueing.

Usability during Config::init

As is, the data transmission API can not be used during Config::init, because the application is blocked until all initialization task methods have returned.

In order to allow data transmission during initialization, Config::init is split into startInit and finishInit, similar to the config's frame functions. The start method returns as soon as all the nodes have been constructed. The finish function blocks until all initialization functions have returned, and returns the success value of the config initialization. Config::init becomes a convenience function calling the start and finish method.

The pseudo-code for sending data during initialization is:

[Application:]
    config->startInit();
    for each init data
        config->broadcastData( data, size );
    config->finishInit();

[Render Clients]
    bool Node::configInit(...)
    {
        ...
        while not all init data received
            data = receiveData(...);
            process data
        ...
    }

Implementation

• Sender Implementation
  • config init reply returns list of node IDs
  • broadcast connects all nodes and caches list of nodes
  • broadcast sends ConfigDataPacket to all connected nodes
• Receiver Implementation
  • subclass CommandQueue into DataQueue
  • push Commands into DataQueue from receiver thread
  • return pointer and size of current Command from DataQueue

API

  void eq::Config::broadcastData( const void* data, uint64_t size );
  [void eq::Node::sendData( const void* data, uint64_t size );]
  const void* eq::Node::receiveData( uint64_t* size );
  const void* eq::Node::tryReceiveData( uint64_t* size );
  bool  eq::Node::hasData() const;

  virtual bool Config::startInit( const uint32_t initID );
  virtual bool Config::finishInit();

Restrictions

Sending data is only supported from the application node. If a reasonable use case for sending data from a render node is presented, this can be relaxed.

Open Issues

Late inits will require resend of the data.