Identifying Threads Used by RTI Connext DDS Professional

 

Note: This article applies to Connext DDS Professional 4.x and above.

RTI Connext DDS Professional uses multiple internal threads for sending and receiving data, maintaining internal state, and calling user code when events occur. This solution explains how these threads can be identified in your system.

Threads used by Connext DDS Professional

A DomainParticipant always creates the following three threads:

  • Database Thread maintains the database of DDS Entities stored in the DomainParticipant. It is responsible for purging the objects marked for deletion when they are no longer needed.
  • Event Thread detects triggered events and acts accordingly, invoking user functions when needed (e.g., when a callback was specified for that specific event).
  • Receive Threads get bytes from transport plugins, then deserialize and store the (meta)data in the receive queue of a DataReader and invoke the  on_data_available() callback. Receive threads are also responsible for processing meta-data (e.g., discovery traffic, ACKs, NACKs).

The actual number of threads depends on the configuration of various QosPolicies as well as the implementation of the transports used by the DomainParticipant to send and receive data. In addition, other threads might be created for specific purposes:

  • Interface Tracking Thread checks if there is new interface data. It notifies the user/application and updates the interface data. Note: this thread is created in support of the IP Mobility feature introduced in 5.3.0.
  • Transport-specific Threads handle the tasks that are specific for a transport (e.g., the TCP Transport plugin creates two threads, one for control and one for events).
  • Asynchronous Publishing Threads handle the data transmission for Asynchronous Publishers.
  • Asynchronous Batch Flushing Threads handle batches of data samples, flushing them when needed. These threads are only created when batching is enabled (see BATCH QosPolicy (DDS Extension)) and  max_flush_delay is different than DURATION_INFINITE.
  • Topic Query Publication Threads publish historical samples in response to a TopicQuery. These threads are only created when topic query dispatch is enabled (see TOPIC_QUERY_DISPATCH_QosPolicy (DDS Extension)). Note: this thread is created in support of the Topic Query feature introduced in 5.3.0.
  • User Threads: in addition, your application may present threads that are not part of Connext DDS Professional. If those threads call a DDS API, Connext DDS Professional will automatically register them (allocate some resources to keep statistics and to handle concurrent access to DDS entities). To free up all the allocated resources, you may need to unregister these threads, as explained in this article from our Knowledge Base.

Further details regarding RTI Connext’s threading model can be found in the RTI Connext DDS Core Libraries User’s Manual (Connext DDS Threading Model section).

How to identify RTI Connext DDS Professional threads in your application

Checking the thread names from the call stack

In Connext DDS Professional 6.0.0 and previous versions, thread names are only available in a subset of architectures. This section lists the correspondence between RTI Connext DDS Professional threads and the functions they run. That way you can use this information to identify RTI Connext DDS Professional threads from the call stack, independently of your architecture. If you are using VxWorks or Integrity, please refer to the section “Checking the thread names at the OS level” below.

This is the correspondence between threads and the functions they run:

  • Database ThreadRTIEventActiveDatabaseThread_loop()
  • (Main) Event Thread: RTIEventActiveGeneratorThread_loop(). Note that this function is generic to all the event threads. That is, all of the event threads run RTIEventActiveGeneratorThread_loop(), which detects and handles events.  For this reason, it can be difficult to distinguish the Main Event Thread from other event threads (such as the Topic Query Publication Event Thread); however, to better make this distinction, you can check whether some (sub)functions are called (for example, the subfunctions related to the Asynchronous Batch Flushing Event Thread and Topic Query Publication Even Thread below).
  • Receive Thread: COMMENDActiveFacadeReceiver_loop(), which calls to a different function depending on what transport is being used to get the (meta)data:
    • Shared Memory: NDDS_Transport_Shmem_receive_rEA()
    • UDP: NDDS_Transport_UDP_receive_rEA()
    • TCP: NDDS_Transport_TCP_receive_rEA()
  • Interface Tracking Thread: RTIOsapiInterfaceTracker_()
  • Transport-Specific Threads:
    • TCP Control Thread: NDDS_Transport_TCPv4_Plugin_threadLoop()
    • TCP Event ThreadRTIEventActiveGeneratorThread_loop() and NDDS_Transport_TCPv4_Plugin_clientOn<event_name>()
  • Asynchronous Publishing Thread: RTIEventJobDispatcherThread_spawnedFnc()
  • Asynchronous Batch Flushing Event ThreadRTIEventActiveGeneratorThread_loop() and PRESPsWriter_onFlushBatch()
  • Topic Query Publication Event ThreadRTIEventActiveGeneratorThread_loop() and PRESPsService_onWriterServiceDispatchActiveTopicQueriesEvent()

As an example, if you are on GNU/Linux, you can run the following command on gdb to get the call stack:

(gdb) thread apply all backtrace

The same information can be seen with Visual Studio, too. To see this information in Visual Studio, you need to select Debug->Windows->Threads, and then do Ctrl+D, T. You will need to add a breakpoint and start the application in debug mode.

 Checking the thread names at the OS level

Note: Currently this method is only available on VxWorks and Integrity.
In future releases, these names will be also available on the following operating systems: Linux, Darwin, iOS, QNX, Windows.

On some systems, it is possible to check the internal name of RTI threads directly at the operating system level. In general, thread names follow this pattern:

 <code>r&lt;tastkType&gt;&lt;domainId&gt;&lt;id&gt;[t&lt;threadIndex&gt;]</code> 

Where:

  • r - indicates this is a thread from RTI
  • <taskType> - the type of thread:
    • Evt - Event Thread
    • Dtb - Database Thread
    • Dsp - Dispatcher (i.e., Asynchronous Publishing) Thread
    • Rxx - Receive Thread, where  xx is the Receive Thread Number (2-digit decimal)
    • Bat - Asynchronous Batch Flushing Thread
    • Tqd - Topic Query Publication Thread
  • <domainId> - the Domain ID (3-digit decimal)
  • <id> - the Participant Index for your DomainParticipant, in one of two different formats:
    • Participant Index set automatically by DDS (most use cases): <applicationId> (hexadecimal)
    • Participant Index set manually via the participant_id (WIRE_PROTOCOL QosPolicy): <participantIndex> (decimal)
  • t<threadIndex> (only for Asynchronous Publishing Threads) - index used to distinguish between asynchronous publishing threads when multiple Publishers are present in a single DomainParticipant (decimal)

The following threads have names that do not follow the above-mentioned convention:

  • RTIOsapiInterfaceTracker_<threadType> - IP Mobility (i.e., Interface Tracking Thread) where  threadType can be either  pollinThread or notification.
  • NDDS_Transport_TCPv4 - TCP Control Thread.

The details on checking the thread names depends on the operating system. The following is an example output from a publisher application running on VxWorks 6.7:

-> taskSpawn "test", 10, 0x8, 150000, publisher_main, 208, 100
value = 33769616 = 0x2034890
-> i

  NAME         ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
----------  ------------ -------- --- ---------- -------- -------- ------- -----
[...]
rR00208529> 3c8e73d       1ba64a4  71 PEND         466340  1bc42c0   3006b     0
rR01208529> 3c8e73d       1bd7a38  71 PEND         466340  1bf5850   3006b     0
rR02208529> 3c8e73d       1bf8288  71 PEND         466340  1c28ce0   3006b     0
rR03208529> 3c8e73d       1bf894c  71 PEND         466340  1c59c60   3006b     0
rR04208529> 3c8e73d       1c6e2d8  71 PEND         466340  1c8c1f0   3006b     0
rR00208529> 3c8e73d       24b86f4  71 PEND         466340  24d6510   3006b     0
rR01208529> 3c8e73d       24e9c74  71 PEND         466340  2507a90   3006b     0
rR02208529> 3c8e73d       251b504  71 PEND         466340  2539420   3006b     0
rR03208529> 3c8e73d       254ce14  71 PEND         466340  256ac30   3006b     0
rR04208529> 3c8e73d       256d7cc  71 PEND         466340  259eb00   3006b     0
rEvt208529> 3c8e73d       1511ae4 110 PEND+T       466340  4063380  3d0004    43
rEvt208529> 3c8e73d       20b1c14 110 PEND+T       466340  20cfc30  3d0004    22
rDtb208529> 3c8e73d       153bc48 120 PEND+T       466340  4017070  3d0004  1546
RTIOsapiIn> 3c8e73d       3ffb810 120 PEND+T       466340  1b92de0  3d0004    13
rDtb208529> 3c8e73d       2079540 120 PEND+T       466340  2080d70   3006b   926
RTIOsapiIn> 3c8e73d       20625ac 120 READY        466340  24a4fa0  3d0004     0
value = 0 = 0x0

Checking the thread names using the Worker’s name

Connext DDS uses the concept of worker as an abstraction for threads. Workers are RTI-specific entities used internally to manage critical sections and to provide access to thread-specific storage. Most of the threads created by Connext DDS have an associated worker. In addition, user threads calling certain APIs from Connext DDS will have a worker associated with them. Workers are given a name when they are created. If you have the proper debug symbols, you can use the worker’s name to identify the thread (on a debugger, for instance).

To check the workers’ name, first we need to locate these workers in the threads. You can do that by selecting a thread and printing its full backtrace. Another option is moving up and down through the frames on the thread’s stack. The worker will be either a local variable or the last argument to one of the RTI functions. Here is an example using gdb on GNU/Linux to identify a thread with the method just described:

(gdb) info thread
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fce700 (LWP 6801) "HelloWorld_publ" __clock_nanosleep (clock_id=<optimized out>, flags=0, req=0x7fffffffcb20, rem=0x7fffffffcb30) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
  2    Thread 0x7ffff6ec1700 (LWP 6805) "HelloWorld_publ" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  3    Thread 0x7ffff66c0700 (LWP 6806) "HelloWorld_publ" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
[...]

(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff6ec1700 (LWP 6805))]

(gdb) backtrace full
[...]
#3  0x0000000000c6095b in RTIEventActiveDatabaseThread_loop (param=0x13fc8c0) at ActiveDatabase.c:156
        timeStr = 0x7ffff6ec0dc0 "{0000003d,00000000}"
        t = 0x13fc8c0
        canBeDeleted = 0
        timeBuf = "{0000003d,00000000}"
        workerName = 0x1351cb0 "rDtb2081a9101"
        METHOD_NAME = 0x100a0d0 "RTIEventActiveDatabaseThread_loop"
[...]

As you can see in the example, workers follow the same naming convention as threads (in some cases, a shortened version of it). Workers associated with user threads use the following convention: U<threadId>, where:

  • U - indicates that this is a User Thread
  • <threadId> - ID given to the thread by the OS
Keywords:

Comments

thnks, Will try it today.