The Dolphin drivers are designed to adapt to the environment they are operating in; therefore, manual configuration is rarely required. The upper limit for memory allocation of the low-level driver is the only setting that may need to be adapted for a cluster, but this is also done automatically during the installation.
Changing parameters in these files can affect reliability and performance of the Dolphin Express interconnect.
dis_irm.conf is located in the lib/modules directory of the DIS installation (default /opt/DIS) and contains options for the hardware driver (dis_irm kernel module). Only a few options are to be modified by the user. These options deal with the memory pre-allocation in the driver:
Changing other values in dis_irm.conf than those described below may cause the interconnect to malfunction. Only do so if instructed by Dolphin support.
Whenever a setting in this file is changed, the driver needs to be reloaded to make the new settings effective. Please note that some of the possible settings are commented out in the dis_irm.conf file. Please remove the leading # to change these settings.
These parameters control memory allocations that are only performed on driver initialization.
| Option Name | Description | Unit | Valid Values | Default Value |
|---|---|---|---|---|
| dis_max_segment_size_megabytes | Sets the maximum size of a memory segment that can be allocated for remote access. Some systems may lock up if too much memory is requested. | MB | integers > 0 | 4 |
| max-vc-number | Maximum number of virtual channels (one virtual channel is needed per remote memory connection; i.e. 2 per SuperSocket connection) | n/a | integers > 0 The upper limit is the consumed memory; values > 16384 are typically not necessary. | 1024 |
These parameters control some driver real-time settings. Changes here are normally only needed if you run a real time application or simulation using the SISCI API.
| Option Name | Description | Unit | Valid Values | Default Value |
|---|---|---|---|---|
| linkWatchdogEnabled | Controls the link watchdog behaviour. The link watchdog is a high availability feature to ensure detection of non operational links. This feature is normally not needed but should be left on for additional high availability. The feature introduces a microsecond level jitter. Should be turned off for real-time applications. | Seconds | 0 : Disabled. Integers > 0 : Watchdog period in seconds. | 10 |
| sessionHeartbeatsEnabled | Controls the session hearbeats. The session heartbeat mechanism is used for end to end internal heartbeating. This feature introduces a mocrosecond level jitter. Should be turned off for real-time applications | n/a | 0 : Disabled 1 : Enabled. | 1 - enabled |
Preallocation of memory is recommended on systems without IOMMU (like x86 and x86_64). The problem is the memory fragmentation over time which can cause problems to allocate large segments of contiguous physical memory after the system has been running for some time. To overcome this situation, options has been added to let the IRM driver allocate blocks of memory upon initialization and to provide memory from this pool under certain conditions for allocation of remotely accessible memory segments..
| Option Name | Description | Unit | Valid Values | Default Value |
|---|---|---|---|---|
| number-of-megabytes-preallocated | Defines the number of MiB memory the IRM shall try to allocate upon initialization. | MB | 0: disable preallocation >0: MB to preallocate in as few blocks as possible | 16 (may be increased by the installer script) |
| use-sub-pools-for-preallocation | If the IRM fails to allocate the amount memory specified by number-of-megabytes-preallocated it will by default repetively decrease the amount and retry until success. By enabling use-sub-pools-for-preallocation the IRM will continue allocate memory (possibly in small chunks), until the amount specified by number-of-megabytes-preallocated is reached. | n/a | 0: disable sub-pools 1: enable sub-pools | 1 |
| block-size-of-preallocated-blocks | To allocate not a single large block, but multiple blocks of the same size, this parameter has to be set to a value > 0. Pre-allocating memory this way is useful if the application to be run on the cluster uses many memory segments of the same (relatively small) size. | bytes | 0: don't preallocate memory in this manner > 0: size in bytes (will be aligned upwards to page size boundary) of each memory block. | 0 |
| number-of-preallocated-blocks | The number of block to be pre-allocated (see previous parameter) | n/a | 0: don't preallocate memory in this manner > 0: number of blocks | 0 |
| minimum-size-to-allocate-from-preallocated-pool | This sets a lower limit on the size of memory segments the IRM may try to allocate from the preallocated pool. The IRM will always request the system for resolving memory request less than this size. Due to the aspect of the need of the preallocation mechanism, there is a "hard" lower limit of one SCI_PAGE (currently 8K). The minimum size is defined in 1K blocks. | KB | 0: always allocate from pre-allocated memory > 0: try to allocate memory that is smaller than this value from non-preallocatd memory | 0 |
| try-first-to-allocate-from-preallocated-pool | Directs the IRM when to try to use memory from the preallocated pool. | n/a | 0: The preallocated memory pool becomes a backup solution, only to be used when the system can't honor a request for additional memory. 1:IRM to prefers to allocate memory from the preallocated pool when possible. | 1 |
| Option Name | Description | Unit | Valid Values | Default Value |
|---|---|---|---|---|
| link-messages-enabled | Control logging of non critical link messages during operation. | n/a | 0: no link messages 1: show link messages | 0 |
| notes-disabled | Control logging of non critical notices during operation. | n/a | 0: show notice messages 1: no notice messages | 1 |
| warn-disabled | Control logging of general warnings during operation. | n/a | 0: show warning messages 1: no warning messages | 0 |
| dis_report_resource_outtages | Control logging of out-of-resource messages during operation. | n/a | 0: no messages 1: show messages | 0 |
| notes-on-log-file-only | Control printing of driver messages to the system console | n/a | 0: also print to console 1: only print to syslog | 0 |
Configuration file for SuperSockets (dis_ssocks) kernel module. If a value different from the default is required, edit and uncomment the appropriate line.
#tx_poll_time=0; #rx_poll_time=30; #min_dma_size=0; #min_short_size=1009; #min_long_size=8192; #address_family=27; #rds_compat=0;
The following keywords are valid:
Transmit poll time [µs]. Default is 0, which means that the CPU does not spin at all when no buffers at the receiving side are available. Instead, it will immediately block until the receiver reads data from these buffers (which makes buffer space available again for sending). The situation of no available receive buffers rarely occurs, and increasing this value is not recommended.
Receive poll time [µs]. Default is 30. Increasing this value may reduce the latency as the CPU will spin longer to wait for new data until it blocks sleeping. Reducing this value will send the CPU to sleep earlier, but this may increase message latency.
Minimum message size for using DMA (0 means no DMA). Default is 0.
Minimum message size for using SHORT protocol. Default and maximum is 1009.
Minimum message size for using LONG protocol. Default is 8192.
AF_SCI address family index. Default value is 27. If not set, the driver will automatically chose another index between 27 and 32 until it finds an unused index. The chosen index can be retrieved via the /proc file system like cat /proc/net/af_sci/family . If this value is set explicitly, this value will be chosen, and no search for unused values is performed.
Generally, this value is only required if SuperSockets should be used explictely without the preload library.
RDS compatibility level. Default is 0.