3 Programming with CRL

Applications using CRL can effect interprocessor communication and synchronization either through a relatively typical set of global operations (barrier, broadcast, reduce) or through operations on regions.

A region is an arbitrarily sized, contiguous area of memory identified by a unique region identifier. A region identifier is a portable and stable name for a region; in order to access a region, a processor node must know the region's region identifier. New regions can be created dynamically (using rgn_create) and can later be removed when they are no longer needed. Blocks of memory comprising distinct regions (those with different region identifiers) are non-overlapping.

In order to access the memory associated with a region, it must be mapped into the local address space. To map a region, one calls rgn_map with the region identifier of the region to be mapped; the function returns a pointer to the base of the region's data area.

CRL also provides operations for sharing regions. Specifically, CRL provides library calls that can be used to delimit operations---portions of the code where regions will be read from or written to. The library calls initiating and terminating operations are required to maintain the consistency of shared data. The coherence model provided by CRL considers entire operations on regions as indivisible units. From this perspective, CRL provides sequential consistency for read and write operations in the same sense that a sequentially-consistent hardware-based DSM does for individual loads and stores.

In order to use CRL functionality, application source files should include the file crl.h. When compiling, users of the TCP/UNIX implementation should make sure that the symbol TCPUNIX is #define-ed before including crl.h (e.g., before the #include or via a compiler directive) to ensure that slight differences in the CRL interface (see Section 6) necessary for the TCP/UNIX version are reflected in the include file. After compilation, application object files should be linked with the CRL object file (libcrl.o) to produce an executable.

3.1 CRL Functions

This section provides a brief description of the functionality provided by CRL 1.0.

3.1.1 Basic CRL Functions and Variables

void crl_init(void)
Initialize the CRL internals; crl_init should be called on all nodes simultaneously. crl_init should be called before any CRL functionality is utilized. (The TCP/UNIX version of CRL 1.0 uses a slightly different interface to crl_init; see Section 6 for details.)

unsigned crl_num_nodes
Contains the number of nodes in the system. ( This may be removed or altered in future releases.)

unsigned crl_self_addr
Contains the index of the node on which a thread is running. Each node has a unique index between 0 and crl_num_nodes-1; this index can be used to allow programs to perform different tasks on different nodes. ( This may be removed or altered in future releases.)

typedef unsigned rid_t
rid_t is a scalar type that is used to represent region identifiers; region identifiers are portable, unique `names' for regions. In this implementation, region identifiers are represented using (32-bit) unsigned integers.

3.1.2 Basic Region Functions

rid_t rgn_create(unsigned size)
Create a new region of size size bytes (initially resident on the calling node) and return a region identifier for the new region. The data area of newly created regions is not initialized.

void rgn_destroy(rid_t rgn_id)
Destroy the region named by region identifier rgn_id. ( In the current implementation, rgn_destroy does nothing.)

rid_t rgn_rid(void *rgn)
Return the region identifier of the region represented by rgn, where rgn is a pointer that was returned by rgn_map (see subsection 3.1.3).

unsigned rgn_size(void *rgn)
Return the size (in bytes) of the region represented by rgn, where rgn is a pointer that was returned by rgn_map (see subsection 3.1.3).

void rgn_flush(void *rgn)
Flush the local copy of the region represented by rgn, where rgn is a pointer that was returned by rgn_map (see subsection 3.1.3). When sensible to do so, rgn_flush both writes back any local changes to a region's home node and sends a protocol message relinquishing read ownership.

 3.1.3 Mapping and Unmapping Region Functions

void *rgn_map(rid_t rgn_id)
Map the region named by rgn_id into the address space of the local node. Return a pointer to the base of the region's data area; the pointer is guaranteed to be doubleword aligned. This pointer is also used to name a mapped region when initiating and terminating operations or unmapping the region. A region can be mapped multiple times on the same node.

The address at which a region is mapped into the local address space is only guaranteed to remain fixed for the duration of a mapping; subsequent mappings of the same region may yield different addresses. Further, two mappings of the same region on different nodes are not guaranteed to be at the same address. Because of this, the only globally portable and stable name for a region is the corresponding region identifier. Thus, applications that need to store a ``pointer'' to shared data (e.g., in another region as part of a distributed, shared data structure) must store the corresponding region's unique identifier (as returned by rgn_create), not the address at which the region is currently mapped. Subsequent references to the data referenced by the region identifier must be preceded by calls to rgn_map (to obtain the address at which the region is mapped) and followed by calls to rgn_unmap (to clean up the mapping).

void rgn_unmap(void *rgn)
Terminates the mapping represented by rgn, where rgn is a pointer that was returned by rgn_map. If a region is mapped multiple times on the calling node, rgn_unmap only terminates one `reference' to the mapped region.

3.1.4 Region Read and Write Operations

void rgn_start_read(void *rgn)
Initiate a read operation on the region represented by rgn, where rgn is a pointer that was returned by rgn_map. The read operation is considered to be in progress from the time rgn_start_read returns until a corresponding call to rgn_end_read; during this time, the contents of the region's data area may be safely read.

If any write operations are in-progress or pending on a region, rgn_start_read will block (not return) until they have been completed. Multiple read operations on the same region may be in progress simultaneously.

void rgn_end_read(void *rgn)
Terminate a read operation on the region represented by rgn, where rgn is a pointer that was returned by rgn_map. If multiple read operations are in progress on the calling node, rgn_end_read only terminates one of them.

void rgn_start_write(void *rgn)
Initiate a write operation on the region represented by rgn, where rgn is a pointer that was returned by rgn_map. The write operation is considered to be in progress from the time rgn_start_write returns until a corresponding call to rgn_end_write; during this time, the contents of the region's data area may be safely read or written.

If any read or write operations are in-progress or pending on a region, rgn_start_write will block (not return) until they have been completed. Because write operations are serialized with respect to all other operations on the same region, no other operations can be in progress during a write operation (including subsequent read or write operations initiated at the node with a write operation already in progress).

void rgn_end_write(void *rgn)
Terminate a write operation on the region represented by rgn, where rgn is a pointer that was returned by rgn_map.

3.1.5 Global Synchronization Functions

void rgn_barrier(void)
Participate in a global barrier with all other nodes. rgn_barrier does not return on any node until it has been called on all nodes.

void rgn_bcast_send(int len, void *buf)
Initiate a global broadcast to all other nodes of len bytes starting at address buf. All other nodes must call rgn_bcast_recv with the same len argument and a pointer to an appropriately-sized buffer.

void rgn_bcast_recv(int len, void *buf)
Receive a global broadcast initiated by another node. The value given for len must match that provided on the node initiating the broadcast and buf must point at an appropriately-sized buffer.

double rgn_reduce_dadd(double arg)
Participate in a global reduction. rgn_reduce_dadd does not return on any node until it has been called on all nodes. Returns the sum of the arg values provided on each node.

double rgn_reduce_dmin(double arg)
Participate in a global reduction. rgn_reduce_dmin does not return on any node until it has been called on all nodes. Returns the minimum of the arg values provided on each node.

double rgn_reduce_dmax(double arg)
Participate in a global reduction. rgn_reduce_dmax does not return on any node until it has been called on all nodes. Returns the maximum of the arg values provided on each node.

 3.1.6 Replacements for Standard Functions

CRL 1.0 provides replacement functions for malloc and free. This is necessary because CRL protocol message handlers use malloc to dynamically allocate memory (when necessary). Since message handlers can interrupt the ``normal'' computation running on a node and standard implementations of malloc and other memory allocation functions are not reentrant, this can lead to serious problems if care isn't taken to ensure that calls to malloc and free are executed atomically.

void *safe_malloc(unsigned nbytes)
When using CRL, calls to malloc should be replaced with calls to safe_malloc. Calls to other malloc-like functions (e.g., calloc, realloc) should either be replaced with calls to safe_malloc or by implementing new safe_xxx functions using safe_malloc as a template.

void safe_free(void *ptr)
When using CRL, calls to free should be replaced with calls to safe_free.