An introduction to distributed communication mechanisms

12 April 2021

Obed N Munoz

Cloud Software Engineer


Interprocesss Communication is at the heart of all distributed systems
Distributed Systems Book


Foundations (1/3)

Layered Protocols
Due the absence of shared memory, all communications in distributed systems is based on sending and receiving (low level) messages.

The OSI reference model

The OSI (Open Systems Interconnection) reference model identifies the various levels involved across all the interconnection levels and gives them standard names and points out which level should do which job.


Foundations (2/3)


Foundations (3/3)


Remote Procedure Calls (RPC) (1/2)

The proposal is to allow programs to call procedures located on other machines. When a process on machine A calls a procedure on machine B, the calling process on A is suspended, and execution of the called procedure takes place on B.

The idea behind RPC is to make a remote procedure call look as much as possible like a local one.


Remote Procedure Calls (RPC) (2/2)


Message-Passing communication

Message-passing is another mechanism to achieve inter-process communication. It's basically based on messages sharing between processes.

Many distributed systems will use the transport-level sockets to communicate. A socket is a communication end point to which an application can write data that are to be sent out over the underlying network, and from which incoming data can be read.

A socket forms an abstraction over the actual port that is used by the local operating system for a specific transport protocol.


Message-Passing communication (Socket opertions)

Servers generally execute the first four operations, normally in the order given. When calling the socket operation, the caller creates a new communication end point for a specific transport protocol.

In clients, a socket must first be created using the socket operation. The connect operation requires that the caller specifies the transport-level address to which a connection request is to be sent.


Advanced transient messaging

Sockets essentially support only TCP or UDP, meaning that any extra facility for messaging needs to be implemented separately by an application programmer.

In order to achieve more flexibility and separation of dutties in terms of communication, we need more advanced approaches for message-oriented communication.

Below some projects that are defining message-passing communication standards:


Scalability Protocols (nanomsg approach) 1/5

Scalability Protocols are basic blocks for building distributed systems. By combining them you can create a vast array of distributed applications.

Scalability protocols are layered on top of the transport layer in the network stack. At the moment, the nanomsg library supports the following transports mechanisms:

Nanomsg currently defines six different scalability protocols: PAIR, REQREP, PIPELINE, BUS, PUBSUB and SURVEY.


Scalability Protocols (nanomsg approach) 2/5

PAIR (Bidirectional Communication) implements simple one-to-one, bidirectional communication between two endpoints. Two nodes can send messages back and forth to each other.

REQREP (Client Requests, Server Replies ) defines a pattern for building stateless services to process user requests. A client sends a request, the server receives the request, does some processing, and returns a response.


Scalability Protocols (nanomsg approach) 3/5

PIPELINE (One-Way Dataflow) provides unidirectional dataflow which is useful for creating load-balanced processing pipelines. A producer node submits work that is distributed among consumer nodes.

BUS (Many-to-Many Communication) allows messages sent from each peer to be delivered to every other peer in the group.


Scalability Protocols (nanomsg approach) 4/5

PUBSUB (Topic Broadcasting) allows publishers to multicast messages to zero or more subscribers. Subscribers, which can connect to multiple publishers, can subscribe to specific topics, allowing them to receive only messages that are relevant to them.


Scalability Protocols (nanomsg approach) 5/5

SURVEY (Ask Group a Question) is similar to PUBSUB in that a message from one node is broadcasted to the entire group, but where it differs is that each node in the group responds to the message. This opens up a wide variety of applications because it allows you to quickly and easily query the state of a large number of systems in one go. The survey respondents must respond within a time window configured by the surveyor.



This material is genereated thanks to some extracts from the following resources:


Thank you

Obed N Munoz

Cloud Software Engineer

Use the left and right arrow keys or click the left and right edges of the page to navigate between slides.
(Press 'H' or navigate to hide this message.)