distrifein

Broadcast and Consensus Protocols

distrifein is a distributed systems project for learning purposes where I have currently implemented broadcast protocols like best effort, reliable and uniform reliable broadcast. Essentially p2p data broadcast(text, images) can be done reliably right now utilising p2p links implemented with TCP. I developed every component in a modular fashion with event bus for message passing between layers. Message passing between processes is handled by p2p links implemented via TCP. This project encapsulates critical concepts from distributed systems, networking, and event-driven architecture, providing a modular foundation for building reliable distributed applications.

  • Distributed Systems:
    • Links:
      • Reasonably Reliable P2P links with TCP(approximately Perfect Links)
    • Broadcast Methods:
      1. Best Effort Broadcast
      2. Reliable Broadcast
      3. Uniform Reliable Broadcast
    • Failure Detectors:
      1. Heartbeat FD
    • Event System
      • Shared Event Bus
      • Subscription and Publishing
  • Networking:
    • Channels with TCP based server: Handles incoming clients
    • TCP based transfer of data with sockets

Let’s start with the architecture of the project. The architecture is based on the concept of processes and layers. The processes are independent and communicate with each other via events. The layers are designed to be modular and only communicate via events.

The elements of a process are mentioned below where I assume we have crash stop processes:

We have these layers also in every process I run where the different layers communicate with each other via events which are managed by shared event bus. All components are designed to be modular and only communicating via events.

One of the positive things about this project it emulates this below mentioned architecture really well where each component is independent, modular and communicates via event based system. We have implemented atleast three broadcast methods namely: Best Effort Broadcast, Reliable Broadcast, Uniform Reliable Broadcast. We also have failure detection via heartbeat based method. Channels and links are implemented with TCP.

Date: 5/12/25

Finally I got around to transferring images via broadcast methods like best effort broadcast, reliable broadcast and uniform reliable broadcast. Image files are in PPM from which when path is specified it is read and send via the event system to low level components. For uniform reliable broadcast, it would be app->urb->beb->link and for reliable broadcast it would be app->rb->beb->link. For best effort broadcast, it would be app->beb->link. The image is then sent to the link layer and then to the other process via TCP.

Each of these methods have their own protocol albeit with the implementations utilizing the same modular components with the event system. There has been a major change in the message format where before the messsages were fragmented in sense that their were many types of messages. Now we have a single message format which handles all the types of messages. It offers a lot of flexibility and modularity to the project albeit with a bit of complexity and space.

Example: image transfer via uniform reliable broadcast :

Displays the image of the uniform reliable broadcast protocol in action where I specify the file to be sent. It first puts the image in pending for itself and then waits for acks from the other nodes to deliver it. SID is the current sender id and Org SID is the original sender id. We wait for acks for all the correct processes to deliver the image before being saved to the disk:

Results: image transfer via uniform reliable broadcast:

I had 3 processes running on my local machine, each with a different port number. The processes were able to communicate with each other and transfer images using the uniform reliable broadcast protocol. Each node had a unique ID and a folder to store the images received. The images were transferred successfully which is shown below:

The transferred files on node 1 are shown below, where each node sent 1 photo to all other nodes. Terrain photo was sent by Node 2, Dead Pixel Correction photo was sent by node 1 and finally the distrifein photo was sent by node 0.

Date: 6/7/25

Finally we have support for handling broadcasting for multiple nodes on different machines in the same LAN. This is a major change from before where broadcasting was limited to localhost. This is a great step towards truly p2p distributed system. Now my raspberry pi can communicate with my laptop and vice versa via text and images.

Now we can broadcast images from Raspberry Pi to Laptop and vice versa!

More details on the repository and logs here: 🔗 multi-node logs

References: