Noon

Noon is a free, open source, secure communication platform. For any one party or organization interested in having their own secure communication platform that internalizes all data and communication and does not rely on any closed/non-open-source third party service or package for any of its functionality. The primary development objective that I had in mind when building the platform was to see whether I could build all the systems and the overall infrastructure required by hand, while learning Typescript/Graphql/React in the process. As for the systems that I'm too dumb to build or would require an inordinate amount of time to develop (like properly functioning video conferencing) I would solely rely on open source packages and libraries. From authentication, to the real time communication, to the presence system, etc... all would be built by me or would be provided by established open source packages/libraries. The only closed third parties that I would have to rely on of course would be the infrastructure providers. For this I went with Digital Ocean for the backend and its microservices and Vercel for the client.

I started prototyping with the functionality of Noon in 2018-2019 using Laravel and Vuejs. I stopped working on it due obligations with work, freelance projects and the country (I'm from Lebanon) absolutely going up in flames in late 2019. A severe economic crisis brought on by a ruling mafia that stole the Lebanese's savings. All my, and my parents savings were locked in the banks, unable to be retrieved. Running out of cash on hand and with my freelance projects all brought to a screeching halt due to that same liquidity crisis, I urgently had to find employment and a steady source of income and was lucky to be picked up by Myki in early 2020.

I picked up work on it again early in 2022, this time deciding that since I was very much interested in learning React/Graphql/Typescript it would be a great opportunity to continue on the platform using that stack. And since I, at that point had the necessary experience with production JS having spent the last 2 years maintaining and refactoring Myki's Nodejs backend which served hundreds of thousands of active users, I took the plunge and restarted development. I owe a considerable amount to Ben Awad and his tutorials for helping me make sense of how to manoeuvre the stack and ecosystem especially as it relates to everything Graphql. I must have finished his 14 hours full stack tutorial 4 times. The first 2 times because I didn't know what the fuck was going on and was just following and writing along in semi zombie mode.

Noon provides the features that people have come to expect from traditional communication and messaging platforms. From private/group chat, to media sharing in conversations, to video conferencing, etc... It is very much still in a very early stage of development, about 60% to 65% done with a lot of work still to be done on optimizing, refactoring and testing. Most the vital components are functional; from the communication, authentication, presence systems to the video conferencing. The main emphasis of development as of 31-01-2022 is on implementing E2EE using Signal's algorithm. The platform is live on https://noon.tube.

In the following sections, I will attempt to explain the various parts that make up the platform.

Noon's internals

The Noon backend is built using Node.js and Express for the server-side runtime environment and routing. It is mostly - refactor into TS ongoing - written in TypeScript. It utilizes Apollo Client for handling GraphQL queries and mutations, allowing for efficient and flexible data retrieval and manipulation. Real-time functionality is implemented using Socket.io for bi-directional communication between the client and server. Redis, as well as being as a caching layer, is also being used to store user sessions when users authenticate as well as Socket.io sessions and messages. PostgreSQL is used as the primary relational database for storing user and conferencing data. Neo4j is utilized for handling profile relationships, allowing for easy querying and traversal of complex social graphs. Elasticsearch is integrated for fast and powerful search capabilities. All of these technologies work together to provide a robust and scalable backend for handling a variety of features such as user management, social networking/conferencing, and search functionality.

The Noon backend is designed as a set of microservices, rather than a monolithic application. The microservices communicate with each other using RabbitMQ as a message broker, utilizing the AMQP protocol. Each microservice is responsible for a specific set of functionality, allowing for increased scalability and maintainability. The microservices include:

• The Media microservice

  This service is implemented using JavaScript, with Nodejs as the runtime environment and Express to serve the static media content. The media microservice is responsible for handling all media-related functionality such as image and audio uploads and processing. It utilizes the sharp package to handle all image related data.

  All media is written to the file system of the virtual private server (VPS) where the microservice exists.

  I have defined routes that correspond to the URLs where the media files are stored. When a user requests a media file by accessing the corresponding URL, Express intercepts the request and retrieves the corresponding media file from the file system. Once the file is retrieved, Express sends it to the user's browser, allowing them to view or download the media.

  In order to more easily adjust the size of the VPS to handle increases in load, I am planning to containerize the media microservice using Docker.

  Once the media microservice is containerized, I plan on using Kubernetes to manage the deployment and scaling of the service. This will allow me to easily adjust the size of the VPS to handle increases in load by scaling the number of replicas of the container.

  To protect media access, I have used nginx as a reverse proxy. By using nginx as a reverse proxy, I can control access to the media by setting up authentication and access control. This ensures that only authorized users can access the media, and helps to protect against unauthorized access and data breaches.

  Overall, by using Express in conjunction with other technologies such as Sharp and Nginx, I have created a powerful and scalable media microservice that can handle a large number of media files, providing a smooth and responsive user experience while ensuring the security and protection of the media.

• The Search microservice

  This service is built using Elasticsearch and is responsible for providing powerful search functionality for the application. Like other design decisions pertaining to the development of Noon, I opted for ES instead of managed services like Algolia. A lot of these design decisions aren't just about an ideological commitment to open source, but there are also fairly practical and technical considerations underpinning them, especially when it comes to scaling and how it is associated to cost. My experience building similar systems has shown me that while these managed services make it easy to experiment and prototype, they actually hold a platform's growth back because the underlying cost of their use is significant. Eventually necessitating re-engineering and refactoring; by deploying a home grown solution.

• The Relay microservice

  This service is built using Javascript and is responsible for relaying emails and notifications to the users of Noon.

• The Noon remote procedure call package

  The remote procedure call (RPC) package is a custom-built package written in Typescript that handles all communication between the microservices. It has two main modules: the client module and the server module.

  The client module: This module is responsible for sending messages to queues. It allows the microservices to make requests to other microservices by sending a message to a specific queue. The message contains the information needed for the other service to process the request, such as the method to be called and any necessary parameters.

  The server module: This module listens to the queues and responds to the messages by sending ACKs (acknowledgements) to the corresponding queues. The ACKs contain the results of the requested operation or any errors that may have occurred.

  Both the client and server modules use RabbitMQ as the message broker and AMQP protocol to communicate. The package is designed to be lightweight and easy to use, making it simple to add new microservices and to communicate between them.

  Additionally, the package utilizes the amqp-connection-manager package which enables retries and timeouts. If the broker is not able to respond within the specified time, the package will retry the request after a certain interval of time. This feature makes the Noon RPC package more robust and fault-tolerant.

  Overall, the Noon RPC package allows for decoupled and efficient communication between the microservices, enabling them to work together seamlessly and providing a smooth and responsive user experience.

• The Neo4j Service

Lets draw on an an example that illustrates how the graph DB fits into the platform. We'll have one user who is attempting to add another user as a friend: it starts with the user searching for the profile who they want to add as a friend. After getting the ES results, the user sends a friend request:

await sendFriendRequest({
  variables: {
    profileUuid: profile.uuid,
  },
})

where `profileUuid` is the uuid of the profile who is going to receive the friend request. The resolver receives the request and calls the graph API with sender and receiver details. The method then initiates the following Cypher query:

tx.run(
  MATCH (p1:Profile {uuid: $sUuid}) 
  MATCH (p2:Profile {uuid: $rUuid}) 
  MERGE (p1)-[friendRequest:FRIEND_REQUEST 
  {uuid: $recipientProfileUuid, username: $recipientProfileUsername }]->(p2) 
  RETURN p1, friendRequest, p2,
  {
    sUuid: senderProfileUuid,
    rUuid: recipientProfileUuid,
    recipientProfileUuid,
    recipientProfileUsername,
  }
)

A FRIEND_REQUEST relationship is created between the two profiles, going from profile1 to profile2, and the details of the receiving profile added to the relationship in order to easily identify who that relationship is targeting. The receiving profile then receives the request and can either deny or accept the friend request. If accepted, an acceptFriendRequest request is sent:

await acceptFriendRequest({
  variables: {
    profileUuid: from,
  },
})

Where `from` is the uuid of the profile who originally sent the friend request. We get that uuid from the websocket payload. The resolver receives the request and then calls the graph API to create that relationship:

tx.run(
  Match (p1:Profile {uuid: $sUuid})  
  Match (p2:Profile {uuid: $rUuid}) 
  Merge (p1)-[friends:FRIENDS 
  {uuid: $recipientProfileUuid, username: $recipientProfileUsername }]->(p2) 
  Merge (p2)-[:FRIENDS {uuid: $senderProfileUuid, username: $senderProfileUsername }]->(p1) 
  WITH p1, friends, p2 
  Match (p1)-[fr:FRIEND_REQUEST]->(p2) 
  DELETE fr 
  RETURN p1, friends, p2,
  {
    sUuid: senderProfileUuid,
    rUuid: recipientProfileUuid,
    recipientProfileUuid,
    recipientProfileUsername,
    senderProfileUsername,
    senderProfileUuid,
  }
)
              

The cypher query creates a bi-directional relationship of FRIENDS between the two profiles and then deletes the existing FRIEND_REQUEST relationship that they have between them. The method responsible for calling the graph method then receives a response, creates a conversation object and responds with the conversation object. The user accepting the friend request then receives the response, updates the store and emits an event to the user who requested indicating that friendship request has been accepted. The user who initiated the friendship request then receives the socket event and also updates the store with the conversation. A new conversation is created for both users and they are now able to converse.

• The Video service

  Jitsi is an open-source video conferencing platform that was chosen for its versatility and ease of deployment. The platform is built on top of WebRTC, a technology that allows for real-time communication through web browsers. This eliminates the need for users to install any additional software, making it easily accessible for anyone with a web browser.

  To deploy Jitsi, a DigitalOcean droplet was provisioned and configured. This droplet, or virtual private server, was set up to run Ubuntu, the operating system recommended by Jitsi. The Jitsi package was then installed on the droplet, along with any necessary dependencies, such as Prosody, an XMPP server used for signaling in the Jitsi platform.

  To secure the Jitsi platform, several measures were taken. A valid SSL certificate was obtained and configured for the Jitsi domain. Firewall rules were also set up to restrict access to the Jitsi server and to only allow necessary traffic. Although, for now, access is unrestricted for development and testing purposes. You can access the Jitsi instance running on the VPS by following the dedicated Noon domain on https://www.noon-vid.com.

The Noon web client is built as a SPA, written in TypeScript using React and Next.js as the framework. It utilizes Redux Toolkit for state management. Real-time functionality is implemented using Socket.io for bi-directional communication between the client and server. GraphQL and Apollo Client are used to handle queries and mutations. GraphQL Codegen is used to automatically generate the GraphQL syntax, reducing the need for manual type definitions. The design of the platform at the present is implemented with Chakra UI, a popular library for building accessible, modular, and customizable UI components. For styling, it uses TailwindCSS, a utility-first CSS framework for rapidly building custom designs with a consistent look and feel. All these technologies are used together to create a high-performance and user-friendly client platform that seamlessly communicates with the backend, providing a smooth and responsive user experience.

The functionality of the client is split up into various components and utility functions, each responsible for a specific set of tasks.

All state in the application is managed using Redux/Redux Toolkit. RTK allows for centralized and predictable state management.

RTK is used to store the logged-in user, to populate the conversations, and to update the conversations with the messages sent, amongst a myriad of other things. Additionally, I've used it to handle various UI changes such as toggling the menu, displaying modals and toasts, and showing loading indicators. By using Redux, I can ensure that the state of the application is consistent and that all changes are handled in a predictable and consistent way.

The Store

The Noon store utilizes a number of different reducers to manage specific aspects of the application's state. Among which:

Users reducer: is responsible for storing and retrieving user authentication state. This includes information such as login credentials, session tokens, and other user-specific data.

Profiles reducer: is responsible for managing the list of profiles for friends and tracking friend requests, blocked/unfriended, etc. It also keeps track of who the logged in user sent friend requests to and received friend requests from.

UI reducer: is responsible for managing the state of the user interface, including managing the active navigation, displaying and hiding UI elements, and handling any UI-related errors or issues.

Chat reducer: is responsible for instantiating and managing conversations, keeping track of active conversations, updating conversations with new messages, etc.

Search reducer: is responsible for managing the search filters, which include setting and updating search queries, handling search results, and managing any search-related errors or issues.

Groups reducer: is responsible for managing the state of groups, including creating new groups, updating existing groups, and handling any group-related errors or issues.

Video reducer: is responsible for instantiating and managing video conversations, keeping track of their state, and handling any video-related errors or issues.

Files reducer: is responsible for managing files, including uploading, downloading, and deleting files, and handling any file-related errors or issues.

The Videobridge2 connection

I integrated the Jitsi package into the client by using the Jitsi Meet SDK. The Noon Video component initializes the connection, via the React component provided by SDK, to Videobridge2 on the VPS instance where Jitsi is configured and running. Once the connection is established, the component prepares an iframe to be embedded into the conversation window where the video call is requested.

Overall, by building Noon as an SPA, I have created a smooth and responsive user experience while also providing the ability to later on learn and adopt React Native to build the mobile applications (a process which I have started a few weeks ago).

On build, a linting and verification process ensures that the code is free of any syntax or semantic errors before deployment. Vercel is used for deployment.