Nodeflake

Nodeflake is a distributed computing framework for Node. It aims to solve the following problems:

• Distributed computing (dividing computation across multiple networked computers) is tedious and inaccessible - most common parallel/distributed programming paradigms have steep learning curves (e.g. C++ and MPI) and actually implementing a system to run them (e.g. setting up an MPI cluster) is no small feat. Nodeflake allows distributed computations to be written in JavaScript, a popular and expressive language with a great development ecosystem.
• Writing separate logic for masters and slaves creates more engineering effort in defining how tasks are distributed. Nodeflake presents a functional programming paradigm that can be seamlessly distributed across a tree topology.
• Distributed computing should enable the generation of rich visualizations, but existing methodologies involve separate technology for processing and visualizing data. Nodeflake abstracts the entire analysis workflow with powerful functions for capturing raw data, processing it for analysis and visualization, and exporting the generated visualizations to browser-ready widgets.
• Distributed computing should be seamless across physical devices and the cloud.

This write-up is a work in progress. I built a fully-functional cluster but the code became a patchwork of network services. The bottlenecks are quite obvious with particular workloads and I might be going back to the drawing board for this project when I pick it back up.