diff --git a/content/docs/dissemination/_index.md b/content/docs/dissemination/_index.md index 1c6b181..e429a7f 100644 --- a/content/docs/dissemination/_index.md +++ b/content/docs/dissemination/_index.md @@ -30,7 +30,7 @@ flowchart TD end Distribution[Decentralized Distribution] - Torrent[Accelerated Peer-to-Peer] + Torrent["Accelerated Peer-to-Peer (P2P)"] subgraph Consumers [Consumption] Users[Data People & Developers] @@ -78,7 +78,7 @@ flowchart TD a11@{animate: true, animation: fast} %% Click Actions - click Sources "https://www.dataforcanada.org/#high-level-overview/" _blank + click Sources "https://www.dataforcanada.org/#high-level-overview" _blank click Processes "https://www.dataforcanada.org/docs/processes/" _blank click Artifacts "https://www.dataforcanada.org/docs/getting_started/" _blank click Metadata "https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/" _blank @@ -119,7 +119,9 @@ Once data products reach a production-ready state, they enter a dissemination fl ### Decentralized Distribution -We are piloting a peer-to-peer technology to maximize infrastructure resilience. By leveraging the [peer-to-peer HTTP consumption feature](https://www.bittorrent.org/beps/bep_0019.html), users will be able to download simultaneously from Source Cooperative, Zenodo, Data for Canada infrastructure, and community peers. This ensures high availability without a single point of failure. Current laboratory work is available in the [Decentralized Distribution Labs](https://github.com/dataforcanada/decentralized-distribution-labs). + + +We are piloting a P2P technology, to maximize infrastructure resilience. By leveraging the [P2P HTTP consumption feature](https://www.bittorrent.org/beps/bep_0019.html), users will be able to download simultaneously from Source Cooperative, Zenodo, Data for Canada infrastructure, and community peers. This ensures high availability without a single point of failure. Current laboratory work is available in the [Decentralized Distribution Labs](https://github.com/dataforcanada/decentralized-distribution-labs). ## Work in the Lab: Smart Nodes @@ -129,7 +131,10 @@ A Smart Node functions as a "set-it-and-forget-it" volunteer server, an automate * **Automated Mirroring:** Unlike a standard download, a Smart Node automatically synchronizes with our central **FAIR Data Catalog**. It intelligently fetches new or "at-risk" datasets to ensure they remain available even if the central portal experiences downtime. * **Volunteer-Powered Resilience:** This model allows partner institutions (ex. universities, research labs) and public volunteers to donate bandwidth and storage. By running a Smart Node, contributors actively protect vital Canadian datasets from being lost or gated. -* **Dynamic Storage Management:** The node software monitors network health to optimize resource usage. Leveraging the P2P technology's capability for **selective piece mapping**, the node does not need to store the entire catalog. Instead, it identifies specific file indices or "rare" pieces within the metadata and sends granular `REQUEST` messages for only those blocks. This allows a node with limited storage (ex. 500GB) to provide critical redundancy for a much larger archive (ex. 50TB) by surgically targeting only the data that is currently under shared. + +* **Dynamic Storage Management:** The node software monitors network health to optimize resource usage. Leveraging the P2P technology's capability for **selective piece mapping**, the node does not need to store the entire catalog. Instead, it identifies specific file indices or "rare" pieces within the metadata and sends granular `REQUEST` messages for only those blocks. This allows a node with limited storage (ex. 500GB) to provide critical redundancy for a much larger archive (ex. 50TB) by surgically targeting only the data that is currently under shared. + + We are currently refining the concepts from [smart-node-transmission](https://github.com/academictorrents/smartnode-transmission) to work seamlessly with our catalog, enabling a fully decentralized data mesh for Canadian geospatial information. @@ -138,7 +143,7 @@ graph TD %% Node Definitions with custom labels Catalog[("FAIR Data Catalog")] SmartNode["Volunteer Smart Node
(Limited Storage Optimization)"] - BTNetwork(["Peer to Peer Network
(Massive Data Pool)"]) + BTNetwork(["P2P Network
(Massive Data Pool)"]) %% The Process Flow Catalog -->|"1. Syncs metadata & identifies 'at-risk' data"| SmartNode diff --git a/content/infrastructure/_index.md b/content/infrastructure/_index.md index b6d34a5..5e3fb0a 100644 --- a/content/infrastructure/_index.md +++ b/content/infrastructure/_index.md @@ -58,7 +58,7 @@ Jurisdiction: USA"] Netcup["VPS 01 Location: Manassas, Virginia, USA Specs: 2.5Gbps / 2.5Gbps, 512GB Flash Storage -Protocol: HTTP & BitTorrent +Protocol: HTTP & P2P Jurisdiction: Germany"] end @@ -87,7 +87,7 @@ Jurisdiction: Switzerland"] %% CONNECTIONS %% --------------------------------------------------------- - NodeTO <==>|BitTorrent Sync P2P| NodeAMS + NodeTO <==>|P2P| NodeAMS IA_SF -.->|Internal Replication| IA_Van NodeTO -.->|HTTP Pull| SourceCoop @@ -104,8 +104,8 @@ Jurisdiction: Switzerland"] | :--- | :--- | :--- | :--- | :--- | | **CDN** | [CDN - Cloudflare Details](https://www.cloudflare.com/plans/) - WAF, CDN (Amortized Annual) | $30.90 | $22.60 | €19.13 | | **CDN Services** | [Object Storage - Cloudflare Details](https://www.cloudflare.com/products/r2/) & [Serverless - Cloudflare Details](https://www.cloudflare.com/en-ca/plans/developer-platform/) (Variable) | $32.71 | $23.93 | €20.26 | -| **Smart Node 01** | [Decentralized Distribution - SlashN Services Details](https://ultra.cc/#plan-pricing) - Dedicated BitTorrent client | $28.98 | $21.21 | €17.95 | -| **Smart Node 02** | [Decentralized Distribution - SlashN Services Details](https://ultra.cc/#plan-pricing) - Dedicated BitTorrent client | $28.98 | $21.21 | €17.95 | +| **Smart Node 01** | [Decentralized Distribution - SlashN Services Details](https://ultra.cc/#plan-pricing) - Dedicated P2P client | $28.98 | $21.21 | €17.95 | +| **Smart Node 02** | [Decentralized Distribution - SlashN Services Details](https://ultra.cc/#plan-pricing) - Dedicated P2P client | $28.98 | $21.21 | €17.95 | | **VPS 01** | [Geospatial Services - Netcup Details](https://www.netcup.com/en/server/root-server) - ARM64 | $14.64 | $10.72 | €9.07 | | **TOTAL** | **Monthly Run Rate** | **$136.21** | **$99.67** | **€84.36** |