mirror of
https://github.com/dataforcanada/d4c-service-main-site.git
synced 2026-06-13 14:00:51 +02:00
Made updates to high-level overview on main page
This commit is contained in:
+23
-14
@@ -4,12 +4,15 @@ toc: false
|
||||
---
|
||||
|
||||
## Mission
|
||||
|
||||
Data for Canada exists to bridge the gap between open data availability and data usability. We curate, clean, and re-engineer high-value Canadian datasets into high-performance, analysis-ready formats for researchers, developers, and systems.
|
||||
|
||||
## The Problem
|
||||
|
||||
Canada creates incredible amounts of open data, from foundational road networks to federal census statistics. However, these datasets are often locked in legacy formats, fragmented portals, or structures that require significant engineering effort to normalize before they can be used. For a researcher or a system developer, the "time-to-insight" is often bottlenecked by data preparation.
|
||||
|
||||
## The Solution
|
||||
|
||||
We act as the transformation layer. We aggregate datasets with permissive licenses and process them into "digestible" standards optimized for modern downstream applications.
|
||||
|
||||
* **For Researchers:** Skip the cleaning phase. Access normalized, documented data ready for analysis.
|
||||
@@ -26,13 +29,13 @@ flowchart TD
|
||||
classDef consumer fill:#f3e5f5,stroke:#8e24aa,stroke-width:2px
|
||||
|
||||
subgraph "Data Sources"
|
||||
StatCan[("Statistical Products")]:::source
|
||||
StatProducts[("Statistical Products")]:::source
|
||||
Orthoimagery[("Orthoimagery")]:::source
|
||||
end
|
||||
|
||||
subgraph "Processing Pipeline"
|
||||
Raw[Raw Data Ingestion<br/>CSVs, Shapefiles, ECW]:::process
|
||||
Transform[Transformation Engine<br/>Open & Closed Source]:::process
|
||||
Transform[Transformation Engine]:::process
|
||||
Opt[Optimization]:::process
|
||||
end
|
||||
|
||||
@@ -43,35 +46,41 @@ flowchart TD
|
||||
end
|
||||
|
||||
subgraph "Distribution Infrastructure"
|
||||
S3_COMPLIANT_STORAGE[S3 Compliant Storage]:::storage
|
||||
Decentralized_Distribution[Decentralized Distribution]:::storage
|
||||
Serverless[Cloudflare Worker<br/>API & Serving]:::storage
|
||||
ObjectStorage[Object Storage]:::storage
|
||||
DecentralizedDistribution[Decentralized Distribution]:::storage
|
||||
Serverless[Serverless Worker<br/>API & Serving]:::storage
|
||||
end
|
||||
|
||||
subgraph "Consumption / End Users"
|
||||
DataSci[DuckDB, Python, QGIS, Jupyter]:::consumer
|
||||
WebApps[Web Applications]:::consumer
|
||||
DataSci[Python]:::consumer
|
||||
DataSci[Python, R, Julia]:::consumer
|
||||
Systems[Systems]:::consumer
|
||||
end
|
||||
|
||||
%% Relationships
|
||||
StatCan --> Raw
|
||||
StatProducts --> Raw
|
||||
Raw --> Transform
|
||||
Transform --> Opt
|
||||
Opt --> Parquet
|
||||
Opt --> PMTiles
|
||||
Opt --> FlatGeoBuf
|
||||
Parquet --> S3_COMPLIANT_STORAGE
|
||||
PMTiles --> S3_COMPLIANT_STORAGE
|
||||
FlatGeoBuf --> S3_COMPLIANT_STORAGE
|
||||
S3_COMPLIANT_STORAGE --> Decentralized_Distribution
|
||||
S3_COMPLIANT_STORAGE --> Serverless
|
||||
Decentralized_Distribution --> Systems
|
||||
Parquet --> ObjectStorage
|
||||
PMTiles --> ObjectStorage
|
||||
FlatGeoBuf --> ObjectStorage
|
||||
ObjectStorage --> DecentralizedDistribution
|
||||
ObjectStorage --> Serverless
|
||||
DecentralizedDistribution --> Systems
|
||||
Serverless --> WebApps
|
||||
Serverless --> DataSci
|
||||
|
||||
%% Concept Annotations
|
||||
Transform -.->|"Join Spatial & Tabular"| Parquet
|
||||
PMTiles -.->|"Stream tiles"| WebApps
|
||||
PMTiles -.->|"Stream"| WebApps
|
||||
FlatGeoBuf -.->|"Stream"|DataSci
|
||||
FlatGeoBuf -.->|"Stream"|WebApps
|
||||
|
||||
click StatProducts "https://www.dataforcanada.org/docs/processes/statistical_products/" _blank
|
||||
click Orthoimagery "https://www.dataforcanada.org/docs/processes/statistical_products/" _blank
|
||||
click DecentralizedDistribution "https://www.dataforcanada.org/docs/dissemination/" _blank
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user