Fixes. Updated high-level overview diagram with priorities: red is highest, then there's orange, and then green. If a shade is darker, it means it is higher priority

This commit is contained in:
Diego Ripley
2026-02-21 16:22:38 -05:00
parent db8a1f3b71
commit cc6135e19a
6 changed files with 34 additions and 17 deletions
@@ -1,5 +1,5 @@
--- ---
title: d4c-datapkg-field-imagery title: Field Imagery
weight: 4 weight: 4
next: /docs/d4c-infra-distribution/ next: /docs/d4c-infra-distribution/
--- ---
@@ -8,20 +8,17 @@ next: /docs/d4c-infra-distribution/
Commercial street-level imagery is often locked behind restrictive licenses, proprietary viewers, and paywalls, making it inaccessible to our target audience. We are building a sovereign, open-source alternative for Canada. Commercial street-level imagery is often locked behind restrictive licenses, proprietary viewers, and paywalls, making it inaccessible to our target audience. We are building a sovereign, open-source alternative for Canada.
By self-hosting a **[Panoramax](https://panoramax.fr/)** instance, we provide a decentralized platform where field imagery is treated as a public utility: fully downloadable, API-accessible, and privacy-compliant. We will be using **[Panoramax](https://panoramax.fr/)** as a base, but will be expanding upon the project to provide a decentralized platform where field imagery is treated as a public utility: fully downloadable, API-accessible, privacy-compliant, and resilient to shifting priorities.
## The Infrastructure ## The Infrastructure
Our field imagery pipeline is built on the **Panoramax** ecosystem, a federated open-source alternative to Google Street View that guarantees data permanence and open access. * **Storage**: High-performance object storage backend for hosting terabytes of 360°, flat field imagery, and oblique imagery.
* **Storage**: High-performance object storage backend for hosting terabytes of 360° and flat field imagery.
* **Federation**: Our instance connects to the global Panoramax federation, ensuring that while the data is hosted in Canada, it is discoverable worldwide through the global panoramax catalog.
## The Processing Pipeline ## The Processing Pipeline
We treat field imagery as a data engineering challenge, ensuring "time-to-insight" is minimized for downstream users. We treat field imagery as a data engineering challenge, ensuring "time-to-insight" is minimized for downstream users.
1. **Ingestion**: Raw imagery is captured using diverse hardware (ex. 360° cameras, mobile rigs, meta glasses, etc.) and ingested by our systems. 1. **Ingestion**: Raw imagery is captured using diverse hardware (ex. 360° cameras, mobile rigs, meta glasses, drones, etc.) and ingested by our systems.
2. **Privacy & Anonymization**: Before publication, all imagery undergoes a rigorous privacy scrub. We utilize automated detection pipelines to blur faces and license plates, ensuring compliance with Canadian privacy standards while maintaining data utility. 2. **Privacy & Anonymization**: Before publication, all imagery undergoes a rigorous privacy scrub. We utilize automated detection pipelines to blur faces and license plates, ensuring compliance with Canadian privacy standards while maintaining data utility.
3. **Standardization**: Images are processed into systems-ready formats, making it ready for analysis. 3. **Standardization**: Images are processed into systems-ready formats, making it ready for analysis.
4. **Metadata Extraction**: We extract and normalize/strip identifiying information (ex. EXIF and GPS telemetry), indexing it into a **[FAIR Catalogue](https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/)**. 4. **Metadata Extraction**: We extract and normalize/strip identifiying information (ex. EXIF and GPS telemetry), indexing it into a **[FAIR Catalogue](https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/)**.
@@ -30,6 +27,5 @@ We treat field imagery as a data engineering challenge, ensuring "time-to-insigh
Unlike commercial platforms that only offer a "view" of the data, we provide the **data itself**. Unlike commercial platforms that only offer a "view" of the data, we provide the **data itself**.
* **API Access**: Full programmatic access via the Panoramax REST API for querying imagery by location, date, or sequence.
* **Bulk Datasets**: Curated dumps of street-level imagery available for computer vision training, asset management, and change detection models. * **Bulk Datasets**: Curated dumps of street-level imagery available for computer vision training, asset management, and change detection models.
* **[FAIR Data Catalogue](https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/) Integration**: Seamless integration with geospatial workflows (ex. DuckDB, QGIS, Python, R, Julia, etc.). * **[FAIR Data Catalogue](https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/) Integration**: Seamless integration with geospatial workflows (ex. DuckDB, QGIS, Python, R, Julia, etc.).
@@ -1,5 +1,5 @@
--- ---
title: d4c-datapkg-foundation title: Foundation
weight: 1 weight: 1
--- ---
@@ -1,5 +1,5 @@
--- ---
title: d4c-datapkg-orthoimagery title: Orthoimagery
weight: 3 weight: 3
prev: /docs/d4c-pkgs/d4c-datapkg-statistical/statistics_canada/census_data/ prev: /docs/d4c-pkgs/d4c-datapkg-statistical/statistics_canada/census_data/
next: /docs/d4c-pkgs/d4c-datapkg-field-imagery/ next: /docs/d4c-pkgs/d4c-datapkg-field-imagery/
@@ -1,5 +1,5 @@
--- ---
title: d4c-datapkg-statistical title: Statistical
weight: 2 weight: 2
toc: true toc: true
prev: /docs/d4c-pkgs/foundation/ prev: /docs/d4c-pkgs/foundation/
+26 -5
View File
@@ -9,9 +9,12 @@ flowchart TD
Foundation@{ shape: lean-l} Foundation@{ shape: lean-l}
Orthoimagery@{ shape: lean-l} Orthoimagery@{ shape: lean-l}
EnvironmentClimate@{ shape: lean-l, label: "Environment, Climate, & Health"} EnvironmentClimate@{ shape: lean-l, label: "Environment, Climate, & Health"}
style EnvironmentClimate fill:#FBC02D,stroke:#F9A825,color:#000000
FieldImagery@{ shape: lean-l, label: "Field Imagery"} FieldImagery@{ shape: lean-l, label: "Field Imagery"}
Elevation@{ shape: lean-l} style FieldImagery fill:#FBC02D,stroke:#F9A825,color:#000000
WebCorpus@{ shape: lean-l, label: "Web Corpus"} WebCorpus@{ shape: lean-l, label: "Web Corpus"}
Elevation@{ shape: lean-l}
style WebCorpus fill:#66BB6A,stroke:#2E7D32,color:#000000
end end
%% --------------------------------------------------------- %% ---------------------------------------------------------
@@ -41,33 +44,46 @@ flowchart TD
%% Box: Vector Tiles (Pastel Orange) %% Box: Vector Tiles (Pastel Orange)
subgraph vt [Vector Tiles] subgraph vt [Vector Tiles]
VectorTiles@{ shape: lean-l, label: "Mapbox Vector Tiles"} VectorTiles@{ shape: lean-l, label: "Mapbox Vector Tiles"}
style VectorTiles fill:#66BB6A,stroke:#2E7D32,color:#000000
NextGenVectorTiles@{ shape: lean-l, label: "Next-Gen Vector Tiles"} NextGenVectorTiles@{ shape: lean-l, label: "Next-Gen Vector Tiles"}
style NextGenVectorTiles fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
end end
%% Box: Visuals (Pastel Blue - No Name) %% Box: Visuals (Pastel Blue - No Name)
subgraph visuals [" "] subgraph visuals [" "]
%% AVIF is not an option since QGIS does not seem to support it (I was surprised when my generated file was not compatible)
WebP@{ shape: lean-l} WebP@{ shape: lean-l}
style WebP fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
JPG@{ shape: lean-l} JPG@{ shape: lean-l}
style JPG fill:#66BB6A,stroke:#2E7D32,color:#000000
PNG@{ shape: lean-l} PNG@{ shape: lean-l}
style PNG fill:#66BB6A,stroke:#2E7D32,color:#000000
end end
%% Box: Portable Databases (Pastel Green) %% Box: Portable Databases (Pastel Green)
subgraph pkg [Portable Databases] subgraph pkg [Portable Databases]
%% Good format, but has some issues. I encountered issues when merging multiple adjoining PMTile files, the author is aware of these issues
PMTiles@{ shape: lean-l} PMTiles@{ shape: lean-l}
%% Compatability with every OS.
%% Fun fact, you don't even need an OS for running SQLite
%% https://sqlite.org/selfcontained.html
SQLite@{ shape: lean-l} SQLite@{ shape: lean-l}
end end
%% Box: Enterprise (Pastel Purple) %% Box: Enterprise (Pastel Purple)
subgraph ent [Enterprise] subgraph ent [Enterprise]
FileGeodatabase@{shape: lean-l, label: "File Geodatabase"} FileGeodatabase@{shape: lean-l, label: "File Geodatabase"}
style FileGeodatabase fill:#fff,stroke:#2E7D32,color:#000000
end end
end end
%% --------------------------------------------------------- %% ---------------------------------------------------------
%% 5. DISTRIBUTION INFRASTRUCTURE %% 5. DISTRIBUTION INFRASTRUCTURE
%% --------------------------------------------------------- %% ---------------------------------------------------------
subgraph di [Distribution Infrastructure] subgraph di [Distribution]
ObjectStorage@{ shape: bow-rect, label: "Object Storage"} %% My goals involve going beyond object storage, so use generic term
ObjectStorage@{ shape: bow-rect, label: "Storage"}
style ObjectStorage fill:#FBC02D,stroke:#F9A825,color:#000000
Metadata@{ shape: rect, label: "FAIR Data Catalogue"} Metadata@{ shape: rect, label: "FAIR Data Catalogue"}
HTTP@{ shape: rect, label: "Systems-Ready Data"} HTTP@{ shape: rect, label: "Systems-Ready Data"}
DecentralizedDistribution@{ shape: rect, label: "Decentralized Distribution"} DecentralizedDistribution@{ shape: rect, label: "Decentralized Distribution"}
@@ -78,6 +94,8 @@ flowchart TD
%% --------------------------------------------------------- %% ---------------------------------------------------------
subgraph ei [Experimental Infrastructure] subgraph ei [Experimental Infrastructure]
GeoSpatialServices@{ shape: rect, label: "Geospatial Services"} GeoSpatialServices@{ shape: rect, label: "Geospatial Services"}
%% Does not make sense. Focus on delivering the data
style GeoSpatialServices fill:#66BB6A,stroke:#2E7D32,color:#000000
end end
%% --------------------------------------------------------- %% ---------------------------------------------------------
@@ -178,6 +196,10 @@ style DecentralizedDistribution fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
style HTTP fill:#B71C1C,stroke:#7F0000,color:#FFFFFF style HTTP fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
style Systems fill:#B71C1C,stroke:#7F0000,color:#FFFFFF style Systems fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
style Metadata fill:#B71C1C,stroke:#7F0000,color:#FFFFFF style Metadata fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
style Statistical fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
style Foundation fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
style Orthoimagery fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
style Elevation fill:#66BB6A,stroke:#2E7D32,color:#000000
%%style df fill:#D32F2F,stroke:#8E0000,color:#FFFFFF %%style df fill:#D32F2F,stroke:#8E0000,color:#FFFFFF
style sot fill:#EF9A9A,stroke:#C62828,color:#000000 style sot fill:#EF9A9A,stroke:#C62828,color:#000000
@@ -202,10 +224,9 @@ style visuals fill:#FBC02D,stroke:#F9A825,color:#000000
style ent fill:#66BB6A,stroke:#2E7D32,color:#000000 style ent fill:#66BB6A,stroke:#2E7D32,color:#000000
style DataSci fill:#D32F2F,stroke:#8E0000,color:#FFFFFF style DataSci fill:#D32F2F,stroke:#8E0000,color:#FFFFFF
style GeoSpatialServices fill:#FFCC80,stroke:#FB8C00,color:#000000
class Foundation,Statistical,Orthoimagery,FieldImagery,EnvironmentClimate,Elevation,WebCorpus linkNode class FieldImagery linkNode
class Parquet,FlatGeoBuf,SQLite,FileGeodatabase,VectorTiles,NextGenVectorTiles,GeoTIFF,Zarr,WebP,PMTiles,JPEGXL,AV1,WARC linkNode class Parquet,FlatGeoBuf,SQLite,FileGeodatabase,VectorTiles,NextGenVectorTiles,GeoTIFF,Zarr,WebP,PMTiles,JPEGXL,AV1,WARC linkNode
%% ========================================================= %% =========================================================
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 77 KiB

After

Width:  |  Height:  |  Size: 78 KiB