mirror of
https://github.com/dataforcanada/d4c-service-main-site.git
synced 2026-06-13 14:00:51 +02:00
Fixes. Updated high-level overview diagram with priorities: red is highest, then there's orange, and then green. If a shade is darker, it means it is higher priority
This commit is contained in:
Diego Ripley
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: d4c-datapkg-field-imagery
|
||||
title: Field Imagery
|
||||
weight: 4
|
||||
next: /docs/d4c-infra-distribution/
|
||||
---
|
||||
@@ -8,20 +8,17 @@ next: /docs/d4c-infra-distribution/
|
||||
|
||||
Commercial street-level imagery is often locked behind restrictive licenses, proprietary viewers, and paywalls, making it inaccessible to our target audience. We are building a sovereign, open-source alternative for Canada.
|
||||
|
||||
By self-hosting a **[Panoramax](https://panoramax.fr/)** instance, we provide a decentralized platform where field imagery is treated as a public utility: fully downloadable, API-accessible, and privacy-compliant.
|
||||
We will be using **[Panoramax](https://panoramax.fr/)** as a base, but will be expanding upon the project to provide a decentralized platform where field imagery is treated as a public utility: fully downloadable, API-accessible, privacy-compliant, and resilient to shifting priorities.
|
||||
|
||||
## The Infrastructure
|
||||
|
||||
Our field imagery pipeline is built on the **Panoramax** ecosystem, a federated open-source alternative to Google Street View that guarantees data permanence and open access.
|
||||
|
||||
* **Storage**: High-performance object storage backend for hosting terabytes of 360° and flat field imagery.
|
||||
* **Federation**: Our instance connects to the global Panoramax federation, ensuring that while the data is hosted in Canada, it is discoverable worldwide through the global panoramax catalog.
|
||||
* **Storage**: High-performance object storage backend for hosting terabytes of 360°, flat field imagery, and oblique imagery.
|
||||
|
||||
## The Processing Pipeline
|
||||
|
||||
We treat field imagery as a data engineering challenge, ensuring "time-to-insight" is minimized for downstream users.
|
||||
|
||||
1. **Ingestion**: Raw imagery is captured using diverse hardware (ex. 360° cameras, mobile rigs, meta glasses, etc.) and ingested by our systems.
|
||||
1. **Ingestion**: Raw imagery is captured using diverse hardware (ex. 360° cameras, mobile rigs, meta glasses, drones, etc.) and ingested by our systems.
|
||||
2. **Privacy & Anonymization**: Before publication, all imagery undergoes a rigorous privacy scrub. We utilize automated detection pipelines to blur faces and license plates, ensuring compliance with Canadian privacy standards while maintaining data utility.
|
||||
3. **Standardization**: Images are processed into systems-ready formats, making it ready for analysis.
|
||||
4. **Metadata Extraction**: We extract and normalize/strip identifiying information (ex. EXIF and GPS telemetry), indexing it into a **[FAIR Catalogue](https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/)**.
|
||||
@@ -30,6 +27,5 @@ We treat field imagery as a data engineering challenge, ensuring "time-to-insigh
|
||||
|
||||
Unlike commercial platforms that only offer a "view" of the data, we provide the **data itself**.
|
||||
|
||||
* **API Access**: Full programmatic access via the Panoramax REST API for querying imagery by location, date, or sequence.
|
||||
* **Bulk Datasets**: Curated dumps of street-level imagery available for computer vision training, asset management, and change detection models.
|
||||
* **[FAIR Data Catalogue](https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/) Integration**: Seamless integration with geospatial workflows (ex. DuckDB, QGIS, Python, R, Julia, etc.).
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: d4c-datapkg-foundation
|
||||
title: Foundation
|
||||
weight: 1
|
||||
---
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: d4c-datapkg-orthoimagery
|
||||
title: Orthoimagery
|
||||
weight: 3
|
||||
prev: /docs/d4c-pkgs/d4c-datapkg-statistical/statistics_canada/census_data/
|
||||
next: /docs/d4c-pkgs/d4c-datapkg-field-imagery/
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: d4c-datapkg-statistical
|
||||
title: Statistical
|
||||
weight: 2
|
||||
toc: true
|
||||
prev: /docs/d4c-pkgs/foundation/
|
||||
|
||||
@@ -9,9 +9,12 @@ flowchart TD
|
||||
Foundation@{ shape: lean-l}
|
||||
Orthoimagery@{ shape: lean-l}
|
||||
EnvironmentClimate@{ shape: lean-l, label: "Environment, Climate, & Health"}
|
||||
style EnvironmentClimate fill:#FBC02D,stroke:#F9A825,color:#000000
|
||||
FieldImagery@{ shape: lean-l, label: "Field Imagery"}
|
||||
Elevation@{ shape: lean-l}
|
||||
style FieldImagery fill:#FBC02D,stroke:#F9A825,color:#000000
|
||||
WebCorpus@{ shape: lean-l, label: "Web Corpus"}
|
||||
Elevation@{ shape: lean-l}
|
||||
style WebCorpus fill:#66BB6A,stroke:#2E7D32,color:#000000
|
||||
end
|
||||
|
||||
%% ---------------------------------------------------------
|
||||
@@ -41,33 +44,46 @@ flowchart TD
|
||||
%% Box: Vector Tiles (Pastel Orange)
|
||||
subgraph vt [Vector Tiles]
|
||||
VectorTiles@{ shape: lean-l, label: "Mapbox Vector Tiles"}
|
||||
style VectorTiles fill:#66BB6A,stroke:#2E7D32,color:#000000
|
||||
NextGenVectorTiles@{ shape: lean-l, label: "Next-Gen Vector Tiles"}
|
||||
style NextGenVectorTiles fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
end
|
||||
|
||||
%% Box: Visuals (Pastel Blue - No Name)
|
||||
subgraph visuals [" "]
|
||||
%% AVIF is not an option since QGIS does not seem to support it (I was surprised when my generated file was not compatible)
|
||||
WebP@{ shape: lean-l}
|
||||
style WebP fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
JPG@{ shape: lean-l}
|
||||
style JPG fill:#66BB6A,stroke:#2E7D32,color:#000000
|
||||
PNG@{ shape: lean-l}
|
||||
style PNG fill:#66BB6A,stroke:#2E7D32,color:#000000
|
||||
end
|
||||
|
||||
%% Box: Portable Databases (Pastel Green)
|
||||
subgraph pkg [Portable Databases]
|
||||
%% Good format, but has some issues. I encountered issues when merging multiple adjoining PMTile files, the author is aware of these issues
|
||||
PMTiles@{ shape: lean-l}
|
||||
%% Compatability with every OS.
|
||||
%% Fun fact, you don't even need an OS for running SQLite
|
||||
%% https://sqlite.org/selfcontained.html
|
||||
SQLite@{ shape: lean-l}
|
||||
end
|
||||
|
||||
%% Box: Enterprise (Pastel Purple)
|
||||
subgraph ent [Enterprise]
|
||||
FileGeodatabase@{shape: lean-l, label: "File Geodatabase"}
|
||||
style FileGeodatabase fill:#fff,stroke:#2E7D32,color:#000000
|
||||
end
|
||||
end
|
||||
|
||||
%% ---------------------------------------------------------
|
||||
%% 5. DISTRIBUTION INFRASTRUCTURE
|
||||
%% ---------------------------------------------------------
|
||||
subgraph di [Distribution Infrastructure]
|
||||
ObjectStorage@{ shape: bow-rect, label: "Object Storage"}
|
||||
subgraph di [Distribution]
|
||||
%% My goals involve going beyond object storage, so use generic term
|
||||
ObjectStorage@{ shape: bow-rect, label: "Storage"}
|
||||
style ObjectStorage fill:#FBC02D,stroke:#F9A825,color:#000000
|
||||
Metadata@{ shape: rect, label: "FAIR Data Catalogue"}
|
||||
HTTP@{ shape: rect, label: "Systems-Ready Data"}
|
||||
DecentralizedDistribution@{ shape: rect, label: "Decentralized Distribution"}
|
||||
@@ -78,6 +94,8 @@ flowchart TD
|
||||
%% ---------------------------------------------------------
|
||||
subgraph ei [Experimental Infrastructure]
|
||||
GeoSpatialServices@{ shape: rect, label: "Geospatial Services"}
|
||||
%% Does not make sense. Focus on delivering the data
|
||||
style GeoSpatialServices fill:#66BB6A,stroke:#2E7D32,color:#000000
|
||||
end
|
||||
|
||||
%% ---------------------------------------------------------
|
||||
@@ -178,6 +196,10 @@ style DecentralizedDistribution fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
style HTTP fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
style Systems fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
style Metadata fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
style Statistical fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
style Foundation fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
style Orthoimagery fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
||||
style Elevation fill:#66BB6A,stroke:#2E7D32,color:#000000
|
||||
|
||||
%%style df fill:#D32F2F,stroke:#8E0000,color:#FFFFFF
|
||||
style sot fill:#EF9A9A,stroke:#C62828,color:#000000
|
||||
@@ -202,10 +224,9 @@ style visuals fill:#FBC02D,stroke:#F9A825,color:#000000
|
||||
|
||||
style ent fill:#66BB6A,stroke:#2E7D32,color:#000000
|
||||
style DataSci fill:#D32F2F,stroke:#8E0000,color:#FFFFFF
|
||||
style GeoSpatialServices fill:#FFCC80,stroke:#FB8C00,color:#000000
|
||||
|
||||
|
||||
class Foundation,Statistical,Orthoimagery,FieldImagery,EnvironmentClimate,Elevation,WebCorpus linkNode
|
||||
class FieldImagery linkNode
|
||||
class Parquet,FlatGeoBuf,SQLite,FileGeodatabase,VectorTiles,NextGenVectorTiles,GeoTIFF,Zarr,WebP,PMTiles,JPEGXL,AV1,WARC linkNode
|
||||
|
||||
%% =========================================================
|
||||
|
||||
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 77 KiB After Width: | Height: | Size: 78 KiB |
Reference in New Issue
Block a user