diff --git a/content/docs/d4c-pkgs/d4c-datapkg-field-imagery.md b/content/docs/d4c-pkgs/d4c-datapkg-field-imagery.md index 1e80e1c..3061edc 100644 --- a/content/docs/d4c-pkgs/d4c-datapkg-field-imagery.md +++ b/content/docs/d4c-pkgs/d4c-datapkg-field-imagery.md @@ -1,5 +1,5 @@ --- -title: d4c-datapkg-field-imagery +title: Field Imagery weight: 4 next: /docs/d4c-infra-distribution/ --- @@ -8,20 +8,17 @@ next: /docs/d4c-infra-distribution/ Commercial street-level imagery is often locked behind restrictive licenses, proprietary viewers, and paywalls, making it inaccessible to our target audience. We are building a sovereign, open-source alternative for Canada. -By self-hosting a **[Panoramax](https://panoramax.fr/)** instance, we provide a decentralized platform where field imagery is treated as a public utility: fully downloadable, API-accessible, and privacy-compliant. +We will be using **[Panoramax](https://panoramax.fr/)** as a base, but will be expanding upon the project to provide a decentralized platform where field imagery is treated as a public utility: fully downloadable, API-accessible, privacy-compliant, and resilient to shifting priorities. ## The Infrastructure -Our field imagery pipeline is built on the **Panoramax** ecosystem, a federated open-source alternative to Google Street View that guarantees data permanence and open access. - -* **Storage**: High-performance object storage backend for hosting terabytes of 360° and flat field imagery. -* **Federation**: Our instance connects to the global Panoramax federation, ensuring that while the data is hosted in Canada, it is discoverable worldwide through the global panoramax catalog. +* **Storage**: High-performance object storage backend for hosting terabytes of 360°, flat field imagery, and oblique imagery. ## The Processing Pipeline We treat field imagery as a data engineering challenge, ensuring "time-to-insight" is minimized for downstream users. -1. **Ingestion**: Raw imagery is captured using diverse hardware (ex. 360° cameras, mobile rigs, meta glasses, etc.) and ingested by our systems. +1. **Ingestion**: Raw imagery is captured using diverse hardware (ex. 360° cameras, mobile rigs, meta glasses, drones, etc.) and ingested by our systems. 2. **Privacy & Anonymization**: Before publication, all imagery undergoes a rigorous privacy scrub. We utilize automated detection pipelines to blur faces and license plates, ensuring compliance with Canadian privacy standards while maintaining data utility. 3. **Standardization**: Images are processed into systems-ready formats, making it ready for analysis. 4. **Metadata Extraction**: We extract and normalize/strip identifiying information (ex. EXIF and GPS telemetry), indexing it into a **[FAIR Catalogue](https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/)**. @@ -30,6 +27,5 @@ We treat field imagery as a data engineering challenge, ensuring "time-to-insigh Unlike commercial platforms that only offer a "view" of the data, we provide the **data itself**. -* **API Access**: Full programmatic access via the Panoramax REST API for querying imagery by location, date, or sequence. * **Bulk Datasets**: Curated dumps of street-level imagery available for computer vision training, asset management, and change detection models. * **[FAIR Data Catalogue](https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/) Integration**: Seamless integration with geospatial workflows (ex. DuckDB, QGIS, Python, R, Julia, etc.). diff --git a/content/docs/d4c-pkgs/d4c-datapkg-foundation.md b/content/docs/d4c-pkgs/d4c-datapkg-foundation.md index 7e2bc6e..bc0e8e5 100644 --- a/content/docs/d4c-pkgs/d4c-datapkg-foundation.md +++ b/content/docs/d4c-pkgs/d4c-datapkg-foundation.md @@ -1,5 +1,5 @@ --- -title: d4c-datapkg-foundation +title: Foundation weight: 1 --- diff --git a/content/docs/d4c-pkgs/d4c-datapkg-orthoimagery.md b/content/docs/d4c-pkgs/d4c-datapkg-orthoimagery.md index 8f75390..f7d9a16 100644 --- a/content/docs/d4c-pkgs/d4c-datapkg-orthoimagery.md +++ b/content/docs/d4c-pkgs/d4c-datapkg-orthoimagery.md @@ -1,5 +1,5 @@ --- -title: d4c-datapkg-orthoimagery +title: Orthoimagery weight: 3 prev: /docs/d4c-pkgs/d4c-datapkg-statistical/statistics_canada/census_data/ next: /docs/d4c-pkgs/d4c-datapkg-field-imagery/ diff --git a/content/docs/d4c-pkgs/d4c-datapkg-statistical/_index.md b/content/docs/d4c-pkgs/d4c-datapkg-statistical/_index.md index 3b570ed..f6ea118 100644 --- a/content/docs/d4c-pkgs/d4c-datapkg-statistical/_index.md +++ b/content/docs/d4c-pkgs/d4c-datapkg-statistical/_index.md @@ -1,5 +1,5 @@ --- -title: d4c-datapkg-statistical +title: Statistical weight: 2 toc: true prev: /docs/d4c-pkgs/foundation/ diff --git a/high-level-overview-homepage.mermaid b/high-level-overview-homepage.mermaid index 05d4288..0fcb58e 100644 --- a/high-level-overview-homepage.mermaid +++ b/high-level-overview-homepage.mermaid @@ -9,9 +9,12 @@ flowchart TD Foundation@{ shape: lean-l} Orthoimagery@{ shape: lean-l} EnvironmentClimate@{ shape: lean-l, label: "Environment, Climate, & Health"} + style EnvironmentClimate fill:#FBC02D,stroke:#F9A825,color:#000000 FieldImagery@{ shape: lean-l, label: "Field Imagery"} - Elevation@{ shape: lean-l} + style FieldImagery fill:#FBC02D,stroke:#F9A825,color:#000000 WebCorpus@{ shape: lean-l, label: "Web Corpus"} + Elevation@{ shape: lean-l} + style WebCorpus fill:#66BB6A,stroke:#2E7D32,color:#000000 end %% --------------------------------------------------------- @@ -41,33 +44,46 @@ flowchart TD %% Box: Vector Tiles (Pastel Orange) subgraph vt [Vector Tiles] VectorTiles@{ shape: lean-l, label: "Mapbox Vector Tiles"} + style VectorTiles fill:#66BB6A,stroke:#2E7D32,color:#000000 NextGenVectorTiles@{ shape: lean-l, label: "Next-Gen Vector Tiles"} + style NextGenVectorTiles fill:#B71C1C,stroke:#7F0000,color:#FFFFFF end %% Box: Visuals (Pastel Blue - No Name) subgraph visuals [" "] + %% AVIF is not an option since QGIS does not seem to support it (I was surprised when my generated file was not compatible) WebP@{ shape: lean-l} + style WebP fill:#B71C1C,stroke:#7F0000,color:#FFFFFF JPG@{ shape: lean-l} + style JPG fill:#66BB6A,stroke:#2E7D32,color:#000000 PNG@{ shape: lean-l} + style PNG fill:#66BB6A,stroke:#2E7D32,color:#000000 end %% Box: Portable Databases (Pastel Green) subgraph pkg [Portable Databases] + %% Good format, but has some issues. I encountered issues when merging multiple adjoining PMTile files, the author is aware of these issues PMTiles@{ shape: lean-l} + %% Compatability with every OS. + %% Fun fact, you don't even need an OS for running SQLite + %% https://sqlite.org/selfcontained.html SQLite@{ shape: lean-l} end %% Box: Enterprise (Pastel Purple) subgraph ent [Enterprise] FileGeodatabase@{shape: lean-l, label: "File Geodatabase"} + style FileGeodatabase fill:#fff,stroke:#2E7D32,color:#000000 end end %% --------------------------------------------------------- %% 5. DISTRIBUTION INFRASTRUCTURE %% --------------------------------------------------------- - subgraph di [Distribution Infrastructure] - ObjectStorage@{ shape: bow-rect, label: "Object Storage"} + subgraph di [Distribution] + %% My goals involve going beyond object storage, so use generic term + ObjectStorage@{ shape: bow-rect, label: "Storage"} + style ObjectStorage fill:#FBC02D,stroke:#F9A825,color:#000000 Metadata@{ shape: rect, label: "FAIR Data Catalogue"} HTTP@{ shape: rect, label: "Systems-Ready Data"} DecentralizedDistribution@{ shape: rect, label: "Decentralized Distribution"} @@ -78,6 +94,8 @@ flowchart TD %% --------------------------------------------------------- subgraph ei [Experimental Infrastructure] GeoSpatialServices@{ shape: rect, label: "Geospatial Services"} + %% Does not make sense. Focus on delivering the data + style GeoSpatialServices fill:#66BB6A,stroke:#2E7D32,color:#000000 end %% --------------------------------------------------------- @@ -178,6 +196,10 @@ style DecentralizedDistribution fill:#B71C1C,stroke:#7F0000,color:#FFFFFF style HTTP fill:#B71C1C,stroke:#7F0000,color:#FFFFFF style Systems fill:#B71C1C,stroke:#7F0000,color:#FFFFFF style Metadata fill:#B71C1C,stroke:#7F0000,color:#FFFFFF +style Statistical fill:#B71C1C,stroke:#7F0000,color:#FFFFFF +style Foundation fill:#B71C1C,stroke:#7F0000,color:#FFFFFF +style Orthoimagery fill:#B71C1C,stroke:#7F0000,color:#FFFFFF +style Elevation fill:#66BB6A,stroke:#2E7D32,color:#000000 %%style df fill:#D32F2F,stroke:#8E0000,color:#FFFFFF style sot fill:#EF9A9A,stroke:#C62828,color:#000000 @@ -202,10 +224,9 @@ style visuals fill:#FBC02D,stroke:#F9A825,color:#000000 style ent fill:#66BB6A,stroke:#2E7D32,color:#000000 style DataSci fill:#D32F2F,stroke:#8E0000,color:#FFFFFF -style GeoSpatialServices fill:#FFCC80,stroke:#FB8C00,color:#000000 -class Foundation,Statistical,Orthoimagery,FieldImagery,EnvironmentClimate,Elevation,WebCorpus linkNode +class FieldImagery linkNode class Parquet,FlatGeoBuf,SQLite,FileGeodatabase,VectorTiles,NextGenVectorTiles,GeoTIFF,Zarr,WebP,PMTiles,JPEGXL,AV1,WARC linkNode %% ========================================================= diff --git a/static/high-level-overview-homepage.svg b/static/high-level-overview-homepage.svg index 5862a54..6e016fd 100644 --- a/static/high-level-overview-homepage.svg +++ b/static/high-level-overview-homepage.svg @@ -1 +1 @@ -

Consumption

Distribution Infrastructure

Data Sources

Experimental Infrastructure

Geospatial Services

Dissemination Formats

Enterprise

File Geodatabase

Long-Term Storage

Parquet

Zarr

GeoTIFF

Next-Gen Video

Next-Gen Imagery

Unstructured Web Data

FAIR Data Catalogue

FlatGeoBuf

Vector Tiles

Mapbox Vector Tiles

Next-Gen Vector Tiles

WebP

JPG

PNG

Portable Databases

PMTiles

SQLite

Statistical

Foundation

Orthoimagery

Environment, Climate, & Health

Field Imagery

Elevation

Web Corpus

Data Packages

Object Storage

FAIR Data Catalogue

Systems-Ready Data

Decentralized Distribution

Data People & Developers

Systems

\ No newline at end of file +

Consumption

Distribution

Data Sources

Experimental Infrastructure

Geospatial Services

Dissemination Formats

Enterprise

File Geodatabase

Long-Term Storage

Parquet

Zarr

GeoTIFF

Next-Gen Video

Next-Gen Imagery

Unstructured Web Data

FAIR Data Catalogue

FlatGeoBuf

Vector Tiles

Mapbox Vector Tiles

Next-Gen Vector Tiles

WebP

JPG

PNG

Portable Databases

PMTiles

SQLite

Statistical

Foundation

Orthoimagery

Environment, Climate, & Health

Field Imagery

Web Corpus

Elevation

Data Packages

Storage

FAIR Data Catalogue

Systems-Ready Data

Decentralized Distribution

Data People & Developers

Systems

\ No newline at end of file