mirror of
https://github.com/dataforcanada/d4c-service-main-site.git
synced 2026-06-13 14:00:51 +02:00
Diego Ripley
63d4614531
- Define how to structure data packages, see https://github.com/dataforcanada/d4c-pkgs/issues/2 and https://github.com/dataforcanada/d4c-pkgs/issues/3 - Implement https://github.com/dataforcanada/d4c-infra-distribution/issues/16 in one of the data packages - Invite community to one simple data package (ex. d4c-datapkg-orthoimagery), there was interest in this event https://discourse.pangeo.io/t/pangeo-showcase-no-shard-feelings-geozarr-rendering-in-qgis-powered-by-gdal-march-4-2026-at-12-pm-et/5526 about converting this dataset into GeoZarr https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/archive/ca-on_province_of_ontario-2024A000235_drape_eastern_ontario_orthoimagery_2024_16cm. Make use of professionals that have taken an interest. - I have further strategy, but these should be priority
219 lines
9.3 KiB
Plaintext
219 lines
9.3 KiB
Plaintext
flowchart TD
|
|
classDef linkNode stroke:#333333,color:#333333,stroke-width:1.5px
|
|
classDef legendRed fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
classDef legendYellow fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
classDef legendGreen fill:#66BB6A,stroke:#2E7D32,color:#000000
|
|
|
|
subgraph ds [Data Sources]
|
|
Statistical@{ shape: lean-l}
|
|
Foundation@{ shape: lean-l}
|
|
%% I have some big ideas for this data package, but it will take some exploratory work
|
|
EnvClimate@{ shape: lean-l, label: "Environment, Climate, & Health"}
|
|
Orthoimagery@{ shape: lean-l}
|
|
FieldImagery@{ shape: lean-l, label: "Field Imagery"}
|
|
Elevation@{ shape: lean-l}
|
|
WebCorpus@{ shape: lean-l, label: "Web Corpus"}
|
|
end
|
|
|
|
DataPkgs@{ shape: rect, label: "Data Packages"}
|
|
|
|
subgraph df [Dissemination Formats]
|
|
subgraph sot [Long-Term Storage]
|
|
Parquet@{ shape: lean-l}
|
|
Zarr@{ shape: lean-l}
|
|
GeoTIFF@{ shape: lean-l}
|
|
JPEGXL@{ shape: lean-l, label: "JPEG XL"}
|
|
AV1@{ shape: lean-l, label: "AV1"}
|
|
%% Commented out since I'm pretty sure this is not ideal file format. Ideal file format is Parquet and other file formats outlined depending on need. For example, let's say we archive media posts from various platforms (ex. X, BlueSky, etc.), there's no need to archive the webpage if we can just parse the content and have significant savings.
|
|
%% If we do archive webpages, I want there to be a deduplicating component similar to BTRFS, The Internet Archive is way too wasteful with the way they archive webpages.
|
|
%%WARC@{ shape: lean-l, label: "Unstructured Web Data"}
|
|
FAIRCat@{ shape: lean-l, label: "FAIR Data Catalogue"}
|
|
end
|
|
|
|
FlatGeoBuf@{ shape: lean-l}
|
|
|
|
subgraph vt [Vector Tiles]
|
|
VectorTiles@{ shape: lean-l, label: "Mapbox Vector Tiles"}
|
|
NextGenVT@{ shape: lean-l, label: "Next-Gen Vector Tiles"}
|
|
end
|
|
|
|
subgraph visuals [" "]
|
|
AVIF@{ shape: lean-l}
|
|
WebP@{ shape: lean-l}
|
|
JPG@{ shape: lean-l}
|
|
PNG@{ shape: lean-l}
|
|
end
|
|
|
|
subgraph pkg [Portable Databases]
|
|
PMTiles@{ shape: lean-l}
|
|
SQLite@{ shape: lean-l}
|
|
end
|
|
|
|
subgraph ent [Enterprise]
|
|
FileGDB@{shape: lean-l, label: "File Geodatabase"}
|
|
end
|
|
end
|
|
|
|
subgraph di [Distribution]
|
|
ObjStorage@{ shape: bow-rect, label: "Storage"}
|
|
Metadata@{ shape: rect, label: "FAIR Data Catalogue"}
|
|
HTTP@{ shape: rect, label: "Systems-Ready Data"}
|
|
DecenDist@{ shape: rect, label: "Decentralized Distribution"}
|
|
end
|
|
|
|
subgraph ei [Experimental Infrastructure]
|
|
Services@{ shape: rect, label: "Services"}
|
|
end
|
|
|
|
subgraph consumption [Consumption]
|
|
DataSci@{ shape: rect, label: "Data People & Developers"}
|
|
Systems@{ shape: rect, label: "Systems"}
|
|
end
|
|
|
|
Statistical e1@<--> DataPkgs
|
|
e1@{animate: true, animation: slow}
|
|
Foundation e2@<--> DataPkgs
|
|
e2@{animate: true, animation: slow}
|
|
Orthoimagery e3@<--> DataPkgs
|
|
e3@{animate: true, animation: slow}
|
|
FieldImagery e7@<--> DataPkgs
|
|
e7@{animate: true, animation: fast}
|
|
EnvClimate e4@<--> DataPkgs
|
|
e4@{animate: true, animation: fast}
|
|
Elevation e5@<--> DataPkgs
|
|
e5@{animate: true, animation: slow}
|
|
WebCorpus e6@<--> DataPkgs
|
|
e6@{animate: true, animation: fast}
|
|
|
|
DataPkgs e8@--> df
|
|
e8@{animate: true, animation: fast}
|
|
|
|
sot e9@<--> FlatGeoBuf
|
|
e9@{animate: true, animation: fast}
|
|
|
|
FlatGeoBuf e10@--> vt
|
|
e10@{animate: true, animation: fast}
|
|
|
|
sot e11@<--> visuals
|
|
e11@{animate: true, animation: slow}
|
|
|
|
vt e12@<--> pkg
|
|
e12@{animate: true, animation: fast}
|
|
|
|
visuals e13@<--> pkg
|
|
e13@{animate: true, animation: slow}
|
|
|
|
sot e14@<--> ent
|
|
e14@{animate: true, animation: slow}
|
|
|
|
visuals e15@--> ent
|
|
e15@{animate: true, animation: slow}
|
|
|
|
df e16@<--> di
|
|
e16@{animate: true, animation: slow}
|
|
|
|
ObjStorage e17@<--> Metadata
|
|
e17@{animate: true, animation: slow}
|
|
Metadata e18@<--> HTTP
|
|
e18@{animate: true, animation: slow}
|
|
|
|
HTTP e19@<--> ei
|
|
e19@{animate: true, animation: slow}
|
|
HTTP e20@<--> DecenDist
|
|
e20@{animate: true, animation: slow}
|
|
HTTP e21@<--> DataSci
|
|
e21@{animate: true, animation: slow}
|
|
|
|
DecenDist e22@--> Systems
|
|
e22@{animate: true, animation: fast}
|
|
DecenDist e23@--> DataSci
|
|
e23@{animate: true, animation: fast}
|
|
|
|
Systems e24@<--> DataSci
|
|
e24@{animate: true, animation: fast}
|
|
ei e25@<--> DataSci
|
|
e25@{animate: true, animation: slow}
|
|
|
|
consumption ~~~ legend
|
|
subgraph legend [Priority Legend]
|
|
direction LR
|
|
L1[High]:::legendRed ~~~ L2[Medium]:::legendYellow ~~~ L3[Low]:::legendGreen
|
|
end
|
|
|
|
style EnvClimate fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
style Orthoimagery fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
style FieldImagery fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
style WebCorpus fill:#66BB6A,stroke:#2E7D32,color:#000000
|
|
style Elevation fill:#66BB6A,stroke:#2E7D32,color:#000000
|
|
|
|
style VectorTiles fill:#66BB6A,stroke:#2E7D32,color:#000000
|
|
style NextGenVT fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
%% This is in the ideal file format. As of 2026-03-05, it is mostly supported across major browsers
|
|
style AVIF fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
style WebP fill:#FFCC80,stroke:#FB8C00,color:#000000
|
|
style JPG fill:#66BB6A,stroke:#2E7D32,color:#000000
|
|
style PNG fill:#66BB6A,stroke:#2E7D32,color:#000000
|
|
style FileGDB fill:#fff,stroke:#2E7D32,color:#000000
|
|
style Services fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
style ObjStorage fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
|
|
style DataPkgs fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
style FAIRCat fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
%% I'm not as concerned about distribution of data. I have made some progress on smart nodes so that's going to be a YUGE release
|
|
style DecenDist fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
style HTTP fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
style Systems fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
style Metadata fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
style Statistical fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
style Foundation fill:#B71C1C,stroke:#7F0000,color:#FFFFFF
|
|
|
|
style sot fill:#EF9A9A,stroke:#C62828,color:#000000
|
|
style Parquet fill:#FFCDD2,stroke:#E57373,color:#000000
|
|
style Zarr fill:#FFCDD2,stroke:#E57373,color:#000000
|
|
style GeoTIFF fill:#FFCC80,stroke:#FB8C00,color:#000000
|
|
style JPEGXL fill:#FFCDD2,stroke:#E57373,color:#000000
|
|
%%style WARC fill:#FFCDD2,stroke:#E57373,color:#000000
|
|
style AV1 fill:#FFCDD2,stroke:#E57373,color:#000000
|
|
|
|
style pkg fill:#FFB74D,stroke:#EF6C00,color:#000000
|
|
style SQLite fill:#EF6C00,stroke:#E65100,color:#000000
|
|
style PMTiles fill:#FFCC80,stroke:#FB8C00,color:#000000
|
|
|
|
style vt fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
style FlatGeoBuf fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
style visuals fill:#FBC02D,stroke:#F9A825,color:#000000
|
|
|
|
style ent fill:#66BB6A,stroke:#2E7D32,color:#000000
|
|
style DataSci fill:#D32F2F,stroke:#8E0000,color:#FFFFFF
|
|
|
|
style legend fill:none,stroke:#999,stroke-width:1px
|
|
|
|
class FieldImagery linkNode
|
|
class Parquet,FlatGeoBuf,SQLite,FileGDB,VectorTiles,NextGenVT,GeoTIFF,Zarr,WebP,PMTiles,JPEGXL,AV1,WARC linkNode
|
|
|
|
click DataPkgs "https://github.com/dataforcanada/d4c-pkgs" _blank
|
|
click Foundation "https://github.com/dataforcanada/d4c-datapkg-foundation" _blank
|
|
click Statistical "https://github.com/dataforcanada/d4c-datapkg-statistical" _blank
|
|
click Orthoimagery "https://github.com/dataforcanada/d4c-datapkg-orthoimagery" _blank
|
|
click FieldImagery "https://github.com/dataforcanada/d4c-datapkg-field-imagery" _blank
|
|
click EnvClimate "https://github.com/dataforcanada/d4c-datapkg-environment-climate-health" _blank
|
|
click Elevation "https://github.com/dataforcanada/d4c-datapkg-elevation" _blank
|
|
click WebCorpus "https://github.com/dataforcanada/d4c-datapkg-web-corpus" _blank
|
|
click Parquet "https://github.com/apache/parquet-format/" _blank
|
|
click FlatGeoBuf "https://flatgeobuf.org/" _blank
|
|
click SQLite "https://www.geopackage.org/" _blank
|
|
click FileGDB "https://gdal.org/en/stable/drivers/vector/openfilegdb.html" _blank
|
|
click VectorTiles "https://github.com/mapbox/vector-tile-spec/" _blank
|
|
click NextGenVT "https://github.com/maplibre/maplibre-tile-spec/" _blank
|
|
click GeoTIFF "https://cogeo.org/" _blank
|
|
click Zarr "https://github.com/zarr-developers/geozarr-spec/" _blank
|
|
click WebP "https://developers.google.com/speed/webp/" _blank
|
|
click PMTiles "https://github.com/protomaps/PMTiles/blob/main/spec/v3/spec.md" _blank
|
|
click JPEGXL "https://jpeg.org/jpegxl/" _blank
|
|
click AV1 "https://aomedia.org/specifications/av1/" _blank
|
|
%%click WARC "https://github.com/iipc/warc-specifications/" _blank
|
|
click FAIRCat "https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/" _blank
|
|
click HTTP "https://www.dataforcanada.org/docs/" _blank
|
|
click DecenDist "https://www.dataforcanada.org/docs/d4c-infra-distribution/" _blank
|
|
click Metadata "https://stac-utils.github.io/stac-geoparquet/latest/spec/stac-geoparquet-spec/" _blank
|