dataforcanada/d4c-datapkg-statistical
1
0
Fork 0
mirror of https://github.com/dataforcanada/d4c-datapkg-statistical.git synced 2026-06-13 14:10:55 +02:00
Code Issues Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
Diego Ripley
2025-05-24 13:37:31 -04:00
commit f93e4d0cec
108 changed files with 11689 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.devcontainer/devcontainer.json
+26
View File
@@ -0,0 +1,26 @@
{
"name": "process-statcan-data",
"dockerComposeFile": "../docker-compose.yml",
"service": "devcontainer",
"workspaceFolder": "/workspace",
"shutdownAction": "stopCompose",
"forwardPorts": [
5432,
8888
],
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-toolsai.jupyter",
"vsls-contrib.gistfs",
"vscode-icons-team.vscode-icons",
"dorzey.vscode-sqlfluff"
],
"settings": {
"python.pythonPath": "/root/.venv/bin/python",
"python.defaultInterpreterPath": "/root/.venv/bin/python"
}
}
}
}
.env
+5
View File
@@ -0,0 +1,5 @@
POSTGRES_PASSWORD=floral-retype-viewless
POSTGRES_USER=statcan
POSTGRES_DB=statcan
# For Jupyter Notebook
DATA_FOLDER=/data
.gitignore
+3
View File
@@ -0,0 +1,3 @@
data/
.ipynb_checkpoints
__pycache__/
Dockerfile
+54
View File
@@ -0,0 +1,54 @@
FROM ghcr.io/osgeo/gdal:ubuntu-full-3.11.0
USER root
RUN apt-get update -y \
&& apt-get -y install gcc g++ make libsqlite3-dev zlib1g-dev
# Utilities
RUN apt-get install -y neovim \
python3-neovim \
htop \
tmux \
git \
aria2 \
postgresql-client \
zip
# tippecanoe
RUN git clone https://github.com/felt/tippecanoe.git \
&& cd tippecanoe \
&& git checkout 2.78.0 \
&& make -j \
&& make install \
&& rm /home/jovyan/tippecanoe -rf
# Install uv
RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
# Create virtual environment and install Python packages
RUN uv venv ~/.venv \
&& cd ~ \
&& uv pip install 'geopandas[all]' duckdb psycopg2-binary jupyterlab lonboard click stats-can openpyxl ordered-set sqlfluff
# Bash Kernel
RUN cd ~ \
&& uv pip install bash_kernel \
&& /root/.venv/bin/python -m bash_kernel.install
# Install DuckDB
RUN mkdir -p ~/.local/bin \
&& curl https://install.duckdb.org | sh
# Install rclone
RUN curl https://rclone.org/install.sh | bash
# When user logs in, we use the spatial virtual environment
RUN echo 'source /root/.venv/bin/activate' > ~/.bashrc \
&& echo 'export PATH="~/.local/bin:${PATH}"' >> ~/.bashrc
RUN mkdir /data
RUN apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& uv cache clean
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) Diego Ripley. All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+34
View File
@@ -0,0 +1,34 @@
## Table of Contents
- [About](#-about)
- [How to Run](#-how-to-run)
- [License](#-license)
## About
**process-statcan-data** is a set of scripts that helps you load all Statistics Canada data required for data analysis. Specifically this loads:
- **Boundaries**: Geographic boundaries from 2001 to 2021.
- **Road Network Files**: Road network files from 2001 to 2021.
- **Health Regions**: Health Regions from 2003 to 2023.
- **National Address Register**: National Address Register files from 2022 to 2024.
- **Census of Population**: Census of Population data from 2001 to 2021.
- **Census of Agriculture**: Census of Agriculture from 2001 to 2021.
- **National Household Survey**: National Household Survey from 2011 to 2016.
## How to Run
We make use of Dev Container to run the code:
```shell
# Clone the repository
git clone https://github.com/dataforcanada/process-statcan-data.git
# Navigate to the project directory
cd process-statcan-data
```
## License
This product is distributed under an MIT license.
[Back to top](#top)
boundaries/README.md
+46
View File
@@ -0,0 +1,46 @@
# TODO
- Process 2023 Federal Electoral Districts
- For `load.sh`
- Finish processing 2001 data
- For `country.sql`
- Create `country_2001` from 2001 geometries. Need to finish `load.sh`
- Add English abbreviation for all years
- Add French abbreviation for all years
- For `geographic_regions_of_canada.sql`
- Add other years (2016, 2011, 2006, 2001)
- Add GRC abbreviation english
- Add GRC abbreviation french
- According to this, Territories DGUID should be `2021A00016` https://www150.statcan.gc.ca/n1/en/geo?geotext=Territories%20%5BRegion%5D&geocode=A00016
- According to the link above, British Columbia DGUID should be `2021A00015`
- For `er_2021`, split `er_name` into English and French components. There's some records that are separated by `/`
- South Coast--Burin Peninsula / Côte-sud--Burin Peninsula
- West Coast--Northern Peninsula--Labrador / Côte-ouest--Northern Peninsula--Labrador
- Prince Edward Island / Île-du-Prince-Édouard
- For `cma_2021`, split `cma_name` into English and French components. There's some records that are separated by `/`
- Greater Sudbury / Grand Sudbury
- Ottawa - Gatineau (Ontario part / partie de l'Ontario)
- For `ccs_2021`, split `ccs_name` into English and French components. There's some records that are separated by `/`
- West Nipissing / Nipissing Ouest
- French River / Rivière des Français
- Greater Sudbury / Grand Sudbury
- The Nation / La Nation
- For `csd_2021`, split `csd_name` into English and French components. There's some records that are separated by `/`
- The Nation / La Nation
- West Nipissing / Nipissing Ouest
- Greater Sudbury / Grand Sudbury
- Beaubassin East / Beaubassin-est
- For `csd_2021`, figure out what level of geography the sac_code and sac_type belongs to so I can name it appropriately
- For `pop_ctr_2021`, split `pop_ctr_name` into English and French components. There's one record that is separated by `/`
- Grand Falls / Grand-Sault
- For `dpl_2021`, split `dpl_name` into English and French components. There's records that are separated by `/`
- Saint Irénée and Alderwood / Saint Irénée et Alderwood
- `Sainte-Anne-de-Kent part B / partie B` - this one would need to be split into `Sainte-Anne-de-Kent part B` and `Sainte-Anne-de-Kentpartie partie B`
boundaries/aggregate_dissemination_areas.sql
+59
View File
@@ -0,0 +1,59 @@
/*
Aggregate Dissemination Areas
*/
/* 2021
Definition here: https://web.archive.org/web/20240731061904/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo053
*/
DROP TABLE IF EXISTS silver.ada_2021;
CREATE TABLE silver.ada_2021 AS
SELECT DISTINCT
cd.country_dguid,
cd.country_en_name,
cd.country_fr_name,
cd.country_en_abbreviation,
cd.country_fr_abbreviation,
cd.grc_dguid,
cd.grc_en_name,
cd.grc_fr_name,
cd.pr_dguid,
cd.pr_en_name,
cd.pr_fr_name,
cd.pr_en_abbreviation,
cd.pr_fr_abbreviation,
cd.pr_iso_code,
cd.car_dguid,
cd.car_en_name,
cd.car_fr_name,
cd.cd_dguid,
cd.cd_name,
cd.cd_type,
cma.cma_dguid,
cma.cma_p_dguid,
cma.cma_name,
cma.cma_type,
dgr.ada_dguid,
ada.geom
FROM silver.dissemination_geographies_relationship_2021 AS dgr
LEFT JOIN silver.cd_2021 AS cd
ON dgr.cd_dguid = cd.cd_dguid
LEFT JOIN silver.cma_2021 AS cma
ON
concat(dgr.cma_dguid, dgr.cma_p_dguid)
= concat(cma.cma_dguid, cma.cma_p_dguid)
LEFT JOIN bronze.lada000a21a_e AS ada
ON dgr.ada_dguid = ada.dguid;
-- Make geometries valid
UPDATE
silver.ada_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX ada_2021_geom_idx ON silver.ada_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/boundary_files_2001.txt
+16
View File
@@ -0,0 +1,16 @@
# https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2001-eng.cfm
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gpr_000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gpr_000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gfed000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gfed000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/ger_000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ger_000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gcd_000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcd_000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gcar000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcar000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gccs000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gccs000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gcsd000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcsd000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gcma000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcma000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gct_000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gct_000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gda_000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gda_000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gdb_000a01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gdb_000a01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gdpl000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gdpl000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gua_000b01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gua_000b01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/gecu000e01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gecu000e01m_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2001/geca000e01m_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/geca000e01m_e.zip
boundaries/boundary_files_2006.txt
+16
View File
@@ -0,0 +1,16 @@
# https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2006-eng.cfm
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gpr_000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gpr_000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gfed000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gfed000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/ger_000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ger_000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gcd_000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcd_000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gcar000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcar000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gccs000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gccs000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gcsd000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcsd000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gcma000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcma000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gct_000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gct_000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gda_000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gda_000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gdb_000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gdb_000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gdpl000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gdpl000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gua_000a06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gua_000a06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/gecu000e06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gecu000e06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2006/geca000e06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/geca000e06a_e.zip
boundaries/boundary_files_2011.txt
+17
View File
@@ -0,0 +1,17 @@
# https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2011-eng.cfm
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gpr_000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gpr_000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gfed000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gfed000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/ger_000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ger_000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gcd_000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcd_000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gcar000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcar000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gccs000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gccs000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gcsd000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcsd000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gcma000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gcma000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gct_000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gct_000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gda_000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gda_000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gdb_000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gdb_000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gdpl000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gdpl000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gpc_000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gpc_000a11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gecu000e11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gecu000e11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/geca000e11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/geca000e11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2011/gfsa000a11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/gfsa000a11a_e.zip
boundaries/boundary_files_2016.txt
+18
View File
@@ -0,0 +1,18 @@
# https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2016-eng.cfm
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lpr_000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lpr_000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lfed000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lfed000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/ler_000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/ler_000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lcd_000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lcd_000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lada000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lada000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lcar000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lcar000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lccs000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lccs000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lcsd000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lcsd000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lcma000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lcma000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lct_000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lct_000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lda_000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lda_000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/ldb_000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/ldb_000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/ldpl000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/ldpl000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lpc_000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lpc_000a16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lecu000e16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lecu000e16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/geca000e16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/geca000e16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2016/lfsa000a16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lfsa000a16a_e.zip
boundaries/boundary_files_2021.txt
+18
View File
@@ -0,0 +1,18 @@
# https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/index2021-eng.cfm?year=21
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lpr_000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lpr_000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lcd_000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lcd_000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lfed000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lfed000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lcsd000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lcsd000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/ldpl000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/ldpl000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lfsa000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lfsa000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/ler_000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/ler_000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lcar000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lcar000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lccs000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lccs000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lcma000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lcma000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lct_000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lct_000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lpc_000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lpc_000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lda_000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lda_000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/ldb_000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/ldb_000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lada000a21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lada000a21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/lecu000e21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/lecu000e21a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/boundaries/2021/leca000e21a_e.zip https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/files-fichiers/leca000e21a_e.zip
boundaries/census_agricultural_regions.sql
+49
View File
@@ -0,0 +1,49 @@
/*
Census Agricultural Regions
*/
/* 2021
Definition here: https://web.archive.org/web/20250401192328/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/definition-eng.cfm?ID=geo006
*/
DROP TABLE IF EXISTS silver.car_2021;
CREATE TABLE silver.car_2021 AS
SELECT DISTINCT
pr.country_dguid,
pr.country_en_name,
pr.country_fr_name,
pr.country_en_abbreviation,
pr.country_fr_abbreviation,
pr.grc_dguid,
pr.grc_en_name,
pr.grc_fr_name,
pr.pr_dguid,
pr.pr_en_name,
pr.pr_fr_name,
pr.pr_en_abbreviation,
pr.pr_fr_abbreviation,
pr.pr_iso_code,
dgr.car_dguid,
car.carename AS car_en_name,
car.carfname AS car_fr_name,
car.geom
FROM
silver.pr_2021 AS pr,
silver.dissemination_geographies_relationship_2021 AS dgr,
bronze.lcar000a21a_e AS car
WHERE
pr.pr_dguid = dgr.pr_dguid
AND car.dguid = dgr.car_dguid;
-- Make geometries valid
UPDATE
silver.car_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX car_2021_geom_idx ON silver.car_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/census_consolidated_subdivisions.sql
+55
View File
@@ -0,0 +1,55 @@
/*
Census Consolidated Subdivisions
*/
/* 2021
Definition here: https://web.archive.org/web/20250401192303/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo007
*/
DROP TABLE IF EXISTS silver.ccs_2021;
CREATE TABLE silver.ccs_2021 AS
SELECT DISTINCT
cd.country_dguid,
cd.country_en_name,
cd.country_fr_name,
cd.country_en_abbreviation,
cd.country_fr_abbreviation,
cd.grc_dguid,
cd.grc_en_name,
cd.grc_fr_name,
cd.pr_dguid,
cd.pr_en_name,
cd.pr_fr_name,
cd.pr_en_abbreviation,
cd.pr_fr_abbreviation,
cd.pr_iso_code,
cd.car_dguid,
cd.car_en_name,
cd.car_fr_name,
cd.cd_dguid,
cd.cd_name,
cd.cd_type,
ccs.dguid AS ccs_dguid,
ccs.ccsname AS ccs_name,
ccs.geom
FROM
silver.cd_2021 AS cd,
silver.dissemination_geographies_relationship_2021 AS dgr,
bronze.lccs000a21a_e AS ccs
WHERE
cd.cd_dguid = dgr.cd_dguid
AND
ccs.dguid = dgr.ccs_dguid;
-- Make geometries valid
UPDATE
silver.ccs_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX ccs_2021_geom_idx ON silver.ccs_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/census_divisions.sql
+57
View File
@@ -0,0 +1,57 @@
/*
Census Divisions
*/
/* 2021
Definition here: https://web.archive.org/web/20250131082459/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/definition-eng.cfm?ID=geo008#moreinfo
*/
DROP TABLE IF EXISTS silver.cd_2021;
CREATE TABLE silver.cd_2021 AS
SELECT DISTINCT
pr.country_dguid,
pr.country_en_name,
pr.country_fr_name,
pr.country_en_abbreviation,
pr.country_fr_abbreviation,
pr.grc_dguid,
pr.grc_en_name,
pr.grc_fr_name,
pr.pr_dguid,
pr.pr_en_name,
pr.pr_fr_name,
pr.pr_en_abbreviation,
pr.pr_fr_abbreviation,
pr.pr_iso_code,
dgr.car_dguid,
car.carename AS car_en_name,
car.carfname AS car_fr_name,
cd.dguid AS cd_dguid,
cd.cdname AS cd_name,
cd.cdtype AS cd_type,
cd.geom
FROM
silver.pr_2021 AS pr,
bronze.lcd_000a21a_e AS cd,
silver.dissemination_geographies_relationship_2021 AS dgr,
bronze.lcar000a21a_e AS car
WHERE
pr.pr_dguid = dgr.pr_dguid
AND cd.dguid = dgr.cd_dguid
AND dgr.car_dguid = car.dguid;
-- Make geometries valid
UPDATE
silver.cd_2021
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX cd_2021_geom_idx ON silver.cd_2021
USING gist (geom) WITH (fillfactor = 100);
/* 2016
Definition here: https://web.archive.org/web/20250304001456/https://www12.statcan.gc.ca/census-recensement/2016/ref/dict/geo008-eng.cfm
*/
boundaries/census_metropolitan_areas.sql
+56
View File
@@ -0,0 +1,56 @@
/*
Census Metropolitan Areas
*/
/* 2021
Definition here: https://web.archive.org/web/20250518133322/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo009
*/
DROP TABLE IF EXISTS silver.cma_2021;
CREATE TABLE silver.cma_2021 AS
SELECT DISTINCT
pr.country_dguid,
pr.country_en_name,
pr.country_fr_name,
pr.country_en_abbreviation,
pr.country_fr_abbreviation,
pr.grc_dguid,
pr.grc_en_name,
pr.grc_fr_name,
pr.pr_dguid,
pr.pr_en_name,
pr.pr_fr_name,
pr.pr_en_abbreviation,
pr.pr_fr_abbreviation,
pr.pr_iso_code,
cma.dguid AS cma_dguid,
cma.dguidp AS cma_p_dguid,
cma.cmaname AS cma_name,
cma.cmatype AS cma_type,
cma.geom
FROM
silver.pr_2021 AS pr,
bronze.lcma000a21a_e AS cma,
silver.dissemination_geographies_relationship_2021 AS dgr
WHERE
pr.pr_dguid = dgr.pr_dguid
AND
concat(
cma.dguid,
cma.dguidp) = concat(
dgr.cma_dguid,
dgr.cma_p_dguid
);
-- Make geometries valid
UPDATE
silver.cma_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX cma_2021_geom_idx ON silver.cma_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/census_subdivisions.sql
+71
View File
@@ -0,0 +1,71 @@
/*
Census Subdivisions
*/
/* 2021
Definition here: https://web.archive.org/web/20240526213705/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo012
*/
DROP TABLE IF EXISTS silver.csd_2021;
CREATE TABLE silver.csd_2021 AS
SELECT DISTINCT
cd.country_dguid,
cd.country_en_name,
cd.country_fr_name,
cd.country_en_abbreviation,
cd.country_fr_abbreviation,
cd.grc_dguid,
cd.grc_en_name,
cd.grc_fr_name,
cd.pr_dguid,
cd.pr_en_name,
cd.pr_fr_name,
cd.pr_en_abbreviation,
cd.pr_fr_abbreviation,
cd.pr_iso_code,
cd.car_dguid,
cd.car_en_name,
cd.car_fr_name,
er.er_dguid,
er.er_name,
cd.cd_dguid,
cd.cd_name,
cd.cd_type,
ccs.ccs_dguid,
ccs.ccs_name,
gaf.cma_dguid,
gaf.cma_p_dguid,
cma.cma_name,
cma.cma_type,
gaf.csd_dguid,
csd.csdname AS csd_name,
csd.csdtype AS csd_type,
gaf.sac_type,
gaf.sac_code,
csd.geom
FROM silver.gaf_2021 AS gaf
LEFT JOIN silver.cma_2021 AS cma
ON
concat(gaf.cma_dguid, gaf.cma_p_dguid)
= concat(cma.cma_dguid, cma.cma_p_dguid)
LEFT JOIN silver.cd_2021 AS cd
ON gaf.cd_dguid = cd.cd_dguid
LEFT JOIN silver.er_2021 AS er
ON gaf.er_dguid = er.er_dguid
LEFT JOIN silver.ccs_2021 AS ccs
ON gaf.ccs_dguid = ccs.ccs_dguid
LEFT JOIN bronze.lcsd000a21a_e AS csd
ON gaf.csd_dguid = csd.dguid;
-- Make geometries valid
UPDATE
silver.csd_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX csd_2021_geom_idx ON silver.csd_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/census_tracts.sql
+52
View File
@@ -0,0 +1,52 @@
/*
Census Tracts
*/
/* 2021
Definition here: https://web.archive.org/web/20241013011815/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo013
*/
DROP TABLE IF EXISTS silver.ct_2021;
CREATE TABLE silver.ct_2021 AS
SELECT DISTINCT
cma.country_dguid,
cma.country_en_name,
cma.country_fr_name,
cma.country_en_abbreviation,
cma.country_fr_abbreviation,
cma.grc_dguid,
cma.grc_en_name,
cma.grc_fr_name,
cma.pr_dguid,
cma.pr_en_name,
cma.pr_fr_name,
cma.pr_en_abbreviation,
cma.pr_fr_abbreviation,
cma.pr_iso_code,
cma.cma_dguid,
cma.cma_p_dguid,
cma.cma_name,
cma.cma_type,
ct.dguid AS ct_dguid,
ct.geom
FROM silver.dissemination_geographies_relationship_2021 AS dgr
LEFT JOIN bronze.lct_000a21a_e AS ct
ON dgr.ct_dguid = ct.dguid
LEFT JOIN silver.cma_2021 AS cma
ON
concat(dgr.cma_dguid, dgr.cma_p_dguid)
= concat(cma.cma_dguid, cma.cma_p_dguid)
WHERE dgr.ct_dguid IS NOT null;
-- Make geometries valid
UPDATE
silver.ct_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX ct_2021_geom_idx ON silver.ct_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/country.sql
+131
View File
@@ -0,0 +1,131 @@
/*
Canada
*/
-- 2021 Canada;
DROP TABLE IF EXISTS silver.country_2021;
CREATE TABLE silver.country_2021 AS
SELECT DISTINCT
'2021A000011124' AS country_dguid,
'Canada' AS country_en_name,
'Canada' AS country_fr_name,
'CAN' AS country_en_abbreviation,
'CAN' AS country_fr_abbreviation,
ST_UNION(geom) AS geom
FROM
bronze.lpr_000a21a_e;
-- Make geometries valid
UPDATE
silver.country_2021
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX country_2021_geom_idx ON silver.country_2021
USING gist (geom) WITH (fillfactor = 100);
-- 2016 Canada;
DROP TABLE IF EXISTS silver.country_2016;
CREATE TABLE silver.country_2016 AS
SELECT DISTINCT
'2016A000011124' AS country_dguid,
'Canada' AS country_en_name,
'Canada' AS country_fr_name,
'CAN' AS country_en_abbreviation,
'CAN' AS country_fr_abbreviation,
ST_UNION(geom) AS geom
FROM
bronze.lpr_000a16a_e;
-- Make geometries valid
UPDATE
silver.country_2016
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX country_2016_geom_idx ON silver.country_2016
USING gist (geom) WITH (fillfactor = 100);
-- 2011 Canada;
DROP TABLE IF EXISTS silver.country_2011;
CREATE TABLE silver.country_2011 AS
SELECT DISTINCT
'2011A000011124' AS country_dguid,
'Canada' AS country_en_name,
'Canada' AS country_fr_name,
'CAN' AS country_en_abbreviation,
'CAN' AS country_fr_abbreviation,
st_union(geom) AS geom
FROM
bronze.gpr_000a11a_e;
-- Make geometries valid
UPDATE
silver.country_2011
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX country_2011_geom_idx ON silver.country_2011
USING gist (geom) WITH (fillfactor = 100);
-- 2006 Canada;
DROP TABLE IF EXISTS silver.country_2006;
CREATE TABLE silver.country_2006 AS
SELECT DISTINCT
'2006A000011124' AS country_dguid,
'Canada' AS country_en_name,
'Canada' AS country_fr_name,
'CAN' AS country_en_abbreviation,
'CAN' AS country_fr_abbreviation,
ST_UNION(geom) AS geom
FROM
bronze.gpr_000a06a_e;
-- Make geometries valid
UPDATE
silver.country_2006
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX country_2006_geom_idx ON silver.country_2006
USING gist (geom) WITH (fillfactor = 100);
-- 2001 Canada;
-- TODO
/*
-- Clean Provinces and Territories layer;
UPDATE
bronze.gpr_000a06a_e
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
DROP TABLE IF EXISTS country_2001;
CREATE TABLE country_2001 AS
SELECT
DISTINCT '2001A000011124' AS country_dguid,
'Canada' AS country_en_name,
'Canada' AS country_fr_name,
'CAN' AS country_en_abbreviation,
'CAN' AS country_fr_abbreviation,
st_union(geom) AS geom
FROM
lpr_000a21a_e;
CREATE INDEX country_2001_geom_idx ON
country_2001
USING GIST(geom) WITH (FILLFACTOR = 100);
*/
boundaries/designated_places.sql
+48
View File
@@ -0,0 +1,48 @@
/*
Designated Places
*/
/* 2021
Definition here: https://web.archive.org/web/20240731061904/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo018
*/
DROP TABLE IF EXISTS silver.dpl_2021;
CREATE TABLE silver.dpl_2021 AS
SELECT DISTINCT
pr.country_dguid,
pr.country_en_name,
pr.country_fr_name,
pr.country_en_abbreviation,
pr.country_fr_abbreviation,
pr.grc_dguid,
pr.grc_en_name,
pr.grc_fr_name,
pr.pr_dguid,
pr.pr_en_name,
pr.pr_fr_name,
pr.pr_en_abbreviation,
pr.pr_fr_abbreviation,
pr.pr_iso_code,
dgr.dpl_dguid,
dpl.dplname AS dpl_name,
dpl.dpltype AS dpl_type,
dpl.geom
FROM silver.dissemination_geographies_relationship_2021 AS dgr
LEFT JOIN silver.pr_2021 AS pr
ON dgr.pr_dguid = pr.pr_dguid
LEFT JOIN bronze.ldpl000a21a_e AS dpl
ON dgr.dpl_dguid = dpl.dguid
WHERE dgr.dpl_dguid IS NOT null;
-- Make geometries valid
UPDATE
silver.dpl_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX dpl_2021_geom_idx ON silver.dpl_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/dissemination_areas.sql
+66
View File
@@ -0,0 +1,66 @@
/*
Dissemination Areas
*/
/* 2021
Definition here: https://web.archive.org/web/20240731061905/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/definition-eng.cfm?ID=geo021
*/
DROP TABLE IF EXISTS silver.da_2021;
CREATE TABLE silver.da_2021 AS
SELECT DISTINCT
csd.country_dguid,
csd.country_en_name,
csd.country_fr_name,
csd.country_en_abbreviation,
csd.country_fr_abbreviation,
csd.grc_dguid,
csd.grc_en_name,
csd.grc_fr_name,
csd.pr_dguid,
csd.pr_en_name,
csd.pr_fr_name,
csd.pr_en_abbreviation,
csd.pr_fr_abbreviation,
csd.pr_iso_code,
csd.car_dguid,
csd.car_en_name,
csd.car_fr_name,
csd.er_dguid,
csd.er_name,
csd.cd_dguid,
csd.cd_name,
csd.cd_type,
csd.ccs_dguid,
csd.ccs_name,
csd.cma_dguid,
csd.cma_p_dguid,
csd.cma_name,
csd.cma_type,
csd.csd_dguid,
csd.csd_name,
csd.csd_type,
csd.sac_type,
csd.sac_code,
dgr.ct_dguid,
dgr.ada_dguid,
dgr.da_dguid,
da.geom
FROM silver.dissemination_geographies_relationship_2021 AS dgr
LEFT JOIN silver.csd_2021 AS csd
ON dgr.csd_dguid = csd.csd_dguid
LEFT JOIN bronze.lda_000a21a_e AS da
ON dgr.da_dguid = da.dguid;
-- Make geometries valid
UPDATE
bronze.lda_000a21a_e
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX da_2021_geom_idx ON silver.da_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/dissemination_blocks.sql
+73
View File
@@ -0,0 +1,73 @@
/*
Dissemination Blocks
*/
/* 2021
Definition here: https://web.archive.org/web/20250212081621/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/definition-eng.cfm?ID=geo014
*/
DROP TABLE IF EXISTS silver.db_2021;
CREATE TABLE silver.db_2021 AS
SELECT DISTINCT
csd.country_dguid,
csd.country_en_name,
csd.country_fr_name,
csd.country_en_abbreviation,
csd.country_fr_abbreviation,
csd.grc_dguid,
csd.grc_en_name,
csd.grc_fr_name,
csd.pr_dguid,
csd.pr_en_name,
csd.pr_fr_name,
csd.pr_en_abbreviation,
csd.pr_fr_abbreviation,
csd.pr_iso_code,
csd.car_dguid,
csd.car_en_name,
csd.car_fr_name,
csd.er_dguid,
csd.er_name,
csd.cd_dguid,
csd.cd_name,
csd.cd_type,
csd.ccs_dguid,
csd.ccs_name,
csd.cma_dguid,
csd.cma_p_dguid,
csd.cma_name,
csd.cma_type,
csd.csd_dguid,
csd.csd_name,
csd.csd_type,
csd.sac_type,
csd.sac_code,
dgr.fed_dguid,
fed.fed_name,
fed.fed_en_name,
fed.fed_fr_name,
dgr.ct_dguid,
dgr.ada_dguid,
dgr.da_dguid,
dgr.db_dguid,
db.geom
FROM silver.dissemination_geographies_relationship_2021 AS dgr
LEFT JOIN silver.csd_2021 AS csd
ON dgr.csd_dguid = csd.csd_dguid
LEFT JOIN silver.fed_2021 AS fed
ON dgr.fed_dguid = fed.fed_dguid
LEFT JOIN bronze.ldb_000a21a_e AS db
ON dgr.db_dguid = db.dguid;
-- Make geometries valid
UPDATE
silver.db_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX db_2021_geom_idx ON silver.db_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/download.sh
Executable
+19
View File
@@ -0,0 +1,19 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/boundaries" ]
then
echo "Making directory ${DATA_FOLDER}/boundaries/"
mkdir -p ${DATA_FOLDER}/boundaries/{input,extracted,output}/{2021,2016,2011,2001}
fi
INPUT_FOLDER="${DATA_FOLDER}/boundaries/input"
echo "Downloading 2021 boundaries"
aria2c -x16 -i "${SCRIPT_DIR}/boundaries/boundary_files_2021.txt" --dir=${INPUT_FOLDER}/2021 --auto-file-renaming=false
echo "Downloading 2016 boundaries"
aria2c -x16 -i "${SCRIPT_DIR}/boundaries/boundary_files_2016.txt" --dir=${INPUT_FOLDER}/2016 --auto-file-renaming=false
echo "Downloading 2011 boundaries"
aria2c -x16 -i "${SCRIPT_DIR}/boundaries/boundary_files_2011.txt" --dir=${INPUT_FOLDER}/2011 --auto-file-renaming=false
echo "Downloading 2006 boundaries"
aria2c -x16 -i "${SCRIPT_DIR}/boundaries/boundary_files_2006.txt" --dir=${INPUT_FOLDER}/2006 --auto-file-renaming=false
echo "Downloading 2001 boundaries"
aria2c -x16 -i "${SCRIPT_DIR}/boundaries/boundary_files_2001.txt" --dir=${INPUT_FOLDER}/2001 --auto-file-renaming=false
boundaries/economic_regions.sql
+49
View File
@@ -0,0 +1,49 @@
/*
Economic Regions
*/
/* 2021
Definition here: https://web.archive.org/web/20250518132130/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo022
*/
DROP TABLE IF EXISTS silver.er_2021;
CREATE TABLE silver.er_2021 AS
SELECT DISTINCT
pr.country_dguid,
pr.country_en_name,
pr.country_fr_name,
pr.country_en_abbreviation,
pr.country_fr_abbreviation,
pr.grc_dguid,
pr.grc_en_name,
pr.grc_fr_name,
pr.pr_dguid,
pr.pr_en_name,
pr.pr_fr_name,
pr.pr_en_abbreviation,
pr.pr_fr_abbreviation,
pr.pr_iso_code,
er.dguid AS er_dguid,
er.ername AS er_name,
er.geom
FROM
silver.pr_2021 AS pr,
silver.dissemination_geographies_relationship_2021 AS dgr,
bronze.ler_000a21a_e AS er
WHERE
pr.pr_dguid = dgr.pr_dguid
AND
er.dguid = dgr.er_dguid;
-- Make geometries valid
UPDATE
silver.er_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX er_2021_geom_idx ON silver.er_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/federal_electoral_districts.sql
+50
View File
@@ -0,0 +1,50 @@
/*
Federal Electoral Districts
*/
/* 2021
Definition here: https://web.archive.org/web/20240731061905/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo025
*/
-- 2021 vintage, 2013 representation order;
DROP TABLE IF EXISTS silver.fed_2021_2013;
CREATE TABLE silver.fed_2021_2013 AS
SELECT DISTINCT
pr.country_dguid,
pr.country_en_name,
pr.country_fr_name,
pr.country_en_abbreviation,
pr.country_fr_abbreviation,
pr.grc_dguid,
pr.grc_en_name,
pr.grc_fr_name,
pr.pr_dguid,
pr.pr_en_name,
pr.pr_fr_name,
pr.pr_en_abbreviation,
pr.pr_fr_abbreviation,
pr.pr_iso_code,
dgr.fed_dguid,
fed.fedname AS fed_name,
fed.fedename AS fed_en_name,
fed.fedfname AS fed_fr_name,
fed.geom
FROM silver.dissemination_geographies_relationship_2021 AS dgr
LEFT JOIN silver.pr_2021 AS pr
ON dgr.pr_dguid = pr.pr_dguid
LEFT JOIN bronze.lfed000a21a_e AS fed
ON dgr.fed_dguid = fed.dguid
WHERE dgr.fed_dguid IS NOT null;
-- Make geometries valid
UPDATE
silver.fed_2021_2013
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX fed_2021_2013_geom_idx ON silver.fed_2021_2013 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/forward_sortation_areas.sql
+43
View File
@@ -0,0 +1,43 @@
/*
Forward Sortation Areas
*/
/* 2021
Definition here: https://web.archive.org/web/20241102112247/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/Definition-eng.cfm?ID=geo036
*/
DROP TABLE IF EXISTS silver.fsa_2021;
CREATE TABLE silver.fsa_2021 AS
SELECT DISTINCT
pr.country_dguid,
pr.country_en_name,
pr.country_fr_name,
pr.country_en_abbreviation,
pr.country_fr_abbreviation,
pr.grc_dguid,
pr.grc_en_name,
pr.grc_fr_name,
pr.pr_dguid,
pr.pr_en_name,
pr.pr_fr_name,
pr.pr_en_abbreviation,
pr.pr_fr_abbreviation,
pr.pr_iso_code,
fsa.dguid AS fsa_dguid,
fsa.geom
FROM bronze.lfsa000a21a_e AS fsa,
silver.pr_2021 AS pr
WHERE concat('2021A0002', fsa.pruid) = pr.pr_dguid;
-- Make geometries valid
UPDATE
silver.fsa_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX fsa_2021_geom_idx ON silver.fsa_2021 USING gist (geom) WITH (
fillfactor = 100
);
boundaries/geographic_regions_of_canada.sql
+471
View File
@@ -0,0 +1,471 @@
/*
Geographic Regions of Canada
*/
/* 2021
Definition here: https://web.archive.org/web/20240624230708/https://www150.statcan.gc.ca/n1/pub/92-195-x/2021001/geo/region/region-eng.htm
*/
-- With geometries;
DROP TABLE IF EXISTS silver.grc_2021;
CREATE TABLE silver.grc_2021 AS
WITH territories AS (
SELECT
'2021A00016' AS grc_dguid,
'Territories' AS grc_en_name,
'Territoires' AS grc_fr_name,
ST_UNION(geom) AS geom
FROM
bronze.lpr_000a21a_e
WHERE
pruid IN ('60', '61', '62')
),
atlantic AS (
SELECT
'2021A00011' AS grc_dguid,
'Atlantic' AS grc_en_name,
'Atlantique' AS grc_fr_name,
ST_UNION(geom) AS geom
FROM
bronze.lpr_000a21a_e
WHERE
pruid IN ('10', '11', '12', '13')
),
prairies AS (
SELECT
'2021A00014' AS grc_dguid,
'Prairies' AS grc_en_name,
'Prairies' AS grc_fr_name,
ST_UNION(geom) AS geom
FROM
bronze.lpr_000a21a_e
WHERE
pruid IN ('48', '47', '46')
),
the_rest AS (
SELECT
CASE
WHEN pruid = '59' THEN '2021A00015'
WHEN pruid = '35' THEN '2021A00013'
WHEN pruid = '24' THEN '2021A00012'
END AS grc_dguid,
prename AS grc_en_name,
prfname AS grc_fr_name,
geom
FROM
bronze.lpr_000a21a_e
WHERE
pruid IN ('59', '35', '24')
),
final AS (
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
territories.*
FROM
territories,
silver.country_2021 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
prairies.*
FROM
prairies,
silver.country_2021 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
atlantic.*
FROM
atlantic,
silver.country_2021 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
the_rest.*
FROM
the_rest,
silver.country_2021 AS country
)
SELECT *
FROM
final;
-- Make geometries valid
UPDATE
silver.grc_2021
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX grc_2021_geom_idx ON silver.grc_2021
USING gist (geom) WITH (fillfactor = 100);
-- 2021 without geometries, and with pr_dguid;
DROP TABLE IF EXISTS silver.grc_pr_2021;
CREATE TABLE silver.grc_pr_2021 AS
WITH territories AS (
SELECT
'2021A00016' AS grc_dguid,
'Territories' AS grc_en_name,
'Territoires' AS grc_fr_name,
dguid AS pr_dguid
FROM
bronze.lpr_000a21a_e
WHERE
pruid IN ('60', '61', '62')
),
atlantic AS (
SELECT
'2021A00011' AS grc_dguid,
'Atlantic' AS grc_en_name,
'Atlantique' AS grc_fr_name,
dguid AS pr_dguid
FROM
bronze.lpr_000a21a_e
WHERE
pruid IN ('10', '11', '12', '13')
),
prairies AS (
SELECT
'2021A00014' AS grc_dguid,
'Prairies' AS grc_en_name,
'Prairies' AS grc_fr_name,
dguid AS pr_dguid
FROM
bronze.lpr_000a21a_e
WHERE
pruid IN ('48', '47', '46')
),
the_rest AS (
SELECT
CASE
WHEN pruid = '59' THEN '2021A00015'
WHEN pruid = '35' THEN '2021A00013'
WHEN pruid = '24' THEN '2021A00012'
END AS grc_dguid,
prename AS grc_en_name,
prfname AS grc_fr_name,
dguid AS pr_dguid
FROM
bronze.lpr_000a21a_e
WHERE
pruid IN ('59', '35', '24')
),
final AS (
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
territories.*
FROM
territories,
silver.country_2021 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
prairies.*
FROM
prairies,
silver.country_2021 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
atlantic.*
FROM
atlantic,
silver.country_2021 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
the_rest.*
FROM
the_rest,
silver.country_2021 AS country
)
SELECT *
FROM
final;
/* 2016
Definition here: https://web.archive.org/web/20240224030001/https://www12.statcan.gc.ca/census-recensement/2016/ref/dict/geo027a-eng.cfm
*/
-- With geometries;
DROP TABLE IF EXISTS silver.grc_2016;
CREATE TABLE silver.grc_2016 AS
WITH territories AS (
SELECT
'2016A00016' AS grc_dguid,
'Territories' AS grc_en_name,
'Territoires' AS grc_fr_name,
ST_UNION(geom) AS geom
FROM
bronze.lpr_000a16a_e
WHERE
pruid IN ('60', '61', '62')
),
atlantic AS (
SELECT
'2016A00011' AS grc_dguid,
'Atlantic' AS grc_en_name,
'Atlantique' AS grc_fr_name,
ST_UNION(geom) AS geom
FROM
bronze.lpr_000a16a_e
WHERE
pruid IN ('10', '11', '12', '13')
),
prairies AS (
SELECT
'2016A00014' AS grc_dguid,
'Prairies' AS grc_en_name,
'Prairies' AS grc_fr_name,
ST_UNION(geom) AS geom
FROM
bronze.lpr_000a16a_e
WHERE
pruid IN ('48', '47', '46')
),
the_rest AS (
SELECT
CASE
WHEN pruid = '59' THEN '2016A00015'
WHEN pruid = '35' THEN '2016A00013'
WHEN pruid = '24' THEN '2016A00012'
END AS grc_dguid,
prename AS grc_en_name,
prfname AS grc_fr_name,
geom
FROM
bronze.lpr_000a16a_e
WHERE
pruid IN ('59', '35', '24')
),
final AS (
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
territories.*
FROM
territories,
silver.country_2016 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
prairies.*
FROM
prairies,
silver.country_2016 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
atlantic.*
FROM
atlantic,
silver.country_2016 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
the_rest.*
FROM
the_rest,
silver.country_2016 AS country
)
SELECT *
FROM
final;
-- Make geometries valid
UPDATE
silver.grc_2016
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX grc_2016_geom_idx ON silver.grc_2016
USING gist (geom) WITH (fillfactor = 100);
-- 2016 without geometries, and with pr_dguid;
DROP TABLE IF EXISTS silver.grc_pr_2016;
CREATE TABLE silver.grc_pr_2016 AS
WITH territories AS (
SELECT
'2016A00016' AS grc_dguid,
'Territories' AS grc_en_name,
'Territoires' AS grc_fr_name,
CONCAT('2016A0002', pruid) AS pr_dguid
FROM
bronze.lpr_000a16a_e
WHERE
pruid IN ('60', '61', '62')
),
atlantic AS (
SELECT
'2016A00011' AS grc_dguid,
'Atlantic' AS grc_en_name,
'Atlantique' AS grc_fr_name,
CONCAT('2016A0002', pruid) AS pr_dguid
FROM
bronze.lpr_000a16a_e
WHERE
pruid IN ('10', '11', '12', '13')
),
prairies AS (
SELECT
'2016A00014' AS grc_dguid,
'Prairies' AS grc_en_name,
'Prairies' AS grc_fr_name,
CONCAT('2016A0002', pruid) AS pr_dguid
FROM
bronze.lpr_000a16a_e
WHERE
pruid IN ('48', '47', '46')
),
the_rest AS (
SELECT
CASE
WHEN pruid = '59' THEN '2016A00015'
WHEN pruid = '35' THEN '2016A00013'
WHEN pruid = '24' THEN '2016A00012'
END AS grc_dguid,
prename AS grc_en_name,
prfname AS grc_fr_name,
CONCAT('2016A0002', pruid) AS pr_dguid
FROM
bronze.lpr_000a16a_e
WHERE
pruid IN ('59', '35', '24')
),
final AS (
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
territories.*
FROM
territories,
silver.country_2016 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
prairies.*
FROM
prairies,
silver.country_2016 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
atlantic.*
FROM
atlantic,
silver.country_2016 AS country
UNION
SELECT
country.country_dguid,
country.country_en_name,
country.country_fr_name,
country.country_en_abbreviation,
country.country_fr_abbreviation,
the_rest.*
FROM
the_rest,
silver.country_2016 AS country
)
SELECT *
FROM
final;
/* 2011
Definition here: https://web.archive.org/web/20240214024306/https://www12.statcan.gc.ca/census-recensement/2011/ref/dict/geo027a-eng.cfm
*/
/*
2006
No definition in 2006 dictionary https://www12.statcan.gc.ca/census-recensement/2006/ref/dict/azindex-eng.cfm
*/
/*
2001
Census Dictionary is available here https://www12.statcan.gc.ca/access_acces/archive.action-eng.cfm?/english/census01/products/reference/dict/appendices/92-378-XIE02002.pdf
*/
boundaries/load.sh
Executable
+290
View File
@@ -0,0 +1,290 @@
#!/bin/bash
import_to_postgis() {
local filepath="$1"
local table_name="$2"
local extra_parameters="${@:3}"
# Handle zip files using GDAL's virtual file system
if [[ "${filepath: -4}" == ".zip" ]]; then
filepath="/vsizip/${filepath}"
fi
echo "Importing ${filepath} into table ${table_name}"
ogr2ogr \
--config PG_USE_COPY YES \
-lco "OVERWRITE=YES" \
-f "PostgreSQL" \
"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432" \
-lco GEOMETRY_NAME=geom \
-progress \
-gt 500000 \
-t_srs EPSG:4326 \
-nln "${table_name}" \
${extra_parameters} \
"${filepath}"
}
# Define input folders
INPUT_FOLDER="${DATA_FOLDER}/boundaries/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/boundaries/extracted"
import_data_2021() {
# Source: https://web.archive.org/web/20230307163203/https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/index2021-eng.cfm?year=21
# Provinces/territories
import_to_postgis "${INPUT_FOLDER}/2021/lpr_000a21a_e.zip" lpr_000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census divisions
import_to_postgis "${INPUT_FOLDER}/2021/lcd_000a21a_e.zip" lcd_000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Federal electoral districts (2013 Representation Order)
import_to_postgis "${INPUT_FOLDER}/2021/lfed000a21a_e.zip" lfed000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census subdivisions
import_to_postgis "${INPUT_FOLDER}/2021/lcsd000a21a_e.zip" lcsd000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Designated places
import_to_postgis "${INPUT_FOLDER}/2021/ldpl000a21a_e.zip" ldpl000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Forward sortation areas
unzip -n "${INPUT_FOLDER}/2021/lfsa000a21a_e.zip" -d "${EXTRACTED_FOLDER}/2021"
import_to_postgis "${EXTRACTED_FOLDER}/2021/lfsa000a21a_e/lfsa000a21a_e.shp" lfsa000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Economic regions
import_to_postgis "${INPUT_FOLDER}/2021/ler_000a21a_e.zip" ler_000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# There's issues with the Census agricultural regions encoding, statcan did not export the shapefile properly
# Census agricultural regions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2021/lcar000a21a_e.zip" lcar000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census consolidated subdivisions
import_to_postgis "${INPUT_FOLDER}/2021/lccs000a21a_e.zip" lccs000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census metropolitan areas and census agglomerations
import_to_postgis "${INPUT_FOLDER}/2021/lcma000a21a_e.zip" lcma000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census tracts
import_to_postgis "${INPUT_FOLDER}/2021/lct_000a21a_e.zip" lct_000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Population centres
import_to_postgis "${INPUT_FOLDER}/2021/lpc_000a21a_e.zip" lpc_000a21a_e "-lco SCHEMA=bronze"
# Dissemination areas
import_to_postgis "${INPUT_FOLDER}/2021/lda_000a21a_e.zip" lda_000a21a_e "-lco SCHEMA=bronze"
# Dissemination blocks
import_to_postgis "${INPUT_FOLDER}/2021/ldb_000a21a_e.zip" ldb_000a21a_e "-lco SCHEMA=bronze"
# Aggregate dissemination areas
import_to_postgis "${INPUT_FOLDER}/2021/lada000a21a_e.zip" lada000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Population ecumene
unzip -n "${INPUT_FOLDER}/2021/lecu000e21a_e.zip" -d "${EXTRACTED_FOLDER}/2021"
import_to_postgis "${EXTRACTED_FOLDER}/2021/lecu000e21a_e.shp" lecu000a21a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Agricultural ecumene
unzip -n "${INPUT_FOLDER}/2021/leca000e21a_e.zip" -d "${EXTRACTED_FOLDER}/2021"
import_to_postgis "${EXTRACTED_FOLDER}/2021/leca000e21a_e.shp" leca000e21a_e "-lco SCHEMA=bronze"
}
import_data_2016() {
# Source: https://web.archive.org/web/20230120140926/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2016-eng.cfm
# Provinces/territories
import_to_postgis "${INPUT_FOLDER}/2016/lpr_000a16a_e.zip" lpr_000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Federal electoral districts (2013 Representation Order)
import_to_postgis "${INPUT_FOLDER}/2016/lfed000a16a_e.zip" lfed000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Economic regions
import_to_postgis "${INPUT_FOLDER}/2016/ler_000a16a_e.zip" ler_000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census divisions
import_to_postgis "${INPUT_FOLDER}/2016/lcd_000a16a_e.zip" lcd_000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Aggregate dissemination areas
import_to_postgis "${INPUT_FOLDER}/2016/lada000a16a_e.zip" lada000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census agricultural regions
import_to_postgis "${INPUT_FOLDER}/2016/lcar000a16a_e.zip" lcar000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census consolidated subdivisions
import_to_postgis "${INPUT_FOLDER}/2016/lccs000a16a_e.zip" lccs000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census subdivisions
import_to_postgis "${INPUT_FOLDER}/2016/lcsd000a16a_e.zip" lcsd000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census metropolitan areas and census agglomerations
import_to_postgis "${INPUT_FOLDER}/2016/lcma000a16a_e.zip" lcma000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census tracts
import_to_postgis "${INPUT_FOLDER}/2016/lct_000a16a_e.zip" lct_000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Dissemination areas
import_to_postgis "${INPUT_FOLDER}/2016/lda_000a16a_e.zip" lda_000a16a_e "-lco SCHEMA=bronze"
# Dissemination blocks
import_to_postgis "${INPUT_FOLDER}/2016/ldb_000a16a_e.zip" ldb_000a16a_e "-lco SCHEMA=bronze"
# Designated places
import_to_postgis "${INPUT_FOLDER}/2016/ldpl000a16a_e.zip" ldpl000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Population centres
import_to_postgis "${INPUT_FOLDER}/2016/lpc_000a16a_e.zip" lpc_000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Population ecumene
unzip -n "${INPUT_FOLDER}/2016/lecu000e16a_e.zip" -d "${EXTRACTED_FOLDER}/2016"
import_to_postgis "${EXTRACTED_FOLDER}/2016/lecu000e16a_e.shp" lecu000e16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Agricultural ecumene
unzip -n "${INPUT_FOLDER}/2016/geca000e16a_e.zip" -d ${EXTRACTED_FOLDER}/2016
import_to_postgis "${EXTRACTED_FOLDER}/2016/lagecu000e16a_e.shp" lagecu000e16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Forward sortation areas
import_to_postgis "${INPUT_FOLDER}/2016/lfsa000a16a_e.zip" lfsa000a16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
}
import_data_2011() {
# Source: https://web.archive.org/web/20230110163150/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2011-eng.cfm
# Provinces/territories
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gpr_000a11a_e.zip" gpr_000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Federal electoral districts (2003 Representation Order)
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gfed000a11a_e.zip" gfed000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Economic regions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/ger_000a11a_e.zip" ger_000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census divisions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gcd_000a11a_e.zip" gcd_000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census agricultural regions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gcar000a11a_e.zip" gcar000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census consolidated subdivisions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gccs000a11a_e.zip" gccs000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census subdivisions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gcsd000a11a_e.zip" gcsd000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census metropolitan areas and census agglomerations
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gcma000a11a_e.zip" gcma000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census tracts
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gct_000a11a_e.zip" gct_000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Dissemination areas
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gda_000a11a_e.zip" gda_000a11a_e "-lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Dissemination blocks
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gdb_000a11a_e.zip" gdb_000a11a_e "-lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Designated places
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gdpl000a11a_e.zip" gdpl000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Population centres
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gpc_000a11a_e.zip" gpc_000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Population ecumene
unzip -n "${INPUT_FOLDER}/2011/gecu000e11a_e.zip" -d "${EXTRACTED_FOLDER}/2011"
import_to_postgis "${EXTRACTED_FOLDER}/2011/gecu000e11a_e.shp" gecu000e11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Agricultural ecumene
unzip -n "${INPUT_FOLDER}/2011/geca000e11a_e.zip" -d "${EXTRACTED_FOLDER}/2011"
import_to_postgis "${EXTRACTED_FOLDER}/2011/geca000e11a_e.shp" geca000e11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Forward sortation areas
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/gfsa000a11a_e.zip" gfsa000a11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
}
import_data_2006() {
# Source: https://web.archive.org/web/20221218043125/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2006-eng.cfm
# Provinces/territories
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/gpr_000a06a_e.zip" gpr_000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Federal electoral districts (2003 Representation Order)
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/gfed000a06a_e.zip" gfed000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Economic regions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/ger_000a06a_e.zip" ger_000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census divisions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/gcd_000a06a_e.zip" gcd_000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census agricultural regions
#import_to_postgis "${INPUT_FOLDER}/2006/gcar000a06a_e.zip" gcar000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census consolidated subdivisions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/gccs000a06a_e.zip" gccs000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census subdivisions
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/gcsd000a06a_e.zip" gcsd000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census metropolitan areas and census agglomerations
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/gcma000a06a_e.zip" gcma000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Census tracts
import_to_postgis "${INPUT_FOLDER}/2006/gct_000a06a_e.zip" gct_000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Dissemination areas
import_to_postgis "${INPUT_FOLDER}/2006/gda_000a06a_e.zip" gda_000a06a_e "-lco SCHEMA=bronze"
# Dissemination blocks
import_to_postgis "${INPUT_FOLDER}/2006/gdb_000a06a_e.zip" gdb_000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Designated places
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/gdpl000a06a_e.zip" gdpl000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Urban areas
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/gua_000a06a_e.zip" gua_000a06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# Population ecumene
unzip -n "${INPUT_FOLDER}/2006/gecu000e06a_e.zip" -d "${EXTRACTED_FOLDER}/2006"
import_to_postgis "${EXTRACTED_FOLDER}/2006/gecu000e06a_e.shp" gecu000e06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Agricultural ecumene
unzip -o "${INPUT_FOLDER}/2006/geca000e06a_e.zip" -d ${EXTRACTED_FOLDER}/2006
import_to_postgis "${EXTRACTED_FOLDER}/2006/geca000e06a_e.shp" geca000e06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
}
import_data_2001() {
# Source: https://web.archive.org/web/20221218043135/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2001-eng.cfm
# TODO: merge geometries by unique identifier for all but the block
# Provinces/Territories
unzip -n "${INPUT_FOLDER}/2001/gpr_000b01m_e.zip" -d ${EXTRACTED_FOLDER}/2001
import_to_postgis "${EXTRACTED_FOLDER}/2001/gpr_000b02m_e/gpr_000b02m_e.MID" gpr_000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Federal Electoral Districts (1996 and 2003 Representation Orders)
unzip -n "${INPUT_FOLDER}/2001/gfed000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gfed000b02m_e/gfed000b02m_e.MID" gfed000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Economic Regions
unzip -n "${INPUT_FOLDER}/2001/ger_000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/ger_000b02m_e/ger_000b02m_e.MID" ger_000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census Divisions
unzip -n "${INPUT_FOLDER}/2001/gcd_000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gcd_000b02m_e/gcd_000b02m_e.MID" gcd_000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census Agricultural Regions
unzip -n "${INPUT_FOLDER}/2001/gcar000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gcar000b03m_e/gcar000b03m_e.mid" gcar000b03m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census Consolidated Subdivisions
unzip -n "${INPUT_FOLDER}/2001/gccs000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gccs000b02m_e/gccs000b02m_e.MID" gccs000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census Subdivisions
unzip -n "${INPUT_FOLDER}/2001/gcsd000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gcsd000b02m_e/gcsd000b02m_e.MID" gcsd000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census Metropolitan Areas and Census Agglomerations
unzip -n "${INPUT_FOLDER}/2001/gcma000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gcma000b02m_e/gcma000b02m_e.MID" gcma000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Census Tracts
unzip -n "${INPUT_FOLDER}/2001/gct_000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gct_000b02m_e/gct_000b02m_e.MID" gct_000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Dissemination Areas
unzip -n "${INPUT_FOLDER}/2001/gda_000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gda_000b02m_e/gda_000b02m_e.MID" gda_000b02m_e_tmp "-lco SCHEMA=bronze"
# Blocks (no need to merge geometries)
unzip -n "${INPUT_FOLDER}/2001/gdb_000a01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gbl_000d02m_e/gbl_000d02m_e.TAB" gbl_000d02m_e_tmp "-lco SCHEMA=bronze"
# Designated Places
unzip -n "${INPUT_FOLDER}/2001/gdpl000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gdpl000b02m_e/gdpl000b02m_e.MID" gdpl000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Urban Areas
unzip -n "${INPUT_FOLDER}/2001/gua_000b01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/gua_000b02m_e/gua_000b02m_e.MID" gua000b02m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# TODO
# Population Ecumene
# Agricultural Ecumene
unzip -n "${INPUT_FOLDER}/2001/geca000e01m_e.zip" -d "${EXTRACTED_FOLDER}/2001"
import_to_postgis "${EXTRACTED_FOLDER}/2001/geca000e03m_e/geca000e03m_e.mid" geca000e03m_e_tmp "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
}
# Execute all import functions
import_data_2021
import_data_2016
import_data_2011
import_data_2006
#import_data_2001
boundaries/organize.sql
+608
View File
@@ -0,0 +1,608 @@
/*
PR
*/
/* 2001 Provinces and Territories */
drop table if exists statcan_pr_2001;
create table statcan_pr_2001 as
select distinct pruid, prename, prfname, preabbr, prfabbr,
concat('2001A0002', pruid) as dguid,
st_union(geom) as geom
from statcan_gpr_000b02m_e_tmp
group by pruid, prename, prfname, preabbr, prfabbr;
create index statcan_pr_2001_geom_idx on statcan_pr_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gpr_000b02m_e_tmp;
/* 2006 Provinces and Territories */
drop table if exists statcan_pr_2006;
create table statcan_pr_2006 as
select pruid, prename, prfname, preabbr, prfabbr,
concat('2006A0002', pruid) as dguid,
geom
from statcan_gpr_000a06a_e;
create index statcan_pr_2006_geom_idx on statcan_pr_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gpr_000b02m_e_tmp;
/* 2011 Provinces and Territories */
drop table if exists statcan_pr_2011;
create table statcan_pr_2011 as
select pruid, prename, prfname, preabbr, prfabbr,
concat('2011A0002', pruid) as dguid,
geom
from statcan_gpr_000a11a_e;
create index statcan_pr_2011_geom_idx on statcan_pr_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gpr_000a11a_e;
/* 2016 Provinces and Territories */
drop table if exists statcan_pr_2016;
create table statcan_pr_2016 as
select pruid, prename, prfname, preabbr, prfabbr,
concat('2016A0002', pruid) as dguid,
geom
from statcan_lpr_000a16a_e;
create index statcan_pr_2016_geom_idx on statcan_pr_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lpr_000a16a_e;
/*
CD
*/
--- 2001 Census Divisions;
drop table if exists statcan_cd_2001;
create table statcan_cd_2001 as
select distinct pruid, cduid, cdname, cdtype,
concat('2001A0003', cduid) as dguid,
st_union(geom) as geom
from statcan_gcd_000b02m_e_tmp
group by pruid, cduid, cdname, cdtype;
create index statcan_cd_2001_geom_idx on statcan_cd_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcd_000b02m_e_tmp;
--- 2006 Census Divisions;
drop table if exists statcan_cd_2006;
create table statcan_cd_2006 as
select pruid, cduid, cdname, cdtype,
concat('2006A0003', cduid) as dguid,
geom
from statcan_gcd_000a06a_e;
create index statcan_cd_2006_geom_idx on statcan_cd_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcd_000a06a_e;
--- 2011 Census Divisions;
drop table if exists statcan_cd_2011;
create table statcan_cd_2011 as
select pruid, cduid, cdname, cdtype, concat('2011A0003', cduid) as dguid,
geom
from statcan_gcd_000a11a_e;
create index statcan_cd_2011_geom_idx on statcan_cd_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcd_000a11a_e;
--- 2016 Census Divisions;
drop table if exists statcan_cd_2016;
create table statcan_cd_2016 as
select pruid, cduid, cdname, cdtype,
concat('2016A0003', cduid) as dguid,
geom
from statcan_lcd_000a16a_e;
create index statcan_cd_2016_geom_idx on statcan_cd_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lcd_000a16a_e;
/*
ER
*/
--- 2001 Economic Regions;
drop table if exists statcan_er_2001;
create table statcan_er_2001 as
select distinct pruid, eruid, ername,
concat('2001S0500', eruid) as dguid,
st_union(geom) as geom
from statcan_ger_000b02m_e_tmp
group by pruid, eruid, ername;
create index statcan_er_2001_geom_idx on statcan_er_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_ger_000b02m_e_tmp;
--- 2006 Economic Regions;
drop table if exists statcan_er_2006;
create table statcan_er_2006 as
select distinct pruid, eruid, ername,
concat('2006S0500', eruid) as dguid, geom
from statcan_ger_000a06a_e;
create index statcan_er_2006_geom_idx on statcan_er_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_ger_000a06a_e;
--- 2011 Economic Regions;
drop table if exists statcan_er_2011;
create table statcan_er_2011 as
select distinct pruid, eruid, ername,
concat('2011S0500', eruid) as dguid, geom
from statcan_ger_000a11a_e;
create index statcan_er_2011_geom_idx on statcan_er_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_ger_000a11a_e;
--- 2016 Economic Regions;
drop table if exists statcan_er_2016;
create table statcan_er_2016 as
select distinct pruid, eruid, ername,
concat('2016S0500', eruid) as dguid, geom
from statcan_ler_000a16a_e;
create index statcan_er_2016_geom_idx on statcan_er_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_ler_000a16a_e;
/*
CCS
*/
--- 2001 Census Consolidated Subdivision;
drop table if exists statcan_ccs_2001;
create table statcan_ccs_2001 as
select distinct pruid, ccsuid, ccsname,
concat('2001S0502', ccsuid) as dguid,
st_union(geom) as geom
from statcan_gccs000b02m_e_tmp
group by pruid, ccsuid, ccsname;
create index statcan_ccs_2001_geom_idx on statcan_ccs_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gccs000b02m_e_tmp;
--- 2006 Census Consolidated Subdivision;
drop table if exists statcan_ccs_2006;
create table statcan_ccs_2006 as
select distinct pruid, ccsuid, ccsname,
concat('2006S0502', ccsuid) as dguid, geom
from statcan_gccs000a06a_e;
create index statcan_ccs_2006_geom_idx on statcan_ccs_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gccs000a06a_e;
--- 2011 Census Consolidated Subdivision;
drop table if exists statcan_ccs_2011;
create table statcan_ccs_2011 as
select distinct pruid, cduid, ccsuid, ccsname,
concat('2011S0502', ccsuid) as dguid, geom
from statcan_gccs000a11a_e;
create index statcan_ccs_2011_geom_idx on statcan_ccs_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gccs000a11a_e;
--- 2016 Census Consolidated Subdivision;
drop table if exists statcan_ccs_2016;
create table statcan_ccs_2016 as
select distinct pruid, cduid, ccsuid, ccsname,
concat('2016S0502', ccsuid) as dguid, geom
from statcan_lcsd000a16a_e;
create index statcan_ccs_2016_geom_idx on statcan_ccs_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lcsd000a16a_e;
/*
CSD
*/
/* 2001 Census Subdivisions */
drop table if exists statcan_csd_2001;
create table statcan_csd_2001 as
select distinct pruid, csduid, csdname, csdtype, eruid,
concat('2001A0005', csduid) as dguid,
st_union(geom) as geom
from statcan_gcsd000b02m_e_tmp
group by pruid, csduid, csdname, csdtype, eruid;
create index statcan_csd_2001_geom_idx on statcan_csd_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcsd000b02m_e_tmp;
/* 2006 Census Subdivisions */
drop table if exists statcan_csd_2006;
create table statcan_csd_2006 as
select pruid, csduid, csdname, csdtype, concat(pruid, cmauid) as cmapuid,
concat('2006A0005', csduid) as dguid,
geom
from statcan_gcsd000a06a_e;
create index statcan_csd_2006_geom_idx on statcan_csd_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcsd000a06a_e;
/* 2011 Census Subdivisions */
drop table if exists statcan_csd_2011;
create table statcan_csd_2011 as
select pruid, cduid, ccsuid, csduid, csdname, csdtype, eruid, cmapuid,
concat('2011A0005', csduid) as dguid,
geom
from statcan_gcsd000a11a_e;
create index statcan_csd_2011_geom_idx on statcan_csd_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcsd000a11a_e;
/* 2016 Census Subdivisions */
drop table if exists statcan_csd_2016;
create table statcan_csd_2016 as
select pruid, cduid, ccsuid, csduid, csdname, csdtype, eruid, cmapuid,
concat('2016A0005', csduid) as dguid,
geom
from statcan_lcsd000a16a_e;
create index statcan_csd_2016_geom_idx on statcan_csd_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lcsd000a16a_e;
/*
FED
*/
--- 2003 Federal Electoral Districts;
drop table if exists statcan_fed_2003;
create table statcan_fed_2003 as
select distinct pruid, feduid, fedname, fedename, fedfname,
concat('2003A0004', feduid) as dguid, geom
from statcan_gfed000a11a_e;
create index statcan_fed_2003_geom_idx on statcan_fed_2003 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gfed000a11a_e;
--- 2013 Federal Electoral Districts;
drop table if exists statcan_fed_2013;
create table statcan_fed_2013 as
select distinct pruid, feduid, fedname, fedename, fedfname,
concat('2013A0004', feduid) as dguid, geom
from statcan_lfed000a21a_e;
create index statcan_fed_2013_geom_idx on statcan_fed_2013 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lfed000a21a_e;
drop table if exists statcan_lfed000a16a_e;
/*
Census Agricultural Regions
*/
--- 2001 Census Agricultural Regions;
drop table if exists statcan_car_2001;
create table statcan_car_2001 as
select distinct pruid, caruid, carname, aguid, water,
concat('2001S0501', caruid) as dguid, geom
from statcan_gcar000b03m_e_tmp;
create index statcan_car_2001_geom_idx on statcan_car_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcar000b03m_e_tmp;
--- 2006 Census Agricultural Regions;
drop table if exists statcan_car_2006;
create table statcan_car_2006 as
select distinct pruid, caruid, carname, aguid,
concat('2006S0501', caruid) as dguid, geom
from statcan_gcar000a06a_e;
create index statcan_car_2006_geom_idx on statcan_car_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcar000a06a_e;
--- 2011 Census Agricultural Regions;
drop table if exists statcan_car_2011;
create table statcan_car_2011 as
select distinct pruid, caruid, carname, aguid,
concat('2011S0501', caruid) as dguid, geom
from statcan_gcar000a11a_e;
create index statcan_car_2011_geom_idx on statcan_car_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcar000a11a_e;
--- 2016 Census Agricultural Regions;
drop table if exists statcan_car_2016;
create table statcan_car_2016 as
select distinct pruid, caruid, carename, carfname,
concat('2016S0501', caruid) as dguid, geom
from statcan_lcar000a16a_e;
create index statcan_car_2016_geom_idx on statcan_car_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lcar000a16a_e;
/*
Designated Places
*/
--- 2001 Designated Places;
drop table if exists statcan_dpl_2001;
create table statcan_dpl_2001 as
select distinct pruid, dpluid, dplname, dpltype,
concat('2001A0006', dpluid) as dguid,
st_union(st_makevalid(geom)) as geom
from statcan_gdpl000b02m_e_tmp
group by pruid, dpluid, dplname, dpltype;
create index statcan_dpl_2001_geom_idx on statcan_dpl_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gdpl000b02m_e_tmp;
--- 2006 Designated Places;
drop table if exists statcan_dpl_2006;
create table statcan_dpl_2006 as
select distinct pruid, csduid, dpluid, dplname, dpltype,
concat('2006A0006', dpluid) as dguid, geom
from statcan_gdpl000a06a_e;
create index statcan_dpl_2006_geom_idx on statcan_dpl_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gdpl000a06a_e;
--- 2011 Designated Places;
drop table if exists statcan_dpl_2011;
create table statcan_dpl_2011 as
select distinct pruid, dpluid, dplname, dpltype,
concat('2011A0006', dpluid) as dguid, geom
from statcan_gdpl000a11a_e;
create index statcan_dpl_2011_geom_idx on statcan_dpl_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gdpl000a11a_e;
--- 2016 Designated Places;
drop table if exists statcan_dpl_2016;
create table statcan_dpl_2016 as
select distinct pruid, dpluid, dplname, dpltype,
concat('2016A0006', dpluid) as dguid, geom
from statcan_ldpl000a16a_e;
create index statcan_dpl_2016_geom_idx on statcan_dpl_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_ldpl000a16a_e;
/*
FSA
*/
--- 2011 FSA;
drop table if exists statcan_fsa_2011;
create table statcan_fsa_2011 as
select distinct pruid, cfsauid, concat('2011A0011', cfsauid) as dguid, geom
from statcan_gfsa000a11a_e;
create index statcan_fsa_2011_geom_idx on statcan_fsa_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gfsa000a11a_e;
--- 2016 FSA;
drop table if exists statcan_fsa_2016;
create table statcan_fsa_2016 as
select distinct pruid, cfsauid, concat('2016A0011', cfsauid) as dguid, geom
from statcan_lfsa000a16a_e;
create index statcan_fsa_2016_geom_idx on statcan_fsa_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lfsa000a16a_e;
/*
CMA
*/
--- 2001 CMA;
drop table if exists statcan_cma_2001;
create table statcan_cma_2001 as
select distinct cmauid, concat(pruid, cmauid) as cmapuid, cmaname, cmatype, pruid,
st_union(geom) as geom
from statcan_gcma000b02m_e_tmp
group by cmaname, cmauid, cmatype, pruid;
create index statcan_cma_2001_geom_idx on statcan_cma_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcma000b02m_e_tmp;
--- 2006 CMA;
drop table if exists statcan_cma_2006;
create table statcan_cma_2006 as
select cmauid as cmapuid, cmaname, cmatype, pruid, geom
from statcan_gcma000a06a_e;
create index statcan_cma_2006_geom_idx on statcan_cma_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcma000a06a_e;
--- 2011 CMA;
drop table if exists statcan_cma_2011;
create table statcan_cma_2011 as
select cmauid, cmapuid, cmaname, cmatype, pruid, geom
from statcan_gcma000a11a_e;
create index statcan_cma_2011_geom_idx on statcan_cma_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gcma000a11a_e;
--- 2016 CMA;
drop table if exists statcan_cma_2016;
create table statcan_cma_2016 as
select cmauid, cmapuid, cmaname, cmatype, pruid, geom
from statcan_lcma000a16a_e;
create index statcan_cma_2016_geom_idx on statcan_cma_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lcma000a16a_e;
/*
Population Centres
*/
--- 2011 Population Centres;
drop table if exists statcan_pc_2011;
create table statcan_pc_2011 as
select distinct pruid, cmapuid, pcuid, pcpuid, pcname, pctype,
concat('2011S0510', pcuid) as dguid, geom
from statcan_gpc_000a11a_e;
create index statcan_pc_2011_geom_idx on statcan_pc_2011 using GIST(geom) with (FILLFACTOR=100);
--- 2016 Population Centres;
drop table if exists statcan_pc_2016;
create table statcan_pc_2016 as
select distinct pruid, cmapuid, pcuid, pcpuid, pcname, pctype,
concat('2016S0510', pcuid) as dguid, geom
from statcan_lpc_000a16a_e;
create index statcan_pc_2016_geom_idx on statcan_pc_2016 using GIST(geom) with (FILLFACTOR=100);
/*
Population Centres Part
*/
--- 2011 Population Centres Part;
drop table if exists statcan_pc_part_2011;
create table statcan_pc_part_2011 as
select distinct pruid, cmapuid, pcuid, pcpuid, pcname, pctype,
concat('2011S0511', pcpuid) as dguid, geom
from statcan_gpc_000a11a_e;
create index statcan_pc_part_2011_geom_idx on statcan_pc_part_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gpc_000a11a_e;
--- 2016 Population Centres Part;
drop table if exists statcan_pc_part_2016;
create table statcan_pc_part_2016 as
select distinct pruid, cmapuid, pcuid, pcpuid, pcname, pctype,
concat('2016S0511', pcpuid) as dguid, geom
from statcan_lpc_000a16a_e;
create index statcan_pc_part_2016_geom_idx on statcan_pc_part_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lpc_000a16a_e;
/*
Census Tracts
*/
--- 2001 Census Tracts;
drop table if exists statcan_ct_2001;
create table statcan_ct_2001 as
select distinct pruid, ctuid, ctname, concat(pruid, cmauid) as cmapuid,
concat('2001S0507', ctuid) as dguid, st_union(geom) as geom
from statcan_gct_000b02m_e_tmp
group by pruid, ctuid, ctname, cmauid;
create index statcan_ct_2001_geom_idx on statcan_ct_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gct_000b02m_e_tmp;
--- 2006 Census Tracts;
drop table if exists statcan_ct_2006;
create table statcan_ct_2006 as
select distinct pruid, ctuid, concat(pruid, cmauid) as cmapuid,
concat('2006S0507', ctuid) as dguid, geom
from statcan_gct_000a06a_e;
create index statcan_ct_2006_geom_idx on statcan_ct_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gct_000a06a_e;
--- 2011 Census Tracts;
drop table if exists statcan_ct_2011;
create table statcan_ct_2011 as
select distinct pruid, ctuid, ctname, cmapuid,
concat('2011S0507', ctuid) as dguid, geom
from statcan_gct_000a11a_e;
create index statcan_ct_2011_geom_idx on statcan_ct_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gct_000a11a_e;
--- 2016 Census Tracts;
drop table if exists statcan_ct_2016;
create table statcan_ct_2016 as
select distinct pruid, ctuid, ctname, cmapuid,
concat('2016S0507', ctuid) as dguid, geom
from statcan_lct_000a16a_e;
create index statcan_ct_2016_geom_idx on statcan_ct_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lct_000a16a_e;
/*
DA
*/
--- 2001 Dissemination Areas;
drop table if exists statcan_da_2001;
create table statcan_da_2001 as
select distinct pruid, csduid, concat(pruid, cmauid) as cmapuid, dauid,
concat('2001S0512', dauid) as dguid,
st_union(geom) as geom
from statcan_gda_000b02m_e_tmp
group by pruid, csduid, cmauid, dauid;
create index statcan_da_2001_geom_idx on statcan_da_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gda_000b02m_e_tmp;
--- 2006 Dissemination Area;
drop table if exists statcan_da_2006;
create table statcan_da_2006 as
select distinct pruid, cduid, ccsuid, csduid, eruid, concat(pruid, cmauid) as cmapuid, ctuid, dauid,
concat('2006S0512', dauid) as dguid, geom
from statcan_gda_000a06a_e;
create index statcan_da_2006_geom_idx on statcan_da_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gda_000a06a_e;
--- 2011 Dissemination Area;
drop table if exists statcan_da_2011;
create table statcan_da_2011 as
select distinct pruid, cduid, ccsuid, csduid, eruid, cmapuid, ctuid, dauid,
concat('2011S0512', dauid) as dguid, geom
from statcan_gda_000a11a_e;
create index statcan_da_2011_geom_idx on statcan_da_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gda_000a11a_e;
--- 2016 Dissemination Area;
drop table if exists statcan_da_2016;
create table statcan_da_2016 as
select distinct pruid, cduid, ccsuid, csduid, eruid, cmapuid, ctuid, adauid, dauid,
concat('2016S0512', dauid) as dguid, geom
from statcan_lda_000a16a_e;
create index statcan_da_2016_geom_idx on statcan_da_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lda_000a16a_e;
/*
DB
*/
--- 2001 Dissemination Blocks;
drop table if exists statcan_db_2001;
create table statcan_db_2001 as
select distinct pruid, csduid, concat(pruid, cmauid) as cmapuid, ctname, dauid, blockuid as dbuid,
concat('2001S0513', blockuid) as dguid, geom
from statcan_gbl_000d02m_e_tmp;
create index statcan_db_2001_geom_idx on statcan_db_2001 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gbl_000d02m_e_tmp;
--- 2006 Dissemination Blocks;
drop table if exists statcan_db_2006;
create table statcan_db_2006 as
select distinct pruid, cduid, ccsuid, csduid, eruid, concat(pruid, cmauid) as cmapuid, ctuid, dauid, dbuid,
concat('2006S0513', dbuid) as dguid, geom
from statcan_gdb_000a06a_e;
create index statcan_db_2006_geom_idx on statcan_db_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gdb_000a06a_e;
--- 2011 Dissemination Blocks;
drop table if exists statcan_db_2011;
create table statcan_db_2011 as
select distinct pruid, cduid, ccsuid, csduid, eruid, feduid,
concat(pruid, cmauid) as cmapuid, ctuid, dauid, dbuid,
concat('2011S0513', dbuid) as dguid, geom
from statcan_gdb_000a11a_e;
create index statcan_db_2011_geom_idx on statcan_db_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_gdb_000a11a_e;
--- 2016 Dissemination Blocks;
drop table if exists statcan_db_2016;
create table statcan_db_2016 as
select distinct pruid, cduid, ccsuid, csduid, eruid, feduid, cmapuid, ctuid, adauid, dauid, dbuid,
concat('2016S0513', dbuid) as dguid, geom
from statcan_ldb_000a16a_e;
create index statcan_db_2016_geom_idx on statcan_db_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_ldb_000a16a_e;
/*
ADA
*/
-- 2016 Aggregate Dissemination Areas;
drop table if exists statcan_ada_2016;
create table statcan_ada_2016 as
select distinct pruid, cduid, adauid,
concat('2016S0516', adauid) as dguid, geom
from statcan_lada000a16a_e;
create index statcan_ada_2016_geom_idx on statcan_ada_2016 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_lada000a16a_e;
boundaries/population_centres.sql
+52
View File
@@ -0,0 +1,52 @@
/*
Population Centres
*/
/* 2021
Definition here: https://web.archive.org/web/20250324222733/https://www12.statcan.gc.ca/census-recensement/2021/ref/dict/az/definition-eng.cfm?ID=geo049a
*/
DROP TABLE IF EXISTS silver.pop_ctr_2021;
CREATE TABLE silver.pop_ctr_2021 AS
SELECT DISTINCT
pr.country_dguid,
pr.country_en_name,
pr.country_fr_name,
pr.country_en_abbreviation,
pr.country_fr_abbreviation,
pr.grc_dguid,
pr.grc_en_name,
pr.grc_fr_name,
pr.pr_dguid,
pr.pr_en_name,
pr.pr_fr_name,
pr.pr_en_abbreviation,
pr.pr_fr_abbreviation,
pr.pr_iso_code,
gaf.pop_ctr_dguid,
gaf.pop_ctr_p_dguid,
pop_ctr.pcname AS pop_ctr_name,
pop_ctr.pctype AS pop_ctr_type,
pop_ctr.pcclass AS pop_ctr_class,
pop_ctr.geom
FROM silver.gaf_2021 AS gaf
LEFT JOIN bronze.lpc_000a21a_e AS pop_ctr
ON
concat(gaf.pop_ctr_dguid, gaf.pop_ctr_p_dguid)
= concat(pop_ctr.dguid, pop_ctr.dguidp)
LEFT JOIN silver.pr_2021 AS pr
ON gaf.pr_dguid = pr.pr_dguid
WHERE gaf.pop_ctr_dguid IS NOT null OR gaf.pop_ctr_p_dguid IS NOT null;
-- Make geometries valid
UPDATE
silver.pop_ctr_2021
SET
geom = st_makevalid(geom)
WHERE
st_isvalid(geom) = 'f';
-- Create spatial index
CREATE INDEX pop_ctr_2021_geom_idx ON silver.pop_ctr_2021 USING gist (
geom
) WITH (fillfactor = 100);
boundaries/process.sh
Executable
+18
View File
@@ -0,0 +1,18 @@
# Standardizes field names, builds up hierarchy for datasets, create Canada and Geographic Regions of Canada
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/country.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/geographic_regions_of_canada.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/provinces_and_territories.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_divisions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/economic_regions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_agricultural_regions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/forward_sortation_areas.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/federal_electoral_districts.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_consolidated_subdivisions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_metropolitan_areas.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_subdivisions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/population_centres.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/designated_places.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_tracts.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/aggregate_dissemination_areas.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/dissemination_areas.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/dissemination_blocks.sql
boundaries/provinces_and_territories.sql
+113
View File
@@ -0,0 +1,113 @@
/*
Provinces and territories
*/
/* 2021
Definition here: https://web.archive.org/web/20240402175445/https://www150.statcan.gc.ca/n1/pub/92-195-x/2021001/geo/prov/prov-eng.htm
*/
DROP TABLE IF EXISTS silver.pr_2021;
CREATE TABLE silver.pr_2021 AS
SELECT DISTINCT
grc.country_dguid,
grc.country_en_name,
grc.country_fr_name,
grc.country_en_abbreviation,
grc.country_fr_abbreviation,
grc.grc_dguid,
grc.grc_en_name,
grc.grc_fr_name,
pr.dguid AS pr_dguid,
pr.prename AS pr_en_name,
pr.prfname AS pr_fr_name,
pr.preabbr AS pr_en_abbreviation,
pr.prfabbr AS pr_fr_abbreviation,
CASE
WHEN pr.pruid = '10' THEN 'NL'
WHEN pr.pruid = '11' THEN 'PE'
WHEN pr.pruid = '12' THEN 'NS'
WHEN pr.pruid = '13' THEN 'NB'
WHEN pr.pruid = '24' THEN 'QC'
WHEN pr.pruid = '35' THEN 'ON'
WHEN pr.pruid = '46' THEN 'MB'
WHEN pr.pruid = '47' THEN 'SK'
WHEN pr.pruid = '48' THEN 'AB'
WHEN pr.pruid = '59' THEN 'BC'
WHEN pr.pruid = '60' THEN 'YT'
WHEN pr.pruid = '61' THEN 'NT'
WHEN pr.pruid = '62' THEN 'NU'
END AS pr_iso_code,
pr.geom
FROM
bronze.lpr_000a21a_e AS pr,
silver.grc_pr_2021 AS grc
WHERE
grc.pr_dguid = pr.dguid;
-- Make geometries valid
UPDATE
silver.pr_2021
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX pr_2021_geom_idx ON pr_2021 USING gist (geom) WITH (
fillfactor = 100
);
/* 2016
Definition here: https://web.archive.org/web/20241104061057/https://www12.statcan.gc.ca/census-recensement/2016/ref/dict/geo038-eng.cfm
*/
DROP TABLE IF EXISTS silver.pr_2016;
CREATE TABLE silver.pr_2016 AS
SELECT DISTINCT
grc.country_dguid,
grc.country_en_name,
grc.country_fr_name,
grc.country_en_abbreviation,
grc.country_fr_abbreviation,
grc.grc_dguid,
grc.grc_en_name,
grc.grc_fr_name,
CONCAT('2016A0002', pr.pruid) AS pr_dguid,
pr.prename AS pr_en_name,
pr.prfname AS pr_fr_name,
pr.preabbr AS pr_en_abbreviation,
pr.prfabbr AS pr_fr_abbreviation,
CASE
WHEN pr.pruid = '10' THEN 'NL'
WHEN pr.pruid = '11' THEN 'PE'
WHEN pr.pruid = '12' THEN 'NS'
WHEN pr.pruid = '13' THEN 'NB'
WHEN pr.pruid = '24' THEN 'QC'
WHEN pr.pruid = '35' THEN 'ON'
WHEN pr.pruid = '46' THEN 'MB'
WHEN pr.pruid = '47' THEN 'SK'
WHEN pr.pruid = '48' THEN 'AB'
WHEN pr.pruid = '59' THEN 'BC'
WHEN pr.pruid = '60' THEN 'YT'
WHEN pr.pruid = '61' THEN 'NT'
WHEN pr.pruid = '62' THEN 'NU'
END AS pr_iso_code,
pr.geom
FROM
bronze.lpr_000a16a_e AS pr,
silver.grc_pr_2016 AS grc
WHERE
grc.pr_dguid = CONCAT('2016A0002', pr.pruid);
-- Make geometries valid
UPDATE
silver.pr_2016
SET
geom = ST_MAKEVALID(geom)
WHERE
ST_ISVALID(geom) = 'f';
-- Create spatial index
CREATE INDEX pr_2016_geom_idx ON pr_2016 USING gist (geom) WITH (
fillfactor = 100
);
census_of_agriculture/README.md
+1
View File
@@ -0,0 +1 @@
- See email that I sent to Statistics Canada titled Reporting Mistakes in Census of Agriculture: Data Linked to Geographic Boundaries for mistakes in the Census of Agriculture data
census_of_agriculture/census_of_agriculture_2016.txt
+9
View File
@@ -0,0 +1,9 @@
# https://open.canada.ca/data/en/dataset/b944bd53-49e5-4a80-83e5-1048d3abf38d
# Download variables descriptions
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2016/CEAG16_VariablesDescriptions_REAG16_EN_FR.xlsx
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2016/CEAG16_AgriculturalOperations_ExploitationsAgricoles_REAG16.gdb.zip
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2016/CEAG16_Crops_Cultures_REAG16.gdb.zip
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2016/CEAG16_FarmOperators_ExploitantsAgricoles_REAG16.gdb.zip
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2016/CEAG16_LivestockPoultryBees_BetailVolailleAbeilles_REAG16.gdb.zip
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2016/CEAG16_UseTenurePractices_UtilisationOccupationPratiques_REAG16.gdb.zip
census_of_agriculture/census_of_agriculture_2021.txt
+9
View File
@@ -0,0 +1,9 @@
# https://open.canada.ca/data/en/dataset/b944bd53-49e5-4a80-83e5-1048d3abf38d
# Download variables descriptions
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2021/CEAG21_VariablesDescriptions_REAG21_EN_FR.xlsx
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2021/CEAG21_AgriculturalOperations_ExploitationsAgricoles_REAG21.gdb.zip
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2021/CEAG21_Crops_Cultures_REAG21.gdb.zip
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2021/CEAG21_FarmOperators_ExploitantsAgricoles_REAG21.gdb.zip
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2021/CEAG21_LivestockPoultryBees_BetailVolailleAbeilles_REAG21.gdb.zip
https://ftp.maps.canada.ca/pub/statcan_statcan/Agriculture_Agriculture/census_of_agriculture-recensement_agriculture/2021/CEAG21_UseTenurePractices_UtilisationOccupationPratiques_REAG21.gdb.zip
census_of_agriculture/download.sh
Executable
+15
View File
@@ -0,0 +1,15 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/census_of_agriculture" ]
then
echo "Making directory ${DATA_FOLDER}/census_of_agriculture/"
mkdir -p ${DATA_FOLDER}/census_of_agriculture/{input,extracted,output}/{2021,2016,2011,2001}
mkdir -p ${DATA_FOLDER}/census_of_agriculture/output/{2021,2016,2011,2001}/{tabular,spatial}
fi
INPUT_FOLDER="${DATA_FOLDER}/census_of_agriculture/input"
echo "Downloading 2021 Census of Agriculture"
aria2c -x16 -i "${SCRIPT_DIR}/census_of_agriculture/census_of_agriculture_2021.txt" --dir=${INPUT_FOLDER}/2021 --auto-file-renaming=false
echo "Downloading 2016 Census of Agriculture"
aria2c -x16 -i "${SCRIPT_DIR}/census_of_agriculture/census_of_agriculture_2016.txt" --dir=${INPUT_FOLDER}/2016 --auto-file-renaming=false
census_of_agriculture/process.sh
Executable
+5
View File
@@ -0,0 +1,5 @@
#!/bin/bash
echo "Processing 2021 Census of Agriculture"
jupyter execute census_of_agriculture/process_2021.ipynb
echo "Processing 2016 Census of Agriculture"
jupyter execute census_of_agriculture/process_2016.ipynb
census_of_agriculture/process_2016.ipynb
+1979
View File
File diff suppressed because it is too large Load Diff
census_of_agriculture/process_2021.ipynb
+1793
View File
File diff suppressed because it is too large Load Diff
census_of_population/README.md
+5
View File
@@ -0,0 +1,5 @@
# TODO
- Get download links for 2001 and 2006 Census
- For `process_2021.ipynb`
- Finish processing CMA
- Finish processing HR and Local health integration networks
census_of_population/census_of_population_files_2001.txt
+4
View File
@@ -0,0 +1,4 @@
### 2001 ###
# https://www12.statcan.gc.ca/english/census01/products/standard/popdwell/Tables.cfm
# This one does not have all geographic levels as CSV: https://www12.statcan.gc.ca/census-recensement/2011/dp-pd/prof/details/download-telecharger/comprehensive/comp-csv-tab-dwnld-tlchrgr.cfm?Lang=E#tabs2011
census_of_population/census_of_population_files_2006.txt
+4
View File
@@ -0,0 +1,4 @@
### 2006 ###
# https://www12.statcan.gc.ca/census-recensement/2006/dp-pd/prof/rel/index-eng.cfm
# This one does not have all geographic levels as CSV: https://www12.statcan.gc.ca/census-recensement/2011/dp-pd/prof/details/download-telecharger/comprehensive/comp-csv-tab-dwnld-tlchrgr.cfm?Lang=E#tabs2011
census_of_population/census_of_population_files_2011.txt
+45
View File
@@ -0,0 +1,45 @@
### 2011 ###
# https://www12.statcan.gc.ca/census-recensement/2011/dp-pd/prof/details/download-telecharger/comprehensive/comp-csv-tab-dwnld-tlchrgr.cfm?Lang=E#tabs2011
# Canada, provinces, territories (PRs) (98-316-XWE2011001-101_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-101_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-101_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-101_CSV.zip
# Census divisions (CDs) (98-316-XWE2011001-701_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-701_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-701_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-701_CSV.zip
# Census subdivisions (CSDs) (98-316-XWE2011001-301_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-301_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-301_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-301_CSV.zip
# Census metropolitan areas (CMAs), tracted census agglomerations (CAs) (98-316-XWE2011001-201_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-201_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-201_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-201_CSV.zip
# Census tracts (CTs) (98-316-XWE2011001-401_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-401_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-401_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-401_CSV.zip
# Federal electoral districts (FEDs) (2003 representation order) (98-316-XWE2011001-501_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-501_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-501_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-501_CSV.zip
# Federal electoral districts (FEDs) (2013 representation order) (98-316-XWE2011001-511_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-511_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-511_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-511_CSV.zip
# Economic regions (ERs) (98-316-XWE2011001-901_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-901_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-901_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-901_CSV.zip
# Designated places (DPLs) (98-316-XWE2011001-1301_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-1301_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-1301_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-1301_CSV.zip
# Population centres (POPCTRs) (98-316-XWE2011001-801_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-801_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-801_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-801_CSV.zip
# Dissemination areas (DAs) (98-316-XWE2011001-1501_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-1501_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-1501_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-1501_CSV.zip
# Dissolved census subdivisions (Dissolved CSDs) (98-316-XWE2011001-1401_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-1401_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-1401_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-1401_CSV.zip
# Forward sortation areas (FSAs) (98-316-XWE2011001-1601_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-1601_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-1601_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-1601_CSV.zip
# Health regions (HRs) (December 2013) (98-316-XWE2011001-1701_CSV.zip)
https://archive.org/download/canadas-2011-census-of-population/98-316-XWE2011001-1701_CSV.zip https://zenodo.org/records/15344413/files/98-316-XWE2011001-1701_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2011/98-316-XWE2011001-1701_CSV.zip
census_of_population/census_of_population_files_2016.txt
+31
View File
@@ -0,0 +1,31 @@
### 2016 ###
# Canada, provinces, territories (PRs), census divisions (CDs), census subdivisions (CSDs) and dissemination areas (DAs) (98-401-X2016044_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016044_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016044_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016044_eng_CSV.zip
# Census metropolitan areas (CMAs), tracted census agglomerations (CAs) and census tracts (CTs) (98-401-X2016043_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016043_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016043_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016043_eng_CSV.zip
# Economic regions (ERs) (98-401-X2016049_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016049_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016049_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016049_eng_CSV.zip
# Population centres (POPCTRs) (98-401-X2016048_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016048_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016048_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016048_eng_CSV.zip
# Canada, provinces, territories and federal electoral districts (FEDs) (2013 Representation Order) (98-401-X2016045_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016045_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016045_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016045_eng_CSV.zip
# Designated places (DPLs) (98-401-X2016047_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016047_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016047_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016047_eng_CSV.zip
# Aggregate dissemination areas (ADAs) (98-401-X2016050_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016050_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016050_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016050_eng_CSV.zip
# Forward Sortation Areas (FSAs) (98-401-X2016046_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016046_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016046_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016046_eng_CSV.zip
# Dissolved Census subdivisions (CSDs) (98-401-X2016057_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016057_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016057_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016057_eng_CSV.zip
# Health Regions (HRs) (98-401-X2016058_eng_CSV.zip)
https://archive.org/download/canadas-2016-census-of-population/98-401-X2016058_eng_CSV.zip https://zenodo.org/records/15343658/files/98-401-X2016058_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2016/98-401-X2016058_eng_CSV.zip
census_of_population/census_of_population_files_2021.txt
+36
View File
@@ -0,0 +1,36 @@
### 2021 ###
# https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/prof/details/download-telecharger.cfm?Lang=E
# Canada, provinces, territories (PRs), census divisions (CDs), census subdivisions (CSDs) and dissemination areas (DAs) (98-401-X2021006_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021006_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021006_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021006_eng_CSV.zip
# Census metropolitan areas (CMAs), tracted census agglomerations (CAs) and census tracts (CTs) (98-401-X2021007_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021007_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021007_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021007_eng_CSV.zip
# Economic regions (ERs) (98-401-X2021008_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021008_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021008_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021008_eng_CSV.zip
# Population centres (POPCTRs) (98-401-X2021009_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021009_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021009_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021009_eng_CSV.zip
# Federal electoral districts, 2013 representation order (98-401-X2021010_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021010_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021010_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021010_eng_CSV.zip
# Federal electoral districts, 2023 representation order (98-401-X2021029_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021029_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021029_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021029_eng_CSV.zip
# Designated places (DPLs) (98-401-X2021011_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021011_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021011_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021011_eng_CSV.zip
# Aggregate dissemination areas (ADAs) (98-401-X2021012_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021012_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021012_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021012_eng_CSV.zip
# Forward Sortation Areas (FSAs) (98-401-X2021013_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021013_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021013_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021013_eng_CSV.zip
# Dissolved census subdivisions (Dissolved CSDs) (98-401-X2021014_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021014_eng_CSV.zip https://zenodo.org/records/15334624/files/98-401-X2021014_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021014_eng_CSV.zip
# Health Regions (HRs) and Home and Community Care Support Services (HCCCSS) 2022 (98-401-X2021015_eng_CSV.zip)
https://archive.org/download/canadas-2021-census-of-population/98-401-X2021015_eng_CSV.zip https://zenodo.org/records/15333781/files/98-401-X2021015_eng_CSV.zip?download=1 https://data.dataforcanada.org/archive/statistics_canada/census_of_population/2021/98-401-X2021015_eng_CSV.zip
census_of_population/download.sh
Executable
+24
View File
@@ -0,0 +1,24 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/census_of_population" ]
then
echo "Making directory ${DATA_FOLDER}/census_of_population/"
mkdir -p ${DATA_FOLDER}/census_of_population/{input,extracted,output}/{2021,2016,2011,2001}
mkdir -p ${DATA_FOLDER}/census_of_population/output/{2021,2016,2011,2001}/{tabular,spatial}
fi
INPUT_FOLDER="${DATA_FOLDER}/census_of_population/input"
echo "Downloading 2021 Census of Population"
aria2c -x16 -i "${SCRIPT_DIR}/census_of_population/census_of_population_files_2021.txt" --dir=${INPUT_FOLDER}/2021 --auto-file-renaming=false
echo "Downloading 2016 Census of Population"
aria2c -x16 -i "${SCRIPT_DIR}/census_of_population/census_of_population_files_2016.txt" --dir=${INPUT_FOLDER}/2016 --auto-file-renaming=false
echo "Downloading 2011 Census of Population"
aria2c -x16 -i "${SCRIPT_DIR}/census_of_population/census_of_population_files_2011.txt" --dir=${INPUT_FOLDER}/2011 --auto-file-renaming=false
#echo "Downloading 2006 Census of Population"
#aria2c -x16 -i "${SCRIPT_DIR}/census_of_population/census_of_population_files_2006.txt" --dir=${INPUT_FOLDER}/2006 --auto-file-renaming=false
#
#echo "Downloading 2001 Census of Population"
#aria2c -x16 -i "${SCRIPT_DIR}/census_of_population/census_of_population_files_2001.txt" --dir=${INPUT_FOLDER}/2001 --auto-file-renaming=false
census_of_population/process.sh
Executable
+155
View File
@@ -0,0 +1,155 @@
#!/bin/bash
INPUT_FOLDER="${DATA_FOLDER}/census_of_population/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/census_of_population/extracted"
process_2021() {
echo "Processing 2021 Census of Population"
jupyter execute census_of_population/process_2021.ipynb
}
extract_2021() {
local INPUT_FOLDER="${INPUT_FOLDER}/2021"
local EXTRACTED_FOLDER="${EXTRACTED_FOLDER}/2021"
echo "Extracting Canada, provinces, territories (PRs), census divisions (CDs), census subdivisions (CSDs) and dissemination areas (DAs) (${INPUT_FOLDER}/98-401-X2021006_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021006_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021006_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021006_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021006_eng_CSV
echo "Extracting Census metropolitan areas (CMAs), tracted census agglomerations (CAs) and census tracts (CTs) (${INPUT_FOLDER}/98-401-X2021007_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021007_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021007_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021007_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021007_eng_CSV
echo "Extracting Economic regions (ERs) (${INPUT_FOLDER}/98-401-X2021008_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021008_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021008_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021008_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021008_eng_CSV
echo "Extracting Population centres (POPCTRs) (${INPUT_FOLDER}/98-401-X2021009_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021009_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021009_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021009_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021009_eng_CSV
echo "Extracting Federal electoral districts, 2013 representation order (${INPUT_FOLDER}/98-401-X2021010_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021010_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021010_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021010_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021010_eng_CSV
echo "Extracting Federal electoral districts, 2023 representation order (${INPUT_FOLDER}/98-401-X2021029_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021029_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021029_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021029_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021029_eng_CSV
echo "Extracting Designated places (DPLs) (${INPUT_FOLDER}/98-401-X2021011_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021011_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021011_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021011_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021011_eng_CSV
echo "Extracting Aggregate dissemination areas (ADAs) (${INPUT_FOLDER}/98-401-X2021012_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021012_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021012_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021012_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021012_eng_CSV
echo "Extracting Forward Sortation Areas (FSAs) (${INPUT_FOLDER}/98-401-X2021013_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021013_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021013_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021013_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021013_eng_CSV
echo "Extracting Dissolved census subdivisions (Dissolved CSDs) (98-401-X2021014_eng_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021014_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021014_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021014_eng_CSV
echo "Extracting Health Regions (HRs) and Home and Community Care Support Services (HCCCSS) 2022 (${INPUT_FOLDER}/98-401-X2021015_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2021009_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2021015_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2021015_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2021015_eng_CSV
}
extract_2016() {
local INPUT_FOLDER="${INPUT_FOLDER}/2016"
local EXTRACTED_FOLDER="${EXTRACTED_FOLDER}/2016"
echo "Extracting Canada, provinces, territories (PRs), census divisions (CDs), census subdivisions (CSDs) and dissemination areas (DAs) (${INPUT_FOLDER}/98-401-X2016044_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016044_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016044_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016044_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016044_eng_CSV
echo "Extracting Census metropolitan areas (CMAs), tracted census agglomerations (CAs) and census tracts (CTs) (${INPUT_FOLDER}/98-401-X2016044_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016044_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016044_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016044_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016044_eng_CSV
echo "Extracting Economic regions (ERs) (${INPUT_FOLDER}/98-401-X2016049_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016049_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016049_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016049_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016049_eng_CSV
echo "Extracting Population centres (POPCTRs) (${INPUT_FOLDER}/98-401-X2016048_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016048_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016048_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016048_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016048_eng_CSV
echo "Extracting Canada, provinces, territories and federal electoral districts (FEDs) (2013 Representation Order) (${INPUT_FOLDER}/98-401-X2016045_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016045_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016045_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016045_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016045_eng_CSV
echo "Extracting Designated places (DPLs) (${INPUT_FOLDER}/98-401-X2016047_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016047_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016047_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016047_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016047_eng_CSV
echo "Extracting Aggregate dissemination areas (ADAs) (${INPUT_FOLDER}/98-401-X2016050_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016050_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016050_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016050_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016050_eng_CSV
echo "Extracting Forward Sortation Areas (FSAs) (${INPUT_FOLDER}/98-401-X2016046_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016046_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016046_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016046_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016046_eng_CSV
echo "Extracting Health Regions (HRs) (${INPUT_FOLDER}/98-401-X2016058_eng_CSV.zip). Extracting to ${EXTRACTED_FOLDER}/98-401-X2016058_eng_CSV"
mkdir -p "${EXTRACTED_FOLDER}/98-401-X2016058_eng_CSV"
unzip -q -n ${INPUT_FOLDER}/98-401-X2016058_eng_CSV.zip -d ${EXTRACTED_FOLDER}/98-401-X2016058_eng_CSV
}
extract_2011() {
local INPUT_FOLDER="${INPUT_FOLDER}/2011"
local EXTRACTED_FOLDER="${EXTRACTED_FOLDER}/2011"
echo "Extracting Canada, provinces, territories (PRs) (98-316-XWE2011001-101_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-101_CSV"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-101_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-101_CSV
echo "Extracting Census divisions (CDs) (98-316-XWE2011001-701_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-701_CSV"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-701_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-701_CSV
echo "Extracting Census metropolitan areas (CMAs), tracted census agglomerations (CAs) (98-316-XWE2011001-201_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-201_CSV"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-201_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-201_CSV
echo "Extracting Census tracts (CTs) (98-316-XWE2011001-401_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-401_CSV"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-401_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-401_CSV
echo "Extracting Federal electoral districts (2003 representation order) (98-316-XWE2011001-501_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-501_CSV.zip"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-501_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-501_CSV.zip
echo "Extracting Federal electoral districts (2013 representation order) (98-316-XWE2011001-511_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-511_CSV"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-511_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-511_CSV
echo "Extracting Economic regions (ERs) (98-316-XWE2011001-901_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-901_CSV"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-901_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-901_CSV
echo "Extracting Designated places (98-316-XWE2011001-1301_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-1301_CSV"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-1301_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-1301_CSV
echo "Extracting Population centres (POPCTRs) (98-316-XWE2011001-801_CSV.zip)"
mkdir -p "${EXTRACTED_FOLDER}/98-316-XWE2011001-801_CSV"
unzip -q -n ${INPUT_FOLDER}/98-316-XWE2011001-801_CSV.zip -d ${EXTRACTED_FOLDER}/98-316-XWE2011001-801_CSV
}
extract_2006() {
local INPUT_FOLDER="${INPUT_FOLDER}/2006"
local EXTRACTED_FOLDER="${EXTRACTED_FOLDER}/2006"
}
extract_2001() {
local INPUT_FOLDER="${INPUT_FOLDER}/2001"
local EXTRACTED_FOLDER="${EXTRACTED_FOLDER}/2001"
}
extract_2021
extract_2016
extract_2011
extract_2006
process_2021
census_of_population/process_2021.ipynb
+1360
View File
File diff suppressed because it is too large Load Diff
dissemination_geographies_relationship_file/download.sh
Executable
+11
View File
@@ -0,0 +1,11 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/dissemination_geographies_relationship_file" ]
then
echo "Making directory ${DATA_FOLDER}/dissemination_geographies_relationship_file/"
mkdir -p ${DATA_FOLDER}/dissemination_geographies_relationship_file/{input,extracted,output}
fi
INPUT_FOLDER="${DATA_FOLDER}/dissemination_geographies_relationship_file/input"
echo "Downloading 2021 dissemiantion geographies relationship file"
aria2c -x16 -i "${SCRIPT_DIR}/dissemination_geographies_relationship_file/files.txt" --dir=${INPUT_FOLDER} --auto-file-renaming=false
dissemination_geographies_relationship_file/files.txt
Executable
+2
View File
@@ -0,0 +1,2 @@
# 2021. Here is the reference guide https://web.archive.org/web/20250413152017/https://www150.statcan.gc.ca/n1/pub/98-26-0003/982600032021001-eng.htm
https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/dguid-idugd/files-fichiers/2021_98260004.zip
dissemination_geographies_relationship_file/load.sh
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/bin/bash
INPUT_FOLDER="${DATA_FOLDER}/dissemination_geographies_relationship_file/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/dissemination_geographies_relationship_file/extracted"
import_2021() {
echo "Unzipping 2021 dissemination geographies relationship file"
unzip -n "${INPUT_FOLDER}/2021_98260004.zip" -d ${EXTRACTED_FOLDER}
python dissemination_geographies_relationship_file/process.py ${EXTRACTED_FOLDER}/2021_98260004.csv
}
import_2021
dissemination_geographies_relationship_file/process.py
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python
# coding: utf-8
import os
import sys
import pandas as pd
from sqlalchemy import create_engine
dgr_2021_csv = sys.argv[1]
DATABASE = os.environ.get("POSTGRES_DB")
USER = os.environ.get("POSTGRES_USER")
PASSWORD = os.environ.get("POSTGRES_PASSWORD")
engine = create_engine(f"postgresql://{USER}:{PASSWORD}@db:5432/{DATABASE}")
"""
Data dictionary is here:
https://web.archive.org/web/20250413152017/https://www150.statcan.gc.ca/n1/pub/98-26-0003/982600032021001-eng.htm
This processes the entire DGUID hierarchy and other useful fields
"""
print(f"Processing {dgr_2021_csv}")
dgr_2021_df = pd.read_csv(dgr_2021_csv)
# Rename columns, remove french portion
dgr_2021_df.rename(columns={
'PRDGUID_PRIDUGD': 'pr_dguid',
'CDDGUID_DRIDUGD': 'cd_dguid',
'FEDDGUID_CEFIDUGD': 'fed_dguid',
'CSDDGUID_SDRIDUGD': 'csd_dguid',
'ERDGUID_REIDUGD': 'er_dguid',
'CARDGUID_RARIDUGD': 'car_dguid',
'CCSDGUID_SRUIDUGD': 'ccs_dguid',
'DADGUID_ADIDUGD': 'da_dguid',
'DBDGUID_IDIDUGD': 'db_dguid',
'ADADGUID_ADAIDUGD': 'ada_dguid',
'DPLDGUID_LDIDUGD': 'dpl_dguid',
'CMAPDGUID_RMRPIDUGD': 'cma_p_dguid',
'CMADGUID_RMRIDUGD': 'cma_dguid',
'CTDGUID_SRIDUGD': 'ct_dguid',
'POPCTRPDGUID_CTRPOPPIDUGD': 'pop_ctr_p_dguid',
'POPCTRDGUID_CTRPOPIDUGD': 'pop_ctr_dguid',
}, inplace=True)
columns_ordered = ['pr_dguid', 'fed_dguid', 'er_dguid', 'car_dguid', 'cd_dguid',
'dpl_dguid', 'ccs_dguid', 'csd_dguid',
'cma_p_dguid', 'cma_dguid',
'pop_ctr_p_dguid', 'pop_ctr_dguid',
'ada_dguid', 'ct_dguid', 'da_dguid', 'db_dguid']
dgr_2021_df = dgr_2021_df.reindex(columns_ordered, axis=1)
print("Loading processed 2021 dissemination geographies relationship file to database as dissemination_geographies_relationship_2021")
dgr_2021_df.to_sql(name='dissemination_geographies_relationship_2021',
con=engine,
index=False,
chunksize=50000,
if_exists='replace',
schema='silver'
)
docker-compose.yml
+32
View File
@@ -0,0 +1,32 @@
name: process-statcan-data
services:
devcontainer:
container_name: devcontainer
user: root
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/workspace
- ./data:/data
env_file: .env
ports:
- 8888:8888
command: sleep infinity
db:
image: postgis/postgis:17-3.5-alpine
container_name: db
restart: unless-stopped
volumes:
- pgdata:/var/lib/postgresql/data
env_file: .env
ports:
- 5432:5432
networks:
default:
name: dev-container
volumes:
pgdata:
experiments/boundaries_spatial_checks.ipynb
+140
View File
@@ -0,0 +1,140 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 30,
"id": "3849cfa2-bb10-4323-95a3-51d205f497d5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Buckaroo has been enabled as the default DataFrame viewer. To return to default dataframe visualization use `from buckaroo import disable; disable()`\n"
]
}
],
"source": [
"import os\n",
"\n",
"import buckaroo\n",
"import geopandas as gpd\n",
"from IPython.core.interactiveshell import InteractiveShell\n",
"from lonboard import viz\n",
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"\n",
"# Enable multiple outputs per cell\n",
"InteractiveShell.ast_node_interactivity = \"all\"\n",
"# Show all columns\n",
"pd.set_option('display.max_columns', None)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "7e8b2e03-594c-4d8d-8914-ff4fe68117ac",
"metadata": {},
"outputs": [],
"source": [
"DATABASE = os.getenv('POSTGRES_DB')\n",
"USER = os.getenv('POSTGRES_USER')\n",
"PASSWORD = os.getenv('POSTGRES_PASSWORD')"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "dca70ec0-35c5-4c51-9e6a-5131ed46522c",
"metadata": {},
"outputs": [],
"source": [
"engine = create_engine(f'postgresql://{USER}:{PASSWORD}@db:5432/{DATABASE}')"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "08935c71-ff7b-4a2a-aa9d-029b4231cc71",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "95c1fb1edb9e440d916fa8cd3496bb46",
"version_major": 2,
"version_minor": 1
},
"text/plain": [
"GeopandasBuckarooWidget(buckaroo_options={'sampled': ['random'], 'auto_clean': ['aggressive', 'conservative'],…"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"sql = \"\"\"\n",
"SELECT * FROM silver.pr_2016;\n",
"\"\"\"\n",
"df = gpd.read_postgis(sql, con=engine)\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "03ff2898-1318-4859-8249-d2c3a05117a7",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "775652f4d632485abaf8babe7c671b25",
"version_major": 2,
"version_minor": 1
},
"text/plain": [
"Map(basemap_style=<CartoBasemap.DarkMatter: 'https://basemaps.cartocdn.com/gl/dark-matter-gl-style/style.json'…"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"viz(df)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bfcbf94c-0d65-422d-90b3-2a8d79ef1c32",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/duckdb_census_of_population.ipynb
+242
View File
@@ -0,0 +1,242 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 17,
"id": "56ac906e",
"metadata": {},
"outputs": [],
"source": [
"import duckdb\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "708e293d",
"metadata": {},
"outputs": [],
"source": [
"con = duckdb.connect()\n",
"con.install_extension(\"spatial\")\n",
"con.load_extension(\"spatial\")"
]
},
{
"cell_type": "markdown",
"id": "5d97e882",
"metadata": {},
"source": [
"# DA"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c1b2ccf6",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b2fd24f9af2c45369a4aa93b249227c0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatProgress(value=0.0, layout=Layout(width='auto'), style=ProgressStyle(bar_color='black'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<duckdb.duckdb.DuckDBPyConnection at 0x1474498ddc30>"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"con.execute(\"\"\"\n",
"DROP TABLE IF EXISTS geo_data;\n",
"CREATE TABLE geo_data AS\n",
"SELECT da.da_dguid, da_cop.* EXCLUDE dguid, da.geom FROM 'https://data.dataforcanada.org/processed/statistics_canada/census_of_population/2021/tabular/da_2021.parquet' AS da_cop,\n",
"'https://data.dataforcanada.org/processed/statistics_canada/boundaries/2021/digital_boundary_files/da_2021.parquet' AS da\n",
"WHERE da_cop.dguid = da.da_dguid\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "33a3819e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[(57936,)]"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"con.execute(\"SELECT count(*) FROM geo_data;\")\n",
"con.fetchall()"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "6d38118b",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ee1b19d9ad3c4a9483ba3cef3600ba8e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatProgress(value=0.0, layout=Layout(width='auto'), style=ProgressStyle(bar_color='black'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<duckdb.duckdb.DuckDBPyConnection at 0x1474498ddc30>"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"con.execute(\"\"\"\n",
"COPY geo_data TO 'da_2021_cop.gdb'\n",
"WITH (\n",
" FORMAT GDAL,\n",
" DRIVER 'OpenFileGDB',\n",
" GEOMETRY_TYPE 'POLYGON',\n",
" SRS 'EPSG:4326'\n",
");\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "76ada8f9",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e4d6cb52cb864a0cac1941bb25315b79",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatProgress(value=0.0, layout=Layout(width='auto'), style=ProgressStyle(bar_color='black'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<duckdb.duckdb.DuckDBPyConnection at 0x1477954f73b0>"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"con.execute(\"\"\"\n",
"COPY geo_data TO 'da_2021_cop.geojson'\n",
"WITH (\n",
" FORMAT GDAL,\n",
" DRIVER 'GeoJSON',\n",
" GEOMETRY_TYPE 'POLYGON',\n",
" SRS 'EPSG:4326'\n",
");\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "c32ab0f8",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c226f348d3f14c0abbeb2bfa907a7db6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatProgress(value=0.0, layout=Layout(width='auto'), style=ProgressStyle(bar_color='black'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<duckdb.duckdb.DuckDBPyConnection at 0x147a2c462b30>"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"con.execute(\"\"\"\n",
"COPY geo_data TO 'da_2021_cop_geom.parquet' (FORMAT PARQUET);\n",
"\"\"\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/export_2021_boundaries.ipynb
+922
View File
@@ -0,0 +1,922 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "b6e053ec",
"metadata": {},
"outputs": [],
"source": [
"DATA_FOLDER=/data\n",
"\n",
"source ../.env"
]
},
{
"cell_type": "markdown",
"id": "12eca225-3d05-4bb7-95fa-7b9df694f53d",
"metadata": {},
"source": [
"# 1. Export Digital Boundary Files"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "22183463",
"metadata": {
"vscode": {
"languageId": "shellscript"
}
},
"outputs": [],
"source": [
"output_folder=\"${DATA_FOLDER}/boundaries/output/2021/digital_boundary_files\"\n",
"mkdir -p ${output_folder}"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "a0372dac",
"metadata": {},
"outputs": [],
"source": [
"files=('country_2021'\n",
" 'grc_2021'\n",
" 'pr_2021'\n",
" 'er_2021'\n",
" 'car_2021'\n",
" 'cd_2021'\n",
" 'ccs_2021'\n",
" 'cma_2021'\n",
" 'csd_2021'\n",
" 'fed_2021_2013'\n",
" 'dpl_2021'\n",
" 'fsa_2021'\n",
" 'pop_ctr_2021'\n",
" 'ct_2021'\n",
" 'da_2021'\n",
" 'db_2021'\n",
" 'ada_2021'\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "04a2c69a-6204-4fa5-b6e2-f0375f51f425",
"metadata": {},
"source": [
"## Export as Parquet"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4c5bb532",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Exporting silver.country_2021 table to /data/boundaries/output/2021/digital_boundary_files/country_2021.parquet\n",
"Exporting silver.grc_2021 table to /data/boundaries/output/2021/digital_boundary_files/grc_2021.parquet\n",
"Exporting silver.pr_2021 table to /data/boundaries/output/2021/digital_boundary_files/pr_2021.parquet\n",
"Exporting silver.er_2021 table to /data/boundaries/output/2021/digital_boundary_files/er_2021.parquet\n",
"Exporting silver.car_2021 table to /data/boundaries/output/2021/digital_boundary_files/car_2021.parquet\n",
"Exporting silver.cd_2021 table to /data/boundaries/output/2021/digital_boundary_files/cd_2021.parquet\n",
"Exporting silver.ccs_2021 table to /data/boundaries/output/2021/digital_boundary_files/ccs_2021.parquet\n",
"Exporting silver.cma_2021 table to /data/boundaries/output/2021/digital_boundary_files/cma_2021.parquet\n",
"Exporting silver.csd_2021 table to /data/boundaries/output/2021/digital_boundary_files/csd_2021.parquet\n",
"Exporting silver.fed_2021_2013 table to /data/boundaries/output/2021/digital_boundary_files/fed_2021_2013.parquet\n",
"Exporting silver.dpl_2021 table to /data/boundaries/output/2021/digital_boundary_files/dpl_2021.parquet\n",
"Exporting silver.fsa_2021 table to /data/boundaries/output/2021/digital_boundary_files/fsa_2021.parquet\n",
"Exporting silver.pop_ctr_2021 table to /data/boundaries/output/2021/digital_boundary_files/pop_ctr_2021.parquet\n",
"Exporting silver.ct_2021 table to /data/boundaries/output/2021/digital_boundary_files/ct_2021.parquet\n",
"Exporting silver.da_2021 table to /data/boundaries/output/2021/digital_boundary_files/da_2021.parquet\n",
"Exporting silver.db_2021 table to /data/boundaries/output/2021/digital_boundary_files/db_2021.parquet\n",
"Exporting silver.ada_2021 table to /data/boundaries/output/2021/digital_boundary_files/ada_2021.parquet\n"
]
}
],
"source": [
"for file in ${files[@]}\n",
"do\n",
" output_file=\"${output_folder}/${file}.parquet\"\n",
" echo \"Exporting silver.${file} table to ${output_file}\" \n",
" ogr2ogr \\\n",
" -lco COMPRESSION=\"ZSTD\" \\\n",
" -lco CREATOR=\"www.dataforcanada.org\" \\\n",
" -lco WRITE_COVERING_BBOX=\"YES\" \\\n",
" -lco SORT_BY_BBOX=\"YES\" \\\n",
" -f Parquet \\\n",
" ${output_file} \\\n",
" \"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432\" \\\n",
" \"silver.${file}\"\n",
"done"
]
},
{
"cell_type": "markdown",
"id": "77af4081-b070-4c78-bbd9-ab2bcbb28881",
"metadata": {},
"source": [
"## Export as FlatGeobuf"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "c012df55",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Exporting country_2021 table to /data/boundaries/output/2021/digital_boundary_files/country_2021.fgb\n",
"Exporting grc_2021 table to /data/boundaries/output/2021/digital_boundary_files/grc_2021.fgb\n",
"Exporting pr_2021 table to /data/boundaries/output/2021/digital_boundary_files/pr_2021.fgb\n",
"Exporting er_2021 table to /data/boundaries/output/2021/digital_boundary_files/er_2021.fgb\n",
"Exporting car_2021 table to /data/boundaries/output/2021/digital_boundary_files/car_2021.fgb\n",
"Exporting cd_2021 table to /data/boundaries/output/2021/digital_boundary_files/cd_2021.fgb\n",
"Exporting ccs_2021 table to /data/boundaries/output/2021/digital_boundary_files/ccs_2021.fgb\n",
"Exporting cma_2021 table to /data/boundaries/output/2021/digital_boundary_files/cma_2021.fgb\n",
"Exporting csd_2021 table to /data/boundaries/output/2021/digital_boundary_files/csd_2021.fgb\n",
"Exporting fed_2021_2013 table to /data/boundaries/output/2021/digital_boundary_files/fed_2021_2013.fgb\n",
"Exporting dpl_2021 table to /data/boundaries/output/2021/digital_boundary_files/dpl_2021.fgb\n",
"Exporting fsa_2021 table to /data/boundaries/output/2021/digital_boundary_files/fsa_2021.fgb\n",
"Exporting pop_ctr_2021 table to /data/boundaries/output/2021/digital_boundary_files/pop_ctr_2021.fgb\n",
"Exporting ct_2021 table to /data/boundaries/output/2021/digital_boundary_files/ct_2021.fgb\n",
"Exporting da_2021 table to /data/boundaries/output/2021/digital_boundary_files/da_2021.fgb\n",
"Exporting db_2021 table to /data/boundaries/output/2021/digital_boundary_files/db_2021.fgb\n",
"Exporting ada_2021 table to /data/boundaries/output/2021/digital_boundary_files/ada_2021.fgb\n"
]
}
],
"source": [
"for file in ${files[@]}\n",
"do\n",
" output_file=\"${output_folder}/${file}.fgb\"\n",
" echo \"Exporting ${file} table to ${output_file}\" \n",
" ogr2ogr \\\n",
" -f FlatGeobuf \\\n",
" -lco TITLE=\"${file}\" \\\n",
" ${output_file} \\\n",
" \"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432\" \\\n",
" \"silver.${file}\" \\\n",
" -nln ${file}\n",
"done"
]
},
{
"cell_type": "markdown",
"id": "b90b3919",
"metadata": {},
"source": [
"## Export as File Geodatabase"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "64780573",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Exporting country_2021 table to /data/boundaries/output/2021/digital_boundary_files/country_2021.gdb\n",
"ERROR 6: Unsupported geometry type\n",
"ERROR 1: Terminating translation prematurely after failed\n",
"translation of layer silver.country_2021 (use -skipfailures to skip errors)\n",
"Exporting grc_2021 table to /data/boundaries/output/2021/digital_boundary_files/grc_2021.gdb\n",
"ERROR 6: Unsupported geometry type\n",
"ERROR 1: Terminating translation prematurely after failed\n",
"translation of layer silver.grc_2021 (use -skipfailures to skip errors)\n",
"Exporting pr_2021 table to /data/boundaries/output/2021/digital_boundary_files/pr_2021.gdb\n",
"Exporting er_2021 table to /data/boundaries/output/2021/digital_boundary_files/er_2021.gdb\n",
"Exporting car_2021 table to /data/boundaries/output/2021/digital_boundary_files/car_2021.gdb\n",
"Exporting cd_2021 table to /data/boundaries/output/2021/digital_boundary_files/cd_2021.gdb\n",
"Exporting ccs_2021 table to /data/boundaries/output/2021/digital_boundary_files/ccs_2021.gdb\n",
"Exporting cma_2021 table to /data/boundaries/output/2021/digital_boundary_files/cma_2021.gdb\n",
"Exporting csd_2021 table to /data/boundaries/output/2021/digital_boundary_files/csd_2021.gdb\n",
"Exporting fed_2021_2013 table to /data/boundaries/output/2021/digital_boundary_files/fed_2021_2013.gdb\n",
"Exporting dpl_2021 table to /data/boundaries/output/2021/digital_boundary_files/dpl_2021.gdb\n",
"Exporting fsa_2021 table to /data/boundaries/output/2021/digital_boundary_files/fsa_2021.gdb\n",
"Exporting pop_ctr_2021 table to /data/boundaries/output/2021/digital_boundary_files/pop_ctr_2021.gdb\n",
"Exporting ct_2021 table to /data/boundaries/output/2021/digital_boundary_files/ct_2021.gdb\n",
"Exporting da_2021 table to /data/boundaries/output/2021/digital_boundary_files/da_2021.gdb\n",
"Exporting db_2021 table to /data/boundaries/output/2021/digital_boundary_files/db_2021.gdb\n",
"Exporting ada_2021 table to /data/boundaries/output/2021/digital_boundary_files/ada_2021.gdb\n"
]
}
],
"source": [
"for file in ${files[@]}\n",
"do\n",
" output_file=\"${output_folder}/${file}.gdb\"\n",
" echo \"Exporting ${file} table to ${output_file}\" \n",
" ogr2ogr \\\n",
" -f OpenFileGDB \\\n",
" -lco TARGET_ARCGIS_VERSION=\"ARCGIS_PRO_3_2_OR_LATER\" \\\n",
" ${output_file} \\\n",
" \"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432\" \\\n",
" \"silver.${file}\" \\\n",
" -nln ${file}\n",
"done"
]
},
{
"cell_type": "markdown",
"id": "69d3a9be-6fa5-437d-b851-f53457d49333",
"metadata": {},
"source": [
"### Zip File Geodatabases"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "69c6203b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Zipping country_2021.gdb\n",
" adding: country_2021.gdb/ (stored 0%)\n",
" adding: country_2021.gdb/gdb (stored 0%)\n",
" adding: country_2021.gdb/timestamps (deflated 98%)\n",
" adding: country_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: country_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: country_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: country_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: country_2021.gdb/a00000003.gdbtable (deflated 42%)\n",
" adding: country_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: country_2021.gdb/a00000004.gdbtable (deflated 44%)\n",
" adding: country_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: country_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: country_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: country_2021.gdb/a00000006.gdbtable (deflated 33%)\n",
" adding: country_2021.gdb/a00000006.gdbtablx (deflated 72%)\n",
" adding: country_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: country_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
"Zipping grc_2021.gdb\n",
" adding: grc_2021.gdb/ (stored 0%)\n",
" adding: grc_2021.gdb/gdb (stored 0%)\n",
" adding: grc_2021.gdb/timestamps (deflated 98%)\n",
" adding: grc_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: grc_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: grc_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: grc_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: grc_2021.gdb/a00000003.gdbtable (deflated 42%)\n",
" adding: grc_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: grc_2021.gdb/a00000004.gdbtable (deflated 44%)\n",
" adding: grc_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: grc_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: grc_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: grc_2021.gdb/a00000006.gdbtable (deflated 33%)\n",
" adding: grc_2021.gdb/a00000006.gdbtablx (deflated 72%)\n",
" adding: grc_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: grc_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
"Zipping pr_2021.gdb\n",
" adding: pr_2021.gdb/ (stored 0%)\n",
" adding: pr_2021.gdb/gdb (stored 0%)\n",
" adding: pr_2021.gdb/timestamps (deflated 98%)\n",
" adding: pr_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: pr_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: pr_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: pr_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: pr_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: pr_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: pr_2021.gdb/a00000004.gdbtable (deflated 77%)\n",
" adding: pr_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: pr_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: pr_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: pr_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: pr_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: pr_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: pr_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: pr_2021.gdb/a00000009.gdbtable (deflated 7%)\n",
" adding: pr_2021.gdb/a00000009.gdbtablx (deflated 98%)\n",
" adding: pr_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: pr_2021.gdb/a00000009.spx (deflated 97%)\n",
"Zipping er_2021.gdb\n",
" adding: er_2021.gdb/ (stored 0%)\n",
" adding: er_2021.gdb/gdb (stored 0%)\n",
" adding: er_2021.gdb/timestamps (deflated 98%)\n",
" adding: er_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: er_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: er_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: er_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: er_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: er_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: er_2021.gdb/a00000004.gdbtable (deflated 78%)\n",
" adding: er_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: er_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: er_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: er_2021.gdb/a00000006.gdbtable (deflated 22%)\n",
" adding: er_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: er_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: er_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: er_2021.gdb/a00000009.gdbtable (deflated 8%)\n",
" adding: er_2021.gdb/a00000009.gdbtablx (deflated 94%)\n",
" adding: er_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: er_2021.gdb/a00000009.spx (deflated 95%)\n",
"Zipping car_2021.gdb\n",
" adding: car_2021.gdb/ (stored 0%)\n",
" adding: car_2021.gdb/gdb (stored 0%)\n",
" adding: car_2021.gdb/timestamps (deflated 98%)\n",
" adding: car_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: car_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: car_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: car_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: car_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: car_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: car_2021.gdb/a00000004.gdbtable (deflated 78%)\n",
" adding: car_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: car_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: car_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: car_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: car_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: car_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: car_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: car_2021.gdb/a00000009.gdbtable (deflated 9%)\n",
" adding: car_2021.gdb/a00000009.gdbtablx (deflated 94%)\n",
" adding: car_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: car_2021.gdb/a00000009.spx (deflated 95%)\n",
"Zipping cd_2021.gdb\n",
" adding: cd_2021.gdb/ (stored 0%)\n",
" adding: cd_2021.gdb/gdb (stored 0%)\n",
" adding: cd_2021.gdb/timestamps (deflated 98%)\n",
" adding: cd_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: cd_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: cd_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: cd_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: cd_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: cd_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: cd_2021.gdb/a00000004.gdbtable (deflated 80%)\n",
" adding: cd_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: cd_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: cd_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: cd_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: cd_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: cd_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: cd_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: cd_2021.gdb/a00000009.gdbtable (deflated 9%)\n",
" adding: cd_2021.gdb/a00000009.gdbtablx (deflated 81%)\n",
" adding: cd_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: cd_2021.gdb/a00000009.spx (deflated 86%)\n",
"Zipping ccs_2021.gdb\n",
" adding: ccs_2021.gdb/ (stored 0%)\n",
" adding: ccs_2021.gdb/gdb (stored 0%)\n",
" adding: ccs_2021.gdb/timestamps (deflated 98%)\n",
" adding: ccs_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: ccs_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: ccs_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: ccs_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: ccs_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: ccs_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: ccs_2021.gdb/a00000004.gdbtable (deflated 81%)\n",
" adding: ccs_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: ccs_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: ccs_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: ccs_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: ccs_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: ccs_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: ccs_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: ccs_2021.gdb/a00000009.gdbtable (deflated 12%)\n",
" adding: ccs_2021.gdb/a00000009.gdbtablx (deflated 52%)\n",
" adding: ccs_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: ccs_2021.gdb/a00000009.spx (deflated 90%)\n",
"Zipping cma_2021.gdb\n",
" adding: cma_2021.gdb/ (stored 0%)\n",
" adding: cma_2021.gdb/gdb (stored 0%)\n",
" adding: cma_2021.gdb/timestamps (deflated 98%)\n",
" adding: cma_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: cma_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: cma_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: cma_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: cma_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: cma_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: cma_2021.gdb/a00000004.gdbtable (deflated 79%)\n",
" adding: cma_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: cma_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: cma_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: cma_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: cma_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: cma_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: cma_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: cma_2021.gdb/a00000009.gdbtable (deflated 9%)\n",
" adding: cma_2021.gdb/a00000009.gdbtablx (deflated 90%)\n",
" adding: cma_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: cma_2021.gdb/a00000009.spx (deflated 88%)\n",
"Zipping csd_2021.gdb\n",
" adding: csd_2021.gdb/ (stored 0%)\n",
" adding: csd_2021.gdb/gdb (stored 0%)\n",
" adding: csd_2021.gdb/timestamps (deflated 98%)\n",
" adding: csd_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: csd_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: csd_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: csd_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: csd_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: csd_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: csd_2021.gdb/a00000004.gdbtable (deflated 85%)\n",
" adding: csd_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: csd_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: csd_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: csd_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: csd_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: csd_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: csd_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: csd_2021.gdb/a00000009.gdbtable (deflated 14%)\n",
" adding: csd_2021.gdb/a00000009.gdbtablx (deflated 53%)\n",
" adding: csd_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: csd_2021.gdb/a00000009.spx (deflated 89%)\n",
"Zipping fed_2021_2013.gdb\n",
" adding: fed_2021_2013.gdb/ (stored 0%)\n",
" adding: fed_2021_2013.gdb/gdb (stored 0%)\n",
" adding: fed_2021_2013.gdb/timestamps (deflated 98%)\n",
" adding: fed_2021_2013.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: fed_2021_2013.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: fed_2021_2013.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: fed_2021_2013.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: fed_2021_2013.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: fed_2021_2013.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: fed_2021_2013.gdb/a00000004.gdbtable (deflated 79%)\n",
" adding: fed_2021_2013.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: fed_2021_2013.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: fed_2021_2013.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: fed_2021_2013.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: fed_2021_2013.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: fed_2021_2013.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: fed_2021_2013.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: fed_2021_2013.gdb/a00000009.gdbtable (deflated 9%)\n",
" adding: fed_2021_2013.gdb/a00000009.gdbtablx (deflated 79%)\n",
" adding: fed_2021_2013.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: fed_2021_2013.gdb/a00000009.spx (deflated 94%)\n",
"Zipping dpl_2021.gdb\n",
" adding: dpl_2021.gdb/ (stored 0%)\n",
" adding: dpl_2021.gdb/gdb (stored 0%)\n",
" adding: dpl_2021.gdb/timestamps (deflated 98%)\n",
" adding: dpl_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: dpl_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: dpl_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: dpl_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: dpl_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: dpl_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: dpl_2021.gdb/a00000004.gdbtable (deflated 78%)\n",
" adding: dpl_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: dpl_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: dpl_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: dpl_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: dpl_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: dpl_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: dpl_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: dpl_2021.gdb/a00000009.gdbtable (deflated 15%)\n",
" adding: dpl_2021.gdb/a00000009.gdbtablx (deflated 55%)\n",
" adding: dpl_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: dpl_2021.gdb/a00000009.spx (deflated 85%)\n",
"Zipping fsa_2021.gdb\n",
" adding: fsa_2021.gdb/ (stored 0%)\n",
" adding: fsa_2021.gdb/gdb (stored 0%)\n",
" adding: fsa_2021.gdb/timestamps (deflated 98%)\n",
" adding: fsa_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: fsa_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: fsa_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: fsa_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: fsa_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: fsa_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: fsa_2021.gdb/a00000004.gdbtable (deflated 77%)\n",
" adding: fsa_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: fsa_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: fsa_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: fsa_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: fsa_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: fsa_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: fsa_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: fsa_2021.gdb/a00000009.gdbtable (deflated 8%)\n",
" adding: fsa_2021.gdb/a00000009.gdbtablx (deflated 54%)\n",
" adding: fsa_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: fsa_2021.gdb/a00000009.spx (deflated 89%)\n",
"Zipping pop_ctr_2021.gdb\n",
" adding: pop_ctr_2021.gdb/ (stored 0%)\n",
" adding: pop_ctr_2021.gdb/gdb (stored 0%)\n",
" adding: pop_ctr_2021.gdb/timestamps (deflated 98%)\n",
" adding: pop_ctr_2021.gdb/a00000001.gdbtable (deflated 33%)\n",
" adding: pop_ctr_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: pop_ctr_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: pop_ctr_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: pop_ctr_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: pop_ctr_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: pop_ctr_2021.gdb/a00000004.gdbtable (deflated 80%)\n",
" adding: pop_ctr_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: pop_ctr_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: pop_ctr_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: pop_ctr_2021.gdb/a00000006.gdbtable (deflated 22%)\n",
" adding: pop_ctr_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: pop_ctr_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: pop_ctr_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: pop_ctr_2021.gdb/a00000009.gdbtable (deflated 10%)\n",
" adding: pop_ctr_2021.gdb/a00000009.gdbtablx (deflated 72%)\n",
" adding: pop_ctr_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: pop_ctr_2021.gdb/a00000009.spx (deflated 85%)\n",
"Zipping ct_2021.gdb\n",
" adding: ct_2021.gdb/ (stored 0%)\n",
" adding: ct_2021.gdb/gdb (stored 0%)\n",
" adding: ct_2021.gdb/timestamps (deflated 98%)\n",
" adding: ct_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: ct_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: ct_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: ct_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: ct_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: ct_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: ct_2021.gdb/a00000004.gdbtable (deflated 80%)\n",
" adding: ct_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: ct_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: ct_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: ct_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: ct_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: ct_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: ct_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: ct_2021.gdb/a00000009.gdbtable (deflated 17%)\n",
" adding: ct_2021.gdb/a00000009.gdbtablx (deflated 52%)\n",
" adding: ct_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: ct_2021.gdb/a00000009.spx (deflated 88%)\n",
"Zipping da_2021.gdb\n",
" adding: da_2021.gdb/ (stored 0%)\n",
" adding: da_2021.gdb/gdb (stored 0%)\n",
" adding: da_2021.gdb/timestamps (deflated 98%)\n",
" adding: da_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: da_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: da_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: da_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: da_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: da_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: da_2021.gdb/a00000004.gdbtable (deflated 85%)\n",
" adding: da_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: da_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: da_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: da_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: da_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: da_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: da_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: da_2021.gdb/a00000009.gdbtable (deflated 31%)\n",
" adding: da_2021.gdb/a00000009.gdbtablx (deflated 45%)\n",
" adding: da_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: da_2021.gdb/a00000009.spx (deflated 88%)\n",
"Zipping db_2021.gdb\n",
" adding: db_2021.gdb/ (stored 0%)\n",
" adding: db_2021.gdb/gdb (stored 0%)\n",
" adding: db_2021.gdb/timestamps (deflated 98%)\n",
" adding: db_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: db_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: db_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: db_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: db_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: db_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: db_2021.gdb/a00000004.gdbtable (deflated 86%)\n",
" adding: db_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: db_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: db_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: db_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: db_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: db_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: db_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: db_2021.gdb/a00000009.gdbtable (deflated 59%)\n",
" adding: db_2021.gdb/a00000009.gdbtablx (deflated 45%)\n",
" adding: db_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: db_2021.gdb/a00000009.spx (deflated 88%)\n",
"Zipping ada_2021.gdb\n",
" adding: ada_2021.gdb/ (stored 0%)\n",
" adding: ada_2021.gdb/gdb (stored 0%)\n",
" adding: ada_2021.gdb/timestamps (deflated 98%)\n",
" adding: ada_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: ada_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: ada_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: ada_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: ada_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: ada_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: ada_2021.gdb/a00000004.gdbtable (deflated 82%)\n",
" adding: ada_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: ada_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: ada_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: ada_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: ada_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: ada_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: ada_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: ada_2021.gdb/a00000009.gdbtable (deflated 13%)\n",
" adding: ada_2021.gdb/a00000009.gdbtablx (deflated 51%)\n",
" adding: ada_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: ada_2021.gdb/a00000009.spx (deflated 89%)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" adding: fed_2021.gdb/a00000004.gdbtable (deflated 79%)\n",
" adding: fed_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: fed_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: fed_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: fed_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: fed_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: fed_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: fed_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: fed_2021.gdb/a00000009.gdbtable (deflated 9%)\n",
" adding: fed_2021.gdb/a00000009.gdbtablx (deflated 79%)\n",
" adding: fed_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: fed_2021.gdb/a00000009.spx (deflated 94%)\n",
"Zipping car_2021.gdb\n",
" adding: car_2021.gdb/ (stored 0%)\n",
" adding: car_2021.gdb/gdb (stored 0%)\n",
" adding: car_2021.gdb/timestamps (deflated 98%)\n",
" adding: car_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: car_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: car_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: car_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: car_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: car_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: car_2021.gdb/a00000004.gdbtable (deflated 78%)\n",
" adding: car_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: car_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: car_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: car_2021.gdb/a00000006.gdbtable (deflated 22%)\n",
" adding: car_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: car_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: car_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: car_2021.gdb/a00000009.gdbtable (deflated 9%)\n",
" adding: car_2021.gdb/a00000009.gdbtablx (deflated 94%)\n",
" adding: car_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: car_2021.gdb/a00000009.spx (deflated 95%)\n",
"Zipping dpl_2021.gdb\n",
" adding: dpl_2021.gdb/ (stored 0%)\n",
" adding: dpl_2021.gdb/gdb (stored 0%)\n",
" adding: dpl_2021.gdb/timestamps (deflated 98%)\n",
" adding: dpl_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: dpl_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: dpl_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: dpl_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: dpl_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: dpl_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: dpl_2021.gdb/a00000004.gdbtable (deflated 78%)\n",
" adding: dpl_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: dpl_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: dpl_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: dpl_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: dpl_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: dpl_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: dpl_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: dpl_2021.gdb/a00000009.gdbtable (deflated 14%)\n",
" adding: dpl_2021.gdb/a00000009.gdbtablx (deflated 55%)\n",
" adding: dpl_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: dpl_2021.gdb/a00000009.spx (deflated 83%)\n",
"Zipping fsa_2021.gdb\n",
" adding: fsa_2021.gdb/ (stored 0%)\n",
" adding: fsa_2021.gdb/gdb (stored 0%)\n",
" adding: fsa_2021.gdb/timestamps (deflated 98%)\n",
" adding: fsa_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: fsa_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: fsa_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: fsa_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: fsa_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: fsa_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: fsa_2021.gdb/a00000004.gdbtable (deflated 77%)\n",
" adding: fsa_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: fsa_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: fsa_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: fsa_2021.gdb/a00000006.gdbtable (deflated 22%)\n",
" adding: fsa_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: fsa_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: fsa_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: fsa_2021.gdb/a00000009.gdbtable (deflated 8%)\n",
" adding: fsa_2021.gdb/a00000009.gdbtablx (deflated 54%)\n",
" adding: fsa_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: fsa_2021.gdb/a00000009.spx (deflated 89%)\n",
"Zipping cma_2021.gdb\n",
" adding: cma_2021.gdb/ (stored 0%)\n",
" adding: cma_2021.gdb/gdb (stored 0%)\n",
" adding: cma_2021.gdb/timestamps (deflated 98%)\n",
" adding: cma_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: cma_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: cma_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: cma_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: cma_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: cma_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: cma_2021.gdb/a00000004.gdbtable (deflated 79%)\n",
" adding: cma_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: cma_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: cma_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: cma_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: cma_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: cma_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: cma_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: cma_2021.gdb/a00000009.gdbtable (deflated 9%)\n",
" adding: cma_2021.gdb/a00000009.gdbtablx (deflated 89%)\n",
" adding: cma_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: cma_2021.gdb/a00000009.spx (deflated 88%)\n",
"Zipping pc_2021.gdb\n",
" adding: pc_2021.gdb/ (stored 0%)\n",
" adding: pc_2021.gdb/gdb (stored 0%)\n",
" adding: pc_2021.gdb/timestamps (deflated 98%)\n",
" adding: pc_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: pc_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: pc_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: pc_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: pc_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: pc_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: pc_2021.gdb/a00000004.gdbtable (deflated 80%)\n",
" adding: pc_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: pc_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: pc_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: pc_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: pc_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: pc_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: pc_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: pc_2021.gdb/a00000009.gdbtable (deflated 10%)\n",
" adding: pc_2021.gdb/a00000009.gdbtablx (deflated 72%)\n",
" adding: pc_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: pc_2021.gdb/a00000009.spx (deflated 85%)\n",
"Zipping ct_2021.gdb\n",
" adding: ct_2021.gdb/ (stored 0%)\n",
" adding: ct_2021.gdb/gdb (stored 0%)\n",
" adding: ct_2021.gdb/timestamps (deflated 98%)\n",
" adding: ct_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: ct_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: ct_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: ct_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: ct_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: ct_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: ct_2021.gdb/a00000004.gdbtable (deflated 80%)\n",
" adding: ct_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: ct_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: ct_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: ct_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: ct_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: ct_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: ct_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: ct_2021.gdb/a00000009.gdbtable (deflated 17%)\n",
" adding: ct_2021.gdb/a00000009.gdbtablx (deflated 52%)\n",
" adding: ct_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: ct_2021.gdb/a00000009.spx (deflated 88%)\n",
"Zipping da_2021.gdb\n",
" adding: da_2021.gdb/ (stored 0%)\n",
" adding: da_2021.gdb/gdb (stored 0%)\n",
" adding: da_2021.gdb/timestamps (deflated 98%)\n",
" adding: da_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: da_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: da_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: da_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: da_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: da_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: da_2021.gdb/a00000004.gdbtable (deflated 85%)\n",
" adding: da_2021.gdb/a00000004.gdbtablx (deflated 99%)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" adding: da_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: da_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: da_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: da_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: da_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: da_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: da_2021.gdb/a00000009.gdbtable (deflated 31%)\n",
" adding: da_2021.gdb/a00000009.gdbtablx (deflated 45%)\n",
" adding: da_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: da_2021.gdb/a00000009.spx (deflated 88%)\n",
"Zipping db_2021.gdb\n",
" adding: db_2021.gdb/ (stored 0%)\n",
" adding: db_2021.gdb/gdb (stored 0%)\n",
" adding: db_2021.gdb/timestamps (deflated 98%)\n",
" adding: db_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: db_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: db_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: db_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: db_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: db_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: db_2021.gdb/a00000004.gdbtable (deflated 86%)\n",
" adding: db_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: db_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: db_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: db_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: db_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: db_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: db_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: db_2021.gdb/a00000009.gdbtable (deflated 59%)\n",
" adding: db_2021.gdb/a00000009.gdbtablx (deflated 45%)\n",
" adding: db_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: db_2021.gdb/a00000009.spx (deflated 88%)\n",
"Zipping ada_2021.gdb\n",
" adding: ada_2021.gdb/ (stored 0%)\n",
" adding: ada_2021.gdb/gdb (stored 0%)\n",
" adding: ada_2021.gdb/timestamps (deflated 98%)\n",
" adding: ada_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: ada_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: ada_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: ada_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: ada_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: ada_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: ada_2021.gdb/a00000004.gdbtable (deflated 82%)\n",
" adding: ada_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: ada_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: ada_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: ada_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: ada_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: ada_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: ada_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: ada_2021.gdb/a00000009.gdbtable (deflated 13%)\n",
" adding: ada_2021.gdb/a00000009.gdbtablx (deflated 51%)\n",
" adding: ada_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: ada_2021.gdb/a00000009.spx (deflated 89%)\n",
"Zipping pn_2021.gdb\n",
" adding: pn_2021.gdb/ (stored 0%)\n",
" adding: pn_2021.gdb/gdb (stored 0%)\n",
" adding: pn_2021.gdb/timestamps (deflated 98%)\n",
" adding: pn_2021.gdb/a00000001.gdbtable (deflated 34%)\n",
" adding: pn_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: pn_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: pn_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: pn_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: pn_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: pn_2021.gdb/a00000004.gdbtable (deflated 87%)\n",
" adding: pn_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: pn_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: pn_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: pn_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: pn_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: pn_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: pn_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: pn_2021.gdb/a00000009.gdbtable (deflated 93%)\n",
" adding: pn_2021.gdb/a00000009.gdbtablx (deflated 46%)\n",
" adding: pn_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: pn_2021.gdb/a00000009.spx (deflated 84%)\n",
"Zipping road_2021.gdb\n",
" adding: road_2021.gdb/ (stored 0%)\n",
" adding: road_2021.gdb/gdb (stored 0%)\n",
" adding: road_2021.gdb/timestamps (deflated 98%)\n",
" adding: road_2021.gdb/a00000001.gdbtable (deflated 35%)\n",
" adding: road_2021.gdb/a00000001.gdbtablx (deflated 99%)\n",
" adding: road_2021.gdb/a00000002.gdbtable (deflated 68%)\n",
" adding: road_2021.gdb/a00000002.gdbtablx (deflated 97%)\n",
" adding: road_2021.gdb/a00000003.gdbtable (deflated 56%)\n",
" adding: road_2021.gdb/a00000003.gdbtablx (deflated 99%)\n",
" adding: road_2021.gdb/a00000004.gdbtable (deflated 90%)\n",
" adding: road_2021.gdb/a00000004.gdbtablx (deflated 99%)\n",
" adding: road_2021.gdb/a00000005.gdbtable (deflated 37%)\n",
" adding: road_2021.gdb/a00000005.gdbtablx (deflated 97%)\n",
" adding: road_2021.gdb/a00000006.gdbtable (deflated 23%)\n",
" adding: road_2021.gdb/a00000006.gdbtablx (deflated 99%)\n",
" adding: road_2021.gdb/a00000007.gdbtable (deflated 47%)\n",
" adding: road_2021.gdb/a00000007.gdbtablx (deflated 98%)\n",
" adding: road_2021.gdb/a00000009.gdbtable (deflated 79%)\n",
" adding: road_2021.gdb/a00000009.gdbtablx (deflated 45%)\n",
" adding: road_2021.gdb/a00000009.gdbindexes (deflated 37%)\n",
" adding: road_2021.gdb/a00000009.spx (deflated 85%)\n"
]
}
],
"source": [
"cd ${output_folder}\n",
"for file in ${files[@]}\n",
"do\n",
" output_file=\"${file}.gdb\"\n",
" echo \"Zipping ${output_file}\"\n",
" zip -r \"${output_file}.zip\" ${output_file}\n",
" rm -rf ${output_file}\n",
"done"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1bde70a8-e4ae-450b-9929-de35970172ab",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Bash",
"language": "bash",
"name": "bash"
},
"language_info": {
"codemirror_mode": "shell",
"file_extension": ".sh",
"mimetype": "text/x-sh",
"name": "bash"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/index.html
Executable
+88
View File
@@ -0,0 +1,88 @@
<!DOCTYPE html>
<html lang="en">
<head>
<title>Add a vector tile source</title>
<meta property="og:description" content="Add a vector source to a map." />
<meta charset='utf-8'>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel='stylesheet' href='https://unpkg.com/maplibre-gl@5.2.0/dist/maplibre-gl.css' />
<script src='https://unpkg.com/maplibre-gl@5.2.0/dist/maplibre-gl.js'></script>
<style>
body {
margin: 0;
padding: 0;
}
html,
body,
#map {
height: 100%;
}
</style>
</head>
<body>
<div id="map"></div>
<script>
const map = new maplibregl.Map({
container: 'map',
style:
'https://api.maptiler.com/maps/streets/style.json?key=xBjamUQUPDqLGNMjqo53',
zoom: 10,
center: [-79.3064, 43.7306]
});
map.on('style.load', () => {
map.setProjection({
type: 'globe', // Set projection to globe
});
});
// This is the metadata http://192.168.0.131:3000/test
map.on('load', () => {
map.addSource('my-vector-tiles', {
type: 'vector',
tiles: ['http://192.168.0.131:3000/test/{z}/{x}/{y}'],
minzoom: 10,
maxzoom: 10
});
map.addLayer({
'id': 'my-layer',
'type': 'fill',
'source': 'my-vector-tiles',
'source-layer': 'test', // This must match the layer name in Martin
'paint': {
"fill-outline-color": "#000",
"fill-antialias": true,
'fill-color': [
'interpolate', ['linear'], ['get', 'count_total_1'],
0, '#ffffff',
45276, '#ffbfbf',
175383, '#ff8080',
438366, '#ff4040',
1306784, '#ff0000'
],
'fill-opacity': 0.7,
},
//filter: ["==", "dguid", ""], // Initially empty filter
});
});
map.on('click', 'my-layer', (e) => {
// Check if any features were clicked
if (e.features.length > 0) {
// Retrieve the first clicked feature
const feature = e.features[0];
const dguid = e.features[0].properties.dguid;
const properties = feature.properties;
//console.log(properties['dguid'], properties['count_total_1']);
//map.setFilter("my-layer", ["==", "dguid", dguid]);
}
});
</script>
</body>
</html>
experiments/index_da.html
Executable
+90
View File
@@ -0,0 +1,90 @@
<!DOCTYPE html>
<html lang="en">
<head>
<title>Add a vector tile source</title>
<meta property="og:description" content="Add a vector source to a map." />
<meta charset='utf-8'>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel='stylesheet' href='https://unpkg.com/maplibre-gl@5.2.0/dist/maplibre-gl.css' />
<script src='https://unpkg.com/maplibre-gl@5.2.0/dist/maplibre-gl.js'></script>
<style>
body {
margin: 0;
padding: 0;
}
html,
body,
#map {
height: 100%;
}
</style>
</head>
<body>
<div id="map"></div>
<script>
const map = new maplibregl.Map({
container: 'map',
style:
'https://api.maptiler.com/maps/streets/style.json?key=xBjamUQUPDqLGNMjqo53',
zoom: 10,
center: [-79.3064, 43.7306],
hash: true,
maxZoom: 18
});
map.on('style.load', () => {
map.setProjection({
type: 'globe', // Set projection to globe
});
});
// This is the metadata http://192.168.0.131:3000/test_da
map.on('load', () => {
map.addSource('my-vector-tiles', {
type: 'vector',
tiles: ['http://192.168.0.131:3000/test_da/{z}/{x}/{y}'],
minzoom: 10,
maxzoom: 14
});
map.addLayer({
'id': 'my-layer',
'type': 'fill',
'source': 'my-vector-tiles',
'source-layer': 'test_da', // This must match the layer name in Martin
'paint': {
"fill-outline-color": "#000",
"fill-antialias": true,
'fill-color': [
'interpolate', ['linear'], ['get', 'count_total_1'],
0, '#ffffff',
200, '#ffbfbf',
400, '#ff8080',
600, '#ff4040',
800, '#ff0000'
],
'fill-opacity': 0.7,
},
//filter: ["==", "dguid", ""], // Initially empty filter
});
});
map.on('click', 'my-layer', (e) => {
// Check if any features were clicked
if (e.features.length > 0) {
// Retrieve the first clicked feature
const feature = e.features[0];
const dguid = e.features[0].properties.dguid;
const properties = feature.properties;
console.log(properties['dguid'], properties['count_total_1']);
//map.setFilter("my-layer", ["==", "dguid", dguid]);
}
});
</script>
</body>
</html>
experiments/lonboard_duckdb.ipynb
+152
View File
@@ -0,0 +1,152 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "a1aa3974-2590-4c35-8278-29bf8269a3b5",
"metadata": {},
"outputs": [],
"source": [
"import duckdb\n",
"from lonboard import viz"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "68fc8e11-6463-4ad5-b70e-61910cb6cfbe",
"metadata": {},
"outputs": [],
"source": [
"con = duckdb.connect()\n",
"con.install_extension(\"spatial\")\n",
"con.load_extension(\"spatial\")"
]
},
{
"cell_type": "markdown",
"id": "10c0d819-c86a-4c6c-afed-396136685d0d",
"metadata": {},
"source": [
"# 2021 Dissemination Blocks"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7d7ee1da-4fa7-4d66-bac1-b2e384bd4110",
"metadata": {},
"outputs": [],
"source": [
"sql = \"SELECT * FROM 'https://files.sisyphus.ca/db_2021.parquet';\"\n",
"query = con.sql(sql)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "da57c688-ca49-4fa7-8178-58e0cc9cb351",
"metadata": {},
"outputs": [],
"source": [
"viz(query, con=con)"
]
},
{
"cell_type": "markdown",
"id": "3efc0288-73ea-4b19-b0d1-261c863bf330",
"metadata": {},
"source": [
"# 2021 Road Network File"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "70b951ad-9b8f-47b6-ad6d-8f6e2ae77a56",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "af33fca355de497586bc9d1e80cef1a7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatProgress(value=0.0, layout=Layout(width='auto'), style=ProgressStyle(bar_color='black'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "31ed9c9423ee4601a4ec42b9a787aa44",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatProgress(value=0.0, layout=Layout(width='auto'), style=ProgressStyle(bar_color='black'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "1389e30ae8964a8f9af5a1cf3899077c",
"version_major": 2,
"version_minor": 1
},
"text/plain": [
"Map(basemap_style=<CartoBasemap.DarkMatter: 'https://basemaps.cartocdn.com/gl/dark-matter-gl-style/style.json'…"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sql = \"\"\"\n",
"SELECT * FROM 'https://files.sisyphus.ca/road_2021.parquet'\n",
"WHERE csd_name_left in ('Toronto', 'Ottawa') OR csd_name_right in ('Toronto', 'Ottawa');\n",
"\"\"\"\n",
"query = con.sql(sql)\n",
"viz(query, con=con)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c0006e63-37c9-4dcc-941e-8f247beaa563",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/make_tiles.sh
+3
View File
File diff suppressed because one or more lines are too long
experiments/make_tiles_da.sh
+1
View File
File diff suppressed because one or more lines are too long
experiments/vector_tiles.ipynb
+110
View File
@@ -0,0 +1,110 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "d3e03ab2",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"id": "a9210f39",
"metadata": {},
"source": [
"# This is required for tippecanoe to guess the proper types for the fields"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1cd097e5",
"metadata": {},
"outputs": [],
"source": [
"tippecanoe_df_types = pd.DataFrame(da_df_cop_df.dtypes)\n",
"\n",
"attributes_types = [] \n",
"for row in tippecanoe_df_types.itertuples():\n",
" # This is what it should look like --attribute-type=count_total_1:int\n",
" column_name = row[0]\n",
" column_type = str(row[1])\n",
" \n",
" if 'Int' in column_type:\n",
" parameter = f\"--attribute-type={column_name}:int\"\n",
" attributes_types.append(parameter)\n",
" elif 'Float' in column_type:\n",
" parameter = f\"--attribute-type={column_name}:float\"\n",
" attributes_types.append(parameter) "
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "d8b20754",
"metadata": {},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'attributes_types' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[2], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;241m.\u001b[39mjoin(\u001b[43mattributes_types\u001b[49m))\n",
"\u001b[0;31mNameError\u001b[0m: name 'attributes_types' is not defined"
]
}
],
"source": [
"print(' '.join(attributes_types))"
]
},
{
"cell_type": "markdown",
"id": "160d5b8f",
"metadata": {},
"source": [
"# Then we edit the metadata for the generated vector tiles to remove the long generator_options\n",
"\n",
"Edit metadata.json and remove generator_options\n",
"```\n",
"pmtiles show test.pmtiles --metadata > metadata.json\n",
"pmtiles edit test.pmtiles --metadata=metadata.json\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9280e3d2",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
geographic_attribute_file/download.sh
Executable
+11
View File
@@ -0,0 +1,11 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/geographic_attribute_file" ]
then
echo "Making directory ${DATA_FOLDER}/geographic_attribute_file/"
mkdir -p ${DATA_FOLDER}/geographic_attribute_file/{input,extracted,output}
fi
INPUT_FOLDER="${DATA_FOLDER}/geographic_attribute_file/input"
echo "Downloading 2021 geographic attribute file"
aria2c -x16 -i "${SCRIPT_DIR}/geographic_attribute_file/files.txt" --dir=${INPUT_FOLDER} --auto-file-renaming=false
geographic_attribute_file/files.txt
Executable
+2
View File
@@ -0,0 +1,2 @@
# 2021. Here is the reference guide https://www150.statcan.gc.ca/n1/pub/92-151-g/92-151-g2021001-eng.htm
https://www12.statcan.gc.ca/census-recensement/2021/geo/aip-pia/attribute-attribs/files-fichiers/2021_92-151_X.zip
geographic_attribute_file/load.sh
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/bin/bash
INPUT_FOLDER="${DATA_FOLDER}/geographic_attribute_file/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/geographic_attribute_file/extracted"
import_2021() {
echo "Extracting ${INPUT_FOLDER}/2021_92-151_X.zip"
unzip -q -n ${INPUT_FOLDER}/2021_92-151_X.zip -d ${EXTRACTED_FOLDER}
python geographic_attribute_file/process.py ${EXTRACTED_FOLDER}/2021_92-151_X.csv
}
import_2021
geographic_attribute_file/process.py
Executable
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env python
# coding: utf-8
import os
import sys
import numpy as np
import pandas as pd
from sqlalchemy import create_engine
filename = sys.argv[1]
DATABASE = os.environ.get("POSTGRES_DB")
USER = os.environ.get("POSTGRES_USER")
PASSWORD = os.environ.get("POSTGRES_PASSWORD")
engine = create_engine(f"postgresql://{USER}:{PASSWORD}@db:5432/{DATABASE}")
"""
Data dictionary is here:
https://web.archive.org/web/20240918110218/https://www150.statcan.gc.ca/n1/pub/92-151-g/2021001/tbl/tbl4_1-eng.htm
This processes the entire DGUID hierarchy and other useful fields
"""
print(f"Processing {filename}")
params = {
'filepath_or_buffer': filename,
'encoding': 'latin-1',
'usecols': ['PRDGUID_PRIDUGD', 'CDDGUID_DRIDUGD',
'FEDDGUID_CEFIDUGD', 'CSDDGUID_SDRIDUGD',
'DPLDGUID_LDIDUGD', 'ERDGUID_REIDUGD',
'CCSDGUID_SRUIDUGD', 'SACTYPE_CSSGENRE', 'SACCODE_CSSCODE', 'CMAPDGUID_RMRPIDUGD', 'CMADGUID_RMRIDUGD',
'CMATYPE_RMRGENRE', 'CTDGUID_SRIDUGD', 'POPCTRRAPDGUID_CTRPOPRRPIDUGD', 'POPCTRRADGUID_CTRPOPRRIDUGD',
'DADGUID_ADIDUGD', 'DBDGUID_IDIDUGD',
# 2021 Block population, private dwellings, usual residents
'DBPOP2021_IDPOP2021', 'DBTDWELL2021_IDTLOG2021', 'DBURDWELL2021_IDRHLOG2021',
# 2021 Census Indian reserve refusal flag
'DBIR2021_IDRI2021',
'ADADGUID_ADAIDUGD'
],
# Apparently they have to be int64 because there's NA values
'dtype': {
'DBPOP2021_IDPOP2021': "Int64",
'DBTDWELL2021_IDTLOG2021': "Int64",
'DBURDWELL2021_IDRHLOG2021': "Int64"
}
}
gaf_2021_df = pd.read_csv(**params)
# Rename columns, replace french portion
gaf_2021_df.rename(columns={
'PRDGUID_PRIDUGD': 'pr_dguid',
'CDDGUID_DRIDUGD': 'cd_dguid',
'FEDDGUID_CEFIDUGD': 'fed_dguid',
'CSDDGUID_SDRIDUGD': 'csd_dguid',
'DPLDGUID_LDIDUGD': 'dpl_dguid',
'ERDGUID_REIDUGD': 'er_dguid',
'CCSDGUID_SRUIDUGD': 'ccs_dguid',
'SACTYPE_CSSGENRE': 'sac_type',
'SACCODE_CSSCODE': 'sac_code',
'CMAPDGUID_RMRPIDUGD': 'cma_p_dguid',
'CMADGUID_RMRIDUGD': 'cma_dguid',
'CTDGUID_SRIDUGD': 'ct_dguid',
'POPCTRRAPDGUID_CTRPOPRRPIDUGD': 'pop_ctr_p_dguid',
'POPCTRRADGUID_CTRPOPRRIDUGD': 'pop_ctr_dguid',
'DADGUID_ADIDUGD': 'da_dguid',
'DBDGUID_IDIDUGD': 'db_dguid',
'DBPOP2021_IDPOP2021': 'db_pop_2021',
# This one needs work
'DBTDWELL2021_IDTLOG2021': 'db_total_private_dwell_2021',
# I don't particularly like this one
'DBURDWELL2021_IDRHLOG2021': 'db_usual_residents_dwellings_2021',
'DBIR2021_IDRI2021': 'db_ir_2021',
'ADADGUID_ADAIDUGD': 'ada_dguid'
}, inplace=True)
columns_ordered = ['pr_dguid', 'fed_dguid', 'er_dguid', 'cd_dguid',
'dpl_dguid', 'ccs_dguid', 'csd_dguid', 'sac_type', 'sac_code',
'cma_p_dguid', 'cma_dguid',
'pop_ctr_p_dguid', 'pop_ctr_dguid',
'ada_dguid', 'ct_dguid', 'da_dguid', 'db_dguid',
'db_pop_2021', 'db_total_private_dwell_2021', 'db_usual_residents_dwellings_2021', 'db_ir_2021']
gaf_2021_df = gaf_2021_df.reindex(columns_ordered, axis=1)
print("Loading 2021 geographic attribute file to PostgreSQL as gaf_2021")
gaf_2021_df.to_sql(name='gaf_2021',
con=engine,
index=False,
chunksize=50000,
if_exists='replace',
schema='silver'
)
geosuite/download.sh
Executable
+11
View File
@@ -0,0 +1,11 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/geosuite" ]
then
echo "Making directory ${DATA_FOLDER}/geosuite/"
mkdir -p ${DATA_FOLDER}/geosuite/{input,extracted,output}
fi
INPUT_FOLDER="${DATA_FOLDER}/geosuite/input"
echo "Downloading geosuite files"
aria2c -x16 -i "${SCRIPT_DIR}/geosuite/files.txt" --dir=${INPUT_FOLDER} --auto-file-renaming=false
geosuite/files.txt
Executable
+10
View File
@@ -0,0 +1,10 @@
# 2021. Here is the reference guide https://web.archive.org/web/20240809014903/https://www150.statcan.gc.ca/n1/pub/92-150-g/92-150-g2021001-eng.htm
https://www12.statcan.gc.ca/census-recensement/2021/geo/aip-pia/geosuite/files-fichiers/2021_92-150-X_eng.zip
# 2016. Here is the reference guide https://web.archive.org/web/20250115043056/https://www150.statcan.gc.ca/n1/pub/92-150-g/92-150-g2016002-eng.htm
https://www12.statcan.gc.ca/census-recensement/2016/geo/ref/geosuite/files-fichiers/GeoSuite_2016_92-150_XBB_eng.zip
# 2011. Here is the reference guide https://www150.statcan.gc.ca/n1/pub/92-150-g/92-150-g2011001-eng.htm
https://www12.statcan.gc.ca/census-recensement/2011/geo/ref/files-fichiers/2011_92-150_XBB_eng.zip
# 2006. Here is the reference guide
https://www12.statcan.gc.ca/census-recensement/2011/geo/ref/files-fichiers/2006_92-150_XBB_eng.zip
# 2001. Here is the reference guide
https://www12.statcan.gc.ca/census-recensement/2011/geo/ref/files-fichiers/92F0150WCB2001000.zip
geosuite/load.sh
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/bin/bash
INPUT_FOLDER="${DATA_FOLDER}/geosuite/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/geosuite/extracted"
import_2021() {
echo "Unzipping 2021 geosuite data"
unzip -n "${INPUT_FOLDER}/2021_92-150-X_eng.zip" -d ${EXTRACTED_FOLDER}
python geosuite/process.py ${EXTRACTED_FOLDER}/2021_92-150-X_eng/PN.csv
}
import_2021
geosuite/problems.ipynb
+283
View File
@@ -0,0 +1,283 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 5,
"id": "dacb31a5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Reading /home/jovyan/work/process-statcan-data/data/geosuite/extracted/2021_92-150-X_eng/PN.csv\n"
]
}
],
"source": [
"#!/usr/bin/env python\n",
"# coding: utf-8\n",
"import os\n",
"import sys\n",
"\n",
"import geopandas as gpd\n",
"import pandas as pd\n",
"from sqlalchemy import create_engine\n",
"from sqlalchemy import text\n",
"\n",
"placenames_2021_csv = \"/home/jovyan/work/process-statcan-data/data/geosuite/extracted/2021_92-150-X_eng/PN.csv\"\n",
"\n",
"DATABASE = os.environ.get(\"POSTGRES_DB\")\n",
"HOST = os.environ.get(\"WAREHOUSE_PG_HOST\")\n",
"USER = os.environ.get(\"POSTGRES_USER\")\n",
"PASSWORD = os.environ.get(\"POSTGRES_PASSWORD\")\n",
"\n",
"#engine = create_engine(f\"postgresql://{USER}:{PASSWORD}@{HOST}:5432/{DATABASE}\")\n",
"\n",
"print(f\"Reading {placenames_2021_csv}\")\n",
"placenames = pd.read_csv(filepath_or_buffer=placenames_2021_csv,\n",
" encoding='latin-1',\n",
" usecols=['PNdguid', 'PNname', 'PNsource', 'PNrplat', 'PNrplong'])\n",
"\n",
"placenames.rename(columns={\n",
" 'PNdguid': 'pn_dguid',\n",
" 'PNname': 'pn_name',\n",
" 'PNsource': 'pn_source',\n",
" 'PNrplat': 'latitude',\n",
" 'PNrplong': 'longitude'\n",
"}, inplace=True)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "d2d4d385",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"rec.array([(4269, '2021S0515005422', 'Cascapédia\\x96Saint-Jules', 1, 48.25, -65.9166667)],\n",
" dtype=[('index', '<i8'), ('pn_dguid', 'O'), ('pn_name', 'O'), ('pn_source', '<i8'), ('latitude', '<f8'), ('longitude', '<f8')])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"placenames[placenames['pn_dguid'] == '2021S0515005422'].to_records()"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "5110c35c",
"metadata": {},
"outputs": [],
"source": [
"special_unicodes = []\n",
"for record in placenames.to_records():\n",
" pn_dguid = record[1]\n",
" pn_name = record[2]\n",
" if r'\\x' in repr(pn_name):\n",
" special_unicodes.append((pn_dguid, pn_name))"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "1d880cb1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"19"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#print(special_unicodes)\n",
"len(special_unicodes)\n",
"#dguids_affected = [x[0] for x in special_unicodes]"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "7b320444",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['2021S0515005422',\n",
" '2021S0515007864',\n",
" '2021S0515017557',\n",
" '2021S0515019487',\n",
" '2021S0515019731',\n",
" '2021S0515022795',\n",
" '2021S0515024311',\n",
" '2021S0515028429',\n",
" '2021S0515030028',\n",
" '2021S0515030168',\n",
" '2021S0515030432',\n",
" '2021S0515031197',\n",
" '2021S0515031295',\n",
" '2021S0515031660',\n",
" '2021S0515032370',\n",
" '2021S0515038300',\n",
" '2021S0515038389',\n",
" '2021S0515040448',\n",
" '2021S0515040522']"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dguids_affected"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "b36377ad",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r'\\x' in r'Cascapédia\\x96Saint-Jules'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a68fe5c3",
"metadata": {},
"outputs": [],
"source": [
"print(\"Creating geodataframe from placenames file\")\n",
"gdf = gpd.GeoDataFrame(\n",
" placenames, \n",
" geometry=gpd.points_from_xy(placenames.longitude,\n",
" placenames.latitude),\n",
" crs=\"EPSG:4326\"\n",
")\n",
"\n",
"print(\"Dropping 'latitude', 'longitude' from geodataframe\")\n",
"gdf.drop(columns=[\"latitude\", \"longitude\"], \n",
" inplace=True)\n",
"\n",
"print(f\"Loading geodatframe to PostgreSQL as statcan_pn_2021_tmp\")\n",
"gdf.to_postgis(name=f\"statcan_pn_2021_tmp\", \n",
" con=engine,\n",
" chunksize=150000,\n",
" if_exists='replace')\n",
"\n",
"print(\"Creating statcan_pn_2021\")\n",
"sql = \"\"\"\n",
"DROP TABLE IF EXISTS statcan_pn_2021;\n",
"\n",
"CREATE TABLE statcan_pn_2021 AS\n",
"SELECT \n",
"db.country_dguid,\n",
"db.country_en_name, \n",
"db.country_fr_name,\n",
"db.country_en_abbreviation,\n",
"db.country_fr_abbreviation,\n",
"db.grc_dguid,\n",
"db.grc_en_name,\n",
"db.grc_fr_name,\n",
"db.pr_dguid,\n",
"db.pr_en_name,\n",
"db.pr_fr_name,\n",
"db.pr_en_abbreviation,\n",
"db.pr_fr_abbreviation,\n",
"db.pr_iso_code,\n",
"db.car_dguid,\n",
"db.car_en_name,\n",
"db.car_fr_name,\n",
"db.er_dguid,\n",
"db.er_name,\n",
"db.cd_dguid,\n",
"db.cd_name,\n",
"db.cd_type,\n",
"db.ccs_dguid,\n",
"db.ccs_name,\n",
"db.cma_dguid,\n",
"db.cma_p_dguid,\n",
"db.cma_name,\n",
"db.cma_type,\n",
"db.csd_dguid,\n",
"db.csd_name,\n",
"db.csd_type,\n",
"db.sac_type,\n",
"db.sac_code,\n",
"db.fed_dguid,\n",
"db.fed_name,\n",
"db.fed_en_name,\n",
"db.fed_fr_name,\n",
"db.ct_dguid,\n",
"db.ada_dguid,\n",
"db.da_dguid,\n",
"db.db_dguid,\n",
"placenames.pn_dguid,\n",
"placenames.pn_name,\n",
"placenames.pn_source,\n",
"placenames.geometry as geom\n",
"FROM statcan_pn_2021_tmp as placenames,\n",
" statcan_db_2021 as db\n",
"WHERE ST_Intersects(placenames.geometry, db.geom);\n",
"\n",
"CREATE INDEX statcan_pn_2021_geom_idx ON\n",
"statcan_pn_2021 \n",
"\tUSING GIST(geom) WITH (FILLFACTOR = 100);\n",
"\"\"\"\n",
"\n",
"with engine.connect() as conn:\n",
" conn.execute(text(sql))\n",
" conn.commit()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
geosuite/process.py
+113
View File
@@ -0,0 +1,113 @@
#!/usr/bin/env python
# coding: utf-8
import os
import sys
import geopandas as gpd
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy import text
placenames_2021_csv = sys.argv[1]
DATABASE = os.environ.get("POSTGRES_DB")
USER = os.environ.get("POSTGRES_USER")
PASSWORD = os.environ.get("POSTGRES_PASSWORD")
engine = create_engine(f"postgresql://{USER}:{PASSWORD}@db:5432/{DATABASE}")
print(f"Reading {placenames_2021_csv}")
placenames = pd.read_csv(filepath_or_buffer=placenames_2021_csv,
encoding='ISO-8859-1',
usecols=['PNdguid', 'PNname', 'PNsource', 'PNrplat', 'PNrplong'])
placenames.rename(columns={
'PNdguid': 'pn_dguid',
'PNname': 'pn_name',
'PNsource': 'pn_source',
'PNrplat': 'latitude',
'PNrplong': 'longitude'
}, inplace=True)
print("Creating geodataframe from placenames file")
gdf = gpd.GeoDataFrame(
placenames,
geometry=gpd.points_from_xy(placenames.longitude,
placenames.latitude),
crs="EPSG:4326"
)
print("Dropping 'latitude', 'longitude' from geodataframe")
gdf.drop(columns=["latitude", "longitude"],
inplace=True)
print(f"Loading geodataframe to PostgreSQL as bronze.pn_2021_tmp")
gdf.to_postgis(name=f"pn_2021_tmp",
con=engine,
chunksize=150000,
if_exists='replace',
schema='bronze')
print("Creating silver.pn_2021")
sql = """
DROP TABLE IF EXISTS silver.pn_2021;
CREATE TABLE silver.pn_2021 AS
SELECT
db.country_dguid,
db.country_en_name,
db.country_fr_name,
db.country_en_abbreviation,
db.country_fr_abbreviation,
db.grc_dguid,
db.grc_en_name,
db.grc_fr_name,
db.pr_dguid,
db.pr_en_name,
db.pr_fr_name,
db.pr_en_abbreviation,
db.pr_fr_abbreviation,
db.pr_iso_code,
db.car_dguid,
db.car_en_name,
db.car_fr_name,
db.er_dguid,
db.er_name,
db.cd_dguid,
db.cd_name,
db.cd_type,
db.ccs_dguid,
db.ccs_name,
db.cma_dguid,
db.cma_p_dguid,
db.cma_name,
db.cma_type,
db.csd_dguid,
db.csd_name,
db.csd_type,
db.sac_type,
db.sac_code,
db.fed_dguid,
db.fed_name,
db.fed_en_name,
db.fed_fr_name,
db.ct_dguid,
db.ada_dguid,
db.da_dguid,
db.db_dguid,
placenames.pn_dguid,
placenames.pn_name,
placenames.pn_source,
placenames.geometry as geom
FROM bronze.pn_2021_tmp as placenames,
silver.db_2021 as db
WHERE ST_Intersects(placenames.geometry, db.geom);
CREATE INDEX pn_2021_geom_idx ON
silver.pn_2021
USING GIST(geom) WITH (FILLFACTOR = 100);
"""
with engine.connect() as conn:
conn.execute(text(sql))
conn.commit()
health_regions/download.sh
Executable
+23
View File
@@ -0,0 +1,23 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/health_regions" ]
then
echo "Making directory ${DATA_FOLDER}/health_regions/"
mkdir -p ${DATA_FOLDER/health_regions/{input,extracted,output}
fi
aria2c -x16 --out=hr2024001.zip "https://www150.statcan.gc.ca/n1/pub/82-402-x/2024001/hrbf-flrs/digi/ArcGIS/HR_000a23a_e.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2023001.zip "https://www150.statcan.gc.ca/n1/en/pub/82-402-x/2023001/hrbf-flrs/digi/ArcGIS/HR_000a22a_e.zip?st=P3vL0l_a" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2018001.zip "https://www150.statcan.gc.ca/n1/en/pub/82-402-x/2018001/data-donnees/boundary-limites/arcinfo/HR_000a18a-eng.zip?st=NgmRAZb7" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2017001.zip "https://www150.statcan.gc.ca/n1/en/pub/82-402-x/2017001/data-donnees/boundary-limites/arcinfo/HR_000a17a_e.zip?st=i4-a3s-U" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2015002.zip "https://www150.statcan.gc.ca/n1/en/pub/82-402-x/2015002/data-donnees/boundary-limites/arcinfo/HRP000b11a_e.zip?st=pdmD3cVj" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2015001.zip "https://www150.statcan.gc.ca/n1/pub/82-402-x/2015001/data-donnees/boundary-limites/arcinfo/HRP000b11a_e.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2013003.zip "https://www150.statcan.gc.ca/n1/en/pub/82-402-x/2013003/data-donnees/boundary-limites/arcinfo/HRP000b11a_e.zip?st=yEQLGVii" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2013002.zip "https://www150.statcan.gc.ca/n1/pub/82-402-x/2013002/data-donnees/boundary-limites/arcinfo/HRP000b11a_e.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2013001.zip "https://www150.statcan.gc.ca/pub/82-402-x/2011001/data-donnees/boundary-limites/arcinfo/hr000b06p-eng.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2011001.zip "https://www150.statcan.gc.ca/pub/82-402-x/2011001/data-donnees/boundary-limites/arcinfo/hr000b06p-eng.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2009001.zip "https://www150.statcan.gc.ca/pub/82-402-x/2009001/data-donnees/boundary-limites/arcinfo/hr000b08pz-eng.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2007001.zip "https://www150.statcan.gc.ca/n1/en/pub/82-402-x/2007001/data/boundary/arcinfo/HR000b07PZ.zip?st=hCrYIpzu" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2006001.zip "https://www150.statcan.gc.ca/n1/en/pub/82-402-x/2006001/data/boundary/arcview/lhr000b05pz.zip?st=TEqojTk8" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2005001.zip "https://www150.statcan.gc.ca/n1/pub/82-402-x/2005001/data/boundary/arcview/lhr000b05pz.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2004001.zip "https://www150.statcan.gc.ca/n1/pub/82-402-x/2004001/data/boundary/arcview/hr000b.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
aria2c -x16 --out=hr2003001.zip "https://www150.statcan.gc.ca/n1/pub/82-402-x/2004001/data/boundary/arcview/hr000b.zip" --dir=${DATA_FOLDER}/health_regions/input --auto-file-renaming=false
health_regions/load.sh
Executable
+48
View File
@@ -0,0 +1,48 @@
#!/bin/bash
import_to_postgis() {
local filepath=$1
local table_name=$2
local extra_parameters=${@:3}
# Virtual file system
if [[ ${filepath: -4} = '.zip' ]]; then
local filepath="/vsizip/${filepath}"
fi
echo "Importing ${filepath}"
ogr2ogr \
--config PG_USE_COPY YES \
-overwrite \
-f "PostgreSQL" \
"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432" \
-lco GEOMETRY_NAME=geom \
-progress \
-gt 500000 \
-t_srs EPSG:4326 \
-nln ${table_name} \
${extra_parameters} \
${filepath}
}
INPUT_FOLDER="${DATA_FOLDER}/health_regions/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/health_regions/extracted"
import_to_postgis ${INPUT_FOLDER}/hr2024001.zip statcan_hr2024001_tmp "-nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2023001.zip statcan_hr2023001_tmp "-nlt PROMOTE_TO_MULTI"
unzip -n ${INPUT_FOLDER}/hr2018001.zip -d ${EXTRACTED_FOLDER}
import_to_postgis ${EXTRACTED_FOLDER}/HR_000a18a_e/HR_000a18a_e.shp statcan_hr2018001_tmp "-select HR_UID,ENGNAME,FRENAME -nlt PROMOTE_TO_MULTI"
unzip -n ${INPUT_FOLDER}/hr2017001.zip -d ${EXTRACTED_FOLDER}
import_to_postgis ${EXTRACTED_FOLDER}/HR_000a17a_e/HR_000a17a_e_Dec2017.shp statcan_hr2017001_tmp "-select HR_UID,ENG_LABEL,FRE_LABEL -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2015002.zip statcan_hr2015002_tmp "-nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2015001.zip statcan_hr2015001_tmp "-select HR_UID,ENG_LABEL,FRE_LABEL -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2013003.zip statcan_hr2013003_tmp "-select HR_UID,ENG_LABEL,FRE_LABEL -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2013002.zip statcan_hr2013002_tmp "-select HR_UID,ENG_LABEL,FRE_LABEL -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2013001.zip statcan_hr2013001_tmp "-select PR_HRUID,ENG_LABEL,FRE_LABEL -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2011001.zip statcan_hr2011001_tmp "-select PR_HRUID,ENG_LABEL,FRE_LABEL -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2009001.zip statcan_hr2009001_tmp "-select HRUID2007,ENG_LABEL,FRE_LABEL -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2007001.zip statcan_hr2007001_tmp "-select PR_HRUID,HRNAME,FRNAME -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2006001.zip statcan_hr2006001_tmp "-select PR_HRUID,HRNAME,FRNAME -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2005001.zip statcan_hr2005001_tmp "-select PR_HRUID,HRNAME,FRNAME -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2004001.zip statcan_hr2004001_tmp "-select PR_HRUID,HRNAME -nlt PROMOTE_TO_MULTI"
import_to_postgis ${INPUT_FOLDER}/hr2003001.zip statcan_hr2003001_tmp "-select PR_HRUID,HRNAME -nlt PROMOTE_TO_MULTI"
health_regions/organize.sql
+126
View File
@@ -0,0 +1,126 @@
-- TODO
-- 2022 Health Regions;
drop table if exists statcan_hr_2022;
create table statcan_hr_2022 as
select hr_uid, engname, frename, dguid, wkb_geometry as geom
from statcan_hr2023001_tmp;
create index statcan_hr_2022_geom_idx on statcan_hr_2022 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2023001_tmp;
-- 2018 Health Regions;
drop table if exists statcan_hr_2018;
create table statcan_hr_2018 as
select hr_uid, engname, frename, concat('2018A007', hr_uid) as dguid, wkb_geometry as geom
from statcan_hr2018001_tmp;
create index statcan_hr_2018_geom_idx on statcan_hr_2018 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2018001_tmp;
-- 2017 Health Regions;
drop table if exists statcan_hr_2017;
create table statcan_hr_2017 as
select hr_uid, eng_label as engname, fre_label as frename, concat('2017A007', hr_uid) as dguid, wkb_geometry as geom
from statcan_hr2017001_tmp;
create index statcan_hr_2017_geom_idx on statcan_hr_2017 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2017001_tmp;
-- 2015 Health Regions;
drop table if exists statcan_hr_2015;
create table statcan_hr_2015 as
select hr_uid, eng_label as engname, fre_label as frename, concat('2015A007', hr_uid) as dguid, wkb_geometry as geom
from statcan_hr2015002_tmp;
create index statcan_hr_2015_geom_idx on statcan_hr_2015 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2015002_tmp;
-- 2013 Health Regions;
drop table if exists statcan_hr_2013;
create table statcan_hr_2013 as
select hr_uid, eng_label as engname, fre_label as frename, concat('2013A007', hr_uid) as dguid, wkb_geometry as geom
from statcan_hr2013003_tmp;
create index statcan_hr_2013_geom_idx on statcan_hr_2013 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2013003_tmp;
drop table if exists statcan_hr2013002_tmp;
drop table if exists statcan_hr2013001_tmp;
-- 2011 Health Regions;
drop table if exists statcan_hr_2011;
create table statcan_hr_2011 as
select pr_hruid as hr_uid, eng_label as engname, fre_label as frename, concat('2011A007', pr_hruid) as dguid, wkb_geometry as geom
from statcan_hr2011001_tmp;
create index statcan_hr_2011_geom_idx on statcan_hr_2011 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2011001_tmp;
-- 2009 Health Regions;
drop table if exists statcan_hr_2009;
create table statcan_hr_2009 as
select hruid2007 as hr_uid, eng_label as engname, fre_label as frename, concat('2009A007', hruid2007) as dguid, wkb_geometry as geom
from statcan_hr2009001_tmp;
create index statcan_hr_2009_geom_idx on statcan_hr_2009 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2009001_tmp;
-- 2007 Health Regions;
drop table if exists statcan_hr_2007;
create table statcan_hr_2007 as
select distinct pr_hruid as hr_uid, hrname as engname, frname as frename, concat('2007A007', pr_hruid) as dguid, wkb_geometry as geom
from statcan_hr2007001_tmp;
update statcan_hr_2007
set frename = engname
where frename is null;
create index statcan_hr_2007_geom_idx on statcan_hr_2007 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2007001_tmp;
-- 2006 Health Regions;
drop table if exists statcan_hr_2006;
create table statcan_hr_2006 as
select distinct pr_hruid as hr_uid, hrname as engname, frname as frename, concat('2006A007', pr_hruid) as dguid, st_union(wkb_geometry) as geom
from statcan_hr2006001_tmp
group by pr_hruid, hrname, frname;
update statcan_hr_2006
set frename = engname
where frename is null;
create index statcan_hr_2006_geom_idx on statcan_hr_2006 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2006001_tmp;
-- 2005 Health Regions;
drop table if exists statcan_hr_2005;
create table statcan_hr_2005 as
select distinct pr_hruid as hr_uid, hrname as engname, frname as frename, concat('2005A007', pr_hruid) as dguid, st_union(wkb_geometry) as geom
from statcan_hr2005001_tmp
group by pr_hruid, hrname, frname;
update statcan_hr_2005
set frename = engname
where frename is null;
create index statcan_hr_2005_geom_idx on statcan_hr_2005 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2005001_tmp;
-- 2004 Health Regions;
drop table if exists statcan_hr_2004;
create table statcan_hr_2004 as
select distinct pr_hruid as hr_uid, hrname as engname, concat('2004A007', pr_hruid) as dguid, st_union(wkb_geometry) as geom
from statcan_hr2004001_tmp
group by pr_hruid, hrname;
create index statcan_hr_2004_geom_idx on statcan_hr_2004 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2004001_tmp;
-- 2003 Health Regions;
drop table if exists statcan_hr_2003;
create table statcan_hr_2003 as
select distinct pr_hruid as hr_uid, hrname as engname, concat('2003A007', pr_hruid) as dguid, st_union(wkb_geometry) as geom
from statcan_hr2003001_tmp
group by pr_hruid, hrname;
create index statcan_hr_2003_geom_idx on statcan_hr_2003 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_hr2003001_tmp;
hydro/README.md
+2
View File
@@ -0,0 +1,2 @@
# TODO
- Process 2021 hydro
hydro/download.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/hydro" ]
then
echo "Making directory ${DATA_FOLDER}/hydro/"
mkdir -p ${DATA_FOLDER}/hydro/{input,output}/{2021,2016,2011,2006}
fi
INPUT_FOLDER="${DATA_FOLDER}/hydro/input"
#echo "Downloading 2021 hydro"
#aria2c -x16 -i "${SCRIPT_DIR}/hydro/hydro_2021.txt" --dir=${INPUT_FOLDER}/2021 --auto-file-renaming=false
echo "Downloading 2016 hydro"
aria2c -x16 -i "${SCRIPT_DIR}/hydro/hydro_2016.txt" --dir=${INPUT_FOLDER}/2016 --auto-file-renaming=false
echo "Downloading 2011 boundaries"
aria2c -x16 -i "${SCRIPT_DIR}/hydro/hydro_2011.txt" --dir=${INPUT_FOLDER}/2011 --auto-file-renaming=false
echo "Downloading 2006 boundaries"
aria2c -x16 -i "${SCRIPT_DIR}/hydro/hydro_2006.txt" --dir=${INPUT_FOLDER}/2006 --auto-file-renaming=false
hydro/hydro_2006.txt
+4
View File
@@ -0,0 +1,4 @@
# https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2006-eng.cfm
https://data.dataforcanada.org/archive/statistics_canada/hydro/2006/ghy_000c06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ghy_000c06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/hydro/2006/ghy_000d06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ghy_000d06a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/hydro/2006/ghy_000f06a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ghy_000f06a_e.zip
hydro/hydro_2011.txt
+4
View File
@@ -0,0 +1,4 @@
# https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2011-eng.cfm
https://data.dataforcanada.org/archive/statistics_canada/hydro/2011/ghy_000c11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ghy_000c11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/hydro/2011/ghy_000d11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ghy_000d11a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/hydro/2011/ghy_000h11a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/ghy_000h11a_e.zip
hydro/hydro_2016.txt
+4
View File
@@ -0,0 +1,4 @@
# https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2016-eng.cfm
https://data.dataforcanada.org/archive/statistics_canada/hydro/2016/lhy_000c16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lhy_000c16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/hydro/2016/lhy_000d16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lhy_000d16a_e.zip
https://data.dataforcanada.org/archive/statistics_canada/hydro/2016/lhy_000h16a_e.zip https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/files-fichiers/2016/lhy_000h16a_e.zip
hydro/hydro_2021.txt
+15
View File
@@ -0,0 +1,15 @@
# https://open.canada.ca/data/en/dataset/9d96e8c9-22fe-4ad2-b5e8-94a6991b744b
# They are the "Prepackaged FGDB files (download directory)", 50K Hydro
https://zenodo.org/records/15350060/files/canvec_50K_AB_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_AB_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_AB_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_BC_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_BC_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_BC_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_MB_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_MB_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_MB_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_NB_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_NB_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_NB_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_NL_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_NL_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_NL_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_NS_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_NS_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_NS_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_NT_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_NT_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_NT_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_NU_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_NU_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_NU_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_ON_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_ON_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_ON_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_PE_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_PE_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_PE_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_QC_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_QC_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_QC_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_SK_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_SK_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_SK_Hydro_fgdb.zip
https://zenodo.org/records/15350060/files/canvec_50K_YT_Hydro_fgdb.zip?download=1 https://data.dataforcanada.org/archive/natural_resources_canada/hydro/2019/canvec_50K_YT_Hydro_fgdb.zip https://archive.org/download/canvec-hydro-50k-2019-07-24/canvec_50K_YT_Hydro_fgdb.zip
hydro/load.sh
Executable
+70
View File
@@ -0,0 +1,70 @@
#!/bin/bash
import_to_postgis() {
local filepath="$1"
local table_name="$2"
local extra_parameters="${@:3}"
# Handle zip files using GDAL's virtual file system
if [[ "${filepath: -4}" == ".zip" ]]; then
filepath="/vsizip/${filepath}"
fi
echo "Importing ${filepath} into table ${table_name}"
ogr2ogr \
--config PG_USE_COPY YES \
-lco "OVERWRITE=YES" \
-f "PostgreSQL" \
"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432" \
-lco GEOMETRY_NAME=geom \
-progress \
-gt 500000 \
-t_srs EPSG:4326 \
-nln "${table_name}" \
${extra_parameters} \
"${filepath}"
}
# Define input folders
INPUT_FOLDER="${DATA_FOLDER}/hydro/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/hydro/extracted"
# Import 2016 hydro data
import_data_2016() {
# Source: https://web.archive.org/web/20230120140926/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2016-eng.cfm
# Lakes and rivers (polygons)
import_to_postgis "${INPUT_FOLDER}/2016/lhy_000c16a_e.zip" lhy_000c16a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Rivers (lines)
import_to_postgis "${INPUT_FOLDER}/2016/lhy_000d16a_e.zip" lhy_000d16a_e "-lco SCHEMA=bronze"
# Coastal waters (polygons)
import_to_postgis "${INPUT_FOLDER}/2016/lhy_000h16a_e.zip" lhy_000h16a_e "-lco SCHEMA=bronze"
}
# Import 2011 hydro data
import_data_2011() {
# Source: https://web.archive.org/web/20230110163150/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2011-eng.cfm
export PGCLIENTENCODING=LATIN-1;
# Lakes and rivers (polygons)
import_to_postgis "${INPUT_FOLDER}/2011/ghy_000c11a_e.zip" ghy_000c11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Rivers (lines)
import_to_postgis "${INPUT_FOLDER}/2011/ghy_000d11a_e.zip" ghy_000d11a_e "-lco SCHEMA=bronze"
# Coastal waters (polygons)
import_to_postgis "${INPUT_FOLDER}/2011/ghy_000h11a_e.zip" ghy_000h11a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
}
# Import 2006 hydro data
import_data_2006() {
# Source: https://web.archive.org/web/20221218043125/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2006-eng.cfm
export PGCLIENTENCODING=LATIN-1;
# Lakes and rivers (polygons)
import_to_postgis "${INPUT_FOLDER}/2006/ghy_000c06a_e.zip" ghy_000c06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
# Rivers (lines)
import_to_postgis "${INPUT_FOLDER}/2006/ghy_000d06a_e.zip" ghy_000d06a_e "-lco SCHEMA=bronze"
# Coastal waters (polygons)
import_to_postgis "${INPUT_FOLDER}/2006/ghy_000f06a_e.zip" ghy_000f06a_e "-nlt PROMOTE_TO_MULTI -lco SCHEMA=bronze"
}
# Execute all import functions
import_data_2016
import_data_2011
import_data_2006
hydro/process.sh
Executable
+18
View File
@@ -0,0 +1,18 @@
# Standardizes field names, builds up hierarchy for datasets, create Canada and Geographic Regions of Canada
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/country.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/geographic_regions_of_canada.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/provinces_and_territories.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_divisions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/economic_regions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_agricultural_regions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/forward_sortation_areas.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/federal_electoral_districts.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_consolidated_subdivisions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_metropolitan_areas.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_subdivisions.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/population_centres.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/designated_places.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/census_tracts.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/aggregate_dissemination_areas.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/dissemination_areas.sql
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -f boundaries/dissemination_blocks.sql
main.sh
Executable
+50
View File
@@ -0,0 +1,50 @@
#!/bin/bash
export SCRIPT_DIR=$(pwd)
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -c 'CREATE SCHEMA IF NOT EXISTS bronze;'
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -c 'CREATE SCHEMA IF NOT EXISTS silver;'
psql "postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@db/$POSTGRES_DB" -c 'CREATE SCHEMA IF NOT EXISTS gold;'
#### 1.0 Dissemination Geographies Relationship File ####
## 1.1 Download Dissemination Geographies Relationship File ##
dissemination_geographies_relationship_file/download.sh
## 1.2 Load Dissemination Geographies Relationship File to PostgreSQL ##
dissemination_geographies_relationship_file/load.sh
#### 2.0 Geographic Attribute File ####
## 2.1 Download Geographic Attribute File ##
geographic_attribute_file/download.sh
## 2.2 Load Geographic Attribute File to PostgreSQL ##
geographic_attribute_file/load.sh
#### 3.0 Hydro ####
## 3.1 Download Hydro ##
hydro/download.sh
## 3.2 Load Hydro ##
hydro/load.sh
#### 3.0 Boundaries ####
## 3.1 Download Boundaries ##
boundaries/download.sh
## 3.2 Load Boundaries ##
boundaries/load.sh
## 3.3 Process Boundaries ##
boundaries/process.sh
#### 4.0 Road Network Files ####
## 4.1 Download Road Network Files ##
road_network_files/download.sh
## 4.2 Load Road Network Files to PostgreSQL ##
road_network_files/load.sh
road_network_files/process.sh
#### 5.0 GeoSuite ####
## 5.1 Download GeoSuite Files ##
geosuite/download.sh
## 5.2 Load Placename layer to PostgreSQL ##
geosuite/load.sh
#### 6.0 Census of Population ####
## 6.1 Download Census of Population ##
census_of_population/download.sh
## 6.2 Process Census of Population ##
census_of_population/process.sh
#### 7.0 Census of Agriculture ####
## 7.1 Download Census of Population ##
census_of_agriculture/download.sh
## 7.2 Process Census of Agriculture ##
census_of_agriculture/process.sh
national_address_register/download.sh
Executable
+9
View File
@@ -0,0 +1,9 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/national_address_register" ]
then
echo "Making directory ${DATA_FOLDER}/national_address_register/"
mkdir -p ${DATA_FOLDER}/national_address_register/{input,extracted,output,scratch}
fi
echo "Downloading national address register files"
aria2c -x16 -i "${SCRIPT_DIR}/national_address_register/national_address_register_files.txt" --dir=$DATA_FOLDER/national_address_register/input --auto-file-renaming=false
national_address_register/load.sh
Executable
+167
View File
@@ -0,0 +1,167 @@
#!/bin/bash
import_to_postgis() {
local filepath=$1
local table_name=$2
local extra_parameters=${@:3}
# Virtual file system
if [[ ${filepath: -4} = '.zip' ]]; then
local filepath="/vsizip/${filepath}"
fi
echo "Importing ${filepath}"
ogr2ogr \
--config PG_USE_COPY YES \
-overwrite \
-f "PostgreSQL" \
"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432" \
-lco GEOMETRY_NAME=geom \
-progress \
-gt 500000 \
-t_srs EPSG:3857 \
-nln ${table_name} \
${extra_parameters} \
${filepath}
}
concatenate_csvs() {
# Concatenates all of the CSVs in the directory
local input_directory=$1
local output_file=$2
for address_file in $(ls ${input_directory}/*.csv);
do
echo "Processing ${address_file}. Adding to ${output_file}"
tail -n +2 $address_file >> ${output_file}
done
}
INPUT_FOLDER="${DATA_FOLDER}/national_address_register/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/national_address_register/extracted"
SCRATCH_FOLDER="${DATA_FOLDER}/national_address_register/scratch"
import_202412() {
# Process 202412
# Extract files
echo "Extracting ${INPUT_FOLDER}/202412.zip"
unzip -q -n ${INPUT_FOLDER}/202412.zip -d ${EXTRACTED_FOLDER}/202412
if [ ! -f ${SCRATCH_FOLDER}/statcan_nar_addresses_202412.csv ]
then
echo "Adding header file to ${SCRATCH_FOLDER}/statcan_nar_addresses_202412.csv"
echo "LOC_GUID,ADDR_GUID,APT_NO_LABEL,CIVIC_NO,CIVIC_NO_SUFFIX,OFFICIAL_STREET_NAME,OFFICIAL_STREET_TYPE,OFFICIAL_STREET_DIR,PROV_CODE,CSD_ENG_NAME,CSD_FRE_NAME,CSD_TYPE_ENG_CODE,CSD_TYPE_FRE_CODE,MAIL_STREET_NAME,MAIL_STREET_TYPE,MAIL_STREET_DIR,MAIL_MUN_NAME,MAIL_PROV_ABVN,MAIL_POSTAL_CODE,BG_DLS_LSD,BG_DLS_QTR,BG_DLS_SCTN,BG_DLS_TWNSHP,BG_DLS_RNG,BG_DLS_MRD,BG_X,BG_Y,BU_N_CIVIC_ADD,BU_USE" > ${SCRATCH_FOLDER}/statcan_nar_addresses_202412.csv
fi
if [ $(head ${SCRATCH_FOLDER}/statcan_nar_addresses_202412.csv | wc -l) -ne 10 ]
then
echo "Appending Addresses CSVs to ${SCRATCH_FOLDER}/statcan_nar_addresses_202412.csv"
concatenate_csvs "${EXTRACTED_FOLDER}/202412/Addresses" "${SCRATCH_FOLDER}/statcan_nar_addresses_202412.csv"
fi
if [ ! -f ${SCRATCH_FOLDER}/statcan_nar_locations_202412.csv ]
then
echo "Adding header file to ${SCRATCH_FOLDER}/statcan_nar_locations_202412.csv"
echo "LOC_GUID,CSD_CODE,FED_CODE,FED_ENG_NAME,FED_FRE_NAME,ER_CODE,ER_ENG_NAME,ER_FRE_NAME,REPPOINT_LATITUDE,REPPOINT_LONGITUDE" > ${SCRATCH_FOLDER}/statcan_nar_locations_202412.csv
fi
if [ $(head ${SCRATCH_FOLDER}/statcan_nar_locations_202412.csv | wc -l) -ne 10 ]
then
echo "Appending Addresses CSVs to ${SCRATCH_FOLDER}/statcan_nar_locations_202412.csv"
concatenate_csvs "${EXTRACTED_FOLDER}/202412/Locations" "${SCRATCH_FOLDER}/statcan_nar_locations_202412.csv"
fi
python national_address_register/process.py ${SCRATCH_FOLDER}/statcan_nar_addresses_202412.csv ${SCRATCH_FOLDER}/statcan_nar_locations_202412.csv 202412 utf-8
}
import_202406() {
# Process 202406
echo "Extracting ${INPUT_FOLDER}/2024.zip"
unzip -q -n ${INPUT_FOLDER}/2024.zip -d ${EXTRACTED_FOLDER}/202406
if [ ! -f ${SCRATCH_FOLDER}/statcan_nar_addresses_202406.csv ]
then
echo "Adding header file to ${SCRATCH_FOLDER}/statcan_nar_addresses_202406.csv"
echo "LOC_GUID,ADDR_GUID,APT_NO_LABEL,CIVIC_NO,CIVIC_NO_SUFFIX,OFFICIAL_STREET_NAME,OFFICIAL_STREET_TYPE,OFFICIAL_STREET_DIR,PROV_CODE,CSD_ENG_NAME,CSD_FRE_NAME,CSD_TYPE_ENG_CODE,CSD_TYPE_FRE_CODE,MAIL_STREET_NAME,MAIL_STREET_TYPE,MAIL_STREET_DIR,MAIL_MUN_NAME,MAIL_PROV_ABVN,MAIL_POSTAL_CODE,BG_DLS_LSD,BG_DLS_QTR,BG_DLS_SCTN,BG_DLS_TWNSHP,BG_DLS_RNG,BG_DLS_MRD,BG_X,BG_Y,BU_N_CIVIC_ADD,BU_USE" > ${SCRATCH_FOLDER}/statcan_nar_addresses_202406.csv
fi
if [ $(head ${SCRATCH_FOLDER}/statcan_nar_addresses_202406.csv | wc -l) -ne 10 ]
then
echo "Appending Addresses CSVs to ${SCRATCH_FOLDER}/statcan_nar_addresses_202406.csv"
concatenate_csvs "${EXTRACTED_FOLDER}/202406/Addresses" "${SCRATCH_FOLDER}/statcan_nar_addresses_202406.csv"
fi
if [ ! -f ${SCRATCH_FOLDER}/statcan_nar_locations_202406.csv ]
then
echo "Adding header file to ${SCRATCH_FOLDER}/statcan_nar_locations_202406.csv"
echo "LOC_GUID,CSD_CODE,FED_CODE,FED_ENG_NAME,FED_FRE_NAME,ER_CODE,ER_ENG_NAME,ER_FRE_NAME,REPPOINT_LATITUDE,REPPOINT_LONGITUDE" > ${SCRATCH_FOLDER}/statcan_nar_locations_202406.csv
fi
if [ $(head ${SCRATCH_FOLDER}/statcan_nar_locations_202406.csv | wc -l) -ne 10 ]
then
echo "Appending Addresses CSVs to ${SCRATCH_FOLDER}/statcan_nar_locations_202406.csv"
concatenate_csvs "${EXTRACTED_FOLDER}/202406/Locations" "${SCRATCH_FOLDER}/statcan_nar_locations_202406.csv"
fi
python national_address_register/process.py ${SCRATCH_FOLDER}/statcan_nar_addresses_202406.csv ${SCRATCH_FOLDER}/statcan_nar_locations_202406.csv 202406 utf-8
}
import_2023() {
# Process 2023
echo "Extracting ${INPUT_FOLDER}/2023.zip"
unzip -q -n ${INPUT_FOLDER}/2023.zip -d ${EXTRACTED_FOLDER}/2023
if [ ! -f ${SCRATCH_FOLDER}/statcan_nar_addresses_2023.csv ]
then
echo "Adding header file to ${SCRATCH_FOLDER}/statcan_nar_addresses_2023.csv"
echo "LOC_GUID,ADDR_GUID,APT_NO_LABEL,CIVIC_NO,CIVIC_NO_SUFFIX,OFFICIAL_STREET_NAME,OFFICIAL_STREET_TYPE,OFFICIAL_STREET_DIR,PROV_CODE,CSD_ENG_NAME,CSD_FRE_NAME,CSD_TYPE_ENG_CODE,CSD_TYPE_FRE_CODE,MAIL_STREET_NAME,MAIL_STREET_TYPE,MAIL_STEET_DIR,MAIL_MUN_NAME,MAIL_PROV_ABVN,MAIL_POSTAL_CODE,BG_DLS_LSD,BG_DLS_QTR,BG_DLS_SCTN,BG_DLS_TWNSHP,BG_DLS_RNG,BG_DLS_MRD,BG_X,BG_Y,BU_N_CIVIC_ADD,BU_USE" > ${SCRATCH_FOLDER}/statcan_nar_addresses_2023.csv
fi
if [ $(head ${SCRATCH_FOLDER}/statcan_nar_addresses_2023.csv | wc -l) -ne 10 ]
then
echo "Appending Addresses CSVs to ${SCRATCH_FOLDER}/statcan_nar_addresses_2023.csv"
concatenate_csvs "${EXTRACTED_FOLDER}/2023/Addresses" "${SCRATCH_FOLDER}/statcan_nar_addresses_2023.csv"
fi
if [ ! -f ${SCRATCH_FOLDER}/statcan_nar_locations_2023.csv ]
then
echo "Adding header file to ${SCRATCH_FOLDER}/statcan_nar_locations_2023.csv"
echo "LOC_GUID,CSD_CODE,FED_2021_CODE,FED_2021_ENG_NAME,FED_2021_FRE_NAME,ER_2021_CODE,ER_2021_ENG_NAME,ER_2021_FRE_NAME,REPPOINT_LATITUDE,REPPOINT_LONGITUDE" > ${SCRATCH_FOLDER}/statcan_nar_locations_2023.csv
fi
if [ $(head ${SCRATCH_FOLDER}/statcan_nar_locations_2023.csv | wc -l) -ne 10 ]
then
echo "Appending Addresses CSVs to ${SCRATCH_FOLDER}/statcan_nar_locations_2023.csv"
concatenate_csvs "${EXTRACTED_FOLDER}/2022/Locations" "${SCRATCH_FOLDER}/statcan_nar_locations_2023.csv"
fi
python national_address_register/process.py ${SCRATCH_FOLDER}/statcan_nar_addresses_2023.csv ${SCRATCH_FOLDER}/statcan_nar_locations_2023.csv 2023 latin-1
}
import_2022() {
# Process 2022
echo "Extracting ${INPUT_FOLDER}/2022.zip"
unzip -q -n ${INPUT_FOLDER}/2022.zip -d ${EXTRACTED_FOLDER}/2022
if [ ! -f ${SCRATCH_FOLDER}/statcan_nar_addresses_2022.csv ]
then
echo "Adding header file to ${SCRATCH_FOLDER}/statcan_nar_addresses_2022.csv"
echo "LOC_GUID,ADDR_GUID,CIVIC_NO,CIVIC_NO_SUFFIX,APT_NO_LABEL,OFFICIAL_STREET_NAME,OFFICIAL_STREET_TYPE,OFFICIAL_STREET_DIR,PROV_CODE,CSD_ENG_NAME,CSD_FRE_NAME,CSD_TYPE_ENG_CODE,CSD_TYPE_FRE_CODE,MAIL_STREET_NAME,MAIL_STREET_TYPE,MAIL_STREET_DIR,MAIL_MUN_NAME,MAIL_POSTAL_CODE,MAIL_PROV_ABVN,BG_DLS_LSD,BG_DLS_QTR,BG_DLS_SCTN,BG_DLS_TWNSHP,BG_DLS_RNG,BG_DLS_MRD,BG_X,BG_Y,BU_N_CIVIC_ADD,BU_USE" > ${SCRATCH_FOLDER}/statcan_nar_addresses_2022.csv
fi
if [ $(head ${SCRATCH_FOLDER}/statcan_nar_addresses_2022.csv | wc -l) -ne 10 ]
then
echo "Appending Addresses CSVs to ${SCRATCH_FOLDER}/statcan_nar_addresses_2022.csv"
concatenate_csvs "${EXTRACTED_FOLDER}/2022/Addresses" "${SCRATCH_FOLDER}/statcan_nar_addresses_2022.csv"
fi
if [ ! -f ${SCRATCH_FOLDER}/statcan_nar_locations_2022.csv ]
then
echo "Adding header file to ${SCRATCH_FOLDER}/statcan_nar_locations_2022.csv"
echo "LOC_GUID,CSD_CODE,FED_2016_CODE,FED_2016_ENG_NAME,FED_2016_FRE_NAME,ER_2016_CODE,ER_2016_ENG_NAME,ER_2016_FRE_NAME,REPPOINT_LATITUDE,REPPOINT_LONGITUDE" > ${SCRATCH_FOLDER}/statcan_nar_locations_2022.csv
fi
if [ $(head ${SCRATCH_FOLDER}/statcan_nar_locations_2022.csv | wc -l) -ne 10 ]
then
echo "Appending Addresses CSVs to ${SCRATCH_FOLDER}/statcan_nar_locations_2022.csv"
concatenate_csvs "${EXTRACTED_FOLDER}/2022/Locations" "${SCRATCH_FOLDER}/statcan_nar_locations_2022.csv"
fi
python national_address_register/process.py ${SCRATCH_FOLDER}/statcan_nar_addresses_2022.csv ${SCRATCH_FOLDER}/statcan_nar_locations_2022.csv 2022 latin-1
}
import_202412
import_202406
import_2023
import_2022
national_address_register/national_address_register_files.txt
+8
View File
@@ -0,0 +1,8 @@
# December 2024
https://www150.statcan.gc.ca/n1/pub/46-26-0002/2022001/202412.zip
# June 2024
https://www150.statcan.gc.ca/n1/pub/46-26-0002/2022001/2024.zip
# 2023
https://www150.statcan.gc.ca/n1/pub/46-26-0002/2022001/2023.zip
# 2022
https://www150.statcan.gc.ca/n1/pub/46-26-0002/2022001/2022.zip
national_address_register/process.py
Executable
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env python
# coding: utf-8
import os
import sys
import geopandas as gpd
import pandas as pd
from sqlalchemy import create_engine
statcan_nar_addresses_csv = sys.argv[1]
statcan_nar_locations_csv = sys.argv[2]
vintage = sys.argv[3]
encoding = sys.argv[4]
print(f"Reading {statcan_nar_addresses_csv}")
statcan_nar_addresses = pd.read_csv(filepath_or_buffer=statcan_nar_addresses_csv,
dtype={
"CIVIC_NO": "Int32",
"PROV_CODE": object,
"BU_USE": "Int8",
"BG_DLS_LSD": object,
"BG_DLS_QTR": object,
"BG_DLS_SCTN": object,
"BG_DLS_TWNSHP": object,
"BG_DLS_RNG": object,
"BG_DLS_MRD": object
},
encoding=encoding)
print(f"Reading {statcan_nar_locations_csv}")
statcan_nar_locations = pd.read_csv(filepath_or_buffer=statcan_nar_locations_csv,
usecols=["LOC_GUID",
"REPPOINT_LATITUDE",
"REPPOINT_LONGITUDE"],
encoding=encoding)
print(f"Combining {statcan_nar_addresses_csv} and {statcan_nar_locations_csv}")
statcan_nar_addresses_combined = pd.merge(statcan_nar_addresses,
statcan_nar_locations,
on="LOC_GUID", how="inner")
del statcan_nar_addresses
del statcan_nar_locations
DATABASE = os.environ.get("POSTGRES_DB")
HOST = os.environ.get("WAREHOUSE_PG_HOST")
USER = os.environ.get("POSTGRES_USER")
PASSWORD = os.environ.get("POSTGRES_PASSWORD")
engine = create_engine(f"postgresql://{USER}:{PASSWORD}@{HOST}:5432/{DATABASE}")
print("Creating geodataframe from combined address file")
gdf = gpd.GeoDataFrame(
statcan_nar_addresses_combined,
geometry=gpd.points_from_xy(statcan_nar_addresses_combined.REPPOINT_LONGITUDE,
statcan_nar_addresses_combined.REPPOINT_LATITUDE),
crs="EPSG:4326"
)
print("Dropping 'REPPOINT_LATITUDE', 'REPPOINT_LONGITUDE' from geodataframe")
gdf.drop(columns=["REPPOINT_LATITUDE", "REPPOINT_LONGITUDE"],
inplace=True)
print("Converting geodataframe to EPSG:3857")
gdf.to_crs(3857, inplace=True)
print(f"Loading geodatframe to PostgreSQL as statcan_nar_addresses_combined_{vintage}")
gdf.to_postgis(name=f"statcan_nar_addresses_combined_{vintage}",
con=engine,
chunksize=150000)
open_databases/files_opendatabase_addresses.txt
+10
View File
@@ -0,0 +1,10 @@
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_AB_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_BC_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_MB_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_NB_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_NT_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_NS_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_ON_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_PE_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_QC_v1.zip
https://www150.statcan.gc.ca/n1/en/pub/46-26-0001/2021001/ODA_SK_v1.zip
open_databases/files_opendatabase_buildings.txt
+8
View File
@@ -0,0 +1,8 @@
https://www150.statcan.gc.ca/n1/en/pub/34-26-0001/2018001/ODB_v2_Alberta.zip?st=0J_AsIyy
https://www150.statcan.gc.ca/n1/en/pub/34-26-0001/2018001/ODB_v2_BritishColumbia.zip
https://www150.statcan.gc.ca/n1/en/pub/34-26-0001/2018001/ODB_v2_NewBrunswick.zip?st=k35-Ygwr
https://www150.statcan.gc.ca/n1/en/pub/34-26-0001/2018001/ODB_v2_NorthwestTerritories.zip?st=SXozU436
https://www150.statcan.gc.ca/n1/en/pub/34-26-0001/2018001/ODB_v2_NovaScotia.zip
https://www150.statcan.gc.ca/n1/en/pub/34-26-0001/2018001/ODB_v2_Ontario.zip
https://www150.statcan.gc.ca/n1/en/pub/34-26-0001/2018001/ODB_v2_Quebec.zip
https://www150.statcan.gc.ca/n1/en/pub/34-26-0001/2018001/ODB_v2_Saskatchewan.zip
open_databases/files_opendatabase_cultural.txt
+1
View File
@@ -0,0 +1 @@
https://www150.statcan.gc.ca/n1/en/pub/21-26-0001/2020001/ODCAF_V1.0.zip
open_databases/files_opendatabase_educational.txt
+1
View File
@@ -0,0 +1 @@
https://www150.statcan.gc.ca/n1/en/pub/37-26-0001/2022001/ODEF_v2.1.zip
open_databases/files_opendatabase_greenhouses.txt
+1
View File
@@ -0,0 +1 @@
https://www150.statcan.gc.ca/n1/en/pub/32-26-0005/2023001/ODG_V1.zip
open_databases/files_opendatabase_healthcare.txt
+1
View File
@@ -0,0 +1 @@
https://www150.statcan.gc.ca/n1/en/pub/13-26-0001/2020001/ODHF_v1.1.zip
open_databases/files_opendatabase_recreation.txt
+1
View File
@@ -0,0 +1 @@
https://www150.statcan.gc.ca/n1/en/pub/21-26-0002/2021001/ODRSF_v1.0.zip
open_databases/import_opendatabase.sh
+96
View File
@@ -0,0 +1,96 @@
#!/bin/bash
DATA_FOLDER=/home/ripledi/Documents/projects/process-statcan-spatial-data/data
source credentials.sh
export_postgis_single() {
local filepath=$1
local table_name=$2
local extra_parameters=${@:3}
# Virtual file system
if [[ ${filepath: -4} = '.zip' ]]; then
local filepath="/vsizip/${filepath}"
fi
echo "Importing ${filepath}"
ogr2ogr \
--config PG_USE_COPY YES \
-overwrite \
-f "PostgreSQL" \
"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432" \
-progress \
-gt 500000 \
-t_srs EPSG:4326 \
-nln ${table_name} \
${extra_parameters} \
${filepath}
}
export_open_database_of_greenhouses() {
export PGCLIENTENCODING=UTF-8;
export_postgis_single ${DATA_FOLDER}/ODG_V1/odg_v1.shp statcan_odg_tmp
}
export_open_database_of_buildings() {
# Open Database of Buildings
export PGCLIENTENCODING=UTF-8;
export_postgis_single ${DATA_FOLDER}/ODB_Alberta/odb_alberta.shp statcan_odb_tmp "-nlt PROMOTE_TO_MULTI"
export_postgis_single ${DATA_FOLDER}/ODB_BritishColumbia/odb_britishcolumbia.shp statcan_odb_tmp "-append -nlt PROMOTE_TO_MULTI"
export_postgis_single ${DATA_FOLDER}/ODB_NewBrunswick/odb_newbrunswick.shp statcan_odb_tmp "-append -nlt PROMOTE_TO_MULTI"
export_postgis_single ${DATA_FOLDER}/ODB_NorthwestTerritories/odb_northwestterritories.shp statcan_odb_tmp "-append -nlt PROMOTE_TO_MULTI"
export_postgis_single ${DATA_FOLDER}/ODB_NovaScotia/odb_novascotia.shp statcan_odb_tmp "-append -nlt PROMOTE_TO_MULTI"
export_postgis_single ${DATA_FOLDER}/ODB_Ontario/odb_ontario.shp statcan_odb_tmp "-append -nlt PROMOTE_TO_MULTI"
export_postgis_single ${DATA_FOLDER}/ODB_Quebec/odb_quebec.shp statcan_odb_tmp "-append -nlt PROMOTE_TO_MULTI"
export_postgis_single ${DATA_FOLDER}/ODB_Saskatchewan/odb_saskatchewan.shp statcan_odb_tmp "-append -nlt PROMOTE_TO_MULTI"
}
export_open_database_of_educational_facilities() {
export PGCLIENTENCODING=LATIN-1;
export_postgis_single ${DATA_FOLDER}/ODEF_v2.1_EN/ODEF_v2_1.csv statcan_odef_tmp "-oo X_POSSIBLE_NAMES=Longitude, -oo Y_POSSIBLE_NAMES=Latitude -s_srs EPSG:4326"
}
export_open_database_of_healthcare_facilities() {
export PGCLIENTENCODING=LATIN-1;
# TODO: process further
# There are issues with the characters in this file, example <97>
export_postgis_single ${DATA_FOLDER}/ODHF_v1.1/odhf_v1.1.csv statcan_odhf_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326"
}
export_open_database_of_cultural_and_art_facilities() {
export PGCLIENTENCODING=LATIN-1;
# TODO: process further
export_postgis_single ${DATA_FOLDER}/ODCAF_V1.0/ODCAF_v1.0.csv statcan_odcaf_tmp "-oo X_POSSIBLE_NAMES=Longitude, -oo Y_POSSIBLE_NAMES=Latitude -s_srs EPSG:4326"
}
export_open_database_of_addresses() {
# PGCLIENTENCODING=UTF-8 seems to have fixed all of the issues
export PGCLIENTENCODING=UTF-8;
export_postgis_single ${DATA_FOLDER}/ODA_AB_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326"
export_postgis_single ${DATA_FOLDER}/ODA_BC_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
export_postgis_single ${DATA_FOLDER}/ODA_MB_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
export_postgis_single ${DATA_FOLDER}/ODA_NB_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
export_postgis_single ${DATA_FOLDER}/ODA_NS_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
export_postgis_single ${DATA_FOLDER}/ODA_NT_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
export_postgis_single ${DATA_FOLDER}/ODA_ON_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
export_postgis_single ${DATA_FOLDER}/ODA_PE_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
export_postgis_single ${DATA_FOLDER}/ODA_QC_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
export_postgis_single ${DATA_FOLDER}/ODA_SK_v1.csv statcan_oda_tmp "-oo X_POSSIBLE_NAMES=longitude, -oo Y_POSSIBLE_NAMES=latitude -s_srs EPSG:4326 -append"
}
export_open_database_of_recreational_and_sport_facilities() {
export PGCLIENTENCODING=LATIN-1;
# TODO: process further
export_postgis_single ${DATA_FOLDER}/ODRSF_V1.0/ODRSF_v1.0.csv statcan_odrsf_tmp "-oo X_POSSIBLE_NAMES=Longitude, -oo Y_POSSIBLE_NAMES=Latitude -s_srs EPSG:4326"
}
#export_open_database_of_greenhouses
#export_open_database_of_buildings
#export_open_database_of_educational_facilities
#export_open_database_of_healthcare_facilities
#export_open_database_of_cultural_and_art_facilities
#export_open_database_of_addresses
export_open_database_of_recreational_and_sport_facilities
open_databases/standardize.sql
+112
View File
@@ -0,0 +1,112 @@
/* Open Databases */
/* Open Database of Greenhouses */
drop table if exists statcan_odg_2023;
create table statcan_odg_2023 as
select b.dguid as prdguid, b.prename as provincenameenglish, a.imagedate, a.datasource as provider, wkb_geometry as geom
from statcan_odg_tmp as a,
statcan_pr_2021 as b
where st_intersects(a.wkb_geometry, b.geom);
create index statcan_odg_2023_geom_idx on statcan_odg_2023 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_odg_tmp;
/* Open Database of Buildings */
create table statcan_odb_2019 as
select b.dguid as csddguid, b.csdname, a.data_prov as data_provider, a.build_id, a.wkb_geometry as geom
from statcan_odb_tmp as a,
statcan_csd_2021 as b
where st_intersects(a.wkb_geometry, b.geom);
create index statcan_odb_2019_geom_idx on statcan_odb_2019 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_odb_tmp;
/* Open Database of Educational Facilities */
drop table if exists statcan_odef_2022;
create table statcan_odef_2022 as
select index, source_id, facility_name, facility_type, authority_name, isced010, isced020, isced1, isced2, isced3, isced4plus, olms_status, unit, street_no, street_name, city, prov_terr,
postal_code, a.pruid, csdname, csduid, geo_source, provider, cmaname, cmauid, wkb_geometry as geom
from statcan_odef_tmp as a,
statcan_pr_2021 as b
where st_intersects(a.wkb_geometry, b.geom);
create index statcan_odef_2022_geom_idx on statcan_odef_2022 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_odef_tmp;
/* Open Database of Healthcare Facilities */
drop table if exists statcan_odhf_2020;
create table statcan_odhf_2020 as
select index, facility_name, source_facility_type, odhf_facility_type, provider, unit, street_no, street_name, postal_code,
b.dguid as csddguid, b.csdname, c.dguid as prdguid, c.prename, a.wkb_geometry as geom
from statcan_odhf_tmp as a,
statcan_csd_2021 as b,
statcan_pr_2021 as c
where a.wkb_geometry is not null
and st_intersects(a.wkb_geometry, b.geom)
and st_intersects(a.wkb_geometry, c.geom);
create index statcan_odhf_2020_geom_idx on statcan_odhf_2020 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_odhf_tmp;
/* Open Database of Cultural and Art Facilities */
drop table if exists statcan_odcaf_2020;
create table statcan_odcaf_2020 as
select index, facility_name, source_facility_type, odcaf_facility_type, provider, unit, street_no,
street_name, postal_code, city, prov_terr, csd_name, csduid, pruid, wkb_geometry as geom
from statcan_odcaf_tmp;
create index statcan_odcaf_2020_geom_idx on statcan_odcaf_2020 using GIST(geom) with (FILLFACTOR=100);
update statcan_odcaf_2020
set facility_name = ''
where facility_name = '..';
update statcan_odcaf_2020
set source_facility_type = ''
where source_facility_type = '..';
update statcan_odcaf_2020
set unit = ''
where unit = '..';
update statcan_odcaf_2020
set street_no = ''
where street_no = '..';
update statcan_odcaf_2020
set street_name = ''
where street_name = '..';
update statcan_odcaf_2020
set postal_code = ''
where postal_code = '..';
update statcan_odcaf_2020
set city = ''
where city = '..';
update statcan_odcaf_2020
set city = ''
where city = '..';
update statcan_odcaf_2020
set prov_terr = ''
where prov_terr = '..';
update statcan_odcaf_2020
set csd_name = ''
where csd_name = '..';
update statcan_odcaf_2020
set csduid = ''
where csduid = '..';
update statcan_odcaf_2020
set pruid = ''
where pruid = '..';
drop table if exists statcan_odcaf_tmp;
/* Open Database of Addresses */
create table statcan_oda_2021 as
select a.id, a.street_no, a.street, a.unit, a.postal_code, b.dguid as csddguid, b.csdname, c.dguid as prdguid, a.provider, wkb_geometry as geom
from statcan_oda_tmp as a,
statcan_csd_2021 as b,
statcan_pr_2021 as c
where st_intersects(a.wkb_geometry, b.geom)
and b.pruid = c.pruid;
create index statcan_oda_2021_geom_idx on statcan_oda_2021 using GIST(geom) with (FILLFACTOR=100);
drop table if exists statcan_oda_tmp;
/* Open Database of Recreational and Sport Facilities */
-- TODO;
road_network_files/README.md
+4
View File
@@ -0,0 +1,4 @@
# TODO
- Process 2006 Geographic Attribute File Road Network
- Process 2001 road network
- Change loading of Census Road Network files as a function
road_network_files/download.sh
Executable
+19
View File
@@ -0,0 +1,19 @@
#!/bin/bash
if [ ! -d "${DATA_FOLDER}/road_network_files" ]
then
echo "Making directory ${DATA_FOLDER}/road_network_files/"
mkdir -p ${DATA_FOLDER}/road_network_files/{input,extracted,output}/{2021,2016,2011,2006}
fi
INPUT_FOLDER="${DATA_FOLDER}/road_network_files/input"
echo "Downloading 2021 road network file"
aria2c -x16 -i "${SCRIPT_DIR}/road_network_files/road_network_file_2021.txt" --dir="${INPUT_FOLDER}/2021" --auto-file-renaming=false
echo "Downloading 2016 road network file"
aria2c -x16 -i "${SCRIPT_DIR}/road_network_files/road_network_file_2016.txt" --dir="${INPUT_FOLDER}/2016" --auto-file-renaming=false
echo "Downloading 2011 road network file"
aria2c -x16 -i "${SCRIPT_DIR}/road_network_files/road_network_file_2011.txt" --dir="${INPUT_FOLDER}/2011" --auto-file-renaming=false
echo "Downloading 2006 road network file"
aria2c -x16 -i "${SCRIPT_DIR}/road_network_files/road_network_file_2006.txt" --dir="${INPUT_FOLDER}/2006" --auto-file-renaming=false
echo "Downloading 2001 road network file"
aria2c -x16 -i "${SCRIPT_DIR}/road_network_files/road_network_file_2001.txt" --dir="${INPUT_FOLDER}/2001" --auto-file-renaming=false
road_network_files/load.sh
Executable
+62
View File
@@ -0,0 +1,62 @@
#!/bin/bash
import_to_postgis() {
local filepath="$1"
local table_name="$2"
local extra_parameters="${@:3}"
# Handle zip files using GDAL's virtual file system
if [[ "${filepath: -4}" == ".zip" ]]; then
filepath="/vsizip/${filepath}"
fi
echo "Importing ${filepath} into table bronze.${table_name}"
ogr2ogr \
--config PG_USE_COPY YES \
-lco "OVERWRITE=YES" \
-f "PostgreSQL" \
"PG:host=db dbname=${POSTGRES_DB} user=${POSTGRES_USER} password=${POSTGRES_PASSWORD} port=5432" \
-lco GEOMETRY_NAME=geom \
-progress \
-gt 500000 \
-t_srs EPSG:4326 \
-nln "${table_name}" \
${extra_parameters} \
"${filepath}"
}
# Define input folders
INPUT_FOLDER="${DATA_FOLDER}/road_network_files/input"
EXTRACTED_FOLDER="${DATA_FOLDER}/road_network_files/extracted"
### 2021 Road Network File ###
# https://web.archive.org/web/20230307163203/https://www12.statcan.gc.ca/census-recensement/2021/geo/sip-pis/boundary-limites/index2021-eng.cfm?year=21
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2021/lrnf000r21a_e.zip" lrnf000r21a_e "-lco COLUMN_TYPES=afl_val=integer,atl_val=integer,afr_val=integer,atr_val=integer -lco SCHEMA=bronze"
unset PGCLIENTENCODING
### 2016 Road Network File ###
# https://web.archive.org/web/20230120140926/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2016-eng.cfm
import_to_postgis "${INPUT_FOLDER}/2016/lrnf000r16a_e.zip" lrnf000r16a_e "-lco COLUMN_TYPES=afl_val=integer,atl_val=integer,afr_val=integer,atr_val=integer -lco SCHEMA=bronze"
### 2011 Road Network File ###
# https://web.archive.org/web/20230110163150/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2011-eng.cfm
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2011/grnf000r11a_e.zip" grnf000r11a_e "-lco COLUMN_TYPES=afl_val=integer,atl_val=integer,afr_val=integer,atr_val=integer -lco SCHEMA=bronze"
unset PGCLIENTENCODING
### 2006 Road Network File ###
# https://web.archive.org/web/20221218043125/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2006-eng.cfm
export PGCLIENTENCODING=LATIN-1;
import_to_postgis "${INPUT_FOLDER}/2006/grnf000r06a_e.zip" grnf000r06a_e "-lco COLUMN_TYPES=addr_fm_le=integer,addr_to_le=integer,addr_fm_rg=integer,addr_to_rg=integer -lco SCHEMA=bronze"
unset PGCLIENTENCODING
# TODO - get working
### 2001 Road Network Files ###
# https://web.archive.org/web/20221218043135/https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2001-eng.cfm
# Census Road Network file
#unzip -n ${INPUT_FOLDER}/grnf000r01m_e.zip -d ${EXTRACTED_FOLDER}
#import_to_postgis ${EXTRACTED_FOLDER}/grnf000r02ml_e.MID statcan_grnf000r02ml_e_tmp
## Census Skeletal Road Network File
#unzip -n ${INPUT_FOLDER}/gsrn000r01m_e.zip -d ${EXTRACTED_FOLDER}
#import_to_postgis ${DATA_FOLDER}/gsrn000r02m_e.MID statcan_gsrn000r02m_e_tmp

Some files were not shown because too many files have changed in this diff Show More