mirror of
https://github.com/felt/tippecanoe.git
synced 2026-11-22 16:31:18 +01:00
Compare commits
96
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6d46578ad8 | ||
|
|
794ae2b1eb | ||
|
|
23667bb8eb | ||
|
|
b3b89e1e07 | ||
|
|
120b8f4e41 | ||
|
|
28efc40e6e | ||
|
|
78661f1cb2 | ||
|
|
8409bf9ce4 | ||
|
|
23c6559b03 | ||
|
|
66e5d66300 | ||
|
|
4822d02f24 | ||
|
|
c5f2f0da34 | ||
|
|
534242887d | ||
|
|
84f6e887a7 | ||
|
|
51fcf142df | ||
|
|
5b18eea673 | ||
|
|
f3b575ead4 | ||
|
|
f44230f5f5 | ||
|
|
40bb4ff732 | ||
|
|
d891ca7289 | ||
|
|
bc3ef87c3f | ||
|
|
50deb9ce63 | ||
|
|
47e774adc4 | ||
|
|
939579e5a2 | ||
|
|
e11583df1c | ||
|
|
bb4f220678 | ||
|
|
266ef3c286 | ||
|
|
1519b50791 | ||
|
|
bd48ba8ea1 | ||
|
|
e796400377 | ||
|
|
655eccfcc3 | ||
|
|
021bb96aeb | ||
|
|
6a8f1b83d8 | ||
|
|
312e1560a3 | ||
|
|
d9a5c2eed7 | ||
|
|
853ada87b5 | ||
|
|
a987197eed | ||
|
|
2b6630c42c | ||
|
|
96f126dd59 | ||
|
|
4e52cbd957 | ||
|
|
e2a7a409c7 | ||
|
|
c5c4698161 | ||
|
|
6a2bce8164 | ||
|
|
ca1aa58a11 | ||
|
|
7d4d264d50 | ||
|
|
cbf222754b | ||
|
|
f957f30f90 | ||
|
|
679a0d62f2 | ||
|
|
5d92a17193 | ||
|
|
e8ca6c6de3 | ||
|
|
02e3bac2c0 | ||
|
|
15a3e313ff | ||
|
|
d7bdbe363b | ||
|
|
d359461e61 | ||
|
|
2c56187530 | ||
|
|
e1c665b35d | ||
|
|
0ca14086a9 | ||
|
|
cc5c1c79df | ||
|
|
26bf08deb2 | ||
|
|
dcc3a11915 | ||
|
|
f7dc7faf31 | ||
|
|
390771f855 | ||
|
|
dee40802fd | ||
|
|
e6d05bc317 | ||
|
|
2ec6180003 | ||
|
|
6778aeac52 | ||
|
|
39ad95ed73 | ||
|
|
2b133a1def | ||
|
|
f00eeb486a | ||
|
|
9e9e21bcc0 | ||
|
|
430d8edd17 | ||
|
|
5cba91981a | ||
|
|
55b9aa501c | ||
|
|
4318e964e3 | ||
|
|
1c3bd5352c | ||
|
|
48e769c94c | ||
|
|
58bfd121a7 | ||
|
|
b5ec3c46f0 | ||
|
|
c1051232ad | ||
|
|
b09a880822 | ||
|
|
ba3e42568c | ||
|
|
87ed674a30 | ||
|
|
0422d77bbe | ||
|
|
20eb9c8dae | ||
|
|
523ca55243 | ||
|
|
605640e138 | ||
|
|
5bd46e7ed5 | ||
|
|
434817bde9 | ||
|
|
da7bbbd6ea | ||
|
|
e43d6dacdd | ||
|
|
afc7ca01a2 | ||
|
|
329dd86bfd | ||
|
|
3d714e32b0 | ||
|
|
7371072e45 | ||
|
|
0fa758c1e3 | ||
|
|
f2127fec97 |
@@ -0,0 +1 @@
|
||||
* @e-n-f @joeherm @migurski @arredond @dnomadb @ChrisLoer @ibesora
|
||||
@@ -5,7 +5,11 @@ on: [push]
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
version: ['Release', 'Debug']
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- run: make
|
||||
- run: sudo apt-get install libsqlite3-dev
|
||||
- run: uname -a; BUILDTYPE=${{ matrix.version }} make
|
||||
- run: make test
|
||||
|
||||
@@ -35,6 +35,7 @@ tile-join
|
||||
tippecanoe-decode
|
||||
tippecanoe-enumerate
|
||||
tippecanoe-json-tool
|
||||
tippecanoe-overzoom
|
||||
unit
|
||||
|
||||
# Tests
|
||||
|
||||
+295
@@ -1,3 +1,298 @@
|
||||
# 2.69.0
|
||||
|
||||
* Fix crash when the first bin gets clipped away
|
||||
|
||||
# 2.68.0
|
||||
|
||||
* Adds `--no-tile-compression` option to `tippecanoe-overzoom`
|
||||
* Make `tippecanoe-overzoom` clip the output tile to the tile buffer after assigning points to bins
|
||||
* Adds `--include`/`-y` option to `tippecanoe-decode` to decode only specified attributes
|
||||
* Cleans up some inconsistent handling of variable tile extents in overzooming code
|
||||
* Speeds up overzooming slightly in `tile-join` by doing less preflighting to discover which child tiles contain features
|
||||
|
||||
# 2.67.0
|
||||
|
||||
* Reduce memory consumption of duplicate attribute names in `serial_feature`
|
||||
* The maxzoom guess calculation now takes into account the number of duplicate feature locations
|
||||
|
||||
# 2.66.0
|
||||
|
||||
* Only bin by ID, not by geometry, if --bin-by-id-list is specified
|
||||
* Do attribute accumulation in overzoom in mvt_value instead of converting to serial_val
|
||||
* Fix bool values read from flatgeobuf sources (#289)
|
||||
|
||||
# 2.65.0
|
||||
|
||||
* Improve spatial distribution of --retain-points-multiplier features
|
||||
* Add --preserve-multiplier-density-threshold option to maintain minimum density of multiplier features
|
||||
|
||||
# 2.64.0
|
||||
|
||||
* Add --bin-by-id to overzoom
|
||||
|
||||
# 2.63.0
|
||||
|
||||
* Top-level null filter now evaluates to true
|
||||
|
||||
# 2.62.6
|
||||
|
||||
* Remove buggy optimization to avoid reclipping in tippecanoe-overzoom
|
||||
|
||||
# 2.62.5
|
||||
|
||||
* More aggressive binning when points fail the point-in-polygon test
|
||||
|
||||
# 2.62.4
|
||||
|
||||
* Fix accumulation of count and mean in overzoom
|
||||
|
||||
# 2.62.3
|
||||
|
||||
* Summary statistics with --accumulate-numeric-attributes make it from tiling through to binning
|
||||
* Prefix can be specified for --accumulate-numeric-attributes
|
||||
* Added --exclude and --exclude-prefix to tippecanoe-overzoom
|
||||
|
||||
# 2.62.2
|
||||
|
||||
* Pass feature ID through with bins
|
||||
|
||||
# 2.62.1
|
||||
|
||||
* More work in progress on binning point features in overzoom
|
||||
|
||||
# 2.62.0
|
||||
|
||||
* Fix another bad interaction, this time between dropping-as-needed and --limit-tile-feature-count
|
||||
|
||||
# 2.61.0
|
||||
|
||||
* Added --calculate-feature-index option
|
||||
* Added "count" accumulation type to --accumulate-attribute
|
||||
* Work in progress on binning of point features in overzoom. Not ready for use yet.
|
||||
|
||||
# 2.60.0
|
||||
|
||||
* Fix bad interaction between --retain-points-multiplier and stopping early when the tile feature limit is reached
|
||||
* Fix another bad interaction between --retain-points-multiplier, dropping-as-needed, and variable depth tile pyramids
|
||||
* Add optional BUILD_INFO string to version
|
||||
* Reorder overzoom logic to clip before dealing with multiplier and filters
|
||||
* When --generate-variable-depth-tile-pyramid is in use, report the actual highest zoom generated as tileset maxzoom
|
||||
|
||||
# 2.59.0
|
||||
|
||||
* Correct `antimeridian_adjusted_bounds` latitude calculation when vertices extend beyond the edge of the Mercator plane
|
||||
|
||||
# 2.58.0
|
||||
|
||||
* Add --generate-variable-depth-tile-pyramid option
|
||||
* Add --line-simplification and --tiny-polygon-size options to tippecanoe-overzoom
|
||||
* Adjust tile feature limit for --retain-points-multiplier
|
||||
* Tune convergence rate for --coalesce-densest and --drop-densest
|
||||
* Fix overreported drop and coalesce counts in strategies
|
||||
|
||||
# 2.57.0
|
||||
|
||||
* Add multi-tile input to tippecanoe-overzoom
|
||||
|
||||
# 2.56.0
|
||||
|
||||
* Rework --coalesce-densest-as-needed and --drop-densest-as-needed to look better
|
||||
* Add --maximum-string-attribute-length option
|
||||
|
||||
# 2.55.0
|
||||
|
||||
* Fix hash collisions in the string pool
|
||||
|
||||
# 2.53.0
|
||||
|
||||
* Stop trying to add features to the tile after the feature limit is reached
|
||||
|
||||
# 2.52.0
|
||||
|
||||
* Fix accidental loss (at all zooms) of features that specify an explicit minzoom
|
||||
|
||||
# 2.51.0
|
||||
|
||||
* Fix null behavior in "in" expressions
|
||||
* But they are back to not using unidecode
|
||||
|
||||
# 2.50.0
|
||||
|
||||
* FSL-style "in" expressions use unidecode again
|
||||
|
||||
# 2.49.0
|
||||
|
||||
* FSL-style "in" expressions now allow numeric comparisons, but they no longer use unidecode to remove diacritics.
|
||||
|
||||
# 2.48.0
|
||||
|
||||
* Fix some undefined behavior bugs, one of which results in slight changes to line simplification choices
|
||||
|
||||
# 2.47.0
|
||||
|
||||
* Stabilize feature order in tippecanoe-overzoom when --preserve-feature-order is specified but the sequence attribute is not present
|
||||
|
||||
# 2.46.0
|
||||
|
||||
* Polygon dust returns to having the attributes of the contributing feature nearest the placeholder instead of the contributing feature with the largest area.
|
||||
|
||||
# 2.45.0
|
||||
|
||||
* Adjust tile size limit with --retain-points-multiplier dynamically within each tile, to allow multiplier features at high zooms if other features are being dropped as-needed
|
||||
|
||||
# 2.44.0
|
||||
|
||||
* Add --unidecode-data option to allow case-insensitive filter comparisons of transliterated strings
|
||||
|
||||
# 2.43.0
|
||||
|
||||
* Change -fraction-as-needed feature dropping to be consistent across tiles and zoom levels, and to follow the same pattern as point dropping by zoom level
|
||||
* With -as-needed feature dropping, drop or retain entire multiplier clusters instead of individual features
|
||||
* Sort the features within each multiplier cluster by its retention priority, for more consistency between zoom levels in filtered feature choice
|
||||
|
||||
# 2.42.0
|
||||
|
||||
* Improve tiling speed
|
||||
* Generate tilestats for the --retain-points-multiplier attributes
|
||||
|
||||
# 2.41.3
|
||||
|
||||
* Performance optimizations to tile reading, writing, and overzooming
|
||||
* Automatically vary the tile size limit by zoom level to match the intended retain-points-multiplier multiplication
|
||||
* Fix decompression error when bailing out because a tile can't be made small enough
|
||||
* Search harder for a feature size threshold to make the tile small enough before giving up
|
||||
|
||||
# 2.41.2
|
||||
|
||||
* Add --accumulate-attribute to tippecanoe-overzoom
|
||||
* Go back to ordering features within each multiplier cluster spatially, not in the order specified for tile feature order
|
||||
|
||||
# 2.41.1
|
||||
|
||||
* Make --preserve-input-order, --order-by, --order-descending-by, --order-smallest-first, and --order-largest-first cooperate with --retain-points-multiplier. The clusters will be ordered by their lead feature in the specified sequence. The other features in each cluster will continue to be physically near the lead feature, but ordered as specified within the cluster.
|
||||
|
||||
# 2.41.0
|
||||
|
||||
* Add Felt-style expression support for -j feature filters
|
||||
* Add --retain-points-multiplier option
|
||||
* Add tippecanoe_decisions metadata field to record basezoom, drop rate, and multiplier
|
||||
* Add multiplier thinning (-m) and feature filters (-j) to tippecanoe-overzoom
|
||||
|
||||
# 2.40.0
|
||||
|
||||
* Slightly reduce compression aggressiveness to improve as-needed dropping speed
|
||||
|
||||
# 2.39.0
|
||||
|
||||
* Reduce memory usage during tiling
|
||||
|
||||
# 2.38.0
|
||||
|
||||
* Tolerate polygon rings with insuffiently many points in input
|
||||
|
||||
# 2.37.1
|
||||
|
||||
* Reduce maximum memory used for vertex sorting
|
||||
|
||||
# 2.37.0
|
||||
|
||||
* Speed up tile-join overzooming and make it use less memory, by not including empty child tiles in the enumeration
|
||||
|
||||
# 2.36.0
|
||||
|
||||
* Make tile-join distrust the source tilesets' metadata maxzoom and minzoom
|
||||
* Add a special case in --detect-longitude-wraparound not to wrap around jumps of exactly 360°
|
||||
|
||||
# 2.35.0
|
||||
|
||||
* Fix a bug in --detect-longitude-wraparound when there are multiple rings
|
||||
|
||||
# 2.34.1
|
||||
|
||||
* Further improvements to tile-join speed
|
||||
|
||||
# 2.34.0
|
||||
|
||||
* Improve speed of overzooming in tile-join
|
||||
|
||||
# 2.33.0
|
||||
|
||||
* Further reduce memory usage of --no-simplification-of-shared-nodes by calculating the list of shared nodes globally using temporary files rather than in memory for each individual tile
|
||||
* Make --no-simplification-of-shared-nodes behave for LineStrings as it does for Polygons, preventing simplification only at crossings, convergences, and divergences, not at every point along collinear segments
|
||||
|
||||
# 2.32.1
|
||||
|
||||
* Reduce memory usage of --no-simplification-of-shared-nodes for polygons
|
||||
|
||||
# 2.32.0
|
||||
|
||||
* Extend --no-simplification-of-shared-nodes to also simplify shared polygon borders consistently
|
||||
|
||||
# 2.31.0
|
||||
|
||||
* Fix tile-join crash when trying to join empty tilesets
|
||||
* Add --no-tiny-polygon-reduction-at-maximum-zoom option
|
||||
|
||||
# 2.30.1
|
||||
|
||||
* Fix spurious reports of tiny polygons and 0-length LineStrings in "strategies"
|
||||
|
||||
# 2.30.0
|
||||
|
||||
* Add --extend-zooms-if-still-dropping-maximum option
|
||||
* Add --overzoom option to tile-join
|
||||
|
||||
# 2.29.0
|
||||
|
||||
* Add tippecanoe-overzoom tool
|
||||
|
||||
# 2.28.1
|
||||
|
||||
* Allow --set-attribute to override an existing attribute value
|
||||
|
||||
# 2.28.0
|
||||
|
||||
* Add --preserve-point-density-threshold option to reduce blank areas of the map at low zooms
|
||||
* Fix tile-join bug where use of --read-from would also accidentally enable --quiet
|
||||
|
||||
# 2.27.0
|
||||
|
||||
* Do more of line simplification in integer coordinates, to make behavior consistent across platforms
|
||||
* Reduce excessive logging during pmtiles conversion
|
||||
* Add --set-attribute option
|
||||
* Accept JSON form of --accumulate-attribute
|
||||
|
||||
# 2.26.1
|
||||
|
||||
* Avoid crashing if there is a polygon ring with only one point
|
||||
|
||||
# 2.26.0
|
||||
|
||||
* Fix bugs in --no-simplification-of-shared-nodes
|
||||
* Updated dockerfile (jtmiclat)
|
||||
* Set build options to use C++-17 (james2432)
|
||||
* Use std::fpclassify instead of plain fpclassify (james)
|
||||
* Fix pmtiles warnings (bdon)
|
||||
|
||||
# 2.25.0
|
||||
|
||||
* Add `--include`/`-y` option to tile-join
|
||||
|
||||
# 2.24.0
|
||||
|
||||
* Add --cluster-maxzoom option to limit zoom levels that receive clustering
|
||||
* Add `point_count_abbreviated` attribute to clustered features, for consistency with supercluster
|
||||
* Makefile changes to support FreeBSD
|
||||
* Add -r option to tile-join to provide a file containing a list of input files
|
||||
* Add antimeridian_adjusted_bounds field to tileset metadata
|
||||
|
||||
## 2.23.0
|
||||
|
||||
* Remove the concept of "separate metadata." Features now always directly reference their keys and values rather than going through a second level of indirection.
|
||||
* Limit the size of the string pool to 1/5 the size of memory, to prevent thrashing during feature ingestion.
|
||||
* Avoid using writeable memory maps. Instead, explicitly copy data in and out of memory.
|
||||
* Compress streams of features in the temporary files, to reduce disk usage and I/O latency
|
||||
|
||||
## 2.22.0
|
||||
|
||||
* Speed up feature dropping by removing unnecessary search for other small features
|
||||
|
||||
+13
-12
@@ -1,19 +1,20 @@
|
||||
# Start from ubuntu
|
||||
FROM ubuntu:16.04
|
||||
FROM ubuntu:22.04 AS tippecanoe-builder
|
||||
|
||||
# Update repos and install dependencies
|
||||
RUN apt-get update \
|
||||
&& apt-get -y upgrade \
|
||||
&& apt-get -y install build-essential libsqlite3-dev zlib1g-dev
|
||||
&& apt-get -y install make gcc g++ libsqlite3-dev zlib1g-dev
|
||||
|
||||
# Create a directory and copy in all files
|
||||
RUN mkdir -p /tmp/tippecanoe-src
|
||||
WORKDIR /tmp/tippecanoe-src
|
||||
COPY . /tmp/tippecanoe-src
|
||||
WORKDIR /tmp/tippecanoe-src
|
||||
|
||||
# Build tippecanoe
|
||||
RUN make \
|
||||
&& make install
|
||||
RUN make
|
||||
|
||||
# Run the tests
|
||||
CMD make test
|
||||
|
||||
# Using multistage build reduces the docker image size by alot by only copying the needed binaries
|
||||
FROM ubuntu:22.04
|
||||
RUN apt-get update \
|
||||
&& apt-get -y install libsqlite3-0 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
COPY --from=tippecanoe-builder /tmp/tippecanoe-src/tippecanoe* /usr/local/bin/
|
||||
COPY --from=tippecanoe-builder /tmp/tippecanoe-src/tile-join /usr/local/bin/
|
||||
WORKDIR /app
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
FROM centos:7
|
||||
FROM centos:centos8
|
||||
|
||||
RUN yum install -y make sqlite-devel zlib-devel bash git gcc-c++
|
||||
|
||||
|
||||
@@ -1,35 +1,47 @@
|
||||
PREFIX ?= /usr/local
|
||||
MANDIR ?= $(PREFIX)/share/man/man1/
|
||||
BUILDTYPE ?= Release
|
||||
SHELL = /bin/bash
|
||||
BUILD_INFO ?=
|
||||
SHELL = /bin/sh
|
||||
|
||||
|
||||
# inherit from env if set
|
||||
CC := $(CC)
|
||||
CXX := $(CXX)
|
||||
CFLAGS := $(CFLAGS)
|
||||
CXXFLAGS := $(CXXFLAGS) -std=c++11
|
||||
CFLAGS := $(CFLAGS) -fPIE -DBUILD_INFO=$(BUILD_INFO)
|
||||
CXXFLAGS := $(CXXFLAGS) -std=c++17 -fPIE -DBUILD_INFO=$(BUILD_INFO)
|
||||
LDFLAGS := $(LDFLAGS)
|
||||
WARNING_FLAGS := -Wall -Wshadow -Wsign-compare -Wextra -Wunreachable-code -Wuninitialized -Wshadow
|
||||
RELEASE_FLAGS := -O3 -DNDEBUG
|
||||
DEBUG_FLAGS := -O0 -DDEBUG -fno-inline-functions -fno-omit-frame-pointer
|
||||
|
||||
|
||||
OS := $(shell uname -s)
|
||||
ifeq ($(OS),FreeBSD)
|
||||
ADVSHELL = /usr/local/bin/bash
|
||||
else
|
||||
ADVSHELL = /bin/bash
|
||||
endif
|
||||
|
||||
|
||||
ifeq ($(BUILDTYPE),Release)
|
||||
FINAL_FLAGS := -g $(WARNING_FLAGS) $(RELEASE_FLAGS)
|
||||
else
|
||||
FINAL_FLAGS := -g $(WARNING_FLAGS) $(DEBUG_FLAGS)
|
||||
endif
|
||||
|
||||
all: tippecanoe tippecanoe-enumerate tippecanoe-decode tile-join unit tippecanoe-json-tool
|
||||
all: tippecanoe tippecanoe-enumerate tippecanoe-decode tile-join unit tippecanoe-json-tool tippecanoe-overzoom
|
||||
|
||||
docs: man/tippecanoe.1
|
||||
|
||||
install: tippecanoe tippecanoe-enumerate tippecanoe-decode tile-join tippecanoe-json-tool
|
||||
install: tippecanoe tippecanoe-enumerate tippecanoe-decode tile-join tippecanoe-json-tool tippecanoe-overzoom
|
||||
mkdir -p $(PREFIX)/bin
|
||||
mkdir -p $(MANDIR)
|
||||
cp tippecanoe $(PREFIX)/bin/tippecanoe
|
||||
cp tippecanoe-enumerate $(PREFIX)/bin/tippecanoe-enumerate
|
||||
cp tippecanoe-decode $(PREFIX)/bin/tippecanoe-decode
|
||||
cp tippecanoe-json-tool $(PREFIX)/bin/tippecanoe-json-tool
|
||||
cp tippecanoe-overzoom $(PREFIX)/bin/tippecanoe-overzoom
|
||||
cp tile-join $(PREFIX)/bin/tile-join
|
||||
cp man/tippecanoe.1 $(MANDIR)/tippecanoe.1
|
||||
|
||||
@@ -47,7 +59,7 @@ C = $(wildcard *.c) $(wildcard *.cpp)
|
||||
INCLUDES = -I/usr/local/include -I.
|
||||
LIBS = -L/usr/local/lib
|
||||
|
||||
tippecanoe: geojson.o jsonpull/jsonpull.o tile.o pool.o mbtiles.o geometry.o projection.o memfile.o mvt.o serial.o main.o text.o dirtiles.o pmtiles_file.o plugin.o read_json.o write_json.o geobuf.o flatgeobuf.o evaluator.o geocsv.o csv.o geojson-loop.o json_logger.o visvalingam.o
|
||||
tippecanoe: geojson.o jsonpull/jsonpull.o tile.o pool.o mbtiles.o geometry.o projection.o memfile.o mvt.o serial.o main.o text.o dirtiles.o pmtiles_file.o plugin.o read_json.o write_json.o geobuf.o flatgeobuf.o evaluator.o geocsv.o csv.o geojson-loop.o json_logger.o visvalingam.o compression.o clip.o sort.o attribute.o thread.o shared_borders.o
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
tippecanoe-enumerate: enumerate.o
|
||||
@@ -56,13 +68,16 @@ tippecanoe-enumerate: enumerate.o
|
||||
tippecanoe-decode: decode.o projection.o mvt.o write_json.o text.o jsonpull/jsonpull.o dirtiles.o pmtiles_file.o
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3
|
||||
|
||||
tile-join: tile-join.o projection.o mbtiles.o mvt.o memfile.o dirtiles.o jsonpull/jsonpull.o text.o evaluator.o csv.o write_json.o pmtiles_file.o
|
||||
tile-join: tile-join.o projection.o mbtiles.o mvt.o memfile.o dirtiles.o jsonpull/jsonpull.o text.o evaluator.o csv.o write_json.o pmtiles_file.o clip.o attribute.o thread.o
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
tippecanoe-json-tool: jsontool.o jsonpull/jsonpull.o csv.o text.o geojson-loop.o
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
unit: unit.o text.o
|
||||
unit: unit.o text.o sort.o mvt.o projection.o clip.o attribute.o jsonpull/jsonpull.o evaluator.o
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
tippecanoe-overzoom: overzoom.o mvt.o clip.o evaluator.o jsonpull/jsonpull.o text.o attribute.o read_json.o projection.o
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
-include $(wildcard *.d)
|
||||
@@ -77,19 +92,36 @@ clean:
|
||||
rm -f ./tippecanoe ./tippecanoe-* ./tile-join ./unit *.o *.d */*.o */*.d tests/**/*.mbtiles tests/**/*.check
|
||||
|
||||
indent:
|
||||
clang-format -i -style="{BasedOnStyle: Google, IndentWidth: 8, UseTab: Always, AllowShortIfStatementsOnASingleLine: false, ColumnLimit: 0, ContinuationIndentWidth: 8, SpaceAfterCStyleCast: true, IndentCaseLabels: false, AllowShortBlocksOnASingleLine: false, AllowShortFunctionsOnASingleLine: false, SortIncludes: false}" $(C) $(H)
|
||||
clang-format -i -style="{BasedOnStyle: Google, IndentWidth: 8, UseTab: Always, AllowShortIfStatementsOnASingleLine: false, ColumnLimit: 0, ContinuationIndentWidth: 8, SpaceAfterCStyleCast: true, IndentCaseLabels: false, AllowShortBlocksOnASingleLine: false, AllowShortFunctionsOnASingleLine: false, SortIncludes: false}" $(filter-out flatgeobuf.cpp,$(C)) $(H) jsonpull/*.[ch]
|
||||
|
||||
TESTS = $(wildcard tests/*/out/*.json)
|
||||
SPACE = $(NULL) $(NULL)
|
||||
|
||||
test: tippecanoe tippecanoe-decode $(addsuffix .check,$(TESTS)) raw-tiles-test parallel-test pbf-test join-test enumerate-test decode-test join-filter-test unit json-tool-test allow-existing-test csv-test layer-json-test pmtiles-test decode-pmtiles-test
|
||||
test: tippecanoe tippecanoe-decode $(addsuffix .check,$(TESTS)) raw-tiles-test parallel-test pbf-test join-test enumerate-test decode-test join-filter-test unit json-tool-test allow-existing-test csv-test layer-json-test pmtiles-test decode-pmtiles-test overzoom-test accumulate-test
|
||||
./unit
|
||||
|
||||
suffixes = json json.gz
|
||||
|
||||
# Work around Makefile and filename punctuation limits: _ for space, @ for :, % for /
|
||||
# Work around Makefile and filename punctuation limits:
|
||||
# _ for argument-separator space
|
||||
# %20 for quoted space
|
||||
# %22 for quoted quote
|
||||
# %2f for /
|
||||
# %3a for :
|
||||
# %5f for _
|
||||
# %7b for {
|
||||
|
||||
testargs = \
|
||||
$(subst %20,' ',\
|
||||
$(subst %22,'"',\
|
||||
$(subst %3a,:,\
|
||||
$(subst %2f,/,\
|
||||
$(subst %7b,'{',\
|
||||
$(subst %5f,'_',\
|
||||
$(subst _, ,$(1))))))))
|
||||
|
||||
%.json.check:
|
||||
./tippecanoe -q -a@ -f -o $@.mbtiles $(subst @,:,$(subst %,/,$(subst _, ,$(patsubst %.json.check,%,$(word 4,$(subst /, ,$@)))))) $(foreach suffix,$(suffixes),$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix)))) < /dev/null
|
||||
./tippecanoe -q -a@ -f -o $@.mbtiles $(call testargs,$(patsubst %.json.check,%,$(word 4,$(subst /, ,$@)))) $(foreach suffix,$(suffixes),$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix)))) < /dev/null
|
||||
./tippecanoe-decode -x generator $@.mbtiles > $@.out
|
||||
cmp $@.out $(patsubst %.check,%,$@)
|
||||
rm $@.out $@.mbtiles
|
||||
@@ -106,12 +138,12 @@ fewer-tests: tippecanoe tippecanoe-decode geobuf-test raw-tiles-test parallel-te
|
||||
%.json.checkbuf:
|
||||
for i in $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.json); do ./tippecanoe-json-tool -w $$i | ./node_modules/geobuf/bin/json2geobuf > $$i.geobuf; done
|
||||
for i in $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.json.gz); do gzip -dc $$i | ./tippecanoe-json-tool -w | ./node_modules/geobuf/bin/json2geobuf > $$i.geobuf; done
|
||||
./tippecanoe -q -a@ -f -o $@.mbtiles $(subst @,:,$(subst %,/,$(subst _, ,$(patsubst %.json.checkbuf,%,$(word 4,$(subst /, ,$@)))))) $(foreach suffix,$(suffixes),$(addsuffix .geobuf,$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix))))) < /dev/null
|
||||
./tippecanoe -q -a@ -f -o $@.mbtiles $(call testargs,$(patsubst %.json.checkbuf,%,$(word 4,$(subst /, ,$@)))) $(foreach suffix,$(suffixes),$(addsuffix .geobuf,$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix))))) < /dev/null
|
||||
./tippecanoe-decode -x generator $@.mbtiles | sed 's/checkbuf/check/g' | sed 's/\.geobuf//g' > $@.out
|
||||
cmp $@.out $(patsubst %.checkbuf,%,$@)
|
||||
rm $@.out $@.mbtiles
|
||||
|
||||
parallel-test:
|
||||
parallel-test: $(eval SHELL:=$(ADVSHELL))
|
||||
mkdir -p tests/parallel
|
||||
perl -e 'for ($$i = 0; $$i < 20; $$i++) { $$lon = rand(360) - 180; $$lat = rand(180) - 90; $$k = rand(1); $$v = rand(1); print "{ \"type\": \"Feature\", \"properties\": { \"yes\": \"no\", \"who\": 1, \"$$k\": \"$$v\" }, \"geometry\": { \"type\": \"Point\", \"coordinates\": [ $$lon, $$lat ] } }\n"; }' > tests/parallel/in1.json
|
||||
perl -e 'for ($$i = 0; $$i < 300000; $$i++) { $$lon = rand(360) - 180; $$lat = rand(180) - 90; print "{ \"type\": \"Feature\", \"properties\": { }, \"geometry\": { \"type\": \"Point\", \"coordinates\": [ $$lon, $$lat ] } }\n"; }' > tests/parallel/in2.json
|
||||
@@ -138,7 +170,7 @@ parallel-test:
|
||||
cmp tests/parallel/linear-file.json tests/parallel/parallel-pipes.json
|
||||
rm tests/parallel/*.mbtiles tests/parallel/*.json
|
||||
|
||||
raw-tiles-test:
|
||||
raw-tiles-test: tippecanoe tippecanoe-decode tile-join
|
||||
./tippecanoe -q -f -e tests/raw-tiles/raw-tiles -r1 -pC tests/raw-tiles/hackspots.geojson
|
||||
./tippecanoe-decode -x generator tests/raw-tiles/raw-tiles > tests/raw-tiles/raw-tiles.json.check
|
||||
cmp tests/raw-tiles/raw-tiles.json.check tests/raw-tiles/raw-tiles.json
|
||||
@@ -156,7 +188,7 @@ raw-tiles-test:
|
||||
cmp tests/raw-tiles/nothing.json.check tests/raw-tiles/nothing.json
|
||||
rm -r tests/raw-tiles/nothing tests/raw-tiles/nothing.json.check
|
||||
|
||||
pmtiles-test:
|
||||
pmtiles-test: tippecanoe tippecanoe-decode tile-join
|
||||
./tippecanoe -q -f -o tests/pmtiles/hackspots.pmtiles -r1 -pC tests/raw-tiles/hackspots.geojson
|
||||
./tippecanoe-decode -x generator tests/pmtiles/hackspots.pmtiles > tests/pmtiles/hackspots.json.check
|
||||
cmp tests/pmtiles/hackspots.json.check tests/pmtiles/hackspots.json
|
||||
@@ -193,7 +225,7 @@ pmtiles-test:
|
||||
cmp tests/raw-tiles/nothing.json.check tests/raw-tiles/nothing.json
|
||||
rm -r tests/raw-tiles/nothing.pmtiles tests/raw-tiles/nothing.json.check
|
||||
|
||||
decode-test:
|
||||
decode-test: tippecanoe tippecanoe-decode
|
||||
mkdir -p tests/muni/decode
|
||||
./tippecanoe -q -z11 -Z11 -f -o tests/muni/decode/multi.mbtiles tests/muni/*.json
|
||||
./tippecanoe-decode -x generator -l subway tests/muni/decode/multi.mbtiles > tests/muni/decode/multi.mbtiles.json.check
|
||||
@@ -210,7 +242,7 @@ decode-test:
|
||||
cmp tests/muni/decode/multi.mbtiles.stats.json.check tests/muni/decode/multi.mbtiles.stats.json
|
||||
rm -f tests/muni/decode/multi.mbtiles.json.check tests/muni/decode/multi.mbtiles tests/muni/decode/multi.mbtiles.pipeline.json.check tests/muni/decode/multi.mbtiles.stats.json.check tests/muni/decode/multi.mbtiles.onetile.json.check
|
||||
|
||||
decode-pmtiles-test:
|
||||
decode-pmtiles-test: tippecanoe tippecanoe-decode
|
||||
mkdir -p tests/muni/decode
|
||||
./tippecanoe -q -z11 -Z11 -f -o tests/muni/decode/multi.pmtiles tests/muni/*.json
|
||||
./tippecanoe-decode -x generator -l subway tests/muni/decode/multi.pmtiles | sed 's/pmtiles/mbtiles/g' > tests/muni/decode/multi.pmtiles.json.check
|
||||
@@ -227,7 +259,7 @@ decode-pmtiles-test:
|
||||
cmp tests/muni/decode/multi.pmtiles.stats.json.check tests/muni/decode/multi.mbtiles.stats.json
|
||||
rm -f tests/muni/decode/multi.pmtiles.json.check tests/muni/decode/multi.pmtiles tests/muni/decode/multi.pmtiles.pipeline.json.check tests/muni/decode/multi.pmtiles.stats.json.check tests/muni/decode/multi.pmtiles.onetile.json.check
|
||||
|
||||
pbf-test:
|
||||
pbf-test: tippecanoe-decode
|
||||
./tippecanoe-decode -x generator tests/pbf/11-328-791.vector.pbf 11 328 791 > tests/pbf/11-328-791.vector.pbf.out
|
||||
cmp tests/pbf/11-328-791.json tests/pbf/11-328-791.vector.pbf.out
|
||||
rm tests/pbf/11-328-791.vector.pbf.out
|
||||
@@ -235,13 +267,129 @@ pbf-test:
|
||||
cmp tests/pbf/11-328-791.3857.json tests/pbf/11-328-791.3857.vector.pbf.out
|
||||
rm tests/pbf/11-328-791.3857.vector.pbf.out
|
||||
|
||||
enumerate-test:
|
||||
enumerate-test: tippecanoe tippecanoe-enumerate
|
||||
./tippecanoe -q -z5 -f -o tests/ne_110m_admin_0_countries/out/enum.mbtiles tests/ne_110m_admin_0_countries/in.json.gz
|
||||
./tippecanoe-enumerate tests/ne_110m_admin_0_countries/out/enum.mbtiles > tests/ne_110m_admin_0_countries/out/enum.check
|
||||
cmp tests/ne_110m_admin_0_countries/out/enum.check tests/ne_110m_admin_0_countries/out/enum
|
||||
rm tests/ne_110m_admin_0_countries/out/enum.mbtiles tests/ne_110m_admin_0_countries/out/enum.check
|
||||
|
||||
join-test: tile-join
|
||||
overzoom-test: tippecanoe-overzoom
|
||||
# Basic operation
|
||||
./tippecanoe-overzoom -o tests/pbf/13-1310-3166.pbf tests/pbf/11-327-791.pbf 11/327/791 13/1310/3166
|
||||
./tippecanoe-decode tests/pbf/13-1310-3166.pbf 13 1310 3166 > tests/pbf/13-1310-3166.pbf.json.check
|
||||
cmp tests/pbf/13-1310-3166.pbf.json.check tests/pbf/13-1310-3166.pbf.json
|
||||
rm tests/pbf/13-1310-3166.pbf tests/pbf/13-1310-3166.pbf.json.check
|
||||
# Make sure feature order is stable
|
||||
# Large -b20 tile buffer to prevent altering the geometries by clipping
|
||||
./tippecanoe-overzoom -b20 --preserve-input-order -o tests/pbf/11-327-791-out.pbf tests/pbf/11-327-791.pbf 11/327/791 11/327/791
|
||||
./tippecanoe-decode tests/pbf/11-327-791.pbf 11 327 791 > tests/pbf/11-327-791.json
|
||||
./tippecanoe-decode tests/pbf/11-327-791-out.pbf 11 327 791 > tests/pbf/11-327-791-out.json
|
||||
cmp tests/pbf/11-327-791.json tests/pbf/11-327-791-out.json
|
||||
rm tests/pbf/11-327-791.json tests/pbf/11-327-791-out.json tests/pbf/11-327-791-out.pbf
|
||||
# Basic operation, multiple input form
|
||||
./tippecanoe-overzoom -o tests/pbf/13-1310-3166.pbf -t 13/1310/3166 tests/pbf/11-327-791.pbf 11/327/791
|
||||
./tippecanoe-decode tests/pbf/13-1310-3166.pbf 13 1310 3166 > tests/pbf/13-1310-3166.pbf.json.check
|
||||
cmp tests/pbf/13-1310-3166.pbf.json.check tests/pbf/13-1310-3166.pbf.json
|
||||
rm tests/pbf/13-1310-3166.pbf tests/pbf/13-1310-3166.pbf.json.check
|
||||
# Multiple inputs, no compression
|
||||
./tippecanoe-overzoom --no-tile-compression -o tests/pbf/13-1310-3166-ne.pbf -t 13/1310/3166 tests/pbf/11-327-791.pbf 11/327/791 tests/pbf/0-0-0.pbf 0/0/0
|
||||
./tippecanoe-decode tests/pbf/13-1310-3166-ne.pbf 13 1310 3166 > tests/pbf/13-1310-3166-ne.pbf.json.check
|
||||
cmp tests/pbf/13-1310-3166-ne.pbf.json.check tests/pbf/13-1310-3166-ne.pbf.json
|
||||
rm tests/pbf/13-1310-3166-ne.pbf tests/pbf/13-1310-3166-ne.pbf.json.check
|
||||
# Different detail and buffer, and attribute stripping
|
||||
./tippecanoe-overzoom -d8 -b30 -y NAME -y name -y scalerank -o tests/pbf/13-1310-3166-8-30.pbf tests/pbf/11-327-791.pbf 11/327/791 13/1310/3166
|
||||
./tippecanoe-decode tests/pbf/13-1310-3166-8-30.pbf 13 1310 3166 > tests/pbf/13-1310-3166-8-30.pbf.json.check
|
||||
cmp tests/pbf/13-1310-3166-8-30.pbf.json.check tests/pbf/13-1310-3166-8-30.pbf.json
|
||||
rm tests/pbf/13-1310-3166-8-30.pbf tests/pbf/13-1310-3166-8-30.pbf.json.check
|
||||
# No features in child tile
|
||||
./tippecanoe-overzoom -o tests/pbf/14-2616-6331.pbf tests/pbf/11-327-791.pbf 11/327/791 14/2616/6331
|
||||
cmp tests/pbf/14-2616-6331.pbf /dev/null
|
||||
rm tests/pbf/14-2616-6331.pbf
|
||||
# Thinning
|
||||
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
|
||||
# 9 of them survive as the best of each cluster of 30
|
||||
# ./tippecanoe -z1 -r30 --retain-points-multiplier 30 -f -e out.dir tests/ne_110m_populated_places/in.json
|
||||
# cp out.dir/0/0/0.pbf tests/pbf/0-0-0-pop.pbf
|
||||
./tippecanoe-overzoom -y NAME -m -o tests/pbf/0-0-0-pop-filtered.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/pbf/0-0-0-pop-filtered.pbf 0 0 0 > tests/pbf/0-0-0-pop-filtered.pbf.json.check
|
||||
cmp tests/pbf/0-0-0-pop-filtered.pbf.json.check tests/pbf/0-0-0-pop-filtered.pbf.json
|
||||
rm tests/pbf/0-0-0-pop-filtered.pbf tests/pbf/0-0-0-pop-filtered.pbf.json.check
|
||||
# Thinning with accumulation
|
||||
./tippecanoe-overzoom -y NAME -m --accumulate-attribute NAME:comma -o tests/pbf/0-0-0-pop-accum.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/pbf/0-0-0-pop-accum.pbf 0 0 0 > tests/pbf/0-0-0-pop-accum.pbf.json.check
|
||||
cmp tests/pbf/0-0-0-pop-accum.pbf.json.check tests/pbf/0-0-0-pop-accum.pbf.json
|
||||
rm tests/pbf/0-0-0-pop-accum.pbf tests/pbf/0-0-0-pop-accum.pbf.json.check
|
||||
# Filtering
|
||||
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
|
||||
# 27 of them match the filter and are retained
|
||||
./tippecanoe-overzoom -y NAME -j'{"*":["SCALERANK","eq",0]}' -o tests/pbf/0-0-0-pop-expr.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/pbf/0-0-0-pop-expr.pbf 0 0 0 > tests/pbf/0-0-0-pop-expr.pbf.json.check
|
||||
cmp tests/pbf/0-0-0-pop-expr.pbf.json.check tests/pbf/0-0-0-pop-expr.pbf.json
|
||||
rm tests/pbf/0-0-0-pop-expr.pbf tests/pbf/0-0-0-pop-expr.pbf.json.check
|
||||
# Filtering with multiplier
|
||||
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
|
||||
# 8 features survive into the output, from 9 clusters of 30
|
||||
./tippecanoe-overzoom -y NAME -y SCALERANK -j'{"*":["SCALERANK","eq",0]}' -m -o tests/pbf/0-0-0-filter-mult.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/pbf/0-0-0-filter-mult.pbf 0 0 0 > tests/pbf/0-0-0-filter-mult.pbf.json.check
|
||||
cmp tests/pbf/0-0-0-filter-mult.pbf.json.check tests/pbf/0-0-0-filter-mult.pbf.json
|
||||
rm tests/pbf/0-0-0-filter-mult.pbf tests/pbf/0-0-0-filter-mult.pbf.json.check
|
||||
# Filtering with multiplier and preserve-input-order
|
||||
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
|
||||
./tippecanoe-overzoom -y NAME -y SCALERANK -j'{"*":["NAME","cn","e"]}' -m --preserve-input-order -o tests/pbf/0-0-0-filter-mult-order.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/pbf/0-0-0-filter-mult-order.pbf 0 0 0 > tests/pbf/0-0-0-filter-mult-order.pbf.json.check
|
||||
cmp tests/pbf/0-0-0-filter-mult-order.pbf.json.check tests/pbf/0-0-0-filter-mult-order.pbf.json
|
||||
rm tests/pbf/0-0-0-filter-mult-order.pbf tests/pbf/0-0-0-filter-mult-order.pbf.json.check
|
||||
# Test that overzooming with a multiplier exactly reverses the effect of tiling with a multiplier
|
||||
./tippecanoe -q -z5 --preserve-point-density-threshold 8 --retain-points-multiplier 3 -f -e tests/muni/out/out.dir tests/muni/muni.json
|
||||
./tippecanoe -q -z5 --preserve-point-density-threshold 8 -f -o tests/muni/out/out.mbtiles tests/muni/muni.json
|
||||
./tippecanoe-overzoom -m -o tests/muni/out/out.dir/000.pbf tests/muni/out/out.dir/0/0/0.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/muni/out/out.mbtiles 0 0 0 > tests/muni/out/out.dir/direct.json
|
||||
./tippecanoe-decode tests/muni/out/out.dir/000.pbf 0 0 0 > tests/muni/out/out.dir/overzoomed.json
|
||||
cmp tests/muni/out/out.dir/overzoomed.json tests/muni/out/out.dir/direct.json
|
||||
rm -rf tests/muni/out/out.dir tests/muni/out/out.mbtiles tests/muni/out/out.dir/overzoomed.json tests/muni/out/out.dir/direct.json
|
||||
# Test filter with null attribute
|
||||
./tippecanoe-overzoom -j '{"*":["name","ni",[1,5,6,9]]}' -o tests/pbf/12-2145-1391-filter1.pbf tests/pbf/12-2145-1391.pbf 12/2145/1391 12/2145/1391
|
||||
./tippecanoe-decode tests/pbf/12-2145-1391-filter1.pbf 12 2145 1391 > tests/pbf/12-2145-1391-filter1.pbf.json.check
|
||||
cmp tests/pbf/12-2145-1391-filter1.pbf.json.check tests/pbf/12-2145-1391-filter1.pbf.json
|
||||
rm tests/pbf/12-2145-1391-filter1.pbf.json.check tests/pbf/12-2145-1391-filter1.pbf
|
||||
# Test filter with null attribute in "ni" list
|
||||
./tippecanoe-overzoom -j '{"*":["name","ni",[1,5,6,9,null]]}' -o tests/pbf/12-2145-1391-filter2.pbf tests/pbf/12-2145-1391.pbf 12/2145/1391 12/2145/1391
|
||||
./tippecanoe-decode tests/pbf/12-2145-1391-filter2.pbf 12 2145 1391 > tests/pbf/12-2145-1391-filter2.pbf.json.check
|
||||
cmp tests/pbf/12-2145-1391-filter2.pbf.json.check tests/pbf/12-2145-1391-filter2.pbf.json
|
||||
rm tests/pbf/12-2145-1391-filter2.pbf.json.check tests/pbf/12-2145-1391-filter2.pbf
|
||||
# Tiny polygon reduction
|
||||
./tippecanoe-overzoom --line-simplification=5 --tiny-polygon-size=50 -o tests/pbf/countries-0-0-0.pbf.out tests/pbf/countries-0-0-0.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/pbf/countries-0-0-0.pbf.out 0 0 0 > tests/pbf/countries-0-0-0.pbf.out.json.check
|
||||
cmp tests/pbf/countries-0-0-0.pbf.out.json.check tests/pbf/countries-0-0-0.pbf.out.json
|
||||
rm tests/pbf/countries-0-0-0.pbf.out tests/pbf/countries-0-0-0.pbf.out.json.check
|
||||
# Binning
|
||||
./tippecanoe-overzoom -o tests/pbf/bin-11-327-791.pbf.out --assign-to-bins tests/pbf/sf-zips.json tests/pbf/muni-11-327-791.pbf 11/327/791 11/327/791
|
||||
./tippecanoe-decode tests/pbf/bin-11-327-791.pbf.out 11 327 791 > tests/pbf/bin-11-327-791.pbf.out.json.check
|
||||
cmp tests/pbf/bin-11-327-791.pbf.out.json.check tests/pbf/bin-11-327-791.pbf.out.json
|
||||
rm tests/pbf/bin-11-327-791.pbf.out.json.check tests/pbf/bin-11-327-791.pbf.out
|
||||
# Binning by id
|
||||
./tippecanoe-overzoom -o tests/pbf/bin-11-327-791-ids.pbf.out --assign-to-bins tests/pbf/sf-zips.json --bin-by-id bin-ids tests/pbf/yearbuilt.pbf 11/327/791 11/327/791
|
||||
./tippecanoe-decode tests/pbf/bin-11-327-791-ids.pbf.out 11 327 791 > tests/pbf/bin-11-327-791-ids.pbf.out.json.check
|
||||
cmp tests/pbf/bin-11-327-791-ids.pbf.out.json.check tests/pbf/bin-11-327-791-ids.pbf.out.json
|
||||
rm tests/pbf/bin-11-327-791-ids.pbf.out.json.check tests/pbf/bin-11-327-791-ids.pbf.out
|
||||
# Binning with longitude wraparound problems
|
||||
./tippecanoe-overzoom -o tests/pbf/0-0-0-pop-2-0-1.pbf.out --accumulate-numeric-attributes=tippecanoe --assign-to-bins tests/pbf/h3-2-0-1.geojson tests/pbf/0-0-0.pbf 2/0/1 2/0/1
|
||||
./tippecanoe-decode tests/pbf/0-0-0-pop-2-0-1.pbf.out 2 0 1 > tests/pbf/0-0-0-pop-2-0-1.pbf.out.json.check
|
||||
cmp tests/pbf/0-0-0-pop-2-0-1.pbf.out.json.check tests/pbf/0-0-0-pop-2-0-1.pbf.out.json
|
||||
rm tests/pbf/0-0-0-pop-2-0-1.pbf.out tests/pbf/0-0-0-pop-2-0-1.pbf.out.json.check
|
||||
./tippecanoe-overzoom -o tests/pbf/0-0-0-pop-1-1-0.pbf.out --accumulate-numeric-attributes=tippecanoe --assign-to-bins tests/pbf/h3-1-1-0.geojson tests/pbf/0-0-0.pbf 1/1/0 1/1/0
|
||||
./tippecanoe-decode tests/pbf/0-0-0-pop-1-1-0.pbf.out 1 1 0 > tests/pbf/0-0-0-pop-1-1-0.pbf.out.json.check
|
||||
cmp tests/pbf/0-0-0-pop-1-1-0.pbf.out.json.check tests/pbf/0-0-0-pop-1-1-0.pbf.out.json
|
||||
rm tests/pbf/0-0-0-pop-1-1-0.pbf.out tests/pbf/0-0-0-pop-1-1-0.pbf.out.json.check
|
||||
./tippecanoe-overzoom -o tests/pbf/0-0-0-pop-0-0-0.pbf.out --accumulate-numeric-attributes=tippecanoe --assign-to-bins tests/pbf/h3-0-0-0.geojson tests/pbf/0-0-0.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/pbf/0-0-0-pop-0-0-0.pbf.out 0 0 0 > tests/pbf/0-0-0-pop-0-0-0.pbf.out.json.check
|
||||
cmp tests/pbf/0-0-0-pop-0-0-0.pbf.out.json.check tests/pbf/0-0-0-pop-0-0-0.pbf.out.json
|
||||
rm tests/pbf/0-0-0-pop-0-0-0.pbf.out tests/pbf/0-0-0-pop-0-0-0.pbf.out.json.check
|
||||
# Verify fix for crash
|
||||
./tippecanoe-overzoom '-o' tests/10188-crash/out.pbf '-t' '3/2/2' '--assign-to-bins' 'tests/10188-crash/bins.json' '--bin-by-id-list' 'felt:bin_features' '-b5' 'tests/10188-crash/2-0-0.pbf' '2/0/0'
|
||||
rm tests/10188-crash/out.pbf
|
||||
|
||||
join-test: tippecanoe tippecanoe-decode tile-join
|
||||
./tippecanoe -q -f -z12 -o tests/join-population/tabblock_06001420.mbtiles -YALAND10:'Land area' -L'{"file": "tests/join-population/tabblock_06001420.json", "description": "population"}'
|
||||
./tippecanoe -q -f -Z5 -z10 -o tests/join-population/macarthur.mbtiles -l macarthur tests/join-population/macarthur.json
|
||||
./tile-join -q -f -Z6 -z9 -o tests/join-population/macarthur-6-9.mbtiles tests/join-population/macarthur.mbtiles
|
||||
@@ -329,6 +477,15 @@ join-test: tile-join
|
||||
./tippecanoe-decode -x generator tests/join-population/concat.mbtiles > tests/join-population/concat.mbtiles.json.check
|
||||
cmp tests/join-population/concat.mbtiles.json.check tests/join-population/concat.mbtiles.json
|
||||
rm tests/join-population/concat.mbtiles.json.check tests/join-population/concat.mbtiles tests/join-population/macarthur.mbtiles
|
||||
# Test reading list of input files from file
|
||||
./tippecanoe -q -f -Z5 -z10 -o tests/readfile/macarthur.mbtiles -l macarthur1 tests/join-population/macarthur.json
|
||||
./tippecanoe -q -f -Z5 -z10 -o tests/readfile/macarthur2.mbtiles -l macarthur2 tests/join-population/macarthur2.json
|
||||
./tile-join -q -R macarthur1:one --rename-layer=macarthur2:two -f -o tests/readfile/renamed.mbtiles tests/readfile/macarthur.mbtiles tests/readfile/macarthur2.mbtiles
|
||||
./tippecanoe-decode -x generator -x generator_options tests/readfile/renamed.mbtiles > tests/readfile/renamed.mbtiles.json.check
|
||||
./tile-join -q -R macarthur1:one --rename-layer=macarthur2:two -f -r tests/readfile/readfile.list -o tests/readfile/readfile.mbtiles
|
||||
./tippecanoe-decode -x generator -x generator_options tests/readfile/readfile.mbtiles > tests/readfile/readfile.mbtiles.json.check
|
||||
cmp tests/readfile/renamed.mbtiles.json.check tests/readfile/readfile.mbtiles.json.check
|
||||
rm tests/readfile/renamed.mbtiles.json.check tests/readfile/readfile.mbtiles.json.check tests/readfile/readfile.mbtiles tests/readfile/renamed.mbtiles
|
||||
#`
|
||||
# Make sure empty tilesets work
|
||||
#
|
||||
@@ -344,16 +501,130 @@ join-test: tile-join
|
||||
./tippecanoe-decode -x generator -x generator_options -x name -x description tests/join-population/empty.pmtiles > tests/join-population/empty.out.json.check
|
||||
cmp tests/join-population/empty.out.json.check tests/join-population/empty.out.json
|
||||
rm -f tests/join-population/empty.pmtiles tests/join-population/empty.out.pmtiles tests/join-population/empty.out.json.check
|
||||
# pmtiles again, with --overzoom
|
||||
./tippecanoe -q -z0 -f -o tests/join-population/empty.pmtiles tests/join-population/empty.json
|
||||
./tile-join --overzoom -f -o tests/join-population/empty.out.pmtiles tests/join-population/empty.pmtiles
|
||||
./tippecanoe-decode -x generator -x generator_options -x name -x description tests/join-population/empty.pmtiles > tests/join-population/empty.out.json.check
|
||||
cmp tests/join-population/empty.out.json.check tests/join-population/empty.out.json
|
||||
rm -f tests/join-population/empty.pmtiles tests/join-population/empty.out.pmtiles tests/join-population/empty.out.json.check
|
||||
# dirtiles:
|
||||
./tippecanoe -q -z0 -f -e tests/join-population/empty.dirtiles tests/join-population/empty.json
|
||||
./tile-join -f -e tests/join-population/empty.out.dirtiles tests/join-population/empty.dirtiles
|
||||
./tippecanoe-decode -x generator -x generator_options -x name -x description tests/join-population/empty.dirtiles > tests/join-population/empty.out.json.check
|
||||
cmp tests/join-population/empty.out.json.check tests/join-population/empty.out.json
|
||||
rm -rf tests/join-population/empty.dirtiles tests/join-population/empty.out.dirtiles tests/join-population/empty.out.json.check
|
||||
#
|
||||
# Test overzooming of tilesets with different maxzooms
|
||||
#
|
||||
mkdir -p tests/ne_110m_ocean/join
|
||||
./tippecanoe -q -z2 -f -o tests/ne_110m_ocean/join/ocean.mbtiles tests/ne_110m_ocean/in.json
|
||||
./tippecanoe -q -z4 -d8 -y name -f -o tests/ne_110m_ocean/join/countries.mbtiles tests/ne_110m_admin_0_countries/in.json.gz
|
||||
./tile-join --overzoom -f -o tests/ne_110m_ocean/join/joined.mbtiles tests/ne_110m_ocean/join/ocean.mbtiles tests/ne_110m_ocean/join/countries.mbtiles
|
||||
./tippecanoe-decode -x generator tests/ne_110m_ocean/join/joined.mbtiles > tests/ne_110m_ocean/join/joined.mbtiles.json.check
|
||||
cmp tests/ne_110m_ocean/join/joined.mbtiles.json.check tests/ne_110m_ocean/join/joined.mbtiles.json
|
||||
rm -f tests/ne_110m_ocean/join/ocean.mbtiles tests/ne_110m_ocean/join/countries.mbtiles tests/ne_110m_ocean/join/joined.mbtiles tests/ne_110m_ocean/join/joined.mbtiles.json.check
|
||||
|
||||
accumulate-test:
|
||||
# there are 144 features with POP1950 in the original dataset
|
||||
test `grep '"POP1950": [0-9]' tests/ne_110m_populated_places_nulls/in.json | wc -l` == 144
|
||||
# and 99 without it
|
||||
test `grep '"POP1950": null' tests/ne_110m_populated_places_nulls/in.json | wc -l` == 99
|
||||
./tippecanoe -yNAME -yPOP1950 -yclustered:cluster_size -yclustered:unrelated -q -z3 -r1.75 -b0 -f -e tests/pbf/accum.dir --accumulate-numeric-attributes=clustered --set-attribute '{"clustered:cluster_size":1}' --accumulate-attribute '{"clustered:cluster_size":"sum"}' --retain-points-multiplier 3 tests/ne_110m_populated_places_nulls/in.json
|
||||
# at this drop rate, there are 61 points at z0 that have no POP1950s clustered onto them....
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | wc -l` == 61
|
||||
# 26 of which have no POP1950 at all
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'POP1950' | wc -l` == 26
|
||||
# 35 of which do have POP1950
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | grep 'POP1950' | wc -l` == 35
|
||||
# plus 60 that are clustered
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep 'clustered:count:POP1950' | wc -l` == 60
|
||||
# the 60 clustered POP1950s have a total count of 109
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep 'clustered:count:POP1950' | sed 's/.*"clustered:count:POP1950": //' | awk '{sum += $$1} END {print sum}'` == 109
|
||||
# we have already established that there are 36 bare POP1950s
|
||||
# which makes a total of 144, which is the total count expected
|
||||
#
|
||||
# meanwhile, regular attribute accumulation.
|
||||
# there are 121 features in the z0 tile, and they all have clustered:cluster_size
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep 'clustered:cluster_size' | wc -l` == 121
|
||||
# there are no features that lack it.
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'clustered:cluster_size' | wc -l` == 0
|
||||
# they add up to the 243 original features
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | sed 's/.*clustered:cluster_size": //' | awk '{sum += $$1} END {print sum}'` == 243
|
||||
# Make sure we do *not* accumulate a numeric attribute that already has the magic prefix:
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep sum:clustered:unrelated | wc -l` == 0
|
||||
# But that we *do* preserve those attributes into the output features:
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep clustered:unrelated | wc -l` == 61
|
||||
#
|
||||
# on to the sums:
|
||||
# in the original data set, the POP1950s that are present add up to 161590
|
||||
test `grep '"POP1950": [0-9]' tests/ne_110m_populated_places_nulls/in.json | sed 's/.*"POP1950": //' | awk '{sum += $$1} END {print sum}' ` == 161590
|
||||
# in the z0 tile, the clustered POP1950s add up to 116967
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep 'clustered:sum:POP1950' | sed 's/.*"clustered:sum:POP1950": //' | awk '{sum += $$1} END {print sum}'` == 116967
|
||||
# and the non-clustered ones add up to 44623
|
||||
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'clustered:sum:POP1950' | grep POP1950 | sed 's/.*"POP1950": //' | awk '{sum += $$1} END {print sum}'` == 44623
|
||||
# which is the correct 161590
|
||||
#
|
||||
# OK, so do these still hold after megatile filtering?
|
||||
./tippecanoe-overzoom --accumulate-numeric-attributes=clustered --accumulate-attribute '{"clustered:cluster_size":"sum"}' -m -o tests/pbf/accum-0-0-0.pbf tests/pbf/accum.dir/0/0/0.pbf 0/0/0 0/0/0
|
||||
# Now there are 40 features with POP1950 clusters
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep 'clustered:count:POP1950' | wc -l` == 40
|
||||
# There are 4 with bare POP1950
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | grep 'POP1950' | wc -l` == 4
|
||||
# And 2 with no POP1950 at all
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep -v 'POP1950' | wc -l` == 2
|
||||
# (which is the same as you get if you don't use -retain-points-multiplier when creating the tileset)
|
||||
#
|
||||
# the clustered and megatile-filtered POP1950s add up to 146370
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep 'clustered:sum:POP1950' | sed 's/.*"clustered:sum:POP1950": //' | awk '{sum += $$1} END {print sum}'` == 146370
|
||||
# the non-clustered but megatile-filtered POP1950s add up to 15220
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep -v 'clustered:sum:POP1950' | grep POP1950 | sed 's/.*"POP1950": //' | awk '{sum += $$1} END {print sum}'` == 15220
|
||||
# which add up to 161590 so we have the right global total
|
||||
# Make sure we do *not* accumulate a numeric attribute that already has the magic prefix:
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep sum:clustered:unrelated | wc -l` == 0
|
||||
# But that we *do* preserve those attributes into the output features:
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep clustered:unrelated | wc -l` == 22
|
||||
# the cluster sizes still add up to the 243 original features
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | sed 's/.*clustered:cluster_size": //' | awk '{sum += $$1} END {print sum}'` == 243
|
||||
#
|
||||
# We actually want to serve point tiles without the numeric accumulations,
|
||||
# but with cluster size, so test that combination:
|
||||
./tippecanoe-overzoom --accumulate-attribute '{"clustered:cluster_size":"sum"}' --exclude-prefix clustered:sum --exclude-prefix clustered:count --exclude-prefix clustered:min --exclude-prefix clustered:max --exclude-prefix clustered:mean -m -o tests/pbf/accum-0-0-0.pbf tests/pbf/accum.dir/0/0/0.pbf 0/0/0 0/0/0
|
||||
# There are no POP1950 clusters
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep 'clustered:count:POP1950' | wc -l` == 0
|
||||
# But there are still 28 with bare POP1950
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | grep 'POP1950' | wc -l` == 28
|
||||
# And 18 with no POP1950 at all
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep -v 'POP1950' | wc -l` == 18
|
||||
# which matches the 46 features that you get if you tile without --retain-points-multiplier.
|
||||
# the cluster sizes still add up to the 243 original features
|
||||
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | sed 's/.*clustered:cluster_size": //' | awk '{sum += $$1} END {print sum}'` == 243
|
||||
#
|
||||
# Now on to binning!
|
||||
./tippecanoe-overzoom --assign-to-bins tests/pbf/h3-0-0-0.geojson --accumulate-numeric-attributes=clustered --accumulate-attribute '{"clustered:cluster_size":"sum"}' -o tests/pbf/bins-0-0-0.pbf tests/pbf/accum.dir/0/0/0.pbf 0/0/0 0/0/0
|
||||
# Now there are 30 bins with POP1950 clusters
|
||||
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep 'clustered:count:POP1950' | wc -l` == 41
|
||||
# There are none with bare POP1950 (which is expected; we should only have summary statistics)
|
||||
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | grep 'POP1950' | wc -l` == 0
|
||||
# And 4 with no POP1950 at all
|
||||
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep -v 'POP1950' | wc -l` == 3
|
||||
#
|
||||
# the clustered and megatile-filtered and binned POP1950s add up to 161590
|
||||
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep 'clustered:sum:POP1950' | sed 's/.*"clustered:sum:POP1950": //' | awk '{sum += $$1} END {print sum}'` == 161590
|
||||
# which is the right global total
|
||||
# Make sure we do *not* accumulate a numeric attribute that already has the magic prefix:
|
||||
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep sum:clustered:unrelated | wc -l` == 0
|
||||
# And those attributes do *not* make it onto the bins
|
||||
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep clustered:unrelated | wc -l` == 0
|
||||
# the cluster sizes still add up to the 243 original features
|
||||
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | sed 's/.*clustered:cluster_size": //' | awk '{sum += $$1} END {print sum}'` == 243
|
||||
#
|
||||
#
|
||||
# A tile where the counts and means were previously wrong:
|
||||
./tippecanoe-overzoom --accumulate-numeric-attributes=felt -m -o tests/pbf/yearbuilt-accum.pbf tests/pbf/yearbuilt.pbf 0/0/0 0/0/0
|
||||
./tippecanoe-decode tests/pbf/yearbuilt-accum.pbf 0 0 0 > tests/pbf/yearbuilt-accum.pbf.json.check
|
||||
cmp tests/pbf/yearbuilt-accum.pbf.json.check tests/pbf/yearbuilt-accum.pbf.json
|
||||
|
||||
|
||||
join-filter-test:
|
||||
join-filter-test: tippecanoe tippecanoe-decode tile-join
|
||||
# Comes out different from the direct tippecanoe run because null attributes are lost
|
||||
./tippecanoe -q -z0 -f -o tests/feature-filter/out/all.mbtiles tests/feature-filter/in.json
|
||||
./tile-join -q -J tests/feature-filter/filter -f -o tests/feature-filter/out/filtered.mbtiles tests/feature-filter/out/all.mbtiles
|
||||
@@ -376,7 +647,7 @@ json-tool-test: tippecanoe-json-tool
|
||||
rm -f tests/join-population/tabblock_06001420.json.sort tests/join-population/tabblock_06001420.json.sort.joined
|
||||
rm -f tests/join-population/tabblock_06001420-null.json.sort.joined
|
||||
|
||||
allow-existing-test:
|
||||
allow-existing-test: tippecanoe
|
||||
# Make a tileset
|
||||
./tippecanoe -q -Z0 -z0 -f -o tests/allow-existing/both.mbtiles tests/coalesce-tract/tl_2010_06001_tract10.json
|
||||
# Writing to existing should fail
|
||||
@@ -405,7 +676,7 @@ allow-existing-test:
|
||||
if ./tippecanoe -q -Z10 -z11 -F -o tests/allow-existing/both.pmtiles tests/coalesce-tract/tl_2010_06001_tract10.json; then exit 1; else exit 0; fi
|
||||
rm -r tests/allow-existing/both.pmtiles tests/allow-existing/both.dir.json.check tests/allow-existing/both.dir tests/allow-existing/both.mbtiles.json.check tests/allow-existing/both.mbtiles
|
||||
|
||||
csv-test:
|
||||
csv-test: tippecanoe tippecanoe-decode
|
||||
# Reading from named CSV
|
||||
./tippecanoe -q -zg -f -o tests/csv/out.mbtiles tests/csv/ne_110m_populated_places_simple.csv
|
||||
./tippecanoe-decode -x generator -x generator_options tests/csv/out.mbtiles > tests/csv/out.mbtiles.json.check
|
||||
@@ -422,7 +693,7 @@ csv-test:
|
||||
cmp tests/csv/out.mbtiles.json.check tests/csv/out.mbtiles.json
|
||||
rm -f tests/csv/out.mbtiles.json.check tests/csv/out.mbtiles
|
||||
|
||||
layer-json-test:
|
||||
layer-json-test: tippecanoe tippecanoe-decode
|
||||
# GeoJSON with description and named layer
|
||||
./tippecanoe -q -z0 -r1 -yNAME -f -o tests/layer-json/out.mbtiles -L'{"file":"tests/ne_110m_populated_places/in.json", "description":"World cities", "layer":"places"}'
|
||||
./tippecanoe-decode -x generator -x generator_options tests/layer-json/out.mbtiles > tests/layer-json/out.mbtiles.json.check
|
||||
@@ -440,7 +711,7 @@ layer-json-test:
|
||||
prep-test: $(TESTS)
|
||||
|
||||
tests/%.json: Makefile tippecanoe tippecanoe-decode
|
||||
./tippecanoe -q -a@ -f -o $@.check.mbtiles $(subst @,:,$(subst %,/,$(subst _, ,$(patsubst %.json,%,$(word 4,$(subst /, ,$@)))))) $(foreach suffix,$(suffixes),$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix))))
|
||||
./tippecanoe -q -a@ -f -o $@.check.mbtiles $(call testargs,$(patsubst %.json,%,$(word 4,$(subst /, ,$@)))) $(foreach suffix,$(suffixes),$(sort $(wildcard $(subst $(SPACE),/,$(wordlist 1,2,$(subst /, ,$@)))/*.$(suffix))))
|
||||
./tippecanoe-decode -x generator $@.check.mbtiles > $@
|
||||
cmp $(patsubst %.check,%,$@) $@
|
||||
rm $@.check.mbtiles
|
||||
|
||||
@@ -6,8 +6,11 @@ Builds [vector tilesets](https://github.com/mapbox/vector-tile-spec/) from large
|
||||
|
||||
This is the official home of Tippecanoe, developed and actively maintained by [Erica Fischer](https://github.com/e-n-f) at [Felt](https://felt.com).
|
||||
|
||||
For a self-hosted, API driven version of Tippecanoe, contact a technical sales engineer at sales@felt.com. Felt produces highly performant, automatically projected versions of your data, and utilizes a rendering engine, built on top of [MapLibre GL JS](https://github.com/maplibre/maplibre-gl-js), to style vector and raster data.
|
||||
|
||||
Version 2.0.0 is equivalent to [1.36.0](https://github.com/mapbox/tippecanoe/tree/1.36.0) in the original repository. Thank you Mapbox for the many years of early support.
|
||||
|
||||
|
||||
Intent
|
||||
------
|
||||
|
||||
@@ -32,7 +35,13 @@ significant travel corridor.
|
||||
Installation
|
||||
------------
|
||||
|
||||
For this fork you will need to build from the source repository:
|
||||
The easiest way to install tippecanoe on OSX is with [Homebrew](http://brew.sh/):
|
||||
|
||||
```sh
|
||||
$ brew install tippecanoe
|
||||
```
|
||||
|
||||
On Ubuntu it will usually be easiest to build from the source repository:
|
||||
|
||||
```sh
|
||||
$ git clone https://github.com/felt/tippecanoe.git
|
||||
@@ -53,7 +62,7 @@ $ tippecanoe -o file.mbtiles [options] [file.json file.json.gz file.fgb ...]
|
||||
```
|
||||
|
||||
If no files are specified, it reads GeoJSON from the standard input.
|
||||
If multiple files are specified, each is placed in its own layer.
|
||||
If multiple files are specified, each is placed [in its own layer](#input-files-and-layer-names).
|
||||
|
||||
The GeoJSON features need not be wrapped in a FeatureCollection.
|
||||
You can concatenate multiple GeoJSON features or files together,
|
||||
@@ -341,6 +350,9 @@ Parallel processing will also be automatic if the input file is in FlatGeobuf fo
|
||||
* `-ae` or `--extend-zooms-if-still-dropping`: Increase the maxzoom if features are still being dropped at that zoom level.
|
||||
The detail and simplification options that ordinarily apply only to the maximum zoom level will apply both to the originally
|
||||
specified maximum zoom and to any levels added beyond that.
|
||||
* `--extend-zooms-if-still-dropping-maximum=`_count_: Increase the maxzoom if features are still being dropped at that zoom level
|
||||
by up to _count_ zoom levels.
|
||||
* `-at` or `--generate-variable-depth-tile-pyramid`: Don't produce child tiles for any tile that should be sufficient to be overzoomed to any higher zoom level. Such tiles will be produced with maximum detail and no simplification or polygon cleaning. Tiles with point features below the basezoom or where any features have to be dropped dynamically, or which contain too many features or bytes with full detail, will be written out with normal detail and split into child tiles. Tilesets generated with this option are suitable for use only with tile servers that will find the appropriate tile to overzoom from and will simplify and clean the geometries appropriately before serving the tile.
|
||||
* `-R` _zoom_`/`_x_`/`_y_ or `--one-tile=`_zoom_`/`_x_`/`_y_: Set the minzoom and maxzoom to _zoom_ and produce only
|
||||
the single specified tile at that zoom level.
|
||||
|
||||
@@ -372,8 +384,8 @@ zoom level | precision (ft) | precision (m) | map scale
|
||||
`-z18` | 1.5 in | 4 cm | 1:1250
|
||||
`-z19` | 0.8 in | 2 cm | 1:600
|
||||
`-z20` | 0.4 in | 1 cm | 1:300
|
||||
`-z21` | 0.2 in | 0.5 cm | 1:150
|
||||
`-z22` | 0.1 in | 0.25 cm | 1:75
|
||||
`-z21` | 0.4 in | 1 cm | 1:300
|
||||
`-z22` | 0.4 in | 1 cm | 1:300
|
||||
|
||||
### Tile resolution
|
||||
|
||||
@@ -405,14 +417,18 @@ be reduced to the maximum that can be used with the specified _maxzoom_.
|
||||
that are dropped, coalesced-as-needed, or clustered. The _operation_ may be
|
||||
`sum`, `product`, `mean`, `max`, `min`, `concat`, or `comma`
|
||||
to specify how the named _attribute_ is accumulated onto the attribute of the same name in a feature that does survive.
|
||||
The attributes and operations may also be specified as JSON keys and values: `--accumulate-attribute='{"attr": "operation", "attr2", "operation2"}'`.
|
||||
* `--set-attribute` _attribute_`:`_value_: Set the value of the specified _attribute_ in each feature to the specified _value_. This is mostly useful to give an attribute in each feature an initial value for `--accumulate-attribute`.
|
||||
The attributes and values may also be specified as JSON keys and values: `--set-attribute='{"attr": value, "attr2", value}'`.
|
||||
* `-pe` or `--empty-csv-columns-are-null`: Treat empty CSV columns as nulls rather than as empty strings.
|
||||
* `-aI` or `--convert-stringified-ids-to-numbers`: If a feature ID is the string representation of a number, convert it to a plain number to use as the feature ID.
|
||||
* `--use-attribute-for-id=`*name*: Use the attribute with the specified *name* as if it were specified as the feature ID. (If this attribute is a stringified number, you must also use `-aI` to convert it to a number.)
|
||||
* `-pN` or `--single-precision`: Write double-precision numeric attribute values to tiles as single-precision to reduce tile size.
|
||||
* `--maximum-string-attribute-length`=_length_: Truncate string attributes that exceed the specified length in bytes.
|
||||
|
||||
### Filtering features by attributes
|
||||
|
||||
* `-j` *filter* or `--feature-filter`=*filter*: Check features against a per-layer filter (as defined in the [Mapbox GL Style Specification](https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter)) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer `"*"` apply to all layers. The special variable `$zoom` refers to the current zoom level.
|
||||
* `-j` *filter* or `--feature-filter`=*filter*: Check features against a per-layer filter (as defined in the [Mapbox GL Style Specification](https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter) or in a Felt filter specification still to be finalized) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer `"*"` apply to all layers. The special variable `$zoom` refers to the current zoom level.
|
||||
* `-J` *filter-file* or `--feature-filter-file`=*filter-file*: Like `-j`, but read the filter from a file.
|
||||
|
||||
Example: to find the Natural Earth countries with low `scalerank` but high `LABELRANK`:
|
||||
@@ -450,12 +466,16 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
|
||||
If you use `-Bg`, it will guess a zoom level that will keep at most 50,000 features in the densest tile.
|
||||
You can also specify a marker-width with `-Bg`*width* to allow fewer features in the densest tile to
|
||||
compensate for the larger marker, or `-Bf`*number* to allow at most *number* features in the densest tile.
|
||||
* `--retain-points-multiplier=`_multiple_: Retain the specified multiple of points instead of just the number of points that would ordinarily be retained by the drop rate. These can be thinned out later with the `-m` option to `tippecanoe-overzoom`. The start of each cluster is marked in the feature sequence by the `tippecanoe:retain_points_multiplier_first` attribute. The `--tile-size-limit` will also be extended at low zoom levels to allow for the multiplied features.
|
||||
* `--drop-denser=`_percentage_: When dropping dots at zoom levels below the base zoom, give the specified _percentage_
|
||||
preference to retaining points in sparse areas and dropping points in dense areas.
|
||||
* `--limit-base-zoom-to-maximum-zoom` or `-Pb`: Limit the guessed base zoom not to exceed the maxzoom, even if this would put more than the requested number of features in a base zoom tile.
|
||||
* `-al` or `--drop-lines`: Let "dot" dropping at lower zooms apply to lines too
|
||||
* `-ap` or `--drop-polygons`: Let "dot" dropping at lower zooms apply to polygons too
|
||||
* `-K` _distance_ or `--cluster-distance=`_distance_: Cluster points (as with `--cluster-densest-as-needed`, but without the experimental discovery process) that are approximately within _distance_ of each other. The units are tile coordinates within a nominally 256-pixel tile, so the maximum value of 255 allows only one feature per tile. Values around 10 are probably appropriate for typical marker sizes. See `--cluster-densest-as-needed` below for behavior.
|
||||
* `-k` _zoom_ or `--cluster-maxzoom=`_zoom_: Max zoom on which to cluster points if clustering is enabled.
|
||||
* `-kg` or `--cluster-maxzoom=g`: Set `--cluster-maxzoom=` to `maxzoom - 1` so that all features are visible at the maximum zoom level.
|
||||
* `--preserve-point-density-threshold=`_level_: At the low zoom levels, do not reduce point density below the specified _level_, even if the specfied drop rate would normally call for it, so that low-density areas of the map do not appear blank. The unit is the distance between preserved points, as a fraction of the size of a tile. Values of 32 or 64 are probably appropriate for typical marker sizes.
|
||||
|
||||
### Dropping a fraction of features to keep under tile size limits
|
||||
|
||||
@@ -480,14 +500,15 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
|
||||
* `-ps` or `--no-line-simplification`: Don't simplify lines and polygons
|
||||
* `-pS` or `--simplify-only-low-zooms`: Don't simplify lines and polygons at maxzoom (but do simplify at lower zooms)
|
||||
* `--simplification-at-maximum-zoom=`_scale_: Use the specified _scale_ at maxzoom instead of the standard simplification scale (which still applies at lower zooms)
|
||||
* `-pn` or `--no-simplification-of-shared-nodes`: Don't simplify away nodes that appear in more than one feature or are used multiple times within the same feature, so that the intersection node will not be lost from intersecting roads. (This will not be effective if you also use `--coalesce` or `--detect-shared-borders`.)
|
||||
* `-pn` or `--no-simplification-of-shared-nodes`: Don't simplify away nodes at which LineStrings or Polygon rings converge, diverge, or cross. (This will not be effective if you also use `--coalesce`.) In between intersection nodes, LineString segments or polygon edges will be simplified identically in each feature if possible. Use this instead of `--detect-shared-borders`.
|
||||
* `-pt` or `--no-tiny-polygon-reduction`: Don't combine the area of very small polygons into small squares that represent their combined area.
|
||||
* `-pT` or `--no-tiny-polygon-reduction-at-maximum-zoom`: Combine the area of very small polygons into small squares that represent their combined area only at zoom levels below the maximum.
|
||||
* `--tiny-polygon-size=`_size_: Use the specified _size_ for tiny polygons instead of the default 2. Anything above 6 or so will lead to visible artifacts with the default tile detail.
|
||||
* `-av` or `--visvalingam`: Use Visvalingam's simplification algorithm rather than Douglas-Peucker's.
|
||||
|
||||
### Attempts to improve shared polygon boundaries
|
||||
|
||||
* `-ab` or `--detect-shared-borders`: In the manner of [TopoJSON](https://github.com/mbostock/topojson/wiki/Introduction), detect borders that are shared between multiple polygons and simplify them identically in each polygon. This takes more time and memory than considering each polygon individually.
|
||||
* `-ab` or `--detect-shared-borders`: DEPRECATED. In the manner of [TopoJSON](https://github.com/mbostock/topojson/wiki/Introduction), detect borders that are shared between multiple polygons and simplify them identically in each polygon. This takes more time and memory than considering each polygon individually. Use `no-simplification-of-shared-nodes` instead, which is faster and more correct.
|
||||
* `-aL` or `--grid-low-zooms`: At all zoom levels below _maxzoom_, snap all lines and polygons to a stairstep grid instead of allowing diagonals. You will also want to specify a tile resolution, probably `-D8`. This option provides a way to display continuous parcel, gridded, or binned data at low zooms without overwhelming the tiles with tiny polygons, since features will either get stretched out to the grid unit or lost entirely, depending on how they happened to be aligned in the original data. You probably don't want to use this.
|
||||
|
||||
### Controlling clipping to tile boundaries
|
||||
@@ -512,6 +533,7 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
|
||||
|
||||
* `-ag` or `--calculate-feature-density`: Add a new attribute, `tippecanoe_feature_density`, to each feature, to record how densely features are spaced in that area of the tile. You can use this attribute in the style to produce a glowing effect where points are densely packed. It can range from 0 in the sparsest areas to 255 in the densest.
|
||||
* `-ai` or `--generate-ids`: Add an `id` (a feature ID, not an attribute named `id`) to each feature that does not already have one. There is currently no guarantee that the `id` added will be stable between runs or that it will not conflict with manually-assigned feature IDs. Future versions of Tippecanoe may change the mechanism for allocating IDs.
|
||||
* `-aX` or `--calculate-feature-index`: Add a `tippecanoe:index` field to each feature, giving its index in the quadkey or hilbert sequence.
|
||||
|
||||
### Trying to correct bad source geometry
|
||||
|
||||
@@ -672,7 +694,7 @@ this minimum size and others will not be drawn at all, preserving the total area
|
||||
all of them should have had together.
|
||||
|
||||
Features in the same tile that share the same type and attributes are coalesced
|
||||
together into a single geometry if you use `--coalesce`. You are strongly encouraged to use -x to exclude
|
||||
together into a single geometry if you use `--coalesce`. You are strongly encouraged to use `-x` to exclude
|
||||
any unnecessary attributes to reduce wasted file size.
|
||||
|
||||
If a tile is larger than 500K, it will try encoding that tile at progressively
|
||||
@@ -686,7 +708,7 @@ uses md2man (`gem install md2man`).
|
||||
|
||||
Linux:
|
||||
|
||||
sudo apt-get install build-essential libsqlite3-dev zlib1g-dev
|
||||
sudo apt-get install gcc g++ make libsqlite3-dev zlib1g-dev
|
||||
|
||||
Then build:
|
||||
|
||||
@@ -757,6 +779,12 @@ The options are:
|
||||
* `-o` *out.mbtiles*, *out.pmtiles* or `--output=`*out.mbtiles*: Write the new tiles to the specified .mbtiles file.
|
||||
* `-e` *directory* or `--output-to-directory=`*directory*: Write the new tiles to the specified directory instead of to an mbtiles file.
|
||||
* `-f` or `--force`: Remove *out.mbtiles* if it already exists.
|
||||
* `-r` or `--read-from`: list of input mbtiles to read from.
|
||||
|
||||
### Overzooming
|
||||
|
||||
* `--overzoom`: If one of the source tilesets has a larger maxzoom than the others, scale up tiles from the tilesets with the lower maxzooms so they will all have the same maxzoom in the output tileset.
|
||||
* `--buffer=`_pixels_ or `-b` _pixels_: Set the size of the tile buffer in the overzoomed tiles.
|
||||
|
||||
### Tileset description and attribution
|
||||
|
||||
@@ -781,8 +809,9 @@ The options are:
|
||||
|
||||
### Filtering features and feature attributes
|
||||
|
||||
* `-x` *key* or `--exclude=`*key*: Remove attributes of type *key* from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want.
|
||||
* `-x` *key* or `--exclude=`*key*: Remove attributes named *key* from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want. You can use multiple `-x` options to remove multiple attributes.
|
||||
* `-X` or `--exclude-all`: Remove all attributes from the output.
|
||||
* `-y` *key* or `--include=`*key*: Remove all attributes except for those named *key* from the output. You can use multiple `-y` options to retain multiple attributes.
|
||||
* `-i` or `--if-matched`: Only include features that matched the CSV.
|
||||
* `-j` *filter* or `--feature-filter`=*filter*: Check features against a per-layer filter (as defined in the [Mapbox GL Style Specification](https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter)) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer `"*"` apply to all layers.
|
||||
* `-J` *filter-file* or `--feature-filter-file`=*filter-file*: Like `-j`, but read the filter from a file.
|
||||
@@ -952,3 +981,33 @@ Join block geometries to employment attributes:
|
||||
```
|
||||
$ tippecanoe-json-tool -c in_wac_S000_JT00_2015.csv tl_2010_18157_tabblock10.sort.json > blocks-wac.json
|
||||
```
|
||||
|
||||
tippecanoe-overzoom
|
||||
===================
|
||||
|
||||
The `tippecanoe-overzoom` utility creates a vector tile from one of its parent tiles,
|
||||
clipping and scaling the geometry from the parent tile and excluding features that
|
||||
are clipped away. The idea is that if you create very high resolution tiles
|
||||
(using `--extra-detail`) at a moderate zoom level, you can use `tippecanoe-overzoom`
|
||||
to turn those into moderate detail tiles at high zoom levels, for the benefit of
|
||||
renderers that cannot internally overzoom high-resolution tiles without losing
|
||||
some of the precision. Running:
|
||||
|
||||
tippecanoe-overzoom -o out.mvt.gz in.mvt.gz inz/inx/iny outz/outx/outy
|
||||
|
||||
reads tile `inz/inx/iny` of `in.mvt.gz` and produces tile `outz/outx/outy` of `out.mvt.gz`.
|
||||
|
||||
tippecanoe-overzoom -o out.mvt.gz -t outz/outx/outy in.mvt.gz inz/inx/iny in2.mvt.gz in2z/in2x/in2y in3.mvt.gz in3z/in3x/in3y
|
||||
|
||||
reads tile `inz/inx/iny` of `in.mvt.gz`, tile `in2z/in2x/in2y` of `in2.mvt.gz`, and tile `in3z/in3x/in3y` of `in3.mvt.gz`,
|
||||
and produces tile `outz/outx/outy` of `out.mvt.gz` from them.
|
||||
|
||||
### Options
|
||||
|
||||
* `-b` *buffer*: Set the tile buffer in the output tile (default 5)
|
||||
* `-d` *detail*: Set the detail of the output tile (default 12)
|
||||
* `-y` *attribute*: Retain the specified *attribute* in the output features. All attributes that are not named in a `-y` option will be removed.
|
||||
* `-j` *filter*: Filter features using the same expression syntax as in tippecanoe.
|
||||
* `-m`: If a tile was created with the `--retain-points-multiplier` option, thin the tile back down to its normal feature count during overzooming. The first feature from each cluster will be retained, unless `-j` is used to specify a filter, in which case the first matching filter from each cluster will be retained instead.
|
||||
* `--preserve-input-order`: Restore a set of filtered features to its original input order
|
||||
* `--accumulate-attribute`: Behaves as in `tippecanoe` to sum attributes from the features of a multiplier cluster that are not included in the final output. The attributes from features that are filtered away with `-j` are *not* accumulated onto the output feature.
|
||||
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
#include <string>
|
||||
#include <unordered_map>
|
||||
#include "attribute.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "serial.hpp"
|
||||
#include "jsonpull/jsonpull.h"
|
||||
#include "milo/dtoa_milo.h"
|
||||
|
||||
std::map<std::string, attribute_op> numeric_operations = {
|
||||
{"sum", op_sum},
|
||||
{"min", op_min},
|
||||
{"max", op_max},
|
||||
{"count", op_count},
|
||||
};
|
||||
|
||||
void set_attribute_accum(std::unordered_map<std::string, attribute_op> &attribute_accum, std::string name, std::string type) {
|
||||
attribute_op t;
|
||||
|
||||
if (type == "sum") {
|
||||
t = op_sum;
|
||||
} else if (type == "product") {
|
||||
t = op_product;
|
||||
} else if (type == "mean") {
|
||||
t = op_mean;
|
||||
} else if (type == "max") {
|
||||
t = op_max;
|
||||
} else if (type == "min") {
|
||||
t = op_min;
|
||||
} else if (type == "concat") {
|
||||
t = op_concat;
|
||||
} else if (type == "comma") {
|
||||
t = op_comma;
|
||||
} else if (type == "count") {
|
||||
t = op_count;
|
||||
} else {
|
||||
fprintf(stderr, "Attribute method (%s) must be sum, product, mean, max, min, concat, comma, or count\n", type.c_str());
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
|
||||
attribute_accum.insert(std::pair<std::string, attribute_op>(name, t));
|
||||
}
|
||||
|
||||
void set_attribute_accum(std::unordered_map<std::string, attribute_op> &attribute_accum, const char *arg, char **argv) {
|
||||
if (*arg == '{') {
|
||||
json_pull *jp = json_begin_string(arg);
|
||||
json_object *o = json_read_tree(jp);
|
||||
|
||||
if (o == NULL) {
|
||||
fprintf(stderr, "%s: -E%s: %s\n", *argv, arg, jp->error);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
if (o->type != JSON_HASH) {
|
||||
fprintf(stderr, "%s: -E%s: not a JSON object\n", *argv, arg);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < o->value.object.length; i++) {
|
||||
json_object *k = o->value.object.keys[i];
|
||||
json_object *v = o->value.object.values[i];
|
||||
|
||||
if (k->type != JSON_STRING) {
|
||||
fprintf(stderr, "%s: -E%s: key %zu not a string\n", *argv, arg, i);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
if (v->type != JSON_STRING) {
|
||||
fprintf(stderr, "%s: -E%s: value %zu not a string\n", *argv, arg, i);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
set_attribute_accum(attribute_accum, k->value.string.string, v->value.string.string);
|
||||
}
|
||||
|
||||
json_free(o);
|
||||
json_end(jp);
|
||||
return;
|
||||
}
|
||||
|
||||
const char *s = strchr(arg, ':');
|
||||
if (s == NULL) {
|
||||
fprintf(stderr, "-E%s option must be in the form -Ename:method\n", arg);
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
|
||||
std::string name = std::string(arg, s - arg);
|
||||
std::string type = std::string(s + 1);
|
||||
|
||||
set_attribute_accum(attribute_accum, name, type);
|
||||
}
|
||||
|
||||
template <class T>
|
||||
static void preserve_attribute1(attribute_op const &op, std::string const &key, T const &val, std::vector<std::shared_ptr<std::string>> &full_keys, std::vector<T> &full_values, std::unordered_map<std::string, accum_state> &attribute_accum_state, key_pool &key_pool) {
|
||||
for (size_t i = 0; i < full_keys.size(); i++) {
|
||||
if (key == *full_keys[i]) {
|
||||
switch (op) {
|
||||
case op_sum:
|
||||
full_values[i] = (full_values[i].to_double() + val.to_double());
|
||||
return;
|
||||
|
||||
case op_product:
|
||||
full_values[i] = (full_values[i].to_double() * val.to_double());
|
||||
return;
|
||||
|
||||
case op_max: {
|
||||
double existing = full_values[i].to_double();
|
||||
double maybe = val.to_double();
|
||||
if (maybe > existing) {
|
||||
full_values[i] = val;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
case op_min: {
|
||||
double existing = full_values[i].to_double();
|
||||
double maybe = val.to_double();
|
||||
if (maybe < existing) {
|
||||
full_values[i] = val;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
case op_mean: {
|
||||
auto state = attribute_accum_state.find(key);
|
||||
if (state == attribute_accum_state.end()) {
|
||||
accum_state s;
|
||||
s.sum = full_values[i].to_double() + val.to_double();
|
||||
s.count = 2;
|
||||
attribute_accum_state.insert(std::pair<std::string, accum_state>(key, s));
|
||||
|
||||
full_values[i] = (s.sum / s.count);
|
||||
} else {
|
||||
state->second.sum += val.to_double();
|
||||
state->second.count += 1;
|
||||
|
||||
full_values[i] = (state->second.sum / state->second.count);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
case op_concat:
|
||||
full_values[i].set_string_value(full_values[i].get_string_value() + val.get_string_value());
|
||||
return;
|
||||
|
||||
case op_comma:
|
||||
full_values[i].set_string_value(full_values[i].get_string_value() + "," + val.get_string_value());
|
||||
return;
|
||||
|
||||
case op_count: {
|
||||
auto state = attribute_accum_state.find(key);
|
||||
if (state == attribute_accum_state.end()) { // not already present
|
||||
accum_state s;
|
||||
s.count = 2;
|
||||
attribute_accum_state.insert(std::pair<std::string, accum_state>(key, s));
|
||||
|
||||
full_values[i] = (s.count);
|
||||
} else { // already present, incrementing
|
||||
state->second.count += 1;
|
||||
full_values[i] = (state->second.count);
|
||||
}
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// not found, so we are making a new value
|
||||
|
||||
T v;
|
||||
switch (op) {
|
||||
case op_sum:
|
||||
case op_max:
|
||||
case op_min:
|
||||
v = val;
|
||||
break;
|
||||
|
||||
case op_count: {
|
||||
auto state = attribute_accum_state.find(key);
|
||||
if (state == attribute_accum_state.end()) { // not already present
|
||||
accum_state s;
|
||||
s.count = 1;
|
||||
attribute_accum_state.insert(std::pair<std::string, accum_state>(key, s));
|
||||
|
||||
v = (s.count);
|
||||
} else { // already present, incrementing
|
||||
fprintf(stderr, "preserve_attribute: can't happen (count)\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
default:
|
||||
fprintf(stderr, "can't happen: operation that isn't used by --accumulate-numeric-attributes\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
full_keys.push_back(key_pool.pool(key));
|
||||
full_values.push_back(v);
|
||||
}
|
||||
|
||||
void preserve_attribute(attribute_op const &op, std::string const &key, mvt_value const &val, std::vector<std::shared_ptr<std::string>> &full_keys, std::vector<mvt_value> &full_values, std::unordered_map<std::string, accum_state> &attribute_accum_state, key_pool &key_pool) {
|
||||
preserve_attribute1(op, key, val, full_keys, full_values, attribute_accum_state, key_pool);
|
||||
}
|
||||
|
||||
void preserve_attribute(attribute_op const &op, std::string const &key, serial_val const &val, std::vector<std::shared_ptr<std::string>> &full_keys, std::vector<serial_val> &full_values, std::unordered_map<std::string, accum_state> &attribute_accum_state, key_pool &key_pool) {
|
||||
preserve_attribute1(op, key, val, full_keys, full_values, attribute_accum_state, key_pool);
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
#ifndef ATTRIBUTE_HPP
|
||||
#define ATTRIBUTE_HPP
|
||||
|
||||
#include <vector>
|
||||
#include <unordered_map>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include "mvt.hpp"
|
||||
#include "milo/dtoa_milo.h"
|
||||
|
||||
enum attribute_op {
|
||||
op_sum,
|
||||
op_product,
|
||||
op_mean,
|
||||
op_concat,
|
||||
op_comma,
|
||||
op_max,
|
||||
op_min,
|
||||
op_count,
|
||||
};
|
||||
|
||||
struct accum_state {
|
||||
double sum = 0;
|
||||
double count = 0;
|
||||
};
|
||||
|
||||
struct serial_val;
|
||||
struct key_pool;
|
||||
|
||||
void set_attribute_accum(std::unordered_map<std::string, attribute_op> &attribute_accum, std::string name, std::string type);
|
||||
void set_attribute_accum(std::unordered_map<std::string, attribute_op> &attribute_accum, const char *arg, char **argv);
|
||||
|
||||
void preserve_attribute(attribute_op const &op, std::string const &key, serial_val const &val, std::vector<std::shared_ptr<std::string>> &full_keys, std::vector<serial_val> &full_values, std::unordered_map<std::string, accum_state> &attribute_accum_state, key_pool &key_pool);
|
||||
void preserve_attribute(attribute_op const &op, std::string const &key, mvt_value const &val, std::vector<std::shared_ptr<std::string>> &full_keys, std::vector<mvt_value> &full_values, std::unordered_map<std::string, accum_state> &attribute_accum_state, key_pool &key_pool);
|
||||
|
||||
extern std::map<std::string, attribute_op> numeric_operations;
|
||||
|
||||
#endif
|
||||
+16245
-8793
File diff suppressed because it is too large
Load Diff
+269
@@ -0,0 +1,269 @@
|
||||
#ifdef __APPLE__
|
||||
#define _DARWIN_UNLIMITED_STREAMS
|
||||
#endif
|
||||
|
||||
#include "compression.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "protozero/varint.hpp"
|
||||
#include "serial.hpp"
|
||||
|
||||
void decompressor::begin() {
|
||||
within = true;
|
||||
|
||||
zs.zalloc = NULL;
|
||||
zs.zfree = NULL;
|
||||
zs.opaque = NULL;
|
||||
zs.msg = (char *) "";
|
||||
|
||||
int d = inflateInit(&zs);
|
||||
if (d != Z_OK) {
|
||||
fprintf(stderr, "initialize decompression: %d %s\n", d, zs.msg);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
|
||||
int decompressor::fread(void *p, size_t size, size_t nmemb, std::atomic<long long> *geompos) {
|
||||
zs.next_out = (Bytef *) p;
|
||||
zs.avail_out = size * nmemb;
|
||||
|
||||
while (zs.avail_out > 0) {
|
||||
if (zs.avail_in == 0) {
|
||||
size_t n = ::fread((Bytef *) buf.c_str(), sizeof(char), buf.size(), fp);
|
||||
if (n == 0) {
|
||||
if (within) {
|
||||
fprintf(stderr, "Reached EOF while decompressing\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
zs.next_in = (Bytef *) buf.c_str();
|
||||
zs.avail_in = n;
|
||||
}
|
||||
|
||||
size_t avail_before = zs.avail_in;
|
||||
|
||||
if (within) {
|
||||
int d = inflate(&zs, Z_NO_FLUSH);
|
||||
*geompos += avail_before - zs.avail_in;
|
||||
|
||||
if (d == Z_OK) {
|
||||
// it made some progress
|
||||
} else if (d == Z_STREAM_END) {
|
||||
// it may have made some progress and now we are done
|
||||
within = false;
|
||||
break;
|
||||
} else {
|
||||
fprintf(stderr, "decompression error %d %s\n", d, zs.msg);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
} else {
|
||||
size_t n = std::min(zs.avail_in, zs.avail_out);
|
||||
memcpy(zs.next_out, zs.next_in, n);
|
||||
*geompos += n;
|
||||
|
||||
zs.avail_out -= n;
|
||||
zs.avail_in -= n;
|
||||
zs.next_out += n;
|
||||
zs.next_in += n;
|
||||
}
|
||||
}
|
||||
|
||||
return (size * nmemb - zs.avail_out) / size;
|
||||
}
|
||||
|
||||
void decompressor::end(std::atomic<long long> *geompos) {
|
||||
// "within" means that we haven't received end-of-stream yet,
|
||||
// so consume more compressed data until we get there.
|
||||
// This can be necessary if the caller knows that it is at
|
||||
// the end of the feature stream (because it got a 0-length
|
||||
// feature) but the decompressor doesn't know yet.
|
||||
|
||||
if (within) {
|
||||
while (true) {
|
||||
if (zs.avail_in == 0) {
|
||||
size_t n = ::fread((Bytef *) buf.c_str(), sizeof(char), buf.size(), fp);
|
||||
zs.next_in = (Bytef *) buf.c_str();
|
||||
zs.avail_in = n;
|
||||
}
|
||||
|
||||
zs.avail_out = 0;
|
||||
|
||||
size_t avail_before = zs.avail_in;
|
||||
int d = inflate(&zs, Z_NO_FLUSH);
|
||||
*geompos += avail_before - zs.avail_in;
|
||||
|
||||
if (d == Z_STREAM_END) {
|
||||
break;
|
||||
}
|
||||
if (d == Z_OK) {
|
||||
continue;
|
||||
}
|
||||
|
||||
fprintf(stderr, "decompression: got %d, not Z_STREAM_END\n", d);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
within = false;
|
||||
}
|
||||
|
||||
int d = inflateEnd(&zs);
|
||||
if (d != Z_OK) {
|
||||
fprintf(stderr, "end decompression: %d %s\n", d, zs.msg);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
|
||||
int decompressor::deserialize_ulong_long(unsigned long long *zigzag, std::atomic<long long> *geompos) {
|
||||
*zigzag = 0;
|
||||
int shift = 0;
|
||||
|
||||
while (1) {
|
||||
char c;
|
||||
if (fread(&c, sizeof(char), 1, geompos) != 1) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
if ((c & 0x80) == 0) {
|
||||
*zigzag |= ((unsigned long long) c) << shift;
|
||||
shift += 7;
|
||||
break;
|
||||
} else {
|
||||
*zigzag |= ((unsigned long long) (c & 0x7F)) << shift;
|
||||
shift += 7;
|
||||
}
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
int decompressor::deserialize_long_long(long long *n, std::atomic<long long> *geompos) {
|
||||
unsigned long long zigzag = 0;
|
||||
int ret = deserialize_ulong_long(&zigzag, geompos);
|
||||
*n = protozero::decode_zigzag64(zigzag);
|
||||
return ret;
|
||||
}
|
||||
|
||||
int decompressor::deserialize_int(int *n, std::atomic<long long> *geompos) {
|
||||
long long ll = 0;
|
||||
int ret = deserialize_long_long(&ll, geompos);
|
||||
*n = ll;
|
||||
return ret;
|
||||
}
|
||||
|
||||
int decompressor::deserialize_uint(unsigned *n, std::atomic<long long> *geompos) {
|
||||
unsigned long long v;
|
||||
deserialize_ulong_long(&v, geompos);
|
||||
*n = v;
|
||||
return 1;
|
||||
}
|
||||
|
||||
void compressor::begin() {
|
||||
zs.zalloc = NULL;
|
||||
zs.zfree = NULL;
|
||||
zs.opaque = NULL;
|
||||
zs.msg = (char *) "";
|
||||
|
||||
int d = deflateInit(&zs, Z_DEFAULT_COMPRESSION);
|
||||
if (d != Z_OK) {
|
||||
fprintf(stderr, "initialize compression: %d %s\n", d, zs.msg);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
|
||||
void compressor::compressor::end(std::atomic<long long> *fpos, const char *fname) {
|
||||
std::string buf;
|
||||
buf.resize(5000);
|
||||
|
||||
if (zs.avail_in != 0) {
|
||||
fprintf(stderr, "compression end called with data available\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
zs.next_in = (Bytef *) buf.c_str();
|
||||
zs.avail_in = 0;
|
||||
|
||||
while (true) {
|
||||
zs.next_out = (Bytef *) buf.c_str();
|
||||
zs.avail_out = buf.size();
|
||||
|
||||
int d = deflate(&zs, Z_FINISH);
|
||||
::fwrite_check(buf.c_str(), sizeof(char), zs.next_out - (Bytef *) buf.c_str(), fp, fpos, fname);
|
||||
|
||||
if (d == Z_OK || d == Z_BUF_ERROR) {
|
||||
// it can take several calls to flush out all the buffered data
|
||||
continue;
|
||||
}
|
||||
|
||||
if (d != Z_STREAM_END) {
|
||||
fprintf(stderr, "%s: finish compression: %d %s\n", fname, d, zs.msg);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
zs.next_out = (Bytef *) buf.c_str();
|
||||
zs.avail_out = buf.size();
|
||||
|
||||
int d = deflateEnd(&zs);
|
||||
if (d != Z_OK) {
|
||||
fprintf(stderr, "%s: end compression: %d %s\n", fname, d, zs.msg);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
::fwrite_check(buf.c_str(), sizeof(char), zs.next_out - (Bytef *) buf.c_str(), fp, fpos, fname);
|
||||
}
|
||||
|
||||
int compressor::fclose() {
|
||||
return ::fclose(fp);
|
||||
}
|
||||
|
||||
void compressor::fwrite_check(const char *p, size_t size, size_t nmemb, std::atomic<long long> *fpos, const char *fname) {
|
||||
std::string buf;
|
||||
buf.resize(size * nmemb * 2 + 200);
|
||||
|
||||
zs.next_in = (Bytef *) p;
|
||||
zs.avail_in = size * nmemb;
|
||||
|
||||
while (zs.avail_in > 0) {
|
||||
zs.next_out = (Bytef *) buf.c_str();
|
||||
zs.avail_out = buf.size();
|
||||
|
||||
int d = deflate(&zs, Z_NO_FLUSH);
|
||||
if (d != Z_OK) {
|
||||
fprintf(stderr, "%s: deflate: %d %s\n", fname, d, zs.msg);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
::fwrite_check(buf.c_str(), sizeof(char), zs.next_out - (Bytef *) buf.c_str(), fp, fpos, fname);
|
||||
}
|
||||
}
|
||||
|
||||
void compressor::serialize_ulong_long(unsigned long long val, std::atomic<long long> *fpos, const char *fname) {
|
||||
while (1) {
|
||||
unsigned char b = val & 0x7F;
|
||||
if ((val >> 7) != 0) {
|
||||
b |= 0x80;
|
||||
fwrite_check((const char *) &b, 1, 1, fpos, fname);
|
||||
val >>= 7;
|
||||
} else {
|
||||
fwrite_check((const char *) &b, 1, 1, fpos, fname);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void compressor::serialize_long_long(long long val, std::atomic<long long> *fpos, const char *fname) {
|
||||
unsigned long long zigzag = protozero::encode_zigzag64(val);
|
||||
|
||||
serialize_ulong_long(zigzag, fpos, fname);
|
||||
}
|
||||
|
||||
void compressor::serialize_int(int val, std::atomic<long long> *fpos, const char *fname) {
|
||||
serialize_long_long(val, fpos, fname);
|
||||
}
|
||||
|
||||
void compressor::serialize_uint(unsigned val, std::atomic<long long> *fpos, const char *fname) {
|
||||
serialize_ulong_long(val, fpos, fname);
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
#ifdef __APPLE__
|
||||
#define _DARWIN_UNLIMITED_STREAMS
|
||||
#endif
|
||||
|
||||
#include <stdio.h>
|
||||
#include <string>
|
||||
#include <atomic>
|
||||
#include <zlib.h>
|
||||
|
||||
struct decompressor {
|
||||
FILE *fp = NULL;
|
||||
z_stream zs;
|
||||
std::string buf;
|
||||
|
||||
// from begin() to receiving end-of-stream
|
||||
bool within = false;
|
||||
|
||||
decompressor(FILE *f) {
|
||||
fp = f;
|
||||
buf.resize(5000);
|
||||
|
||||
zs.next_in = (Bytef *) buf.c_str();
|
||||
zs.avail_in = 0;
|
||||
}
|
||||
|
||||
decompressor() {
|
||||
}
|
||||
|
||||
void begin();
|
||||
int fread(void *p, size_t size, size_t nmemb, std::atomic<long long> *geompos);
|
||||
void end(std::atomic<long long> *geompos);
|
||||
int deserialize_ulong_long(unsigned long long *zigzag, std::atomic<long long> *geompos);
|
||||
int deserialize_long_long(long long *n, std::atomic<long long> *geompos);
|
||||
int deserialize_int(int *n, std::atomic<long long> *geompos);
|
||||
int deserialize_uint(unsigned *n, std::atomic<long long> *geompos);
|
||||
};
|
||||
|
||||
struct compressor {
|
||||
FILE *fp = NULL;
|
||||
z_stream zs;
|
||||
|
||||
compressor(FILE *f) {
|
||||
fp = f;
|
||||
}
|
||||
|
||||
compressor() {
|
||||
}
|
||||
|
||||
void begin();
|
||||
void end(std::atomic<long long> *fpos, const char *fname);
|
||||
int fclose();
|
||||
void fwrite_check(const char *p, size_t size, size_t nmemb, std::atomic<long long> *fpos, const char *fname);
|
||||
void serialize_ulong_long(unsigned long long val, std::atomic<long long> *fpos, const char *fname);
|
||||
|
||||
void serialize_long_long(long long val, std::atomic<long long> *fpos, const char *fname);
|
||||
void serialize_int(int val, std::atomic<long long> *fpos, const char *fname);
|
||||
void serialize_uint(unsigned val, std::atomic<long long> *fpos, const char *fname);
|
||||
};
|
||||
+12
-2
@@ -28,6 +28,11 @@
|
||||
int minzoom = 0;
|
||||
int maxzoom = 32;
|
||||
bool force = false;
|
||||
std::set<std::string> include_attr;
|
||||
|
||||
bool progress_time() {
|
||||
return false;
|
||||
}
|
||||
|
||||
void do_stats(mvt_tile &tile, size_t size, bool compressed, int z, unsigned x, unsigned y, json_writer &state) {
|
||||
state.json_write_hash();
|
||||
@@ -213,7 +218,7 @@ void handle(std::string message, int z, unsigned x, unsigned y, std::set<std::st
|
||||
} else if (coordinate_mode == 2) { // integer
|
||||
scale = 1;
|
||||
}
|
||||
layer_to_geojson(layer, z, x, y, !pipeline, pipeline, pipeline, false, 0, 0, 0, !force, state, scale);
|
||||
layer_to_geojson(layer, z, x, y, !pipeline, pipeline, pipeline, false, 0, 0, 0, !force, state, scale, include_attr);
|
||||
|
||||
if (!pipeline) {
|
||||
if (true) {
|
||||
@@ -273,7 +278,7 @@ void decode(char *fname, int z, unsigned x, unsigned y, std::set<std::string> co
|
||||
struct stat st;
|
||||
std::vector<zxy> tiles;
|
||||
|
||||
char *pmtiles_map;
|
||||
char *pmtiles_map = NULL;
|
||||
std::vector<pmtiles::entry_zxy> entries;
|
||||
bool is_pmtiles = false;
|
||||
|
||||
@@ -567,6 +572,7 @@ int main(int argc, char **argv) {
|
||||
{"stats", no_argument, 0, 'S'},
|
||||
{"force", no_argument, 0, 'f'},
|
||||
{"exclude-metadata-row", required_argument, 0, 'x'},
|
||||
{"include", required_argument, 0, 'y'},
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
@@ -626,6 +632,10 @@ int main(int argc, char **argv) {
|
||||
exclude_meta.insert(optarg);
|
||||
break;
|
||||
|
||||
case 'y':
|
||||
include_attr.insert(optarg);
|
||||
break;
|
||||
|
||||
default:
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
+5
-1
@@ -176,7 +176,7 @@ std::vector<zxy> enumerate_dirtiles(const char *fname, int minzoom, int maxzoom)
|
||||
closedir(d1);
|
||||
}
|
||||
|
||||
std::sort(tiles.begin(), tiles.end());
|
||||
std::stable_sort(tiles.begin(), tiles.end());
|
||||
return tiles;
|
||||
}
|
||||
|
||||
@@ -310,6 +310,7 @@ void dir_write_metadata(const char *outdir, const metadata &m) {
|
||||
out(state, "maxzoom", std::to_string(m.maxzoom));
|
||||
out(state, "center", std::to_string(m.center_lon) + "," + std::to_string(m.center_lat) + "," + std::to_string(m.center_z));
|
||||
out(state, "bounds", std::to_string(m.minlon) + "," + std::to_string(m.minlat) + "," + std::to_string(m.maxlon) + "," + std::to_string(m.maxlat));
|
||||
out(state, "antimeridian_adjusted_bounds", std::to_string(m.minlon2) + "," + std::to_string(m.minlat2) + "," + std::to_string(m.maxlon2) + "," + std::to_string(m.maxlat2));
|
||||
out(state, "type", m.type);
|
||||
if (m.attribution.size() > 0) {
|
||||
out(state, "attribution", m.attribution);
|
||||
@@ -317,6 +318,9 @@ void dir_write_metadata(const char *outdir, const metadata &m) {
|
||||
if (m.strategies_json.size() > 0) {
|
||||
out(state, "strategies", m.strategies_json);
|
||||
}
|
||||
if (m.decisions_json.size() > 0) {
|
||||
out(state, "tippecanoe_decisions", m.decisions_json);
|
||||
}
|
||||
out(state, "format", m.format);
|
||||
out(state, "generator", m.generator);
|
||||
out(state, "generator_options", m.generator_options);
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <sys/stat.h>
|
||||
#include "mbtiles.hpp"
|
||||
|
||||
#ifndef DIRTILES_HPP
|
||||
#define DIRTILES_HPP
|
||||
|
||||
+478
-41
@@ -1,39 +1,217 @@
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <map>
|
||||
#include <unordered_map>
|
||||
#include <functional>
|
||||
#include "mvt.hpp"
|
||||
#include "evaluator.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "milo/dtoa_milo.h"
|
||||
#include "text.hpp"
|
||||
|
||||
int compare(mvt_value one, json_object *two, bool &fail) {
|
||||
if (one.type == mvt_string) {
|
||||
static std::string mvt_value_to_string(mvt_value const &one, bool &fail, std::vector<std::string> const &unidecode_data) {
|
||||
switch (one.type) {
|
||||
case mvt_string:
|
||||
if (unidecode_data.size() > 0) {
|
||||
return unidecode_smash(unidecode_data, one.c_str());
|
||||
} else {
|
||||
return one.get_string_value();
|
||||
}
|
||||
case mvt_float:
|
||||
return milo::dtoa_milo(one.numeric_value.float_value);
|
||||
case mvt_double:
|
||||
return milo::dtoa_milo(one.numeric_value.double_value);
|
||||
case mvt_int:
|
||||
return std::to_string(one.numeric_value.int_value);
|
||||
case mvt_uint:
|
||||
return std::to_string(one.numeric_value.uint_value);
|
||||
case mvt_sint:
|
||||
return std::to_string(one.numeric_value.sint_value);
|
||||
case mvt_bool:
|
||||
return one.numeric_value.bool_value ? "true" : "false";
|
||||
case mvt_null:
|
||||
fail = true; // null op string => null
|
||||
return "";
|
||||
case mvt_no_such_key:
|
||||
default:
|
||||
fprintf(stderr, "unhandled mvt_type %d\n", one.type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
|
||||
// Alter the JSON parse tree in place to replace the original match string
|
||||
// with its unidecode-smashed version.
|
||||
//
|
||||
// To avoid repeated re-smashings of the same JSON object, objects that have
|
||||
// already been smashed are marked by setting their refcon to the unidecode data.
|
||||
static void smash(std::vector<std::string> const &unidecode_data, json_object *j) {
|
||||
if (j->value.string.refcon == (void *) &unidecode_data) {
|
||||
return;
|
||||
}
|
||||
|
||||
std::string s = unidecode_smash(unidecode_data, j->value.string.string);
|
||||
j->value.string.string = (char *) realloc(j->value.string.string, s.size() + 1);
|
||||
if (j->value.string.string == NULL) {
|
||||
perror("realloc for unidecode_smash");
|
||||
exit(EXIT_MEMORY);
|
||||
}
|
||||
strcpy(j->value.string.string, s.c_str());
|
||||
j->value.string.refcon = (void *) &unidecode_data;
|
||||
}
|
||||
|
||||
int compare_fsl(mvt_value const &one, json_object *two, bool &fail, std::vector<std::string> const &unidecode_data) {
|
||||
// In FSL expressions, the attribute value is coerced to the type
|
||||
// of the JSON literal value it is being compared to.
|
||||
//
|
||||
// If it cannot be converted, the comparison returns null
|
||||
// (which is distinct from true and false but is falsy if
|
||||
// it is the final output value).
|
||||
|
||||
if (two->type == JSON_NULL) {
|
||||
fail = true; // anything op null => null
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (two->type == JSON_NUMBER) {
|
||||
double lhs;
|
||||
|
||||
switch (one.type) {
|
||||
case mvt_string: {
|
||||
char *endptr = NULL;
|
||||
const char *s = one.c_str();
|
||||
lhs = strtod(s, &endptr);
|
||||
if (endptr == s) {
|
||||
fail = true; // non-numeric-string op number => null
|
||||
return 0;
|
||||
}
|
||||
} break;
|
||||
case mvt_float:
|
||||
lhs = one.numeric_value.float_value;
|
||||
break;
|
||||
case mvt_double:
|
||||
lhs = one.numeric_value.double_value;
|
||||
break;
|
||||
case mvt_int:
|
||||
lhs = one.numeric_value.int_value;
|
||||
break;
|
||||
case mvt_uint:
|
||||
lhs = one.numeric_value.uint_value;
|
||||
break;
|
||||
case mvt_sint:
|
||||
lhs = one.numeric_value.sint_value;
|
||||
break;
|
||||
case mvt_bool:
|
||||
lhs = one.numeric_value.bool_value;
|
||||
break;
|
||||
case mvt_null:
|
||||
fail = true; // null op number => null
|
||||
return 0;
|
||||
default:
|
||||
fprintf(stderr, "unhandled mvt_type %d\n", one.type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
if (lhs < two->value.number.number) {
|
||||
return -1;
|
||||
} else if (lhs > two->value.number.number) {
|
||||
return 1;
|
||||
} else {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
if (two->type == JSON_STRING) {
|
||||
std::string lhs = mvt_value_to_string(one, fail, unidecode_data);
|
||||
|
||||
if (unidecode_data.size() > 0) {
|
||||
smash(unidecode_data, two);
|
||||
}
|
||||
|
||||
return strcmp(lhs.c_str(), two->value.string.string);
|
||||
}
|
||||
|
||||
if (two->type == JSON_TRUE || two->type == JSON_FALSE) {
|
||||
bool lhs;
|
||||
|
||||
switch (one.type) {
|
||||
case mvt_string:
|
||||
lhs = one.get_string_view().size() > 0;
|
||||
break;
|
||||
case mvt_float:
|
||||
lhs = one.numeric_value.float_value != 0;
|
||||
break;
|
||||
case mvt_double:
|
||||
lhs = one.numeric_value.double_value != 0;
|
||||
break;
|
||||
case mvt_int:
|
||||
lhs = one.numeric_value.int_value != 0;
|
||||
break;
|
||||
case mvt_uint:
|
||||
lhs = one.numeric_value.uint_value != 0;
|
||||
break;
|
||||
case mvt_sint:
|
||||
lhs = one.numeric_value.sint_value != 0;
|
||||
break;
|
||||
case mvt_bool:
|
||||
lhs = one.numeric_value.bool_value;
|
||||
break;
|
||||
case mvt_null:
|
||||
fail = true; // null op bool => null
|
||||
return 0;
|
||||
|
||||
case mvt_no_such_key:
|
||||
default:
|
||||
fprintf(stderr, "unhandled mvt_type %d\n", one.type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
bool rhs = two->type == JSON_TRUE;
|
||||
return lhs - rhs;
|
||||
}
|
||||
|
||||
fprintf(stderr, "unhandled JSON type %d\n", two->type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
int compare(mvt_value const &one, json_object *two, bool &fail) {
|
||||
switch (one.type) {
|
||||
case mvt_string:
|
||||
if (two->type != JSON_STRING) {
|
||||
fail = true;
|
||||
return false; // string vs non-string
|
||||
}
|
||||
|
||||
return strcmp(one.string_value.c_str(), two->value.string.string);
|
||||
}
|
||||
return strcmp(one.c_str(), two->value.string.string);
|
||||
|
||||
if (one.type == mvt_double || one.type == mvt_float || one.type == mvt_int || one.type == mvt_uint || one.type == mvt_sint) {
|
||||
case mvt_double:
|
||||
case mvt_float:
|
||||
case mvt_int:
|
||||
case mvt_uint:
|
||||
case mvt_sint:
|
||||
if (two->type != JSON_NUMBER) {
|
||||
fail = true;
|
||||
return false; // number vs non-number
|
||||
}
|
||||
|
||||
double v;
|
||||
if (one.type == mvt_double) {
|
||||
switch (one.type) {
|
||||
case mvt_double:
|
||||
v = one.numeric_value.double_value;
|
||||
} else if (one.type == mvt_float) {
|
||||
break;
|
||||
case mvt_float:
|
||||
v = one.numeric_value.float_value;
|
||||
} else if (one.type == mvt_int) {
|
||||
break;
|
||||
case mvt_int:
|
||||
v = one.numeric_value.int_value;
|
||||
} else if (one.type == mvt_uint) {
|
||||
break;
|
||||
case mvt_uint:
|
||||
v = one.numeric_value.uint_value;
|
||||
} else if (one.type == mvt_sint) {
|
||||
break;
|
||||
case mvt_sint:
|
||||
v = one.numeric_value.sint_value;
|
||||
} else {
|
||||
break;
|
||||
case mvt_no_such_key:
|
||||
default:
|
||||
fprintf(stderr, "Internal error: bad mvt type %d\n", one.type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
@@ -45,32 +223,65 @@ int compare(mvt_value one, json_object *two, bool &fail) {
|
||||
} else {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
if (one.type == mvt_bool) {
|
||||
case mvt_bool:
|
||||
if (two->type != JSON_TRUE && two->type != JSON_FALSE) {
|
||||
fail = true;
|
||||
return false; // bool vs non-bool
|
||||
}
|
||||
|
||||
bool b = two->type != JSON_FALSE;
|
||||
return one.numeric_value.bool_value > b;
|
||||
}
|
||||
{
|
||||
bool b = two->type != JSON_FALSE;
|
||||
return one.numeric_value.bool_value > b;
|
||||
}
|
||||
|
||||
if (one.type == mvt_null) {
|
||||
case mvt_null:
|
||||
if (two->type != JSON_NULL) {
|
||||
fail = true;
|
||||
return false; // null vs non-null
|
||||
}
|
||||
|
||||
return 0; // null equals null
|
||||
|
||||
case mvt_no_such_key:
|
||||
default:
|
||||
break;
|
||||
}
|
||||
|
||||
fprintf(stderr, "Internal error: bad mvt type %d\n", one.type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::set<std::string> &exclude_attributes) {
|
||||
// 0: false
|
||||
// 1: true
|
||||
// -1: incomparable (sql null), treated as false in final output
|
||||
static int eval(std::function<mvt_value(std::string const &)> feature, json_object *f, std::set<std::string> &exclude_attributes, std::vector<std::string> const &unidecode_data) {
|
||||
if (f != NULL) {
|
||||
if (f->type == JSON_TRUE) {
|
||||
return 1;
|
||||
} else if (f->type == JSON_FALSE) {
|
||||
return 0;
|
||||
} else if (f->type == JSON_NULL) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (f->type == JSON_NUMBER) {
|
||||
if (f->value.number.number == 0) {
|
||||
return 0;
|
||||
} else {
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (f->type == JSON_STRING) {
|
||||
if (f->value.string.string[0] == '\0') {
|
||||
return 0;
|
||||
} else {
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (f == NULL || f->type != JSON_ARRAY) {
|
||||
fprintf(stderr, "Filter is not an array: %s\n", json_stringify(f));
|
||||
exit(EXIT_FILTER);
|
||||
@@ -81,6 +292,163 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
exit(EXIT_FILTER);
|
||||
}
|
||||
|
||||
// FSL conjunctions
|
||||
if (f->value.array.length == 3 &&
|
||||
f->value.array.array[1]->type == JSON_STRING &&
|
||||
(strcmp(f->value.array.array[1]->value.string.string, "or") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "and") == 0)) {
|
||||
int lhs;
|
||||
|
||||
if (f->value.array.array[0]->type == JSON_STRING) {
|
||||
// if LHS of a boolean conjunction is a string, it is an attribute reference
|
||||
mvt_value ff = feature(std::string(f->value.array.array[0]->value.string.string));
|
||||
if (ff.type != mvt_no_such_key) {
|
||||
if (ff.type == mvt_bool) {
|
||||
lhs = ff.numeric_value.bool_value;
|
||||
} else {
|
||||
lhs = -1; // not boolean: null
|
||||
}
|
||||
} else {
|
||||
lhs = -1; // not found: null
|
||||
}
|
||||
} else {
|
||||
lhs = eval(feature, f->value.array.array[0], exclude_attributes, unidecode_data);
|
||||
}
|
||||
|
||||
int rhs = eval(feature, f->value.array.array[2], exclude_attributes, unidecode_data);
|
||||
if (lhs < 0 && rhs < 0) {
|
||||
return -1; // null op null => null
|
||||
}
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "or") == 0) {
|
||||
return (lhs > 0) || (rhs > 0);
|
||||
} else {
|
||||
return (lhs > 0) && (rhs > 0);
|
||||
}
|
||||
}
|
||||
|
||||
// FSL comparators
|
||||
if (f->value.array.length == 3 &&
|
||||
f->value.array.array[0]->type == JSON_STRING &&
|
||||
f->value.array.array[1]->type == JSON_STRING &&
|
||||
(strcmp(f->value.array.array[1]->value.string.string, "lt") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "gt") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "le") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "ge") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "eq") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "ne") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "cn") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "nc") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "in") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "ni") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "is") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "isnt") == 0 ||
|
||||
false)) {
|
||||
mvt_value lhs = feature(std::string(f->value.array.array[0]->value.string.string));
|
||||
if (lhs.type == mvt_no_such_key) {
|
||||
lhs.type = mvt_null;
|
||||
}
|
||||
|
||||
if (f->value.array.array[2]->type == JSON_NULL && strcmp(f->value.array.array[1]->value.string.string, "is") == 0) {
|
||||
return lhs.type == mvt_null; // null is null => true, anything is null => false
|
||||
}
|
||||
if (f->value.array.array[2]->type == JSON_NULL && strcmp(f->value.array.array[1]->value.string.string, "isnt") == 0) {
|
||||
return lhs.type != mvt_null; // null isnt null => false, anything isnt null => true
|
||||
}
|
||||
|
||||
bool fail = false;
|
||||
|
||||
if (f->value.array.array[2]->type == JSON_STRING &&
|
||||
(strcmp(f->value.array.array[1]->value.string.string, "cn") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "nc") == 0)) {
|
||||
if (lhs.type == mvt_null) {
|
||||
return -1; // null cn/nc anything => null
|
||||
}
|
||||
|
||||
std::string s = mvt_value_to_string(lhs, fail, unidecode_data);
|
||||
if (fail) {
|
||||
return -1; // null cn anything => false
|
||||
}
|
||||
|
||||
bool contains;
|
||||
if (unidecode_data.size() > 0) {
|
||||
smash(unidecode_data, f->value.array.array[2]);
|
||||
}
|
||||
contains = strstr(s.c_str(), f->value.array.array[2]->value.string.string);
|
||||
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "cn") == 0) {
|
||||
return contains;
|
||||
} else {
|
||||
return !contains;
|
||||
}
|
||||
}
|
||||
|
||||
if (f->value.array.array[2]->type == JSON_ARRAY &&
|
||||
(strcmp(f->value.array.array[1]->value.string.string, "in") == 0 ||
|
||||
strcmp(f->value.array.array[1]->value.string.string, "ni") == 0)) {
|
||||
// no null check here, since null can be in or not in
|
||||
|
||||
bool contains = false;
|
||||
for (size_t i = 0; i < f->value.array.array[2]->value.array.length; i++) {
|
||||
if (lhs.type == mvt_null) {
|
||||
if (f->value.array.array[2]->value.array.array[i]->type == JSON_NULL) {
|
||||
contains = true; // null is null
|
||||
break;
|
||||
} else {
|
||||
contains = false; // null isnt non-null
|
||||
}
|
||||
} else {
|
||||
fail = false;
|
||||
static std::vector<std::string> no_unidecode_data;
|
||||
int cmp = compare_fsl(lhs, f->value.array.array[2]->value.array.array[i], fail, no_unidecode_data);
|
||||
|
||||
if (fail) {
|
||||
contains = false; // non-null isnt null
|
||||
} else if (cmp == 0) {
|
||||
contains = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "in") == 0) {
|
||||
return contains;
|
||||
} else {
|
||||
return !contains;
|
||||
}
|
||||
}
|
||||
|
||||
if (lhs.type == mvt_null) {
|
||||
return -1; // null compared to anything => null
|
||||
}
|
||||
|
||||
int cmp = compare_fsl(lhs, f->value.array.array[2], fail, unidecode_data);
|
||||
if (fail) {
|
||||
return -1; // null
|
||||
}
|
||||
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "eq") == 0) {
|
||||
return cmp == 0;
|
||||
}
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "ne") == 0) {
|
||||
return cmp != 0;
|
||||
}
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "gt") == 0) {
|
||||
return cmp > 0;
|
||||
}
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "ge") == 0) {
|
||||
return cmp >= 0;
|
||||
}
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "lt") == 0) {
|
||||
return cmp < 0;
|
||||
}
|
||||
if (strcmp(f->value.array.array[1]->value.string.string, "le") == 0) {
|
||||
return cmp <= 0;
|
||||
}
|
||||
|
||||
fprintf(stderr, "expression fsl comparison: can't happen %s\n", f->value.array.array[1]->value.string.string);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
if (f->value.array.array[0]->type != JSON_STRING) {
|
||||
fprintf(stderr, "Filter operation is not a string: %s\n", json_stringify(f));
|
||||
exit(EXIT_FILTER);
|
||||
@@ -98,7 +466,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
fprintf(stderr, "\"has\" key is not a string: %s\n", json_stringify(f));
|
||||
exit(EXIT_FILTER);
|
||||
}
|
||||
return feature.count(std::string(f->value.array.array[1]->value.string.string)) != 0;
|
||||
return feature(std::string(f->value.array.array[1]->value.string.string)).type != mvt_no_such_key;
|
||||
}
|
||||
|
||||
if (strcmp(f->value.array.array[0]->value.string.string, "!has") == 0) {
|
||||
@@ -106,7 +474,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
fprintf(stderr, "\"!has\" key is not a string: %s\n", json_stringify(f));
|
||||
exit(EXIT_FILTER);
|
||||
}
|
||||
return feature.count(std::string(f->value.array.array[1]->value.string.string)) == 0;
|
||||
return feature(std::string(f->value.array.array[1]->value.string.string)).type == mvt_no_such_key;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -125,8 +493,8 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
exit(EXIT_FILTER);
|
||||
}
|
||||
|
||||
auto ff = feature.find(std::string(f->value.array.array[1]->value.string.string));
|
||||
if (ff == feature.end()) {
|
||||
mvt_value ff = feature(std::string(f->value.array.array[1]->value.string.string));
|
||||
if (ff.type == mvt_no_such_key) {
|
||||
static bool warned = false;
|
||||
if (!warned) {
|
||||
const char *s = json_stringify(f);
|
||||
@@ -141,7 +509,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
}
|
||||
|
||||
bool fail = false;
|
||||
int cmp = compare(ff->second, f->value.array.array[2], fail);
|
||||
int cmp = compare(ff, f->value.array.array[2], fail);
|
||||
|
||||
if (fail) {
|
||||
static bool warned = false;
|
||||
@@ -192,17 +560,19 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
}
|
||||
|
||||
for (size_t i = 1; i < f->value.array.length; i++) {
|
||||
bool out = eval(feature, f->value.array.array[i], exclude_attributes);
|
||||
int out = eval(feature, f->value.array.array[i], exclude_attributes, unidecode_data);
|
||||
|
||||
if (strcmp(f->value.array.array[0]->value.string.string, "all") == 0) {
|
||||
v = v && out;
|
||||
if (!v) {
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
v = v || out;
|
||||
if (v) {
|
||||
break;
|
||||
if (out >= 0) { // nulls are ignored in boolean and/or expressions
|
||||
if (strcmp(f->value.array.array[0]->value.string.string, "all") == 0) {
|
||||
v = v && out;
|
||||
if (!v) {
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
v = v || out;
|
||||
if (v) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -226,8 +596,8 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
exit(EXIT_FILTER);
|
||||
}
|
||||
|
||||
auto ff = feature.find(std::string(f->value.array.array[1]->value.string.string));
|
||||
if (ff == feature.end()) {
|
||||
mvt_value ff = feature(std::string(f->value.array.array[1]->value.string.string));
|
||||
if (ff.type == mvt_no_such_key) {
|
||||
static bool warned = false;
|
||||
if (!warned) {
|
||||
const char *s = json_stringify(f);
|
||||
@@ -244,7 +614,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
bool found = false;
|
||||
for (size_t i = 2; i < f->value.array.length; i++) {
|
||||
bool fail = false;
|
||||
int cmp = compare(ff->second, f->value.array.array[i], fail);
|
||||
int cmp = compare(ff, f->value.array.array[i], fail);
|
||||
|
||||
if (fail) {
|
||||
static bool warned = false;
|
||||
@@ -281,7 +651,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
exit(EXIT_FILTER);
|
||||
}
|
||||
|
||||
bool ok = eval(feature, f->value.array.array[2], exclude_attributes);
|
||||
bool ok = eval(feature, f->value.array.array[2], exclude_attributes, unidecode_data) > 0;
|
||||
if (!ok) {
|
||||
exclude_attributes.insert(f->value.array.array[1]->value.string.string);
|
||||
}
|
||||
@@ -293,7 +663,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
|
||||
exit(EXIT_FILTER);
|
||||
}
|
||||
|
||||
bool evaluate(std::map<std::string, mvt_value> const &feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes) {
|
||||
bool evaluate(std::function<mvt_value(std::string const &)> feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes, std::vector<std::string> const &unidecode_data) {
|
||||
if (filter == NULL || filter->type != JSON_HASH) {
|
||||
fprintf(stderr, "Error: filter is not a hash: %s\n", json_stringify(filter));
|
||||
exit(EXIT_JSON);
|
||||
@@ -304,12 +674,12 @@ bool evaluate(std::map<std::string, mvt_value> const &feature, std::string const
|
||||
|
||||
f = json_hash_get(filter, layer.c_str());
|
||||
if (ok && f != NULL) {
|
||||
ok = eval(feature, f, exclude_attributes);
|
||||
ok = eval(feature, f, exclude_attributes, unidecode_data) > 0;
|
||||
}
|
||||
|
||||
f = json_hash_get(filter, "*");
|
||||
if (ok && f != NULL) {
|
||||
ok = eval(feature, f, exclude_attributes);
|
||||
ok = eval(feature, f, exclude_attributes, unidecode_data) > 0;
|
||||
}
|
||||
|
||||
return ok;
|
||||
@@ -346,3 +716,70 @@ json_object *parse_filter(const char *s) {
|
||||
json_end(jp);
|
||||
return filter;
|
||||
}
|
||||
|
||||
bool evaluate(std::unordered_map<std::string, mvt_value> const &feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes, std::vector<std::string> const &unidecode_data) {
|
||||
std::function<mvt_value(std::string const &)> getter = [&](std::string const &key) {
|
||||
auto f = feature.find(key);
|
||||
if (f != feature.end()) {
|
||||
return f->second;
|
||||
} else {
|
||||
mvt_value v;
|
||||
v.type = mvt_no_such_key;
|
||||
v.numeric_value.null_value = 0;
|
||||
return v;
|
||||
}
|
||||
};
|
||||
|
||||
return evaluate(getter, layer, filter, exclude_attributes, unidecode_data);
|
||||
}
|
||||
|
||||
bool evaluate(mvt_feature const &feat, mvt_layer const &layer, json_object *filter, std::set<std::string> &exclude_attributes, int z, std::vector<std::string> const &unidecode_data) {
|
||||
std::function<mvt_value(std::string const &)> getter = [&](std::string const &key) {
|
||||
const static std::string dollar_id = "$id";
|
||||
if (key == dollar_id && feat.has_id) {
|
||||
mvt_value v;
|
||||
v.type = mvt_uint;
|
||||
v.numeric_value.uint_value = feat.id;
|
||||
return v;
|
||||
}
|
||||
|
||||
const static std::string dollar_type = "$type";
|
||||
if (key == dollar_type) {
|
||||
mvt_value v;
|
||||
v.type = mvt_string;
|
||||
|
||||
if (feat.type == mvt_point) {
|
||||
const static std::string point = "Point";
|
||||
v.set_string_value(point);
|
||||
} else if (feat.type == mvt_linestring) {
|
||||
const static std::string linestring = "LineString";
|
||||
v.set_string_value(linestring);
|
||||
} else if (feat.type == mvt_polygon) {
|
||||
const static std::string polygon = "Polygon";
|
||||
v.set_string_value(polygon);
|
||||
}
|
||||
return v;
|
||||
}
|
||||
|
||||
const static std::string dollar_zoom = "$zoom";
|
||||
if (key == dollar_zoom) {
|
||||
mvt_value v2;
|
||||
v2.type = mvt_uint;
|
||||
v2.numeric_value.uint_value = z;
|
||||
return v2;
|
||||
}
|
||||
|
||||
for (size_t i = 0; i + 1 < feat.tags.size(); i += 2) {
|
||||
if (layer.keys[feat.tags[i]] == key) {
|
||||
return layer.values[feat.tags[i + 1]];
|
||||
}
|
||||
}
|
||||
|
||||
mvt_value v;
|
||||
v.type = mvt_no_such_key;
|
||||
v.numeric_value.null_value = 0;
|
||||
return v;
|
||||
};
|
||||
|
||||
return evaluate(getter, layer.name, filter, exclude_attributes, unidecode_data);
|
||||
}
|
||||
|
||||
+4
-2
@@ -1,14 +1,16 @@
|
||||
#ifndef EVALUATOR_HPP
|
||||
#define EVALUATOR HPP
|
||||
|
||||
#include <map>
|
||||
#include <unordered_map>
|
||||
#include <string>
|
||||
#include <set>
|
||||
#include "jsonpull/jsonpull.h"
|
||||
#include "mvt.hpp"
|
||||
|
||||
bool evaluate(std::map<std::string, mvt_value> const &feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes);
|
||||
bool evaluate(std::unordered_map<std::string, mvt_value> const &feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes, std::vector<std::string> const &unidecode_data);
|
||||
json_object *parse_filter(const char *s);
|
||||
json_object *read_filter(const char *fname);
|
||||
|
||||
bool evaluate(mvt_feature const &feat, mvt_layer const &layer, json_object *filter, std::set<std::string> &exclude_attributes, int z, std::vector<std::string> const &unidecode_data);
|
||||
|
||||
#endif
|
||||
|
||||
+19
-14
@@ -7,6 +7,7 @@
|
||||
#include "milo/dtoa_milo.h"
|
||||
#include "main.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "thread.hpp"
|
||||
|
||||
static constexpr uint8_t magicbytes[8] = { 0x66, 0x67, 0x62, 0x03, 0x66, 0x67, 0x62, 0x01 };
|
||||
|
||||
@@ -75,13 +76,13 @@ drawvec readGeometry(const FlatGeobuf::Geometry *geometry, FlatGeobuf::GeometryT
|
||||
|
||||
if (geometry_type == FlatGeobuf::GeometryType_Point) {
|
||||
return readPoints(geometry);
|
||||
} if (geometry_type == FlatGeobuf::GeometryType_MultiPoint) {
|
||||
} else if (geometry_type == FlatGeobuf::GeometryType_MultiPoint) {
|
||||
return readPoints(geometry);
|
||||
} if (geometry_type == FlatGeobuf::GeometryType_LineString) {
|
||||
} else if (geometry_type == FlatGeobuf::GeometryType_LineString) {
|
||||
return readLinePart(geometry);
|
||||
} else if (h_geometry_type == FlatGeobuf::GeometryType_MultiLineString) {
|
||||
} else if (geometry_type == FlatGeobuf::GeometryType_MultiLineString) {
|
||||
return readLinePart(geometry);
|
||||
} if (geometry_type == FlatGeobuf::GeometryType_Polygon) {
|
||||
} else if (geometry_type == FlatGeobuf::GeometryType_Polygon) {
|
||||
return readLinePart(geometry);
|
||||
} else if (geometry_type == FlatGeobuf::GeometryType_MultiPolygon) {
|
||||
// if it is a GeometryCollection, parse Parts, ignore XY
|
||||
@@ -131,7 +132,6 @@ void readFeature(const FlatGeobuf::Feature *feature, long long feature_sequence_
|
||||
serial_feature sf;
|
||||
|
||||
sf.layer = layer;
|
||||
sf.layername = layername;
|
||||
sf.segment = sst->segment;
|
||||
if (feature_sequence_id >= 0) {
|
||||
sf.has_id = true;
|
||||
@@ -139,19 +139,20 @@ void readFeature(const FlatGeobuf::Feature *feature, long long feature_sequence_
|
||||
sf.has_id = false;
|
||||
}
|
||||
sf.id = feature_sequence_id;
|
||||
sf.has_tippecanoe_minzoom = false;
|
||||
sf.has_tippecanoe_maxzoom = false;
|
||||
sf.tippecanoe_minzoom = -1;
|
||||
sf.tippecanoe_maxzoom = -1;
|
||||
sf.feature_minzoom = false;
|
||||
sf.seq = (*sst->layer_seq);
|
||||
sf.geometry = dv;
|
||||
sf.t = drawvec_type;
|
||||
|
||||
std::vector<std::string> full_keys;
|
||||
std::vector<std::shared_ptr<std::string>> full_keys;
|
||||
std::vector<serial_val> full_values;
|
||||
key_pool key_pool;
|
||||
|
||||
// assume tabular schema with columns in header
|
||||
size_t p_pos = 0;
|
||||
while (p_pos < feature->properties()->size()) {
|
||||
while (feature->properties() && p_pos < feature->properties()->size()) {
|
||||
uint16_t col_idx;
|
||||
memcpy(&col_idx, feature->properties()->data() + p_pos, sizeof(col_idx));
|
||||
|
||||
@@ -177,7 +178,11 @@ void readFeature(const FlatGeobuf::Feature *feature, long long feature_sequence_
|
||||
sv.type = mvt_bool;
|
||||
uint8_t bool_val;
|
||||
memcpy(&bool_val, feature->properties()->data() + p_pos + sizeof(uint16_t), sizeof(bool_val));
|
||||
sv.s = std::to_string(bool_val);
|
||||
if (bool_val) {
|
||||
sv.s = "true";
|
||||
} else {
|
||||
sv.s = "false";
|
||||
}
|
||||
p_pos += sizeof(uint16_t) + sizeof(bool_val);
|
||||
} else if (col_type == FlatGeobuf::ColumnType_Short) {
|
||||
sv.type = mvt_sint;
|
||||
@@ -239,14 +244,14 @@ void readFeature(const FlatGeobuf::Feature *feature, long long feature_sequence_
|
||||
fprintf(stderr, "flatgeobuf has unsupported column type %u\n", (unsigned int)col_type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
full_keys.push_back(h_column_names[col_idx]);
|
||||
full_keys.push_back(key_pool.pool(h_column_names[col_idx]));
|
||||
full_values.push_back(sv);
|
||||
}
|
||||
|
||||
sf.full_keys = full_keys;
|
||||
sf.full_values = full_values;
|
||||
|
||||
serialize_feature(sst, sf);
|
||||
serialize_feature(sst, sf, layername);
|
||||
}
|
||||
|
||||
struct fgb_queued_feature {
|
||||
@@ -303,7 +308,7 @@ void fgbRunQueue() {
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
if (pthread_create(&pthreads[i], NULL, fgb_run_parse_feature, &qra[i]) != 0) {
|
||||
if (thread_create(&pthreads[i], NULL, fgb_run_parse_feature, &qra[i]) != 0) {
|
||||
perror("pthread_create");
|
||||
exit(EXIT_PTHREAD);
|
||||
}
|
||||
@@ -368,7 +373,7 @@ void parse_flatgeobuf(std::vector<struct serialization_state> *sst, const char *
|
||||
auto h_geometry_type = header->geometry_type();
|
||||
|
||||
long long feature_sequence_id = -1;
|
||||
int index_size = 0;
|
||||
long long index_size = 0;
|
||||
if (node_size > 0) {
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "detected indexed FlatGeobuf: assigning feature IDs by sequence\n");
|
||||
|
||||
+14
-16
@@ -15,6 +15,7 @@
|
||||
#include "jsonpull/jsonpull.h"
|
||||
#include "text.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "thread.hpp"
|
||||
|
||||
#define POINT 0
|
||||
#define MULTIPOINT 1
|
||||
@@ -269,14 +270,14 @@ std::vector<drawvec_type> readGeometry(protozero::pbf_reader &pbf, size_t dim, d
|
||||
return ret;
|
||||
}
|
||||
|
||||
void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<std::string> &keys, struct serialization_state *sst, int layer, std::string layername) {
|
||||
void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<std::string> &keys, struct serialization_state *sst, int layer, std::string layername, key_pool &key_pool) {
|
||||
std::vector<drawvec_type> dv;
|
||||
long long id = 0;
|
||||
bool has_id = false;
|
||||
std::vector<serial_val> values;
|
||||
std::map<std::string, serial_val> other;
|
||||
|
||||
std::vector<std::string> full_keys;
|
||||
std::vector<std::shared_ptr<std::string>> full_keys;
|
||||
std::vector<serial_val> full_values;
|
||||
|
||||
while (pbf.next()) {
|
||||
@@ -337,7 +338,7 @@ void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<s
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
full_keys.push_back(keys[properties[i]]);
|
||||
full_keys.push_back(key_pool.pool(keys[properties[i]]));
|
||||
full_values.push_back(values[properties[i + 1]]);
|
||||
}
|
||||
|
||||
@@ -379,12 +380,11 @@ void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<s
|
||||
serial_feature sf;
|
||||
|
||||
sf.layer = layer;
|
||||
sf.layername = layername;
|
||||
sf.segment = sst->segment;
|
||||
sf.has_id = has_id;
|
||||
sf.id = id;
|
||||
sf.has_tippecanoe_minzoom = false;
|
||||
sf.has_tippecanoe_maxzoom = false;
|
||||
sf.tippecanoe_minzoom = -1;
|
||||
sf.tippecanoe_maxzoom = -1;
|
||||
sf.feature_minzoom = false;
|
||||
sf.seq = *(sst->layer_seq);
|
||||
sf.geometry = dv[i].dv;
|
||||
@@ -400,19 +400,17 @@ void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<s
|
||||
if (o != NULL) {
|
||||
json_object *min = json_hash_get(o, "minzoom");
|
||||
if (min != NULL && (min->type == JSON_NUMBER)) {
|
||||
sf.has_tippecanoe_minzoom = true;
|
||||
sf.tippecanoe_minzoom = integer_zoom(sst->fname, milo::dtoa_milo(min->value.number.number));
|
||||
}
|
||||
|
||||
json_object *max = json_hash_get(o, "maxzoom");
|
||||
if (max != NULL && (max->type == JSON_NUMBER)) {
|
||||
sf.has_tippecanoe_maxzoom = true;
|
||||
sf.tippecanoe_maxzoom = integer_zoom(sst->fname, milo::dtoa_milo(max->value.number.number));
|
||||
}
|
||||
|
||||
json_object *tlayer = json_hash_get(o, "layer");
|
||||
if (tlayer != NULL && (tlayer->type == JSON_STRING)) {
|
||||
sf.layername = tlayer->value.string.string;
|
||||
layername = tlayer->value.string.string;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -420,7 +418,7 @@ void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<s
|
||||
json_end(jp);
|
||||
}
|
||||
|
||||
serialize_feature(sst, sf);
|
||||
serialize_feature(sst, sf, layername);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -436,10 +434,11 @@ struct queue_run_arg {
|
||||
|
||||
void *run_parse_feature(void *v) {
|
||||
struct queue_run_arg *qra = (struct queue_run_arg *) v;
|
||||
key_pool key_pool;
|
||||
|
||||
for (size_t i = qra->start; i < qra->end; i++) {
|
||||
struct queued_feature &qf = feature_queue[i];
|
||||
readFeature(qf.pbf, qf.dim, qf.e, *qf.keys, &(*qf.sst)[qra->segment], qf.layer, qf.layername);
|
||||
readFeature(qf.pbf, qf.dim, qf.e, *qf.keys, &(*qf.sst)[qra->segment], qf.layer, qf.layername, key_pool);
|
||||
}
|
||||
|
||||
return NULL;
|
||||
@@ -465,7 +464,7 @@ void runQueue() {
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
if (pthread_create(&pthreads[i], NULL, run_parse_feature, &qra[i]) != 0) {
|
||||
if (thread_create(&pthreads[i], NULL, run_parse_feature, &qra[i]) != 0) {
|
||||
perror("pthread_create");
|
||||
exit(EXIT_PTHREAD);
|
||||
}
|
||||
@@ -506,17 +505,16 @@ void outBareGeometry(drawvec const &dv, int type, struct serialization_state *ss
|
||||
serial_feature sf;
|
||||
|
||||
sf.layer = layer;
|
||||
sf.layername = layername;
|
||||
sf.segment = sst->segment;
|
||||
sf.has_id = false;
|
||||
sf.has_tippecanoe_minzoom = false;
|
||||
sf.has_tippecanoe_maxzoom = false;
|
||||
sf.tippecanoe_minzoom = -1;
|
||||
sf.tippecanoe_maxzoom = -1;
|
||||
sf.feature_minzoom = false;
|
||||
sf.seq = (*sst->layer_seq);
|
||||
sf.geometry = dv;
|
||||
sf.t = type;
|
||||
|
||||
serialize_feature(sst, sf);
|
||||
serialize_feature(sst, sf, layername);
|
||||
}
|
||||
|
||||
void readFeatureCollection(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<std::string> &keys, std::vector<struct serialization_state> *sst, int layer, std::string layername) {
|
||||
|
||||
+6
-6
@@ -58,6 +58,7 @@ void parse_geocsv(std::vector<struct serialization_state> &sst, std::string fnam
|
||||
}
|
||||
|
||||
size_t seq = 0;
|
||||
key_pool key_pool;
|
||||
while ((s = csv_getline(f)).size() > 0) {
|
||||
std::string err = check_utf8(s);
|
||||
if (err != "") {
|
||||
@@ -89,7 +90,7 @@ void parse_geocsv(std::vector<struct serialization_state> &sst, std::string fnam
|
||||
drawvec dv;
|
||||
dv.push_back(draw(VT_MOVETO, x, y));
|
||||
|
||||
std::vector<std::string> full_keys;
|
||||
std::vector<std::shared_ptr<std::string>> full_keys;
|
||||
std::vector<serial_val> full_values;
|
||||
|
||||
for (size_t i = 0; i < line.size(); i++) {
|
||||
@@ -107,7 +108,7 @@ void parse_geocsv(std::vector<struct serialization_state> &sst, std::string fnam
|
||||
}
|
||||
sv.s = line[i];
|
||||
|
||||
full_keys.push_back(header[i]);
|
||||
full_keys.push_back(key_pool.pool(header[i]));
|
||||
full_values.push_back(sv);
|
||||
}
|
||||
}
|
||||
@@ -115,12 +116,11 @@ void parse_geocsv(std::vector<struct serialization_state> &sst, std::string fnam
|
||||
serial_feature sf;
|
||||
|
||||
sf.layer = layer;
|
||||
sf.layername = layername;
|
||||
sf.segment = sst[0].segment;
|
||||
sf.has_id = false;
|
||||
sf.id = 0;
|
||||
sf.has_tippecanoe_minzoom = false;
|
||||
sf.has_tippecanoe_maxzoom = false;
|
||||
sf.tippecanoe_minzoom = -1;
|
||||
sf.tippecanoe_maxzoom = -1;
|
||||
sf.feature_minzoom = false;
|
||||
sf.seq = *(sst[0].layer_seq);
|
||||
sf.geometry = dv;
|
||||
@@ -128,7 +128,7 @@ void parse_geocsv(std::vector<struct serialization_state> &sst, std::string fnam
|
||||
sf.full_keys = full_keys;
|
||||
sf.full_values = full_values;
|
||||
|
||||
serialize_feature(&sst[0], sf);
|
||||
serialize_feature(&sst[0], sf, layername);
|
||||
}
|
||||
|
||||
if (fname.size() != 0) {
|
||||
|
||||
+1
-1
@@ -29,7 +29,7 @@ static void json_context(json_object *j) {
|
||||
char *s = json_stringify(j);
|
||||
|
||||
if (strlen(s) >= 500) {
|
||||
sprintf(s + 497, "...");
|
||||
snprintf(s + 497, strlen(s) + 1 - 497, "...");
|
||||
}
|
||||
|
||||
fprintf(stderr, "in JSON object %s\n", s);
|
||||
|
||||
+15
-48
@@ -10,7 +10,6 @@
|
||||
#include <sys/stat.h>
|
||||
#include <sys/types.h>
|
||||
#include <sys/mman.h>
|
||||
#include <string.h>
|
||||
#include <fcntl.h>
|
||||
#include <ctype.h>
|
||||
#include <errno.h>
|
||||
@@ -41,7 +40,7 @@
|
||||
#include "milo/dtoa_milo.h"
|
||||
#include "errors.hpp"
|
||||
|
||||
int serialize_geojson_feature(struct serialization_state *sst, json_object *geometry, json_object *properties, json_object *id, int layer, json_object *tippecanoe, json_object *feature, std::string layername) {
|
||||
int serialize_geojson_feature(struct serialization_state *sst, json_object *geometry, json_object *properties, json_object *id, int layer, json_object *tippecanoe, json_object *feature, std::string const &layername) {
|
||||
json_object *geometry_type = json_hash_get(geometry, "type");
|
||||
if (geometry_type == NULL) {
|
||||
static int warned = 0;
|
||||
@@ -81,7 +80,7 @@ int serialize_geojson_feature(struct serialization_state *sst, json_object *geom
|
||||
|
||||
int tippecanoe_minzoom = -1;
|
||||
int tippecanoe_maxzoom = -1;
|
||||
std::string tippecanoe_layername;
|
||||
std::string tippecanoe_layername = layername;
|
||||
|
||||
if (tippecanoe != NULL) {
|
||||
json_object *min = json_hash_get(tippecanoe, "minzoom");
|
||||
@@ -106,7 +105,8 @@ int serialize_geojson_feature(struct serialization_state *sst, json_object *geom
|
||||
if (id->type == JSON_NUMBER) {
|
||||
if (id->value.number.number >= 0) {
|
||||
char *err = NULL;
|
||||
id_value = strtoull(milo::dtoa_milo(id->value.number.number).c_str(), &err, 10);
|
||||
std::string id_number = milo::dtoa_milo(id->value.number.number);
|
||||
id_value = strtoull(id_number.c_str(), &err, 10);
|
||||
|
||||
if (id->value.number.large_unsigned != 0) {
|
||||
id_value = id->value.number.large_unsigned;
|
||||
@@ -182,36 +182,19 @@ int serialize_geojson_feature(struct serialization_state *sst, json_object *geom
|
||||
nprop = properties->value.object.length;
|
||||
}
|
||||
|
||||
std::vector<char *> metakey;
|
||||
metakey.resize(nprop);
|
||||
std::vector<std::shared_ptr<std::string>> full_keys;
|
||||
std::vector<serial_val> values;
|
||||
|
||||
std::vector<std::string> metaval;
|
||||
metaval.resize(nprop);
|
||||
|
||||
std::vector<int> metatype;
|
||||
metatype.resize(nprop);
|
||||
|
||||
size_t m = 0;
|
||||
full_keys.reserve(nprop);
|
||||
values.reserve(nprop);
|
||||
key_pool key_pool;
|
||||
|
||||
for (size_t i = 0; i < nprop; i++) {
|
||||
if (properties->value.object.keys[i]->type == JSON_STRING) {
|
||||
std::string s(properties->value.object.keys[i]->value.string.string);
|
||||
serial_val sv = stringify_value(properties->value.object.values[i], sst->fname, sst->line, feature);
|
||||
|
||||
int type = -1;
|
||||
std::string val;
|
||||
stringify_value(properties->value.object.values[i], type, val, sst->fname, sst->line, feature);
|
||||
|
||||
if (type >= 0) {
|
||||
metakey[m] = properties->value.object.keys[i]->value.string.string;
|
||||
metatype[m] = type;
|
||||
metaval[m] = val;
|
||||
m++;
|
||||
} else {
|
||||
metakey[m] = properties->value.object.keys[i]->value.string.string;
|
||||
metatype[m] = mvt_null;
|
||||
metaval[m] = "null";
|
||||
m++;
|
||||
}
|
||||
full_keys.emplace_back(key_pool.pool(properties->value.object.keys[i]->value.string.string));
|
||||
values.push_back(std::move(sv));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -224,31 +207,15 @@ int serialize_geojson_feature(struct serialization_state *sst, json_object *geom
|
||||
sf.t = mb_geometry[t];
|
||||
sf.has_id = has_id;
|
||||
sf.id = id_value;
|
||||
sf.has_tippecanoe_minzoom = (tippecanoe_minzoom != -1);
|
||||
sf.tippecanoe_minzoom = tippecanoe_minzoom;
|
||||
sf.has_tippecanoe_maxzoom = (tippecanoe_maxzoom != -1);
|
||||
sf.tippecanoe_maxzoom = tippecanoe_maxzoom;
|
||||
sf.geometry = dv;
|
||||
sf.feature_minzoom = 0; // Will be filled in during index merging
|
||||
sf.seq = *(sst->layer_seq);
|
||||
sf.full_keys = std::move(full_keys);
|
||||
sf.full_values = std::move(values);
|
||||
|
||||
if (tippecanoe_layername.size() != 0) {
|
||||
sf.layername = tippecanoe_layername;
|
||||
} else {
|
||||
sf.layername = layername;
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < m; i++) {
|
||||
sf.full_keys.push_back(metakey[i]);
|
||||
|
||||
serial_val sv;
|
||||
sv.type = metatype[i];
|
||||
sv.s = metaval[i];
|
||||
|
||||
sf.full_values.push_back(sv);
|
||||
}
|
||||
|
||||
return serialize_feature(sst, sf);
|
||||
return serialize_feature(sst, sf, tippecanoe_layername);
|
||||
}
|
||||
|
||||
void check_crs(json_object *j, const char *reading) {
|
||||
|
||||
+129
-970
File diff suppressed because it is too large
Load Diff
+70
-9
@@ -3,8 +3,13 @@
|
||||
|
||||
#include <vector>
|
||||
#include <atomic>
|
||||
#include <set>
|
||||
#include <string>
|
||||
#include <sqlite3.h>
|
||||
#include <stdio.h>
|
||||
#include <mvt.hpp>
|
||||
#include "jsonpull/jsonpull.h"
|
||||
#include "attribute.hpp"
|
||||
|
||||
#define VT_POINT 1
|
||||
#define VT_LINE 2
|
||||
@@ -46,6 +51,10 @@ struct draw {
|
||||
}
|
||||
}
|
||||
|
||||
bool operator>(draw const &s) const {
|
||||
return s < *this;
|
||||
}
|
||||
|
||||
bool operator==(draw const &s) const {
|
||||
return y == s.y && x == s.x;
|
||||
}
|
||||
@@ -58,22 +67,23 @@ struct draw {
|
||||
typedef std::vector<draw> drawvec;
|
||||
struct serial_feature;
|
||||
|
||||
drawvec decode_geometry(char **meta, int z, unsigned tx, unsigned ty, long long *bbox, unsigned initial_x, unsigned initial_y);
|
||||
drawvec decode_geometry(const char **meta, int z, unsigned tx, unsigned ty, long long *bbox, unsigned initial_x, unsigned initial_y);
|
||||
void to_tile_scale(drawvec &geom, int z, int detail);
|
||||
drawvec from_tile_scale(drawvec const &geom, int z, int detail);
|
||||
drawvec remove_noop(drawvec geom, int type, int shift);
|
||||
drawvec clip_point(drawvec &geom, int z, long long buffer);
|
||||
drawvec clean_or_clip_poly(drawvec &geom, int z, int buffer, bool clip);
|
||||
drawvec simple_clip_poly(drawvec &geom, int z, int buffer);
|
||||
drawvec clean_or_clip_poly(drawvec &geom, int z, int buffer, bool clip, bool try_scaling);
|
||||
drawvec close_poly(drawvec &geom);
|
||||
drawvec reduce_tiny_poly(drawvec &geom, int z, int detail, bool *reduced, double *accum_area, serial_feature *this_feature, serial_feature *tiny_feature);
|
||||
drawvec reduce_tiny_poly(const drawvec &geom, int z, int detail, bool *still_needs_simplification, bool *reduced_away, double *accum_area, double tiny_polygon_size);
|
||||
int clip(long long *x0, long long *y0, long long *x1, long long *y1, long long xmin, long long ymin, long long xmax, long long ymax);
|
||||
drawvec clip_lines(drawvec &geom, int z, long long buffer);
|
||||
drawvec stairstep(drawvec &geom, int z, int detail);
|
||||
bool point_within_tile(long long x, long long y, int z);
|
||||
int quick_check(long long *bbox, int z, long long buffer);
|
||||
drawvec simplify_lines(drawvec &geom, int z, int detail, bool mark_tile_bounds, double simplification, size_t retain, drawvec const &shared_nodes);
|
||||
drawvec reorder_lines(drawvec &geom);
|
||||
drawvec fix_polygon(drawvec &geom);
|
||||
int quick_check(const long long *bbox, int z, long long buffer);
|
||||
void douglas_peucker(drawvec &geom, int start, int n, double e, size_t kept, size_t retain, bool prevent_simplify_shared_nodes);
|
||||
drawvec simplify_lines(drawvec &geom, int z, int tx, int ty, int detail, bool mark_tile_bounds, double simplification, size_t retain, drawvec const &shared_nodes, struct node *shared_nodes_map, size_t nodepos, std::string const &shared_nodes_bloom);
|
||||
drawvec reorder_lines(const drawvec &geom);
|
||||
drawvec fix_polygon(const drawvec &geom, bool use_winding, bool reverse_winding);
|
||||
std::vector<drawvec> chop_polygon(std::vector<drawvec> &geoms);
|
||||
void check_polygon(drawvec &geom);
|
||||
double get_area(const drawvec &geom, size_t i, size_t j);
|
||||
@@ -81,10 +91,61 @@ double get_mp_area(drawvec &geom);
|
||||
drawvec polygon_to_anchor(const drawvec &geom);
|
||||
drawvec checkerboard_anchors(drawvec const &geom, int tx, int ty, int z, unsigned long long label_point);
|
||||
|
||||
drawvec simple_clip_poly(drawvec &geom, long long x1, long long y1, long long x2, long long y2);
|
||||
drawvec simple_clip_poly(drawvec &geom, int z, int buffer, drawvec &shared_nodes, bool prevent_simplify_shared_nodes);
|
||||
drawvec simple_clip_poly(drawvec &geom, long long x1, long long y1, long long x2, long long y2, bool prevent_simplify_shared_nodes);
|
||||
drawvec simple_clip_poly(drawvec &geom, long long x1, long long y1, long long x2, long long y2,
|
||||
long long ax, long long ay, long long bx, long long by, drawvec &shared_nodes, bool prevent_simplify_shared_nodes);
|
||||
drawvec clip_lines(drawvec &geom, long long x1, long long y1, long long x2, long long y2);
|
||||
drawvec clip_point(drawvec &geom, long long x1, long long y1, long long x2, long long y2);
|
||||
void visvalingam(drawvec &ls, size_t start, size_t end, double threshold, size_t retain);
|
||||
int pnpoly(const drawvec &vert, size_t start, size_t nvert, long long testx, long long testy);
|
||||
double distance_from_line(long long point_x, long long point_y, long long segA_x, long long segA_y, long long segB_x, long long segB_y);
|
||||
|
||||
struct input_tile {
|
||||
std::string tile;
|
||||
int z;
|
||||
int x;
|
||||
int y;
|
||||
};
|
||||
|
||||
struct source_tile {
|
||||
mvt_tile tile;
|
||||
int z;
|
||||
int x;
|
||||
int y;
|
||||
};
|
||||
|
||||
std::string overzoom(std::vector<source_tile> const &tiles, int nz, int nx, int ny,
|
||||
int detail, int buffer,
|
||||
std::set<std::string> const &keep,
|
||||
std::set<std::string> const &exclude,
|
||||
std::vector<std::string> const &exclude_prefix,
|
||||
bool do_compress,
|
||||
std::vector<std::pair<unsigned, unsigned>> *next_overzoomed_tiles,
|
||||
bool demultiply, json_object *filter, bool preserve_input_order,
|
||||
std::unordered_map<std::string, attribute_op> const &attribute_accum,
|
||||
std::vector<std::string> const &unidecode_data, double simplification,
|
||||
double tiny_polygon_size,
|
||||
std::vector<mvt_layer> const &bins, std::string const &bin_by_id_list,
|
||||
std::string const &accumulate_numeric, size_t feature_limit);
|
||||
|
||||
std::string overzoom(std::vector<input_tile> const &tiles, int nz, int nx, int ny,
|
||||
int detail, int buffer,
|
||||
std::set<std::string> const &keep,
|
||||
std::set<std::string> const &exclude,
|
||||
std::vector<std::string> const &exclude_prefix,
|
||||
bool do_compress,
|
||||
std::vector<std::pair<unsigned, unsigned>> *next_overzoomed_tiles,
|
||||
bool demultiply, json_object *filter, bool preserve_input_order,
|
||||
std::unordered_map<std::string, attribute_op> const &attribute_accum,
|
||||
std::vector<std::string> const &unidecode_data, double simplification,
|
||||
double tiny_polygon_size,
|
||||
std::vector<mvt_layer> const &bins, std::string const &bin_by_id_list,
|
||||
std::string const &accumulate_numeric, size_t feature_limit);
|
||||
|
||||
draw center_of_mass_mp(const drawvec &dv);
|
||||
|
||||
void get_quadkey_bounds(long long xmin, long long ymin, long long xmax, long long ymax,
|
||||
unsigned long long *start, unsigned long long *end);
|
||||
|
||||
#endif
|
||||
|
||||
+105
-74
@@ -10,6 +10,15 @@
|
||||
|
||||
#define BUFFER 10000
|
||||
|
||||
struct string {
|
||||
char *buf;
|
||||
size_t n;
|
||||
size_t nalloc;
|
||||
};
|
||||
|
||||
static void string_init(struct string *s);
|
||||
static void string_free(struct string *s);
|
||||
|
||||
json_pull *json_begin(ssize_t (*read)(struct json_pull *, char *buffer, size_t n), void *source) {
|
||||
json_pull *j = malloc(sizeof(json_pull));
|
||||
if (j == NULL) {
|
||||
@@ -33,6 +42,13 @@ json_pull *json_begin(ssize_t (*read)(struct json_pull *, char *buffer, size_t n
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
j->number_buffer = malloc(sizeof(struct string));
|
||||
if (j->number_buffer == NULL) {
|
||||
perror("Out of memory");
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
string_init(j->number_buffer);
|
||||
|
||||
return j;
|
||||
}
|
||||
|
||||
@@ -88,6 +104,9 @@ json_pull *json_begin_string(const char *s) {
|
||||
}
|
||||
|
||||
void json_end(json_pull *p) {
|
||||
string_free(p->number_buffer);
|
||||
free(p->number_buffer);
|
||||
|
||||
json_free(p->root);
|
||||
free(p->buffer);
|
||||
free(p);
|
||||
@@ -103,7 +122,7 @@ static inline int read_wrap(json_pull *j) {
|
||||
return c;
|
||||
}
|
||||
|
||||
#define SIZE_FOR(i, size) ((size_t)((((i) + 7) & ~7) * size))
|
||||
#define SIZE_FOR(i, size) ((size_t) ((((i) + 7) & ~7) * size))
|
||||
|
||||
static json_object *fabricate_object(json_pull *jp, json_object *parent, json_type type) {
|
||||
json_object *o = malloc(sizeof(struct json_object));
|
||||
@@ -215,12 +234,6 @@ json_object *json_hash_get(json_object *o, const char *s) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
struct string {
|
||||
char *buf;
|
||||
size_t n;
|
||||
size_t nalloc;
|
||||
};
|
||||
|
||||
static void string_init(struct string *s) {
|
||||
s->nalloc = 500;
|
||||
s->buf = malloc(s->nalloc);
|
||||
@@ -292,38 +305,45 @@ json_object *json_read_separators(json_pull *j, json_separator_callback cb, void
|
||||
}
|
||||
|
||||
again:
|
||||
/////////////////////////// Whitespace
|
||||
|
||||
do {
|
||||
c = read_wrap(j);
|
||||
if (c == EOF) {
|
||||
if (j->container != NULL) {
|
||||
j->error = "Reached EOF without all containers being closed";
|
||||
}
|
||||
|
||||
return NULL;
|
||||
c = read_wrap(j);
|
||||
if (c == EOF) {
|
||||
if (j->container != NULL) {
|
||||
j->error = "Reached EOF without all containers being closed";
|
||||
}
|
||||
|
||||
// Byte-order mark
|
||||
if (c == 0xEF) {
|
||||
int c2 = peek(j);
|
||||
if (c2 == 0xBB) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
switch (c) {
|
||||
/////////////////////////// Byte order mark
|
||||
|
||||
case 0xEF: {
|
||||
int c2 = peek(j);
|
||||
if (c2 == 0xBB) {
|
||||
c2 = read_wrap(j);
|
||||
c2 = peek(j);
|
||||
if (c2 == 0xBF) {
|
||||
c2 = read_wrap(j);
|
||||
c2 = peek(j);
|
||||
if (c2 == 0xBF) {
|
||||
c2 = read_wrap(j);
|
||||
c = ' ';
|
||||
continue;
|
||||
}
|
||||
c = ' ';
|
||||
goto again;
|
||||
}
|
||||
j->error = "Corrupt byte-order mark found";
|
||||
return NULL;
|
||||
}
|
||||
} while (c == ' ' || c == '\t' || c == '\r' || c == '\n' || c == 0x1E);
|
||||
j->error = "Corrupt byte-order mark found";
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/////////////////////////// Arrays
|
||||
/////////////////////////// Whitespace
|
||||
|
||||
if (c == '[') {
|
||||
case ' ':
|
||||
case '\t':
|
||||
case '\r':
|
||||
case '\n':
|
||||
case 0x1E:
|
||||
goto again;
|
||||
|
||||
/////////////////////////// Arrays
|
||||
|
||||
case '[': {
|
||||
json_object *o = add_object(j, JSON_ARRAY);
|
||||
if (o == NULL) {
|
||||
return NULL;
|
||||
@@ -336,7 +356,9 @@ again:
|
||||
}
|
||||
|
||||
goto again;
|
||||
} else if (c == ']') {
|
||||
}
|
||||
|
||||
case ']': {
|
||||
if (j->container == NULL) {
|
||||
j->error = "Found ] at top level";
|
||||
return NULL;
|
||||
@@ -359,9 +381,9 @@ again:
|
||||
return ret;
|
||||
}
|
||||
|
||||
/////////////////////////// Hashes
|
||||
/////////////////////////// Hashes
|
||||
|
||||
if (c == '{') {
|
||||
case '{': {
|
||||
json_object *o = add_object(j, JSON_HASH);
|
||||
if (o == NULL) {
|
||||
return NULL;
|
||||
@@ -374,7 +396,9 @@ again:
|
||||
}
|
||||
|
||||
goto again;
|
||||
} else if (c == '}') {
|
||||
}
|
||||
|
||||
case '}': {
|
||||
if (j->container == NULL) {
|
||||
j->error = "Found } at top level";
|
||||
return NULL;
|
||||
@@ -397,9 +421,9 @@ again:
|
||||
return ret;
|
||||
}
|
||||
|
||||
/////////////////////////// Null
|
||||
/////////////////////////// Null
|
||||
|
||||
if (c == 'n') {
|
||||
case 'n': {
|
||||
if (read_wrap(j) != 'u' || read_wrap(j) != 'l' || read_wrap(j) != 'l') {
|
||||
j->error = "Found misspelling of null";
|
||||
return NULL;
|
||||
@@ -408,9 +432,9 @@ again:
|
||||
return add_object(j, JSON_NULL);
|
||||
}
|
||||
|
||||
/////////////////////////// NaN
|
||||
/////////////////////////// NaN
|
||||
|
||||
if (c == 'N') {
|
||||
case 'N': {
|
||||
if (read_wrap(j) != 'a' || read_wrap(j) != 'N') {
|
||||
j->error = "Found misspelling of NaN";
|
||||
return NULL;
|
||||
@@ -420,9 +444,9 @@ again:
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/////////////////////////// Infinity
|
||||
/////////////////////////// Infinity
|
||||
|
||||
if (c == 'I') {
|
||||
case 'I': {
|
||||
if (read_wrap(j) != 'n' || read_wrap(j) != 'f' || read_wrap(j) != 'i' ||
|
||||
read_wrap(j) != 'n' || read_wrap(j) != 'i' || read_wrap(j) != 't' ||
|
||||
read_wrap(j) != 'y') {
|
||||
@@ -434,9 +458,9 @@ again:
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/////////////////////////// True
|
||||
/////////////////////////// True
|
||||
|
||||
if (c == 't') {
|
||||
case 't': {
|
||||
if (read_wrap(j) != 'r' || read_wrap(j) != 'u' || read_wrap(j) != 'e') {
|
||||
j->error = "Found misspelling of true";
|
||||
return NULL;
|
||||
@@ -445,9 +469,9 @@ again:
|
||||
return add_object(j, JSON_TRUE);
|
||||
}
|
||||
|
||||
/////////////////////////// False
|
||||
/////////////////////////// False
|
||||
|
||||
if (c == 'f') {
|
||||
case 'f': {
|
||||
if (read_wrap(j) != 'a' || read_wrap(j) != 'l' || read_wrap(j) != 's' || read_wrap(j) != 'e') {
|
||||
j->error = "Found misspelling of false";
|
||||
return NULL;
|
||||
@@ -456,9 +480,9 @@ again:
|
||||
return add_object(j, JSON_FALSE);
|
||||
}
|
||||
|
||||
/////////////////////////// Comma
|
||||
/////////////////////////// Comma
|
||||
|
||||
if (c == ',') {
|
||||
case ',': {
|
||||
if (j->container != NULL) {
|
||||
if (j->container->expect != JSON_COMMA) {
|
||||
j->error = "Found unexpected comma";
|
||||
@@ -479,9 +503,9 @@ again:
|
||||
goto again;
|
||||
}
|
||||
|
||||
/////////////////////////// Colon
|
||||
/////////////////////////// Colon
|
||||
|
||||
if (c == ':') {
|
||||
case ':': {
|
||||
if (j->container == NULL) {
|
||||
j->error = "Found colon at top level";
|
||||
return NULL;
|
||||
@@ -501,71 +525,80 @@ again:
|
||||
goto again;
|
||||
}
|
||||
|
||||
/////////////////////////// Numbers
|
||||
/////////////////////////// Numbers
|
||||
|
||||
if (c == '-' || (c >= '0' && c <= '9')) {
|
||||
struct string val;
|
||||
string_init(&val);
|
||||
case '-':
|
||||
case '0':
|
||||
case '1':
|
||||
case '2':
|
||||
case '3':
|
||||
case '4':
|
||||
case '5':
|
||||
case '6':
|
||||
case '7':
|
||||
case '8':
|
||||
case '9': {
|
||||
j->number_buffer->n = 0;
|
||||
int decimal = 0;
|
||||
|
||||
if (c == '-') {
|
||||
string_append(&val, c);
|
||||
string_append(j->number_buffer, c);
|
||||
c = read_wrap(j);
|
||||
}
|
||||
|
||||
if (c == '0') {
|
||||
string_append(&val, c);
|
||||
string_append(j->number_buffer, c);
|
||||
} else if (c >= '1' && c <= '9') {
|
||||
string_append(&val, c);
|
||||
string_append(j->number_buffer, c);
|
||||
c = peek(j);
|
||||
|
||||
while (c >= '0' && c <= '9') {
|
||||
string_append(&val, read_wrap(j));
|
||||
string_append(j->number_buffer, read_wrap(j));
|
||||
c = peek(j);
|
||||
}
|
||||
}
|
||||
|
||||
if (peek(j) == '.') {
|
||||
string_append(&val, read_wrap(j));
|
||||
string_append(j->number_buffer, read_wrap(j));
|
||||
decimal = 1;
|
||||
|
||||
c = peek(j);
|
||||
if (c < '0' || c > '9') {
|
||||
j->error = "Decimal point without digits";
|
||||
string_free(&val);
|
||||
string_free(j->number_buffer);
|
||||
return NULL;
|
||||
}
|
||||
while (c >= '0' && c <= '9') {
|
||||
string_append(&val, read_wrap(j));
|
||||
string_append(j->number_buffer, read_wrap(j));
|
||||
c = peek(j);
|
||||
}
|
||||
}
|
||||
|
||||
c = peek(j);
|
||||
if (c == 'e' || c == 'E') {
|
||||
string_append(&val, read_wrap(j));
|
||||
string_append(j->number_buffer, read_wrap(j));
|
||||
decimal = 1;
|
||||
|
||||
c = peek(j);
|
||||
if (c == '+' || c == '-') {
|
||||
string_append(&val, read_wrap(j));
|
||||
string_append(j->number_buffer, read_wrap(j));
|
||||
}
|
||||
|
||||
c = peek(j);
|
||||
if (c < '0' || c > '9') {
|
||||
j->error = "Exponent without digits";
|
||||
string_free(&val);
|
||||
string_free(j->number_buffer);
|
||||
return NULL;
|
||||
}
|
||||
while (c >= '0' && c <= '9') {
|
||||
string_append(&val, read_wrap(j));
|
||||
string_append(j->number_buffer, read_wrap(j));
|
||||
c = peek(j);
|
||||
}
|
||||
}
|
||||
|
||||
json_object *n = add_object(j, JSON_NUMBER);
|
||||
if (n != NULL) {
|
||||
n->value.number.number = atof(val.buf);
|
||||
n->value.number.number = atof(j->number_buffer->buf);
|
||||
n->value.number.large_signed = 0;
|
||||
n->value.number.large_unsigned = 0;
|
||||
|
||||
@@ -575,7 +608,7 @@ again:
|
||||
if (!decimal && n->value.number.number > MAX_SAFE_INTEGER) {
|
||||
errno = 0;
|
||||
char *err = NULL;
|
||||
unsigned long long ull = strtoull(val.buf, &err, 10);
|
||||
unsigned long long ull = strtoull(j->number_buffer->buf, &err, 10);
|
||||
if (errno == 0 && (err == NULL || *err == '\0')) {
|
||||
n->value.number.large_unsigned = ull;
|
||||
}
|
||||
@@ -583,22 +616,18 @@ again:
|
||||
if (!decimal && n->value.number.number < MIN_SAFE_INTEGER) {
|
||||
errno = 0;
|
||||
char *err = NULL;
|
||||
long long ll = strtoll(val.buf, &err, 10);
|
||||
long long ll = strtoll(j->number_buffer->buf, &err, 10);
|
||||
if (errno == 0 && (err == NULL || *err == '\0')) {
|
||||
n->value.number.large_signed = ll;
|
||||
}
|
||||
}
|
||||
|
||||
string_free(&val);
|
||||
} else {
|
||||
string_free(&val);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/////////////////////////// Strings
|
||||
/////////////////////////// Strings
|
||||
|
||||
if (c == '"') {
|
||||
case '"': {
|
||||
struct string val;
|
||||
string_init(&val);
|
||||
|
||||
@@ -726,11 +755,13 @@ again:
|
||||
json_object *s = add_object(j, JSON_STRING);
|
||||
if (s != NULL) {
|
||||
s->value.string.string = val.buf;
|
||||
s->value.string.refcon = NULL;
|
||||
} else {
|
||||
string_free(&val);
|
||||
}
|
||||
return s;
|
||||
}
|
||||
}
|
||||
|
||||
j->error = "Found unexpected character";
|
||||
return NULL;
|
||||
|
||||
@@ -38,6 +38,7 @@ typedef struct json_object {
|
||||
|
||||
struct {
|
||||
char *string;
|
||||
void *refcon; // reference constant for caller's use
|
||||
} string;
|
||||
|
||||
struct {
|
||||
@@ -68,6 +69,8 @@ typedef struct json_pull {
|
||||
|
||||
json_object *container;
|
||||
json_object *root;
|
||||
|
||||
struct string *number_buffer;
|
||||
} json_pull;
|
||||
|
||||
json_pull *json_begin_file(FILE *f);
|
||||
|
||||
+1
-1
@@ -114,7 +114,7 @@ std::string sort_quote(const char *s) {
|
||||
for (size_t i = 0; i < utf32.size(); i++) {
|
||||
if (utf32[i] < 0xD800) {
|
||||
char buf[8];
|
||||
sprintf(buf, "\\u%04lu", utf32[i]);
|
||||
snprintf(buf, sizeof(buf), "\\u%04lu", utf32[i]);
|
||||
ret.append(std::string(buf));
|
||||
} else {
|
||||
unsigned long c = utf32[i];
|
||||
|
||||
@@ -34,7 +34,7 @@
|
||||
#include <map>
|
||||
#include <cmath>
|
||||
|
||||
#ifdef __APPLE__
|
||||
#if defined(__APPLE__) || defined(__FreeBSD__)
|
||||
#include <sys/types.h>
|
||||
#include <sys/sysctl.h>
|
||||
#include <sys/param.h>
|
||||
@@ -49,7 +49,6 @@
|
||||
#include "tile.hpp"
|
||||
#include "pool.hpp"
|
||||
#include "projection.hpp"
|
||||
#include "version.hpp"
|
||||
#include "memfile.hpp"
|
||||
#include "main.hpp"
|
||||
#include "geojson.hpp"
|
||||
@@ -64,6 +63,10 @@
|
||||
#include "evaluator.hpp"
|
||||
#include "text.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "read_json.hpp"
|
||||
#include "sort.hpp"
|
||||
#include "attribute.hpp"
|
||||
#include "thread.hpp"
|
||||
|
||||
static int low_detail = 12;
|
||||
static int full_detail = -1;
|
||||
@@ -82,11 +85,20 @@ size_t max_tile_size = 500000;
|
||||
size_t max_tile_features = 200000;
|
||||
int cluster_distance = 0;
|
||||
int tiny_polygon_size = 2;
|
||||
int cluster_maxzoom = MAX_ZOOM;
|
||||
long justx = -1, justy = -1;
|
||||
std::string attribute_for_id = "";
|
||||
size_t limit_tile_feature_count = 0;
|
||||
size_t limit_tile_feature_count_at_maxzoom = 0;
|
||||
unsigned int drop_denser = 0;
|
||||
std::map<std::string, serial_val> set_attributes;
|
||||
unsigned long long preserve_point_density_threshold = 0;
|
||||
unsigned long long preserve_multiplier_density_threshold = 0;
|
||||
long long extend_zooms_max = 0;
|
||||
int retain_points_multiplier = 1;
|
||||
std::vector<std::string> unidecode_data;
|
||||
size_t maximum_string_attribute_length = 0;
|
||||
std::string accumulate_numeric;
|
||||
|
||||
std::vector<order_field> order_by;
|
||||
bool order_reverse;
|
||||
@@ -116,7 +128,8 @@ void checkdisk(std::vector<struct reader> *r) {
|
||||
for (size_t i = 0; i < r->size(); i++) {
|
||||
// Pool and tree are used once.
|
||||
// Geometry and index will be duplicated during sorting and tiling.
|
||||
used += 2 * (*r)[i].geompos + 2 * (*r)[i].indexpos + (*r)[i].poolfile->off + (*r)[i].treefile->off;
|
||||
used += 2 * (*r)[i].geompos + 2 * (*r)[i].indexpos + (*r)[i].poolfile->off + (*r)[i].treefile->off +
|
||||
(*r)[i].vertexpos + (*r)[i].nodepos;
|
||||
}
|
||||
|
||||
static int warned = 0;
|
||||
@@ -274,7 +287,7 @@ struct drop_state {
|
||||
double gap;
|
||||
unsigned long long previndex;
|
||||
double interval;
|
||||
double seq;
|
||||
double seq; // floating point because interval is
|
||||
};
|
||||
|
||||
struct drop_densest {
|
||||
@@ -289,8 +302,6 @@ struct drop_densest {
|
||||
|
||||
int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, double gamma) {
|
||||
int feature_minzoom = 0;
|
||||
unsigned xx, yy;
|
||||
decode_index(ix->ix, &xx, &yy);
|
||||
|
||||
if (gamma >= 0 && (ix->t == VT_POINT ||
|
||||
(additional[A_LINE_DROP] && ix->t == VT_LINE) ||
|
||||
@@ -299,11 +310,40 @@ int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, d
|
||||
ds[i].seq++;
|
||||
}
|
||||
for (ssize_t i = maxzoom; i >= 0; i--) {
|
||||
if (ds[i].seq >= 0) {
|
||||
ds[i].seq -= ds[i].interval;
|
||||
} else {
|
||||
if (ds[i].seq < 0) {
|
||||
feature_minzoom = i + 1;
|
||||
|
||||
// The feature we are pushing out
|
||||
// appears in zooms i + 1 through maxzoom,
|
||||
// so track where that was so we can make sure
|
||||
// not to cluster something else that is *too*
|
||||
// far away into it.
|
||||
for (ssize_t j = i + 1; j <= maxzoom; j++) {
|
||||
ds[j].previndex = ix->ix;
|
||||
}
|
||||
|
||||
break;
|
||||
} else {
|
||||
ds[i].seq -= ds[i].interval;
|
||||
}
|
||||
}
|
||||
|
||||
// If this feature has been chosen only for a high zoom level,
|
||||
// check whether at a low zoom level it is nevertheless too far
|
||||
// from the last feature chosen for that low zoom, in which case
|
||||
// we will go ahead and push it out.
|
||||
|
||||
if (preserve_point_density_threshold > 0) {
|
||||
for (ssize_t i = 0; i < feature_minzoom && i < maxzoom; i++) {
|
||||
if (ix->ix - ds[i].previndex > ((1LL << (32 - i)) / preserve_point_density_threshold) * ((1LL << (32 - i)) / preserve_point_density_threshold)) {
|
||||
feature_minzoom = i;
|
||||
|
||||
for (ssize_t j = i; j <= maxzoom; j++) {
|
||||
ds[j].previndex = ix->ix;
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -412,7 +452,7 @@ void *run_sort(void *v) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
|
||||
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::unordered_map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
|
||||
long long segs[CPUS + 1];
|
||||
segs[0] = 0;
|
||||
segs[CPUS] = len;
|
||||
@@ -444,10 +484,10 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
|
||||
sst.resize(CPUS);
|
||||
|
||||
pthread_t pthreads[CPUS];
|
||||
std::vector<std::set<type_and_string> > file_subkeys;
|
||||
std::vector<std::set<serial_val> > file_subkeys;
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
file_subkeys.push_back(std::set<type_and_string>());
|
||||
file_subkeys.push_back(std::set<serial_val>());
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
@@ -482,7 +522,7 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
if (pthread_create(&pthreads[i], NULL, run_parse_json, &pja[i]) != 0) {
|
||||
if (thread_create(&pthreads[i], NULL, run_parse_json, &pja[i]) != 0) {
|
||||
perror("pthread_create");
|
||||
exit(EXIT_PTHREAD);
|
||||
}
|
||||
@@ -613,7 +653,7 @@ struct read_parallel_arg {
|
||||
unsigned *initial_y = NULL;
|
||||
std::string layername = "";
|
||||
bool uses_gamma = false;
|
||||
std::map<std::string, int> const *attribute_types = NULL;
|
||||
std::unordered_map<std::string, int> const *attribute_types = NULL;
|
||||
double *dist_sum = NULL;
|
||||
size_t *dist_count = NULL;
|
||||
double *area_sum = NULL;
|
||||
@@ -657,7 +697,7 @@ void *run_read_parallel(void *v) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::atomic<int> *is_parsing, pthread_t *parallel_parser, bool &parser_created, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > &layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
|
||||
void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::atomic<int> *is_parsing, pthread_t *parallel_parser, bool &parser_created, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > &layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::unordered_map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
|
||||
// This has to kick off an intermediate thread to start the parser threads,
|
||||
// so the main thread can get back to reading the next input stage while
|
||||
// the intermediate thread waits for the completion of the parser threads.
|
||||
@@ -699,7 +739,7 @@ void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::ato
|
||||
rpa->want_dist = want_dist;
|
||||
rpa->filters = filters;
|
||||
|
||||
if (pthread_create(parallel_parser, NULL, run_read_parallel, rpa) != 0) {
|
||||
if (thread_create(parallel_parser, NULL, run_read_parallel, rpa) != 0) {
|
||||
perror("pthread_create");
|
||||
exit(EXIT_PTHREAD);
|
||||
}
|
||||
@@ -722,9 +762,9 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
||||
sub_geompos[i] = 0;
|
||||
|
||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
||||
sprintf(geomname, "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
||||
sprintf(indexname, "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
|
||||
geomfds[i] = mkstemp_cloexec(geomname);
|
||||
if (geomfds[i] < 0) {
|
||||
@@ -895,7 +935,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
||||
}
|
||||
|
||||
for (size_t a = 0; a < CPUS; a++) {
|
||||
if (pthread_create(&pthreads[a], NULL, run_sort, &args[a]) != 0) {
|
||||
if (thread_create(&pthreads[a], NULL, run_sort, &args[a]) != 0) {
|
||||
perror("pthread_create");
|
||||
exit(EXIT_PTHREAD);
|
||||
}
|
||||
@@ -1112,7 +1152,7 @@ void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FI
|
||||
}
|
||||
}
|
||||
|
||||
void choose_first_zoom(long long *file_bbox, std::vector<struct reader> &readers, unsigned *iz, unsigned *ix, unsigned *iy, int minzoom, int buffer) {
|
||||
void choose_first_zoom(long long *file_bbox, long long *file_bbox1, long long *file_bbox2, std::vector<struct reader> &readers, unsigned *iz, unsigned *ix, unsigned *iy, int minzoom, int buffer) {
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
if (readers[i].file_bbox[0] < file_bbox[0]) {
|
||||
file_bbox[0] = readers[i].file_bbox[0];
|
||||
@@ -1126,6 +1166,16 @@ void choose_first_zoom(long long *file_bbox, std::vector<struct reader> &readers
|
||||
if (readers[i].file_bbox[3] > file_bbox[3]) {
|
||||
file_bbox[3] = readers[i].file_bbox[3];
|
||||
}
|
||||
|
||||
file_bbox1[0] = std::min(file_bbox1[0], readers[i].file_bbox1[0]);
|
||||
file_bbox1[1] = std::min(file_bbox1[1], readers[i].file_bbox1[1]);
|
||||
file_bbox1[2] = std::max(file_bbox1[2], readers[i].file_bbox1[2]);
|
||||
file_bbox1[3] = std::max(file_bbox1[3], readers[i].file_bbox1[3]);
|
||||
|
||||
file_bbox2[0] = std::min(file_bbox2[0], readers[i].file_bbox2[0]);
|
||||
file_bbox2[1] = std::min(file_bbox2[1], readers[i].file_bbox2[1]);
|
||||
file_bbox2[2] = std::max(file_bbox2[2], readers[i].file_bbox2[2]);
|
||||
file_bbox2[3] = std::max(file_bbox2[3], readers[i].file_bbox2[3]);
|
||||
}
|
||||
|
||||
// If the bounding box extends off the plane on either side,
|
||||
@@ -1163,7 +1213,39 @@ void choose_first_zoom(long long *file_bbox, std::vector<struct reader> &readers
|
||||
}
|
||||
}
|
||||
|
||||
std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzoom, int basezoom, double basezoom_marker_width, sqlite3 *outdb, const char *outdir, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, json_object *filter, double droprate, int buffer, const char *tmpdir, double gamma, int read_parallel, int forcetable, const char *attribution, bool uses_gamma, long long *file_bbox, const char *prefilter, const char *postfilter, const char *description, bool guess_maxzoom, std::map<std::string, int> const *attribute_types, const char *pgm, std::map<std::string, attribute_op> const *attribute_accum, std::map<std::string, std::string> const &attribute_descriptions, std::string const &commandline, int minimum_maxzoom) {
|
||||
int vertexcmp(const void *void1, const void *void2) {
|
||||
vertex *v1 = (vertex *) void1;
|
||||
vertex *v2 = (vertex *) void2;
|
||||
|
||||
if (v1->mid < v2->mid) {
|
||||
return -1;
|
||||
}
|
||||
if (v1->mid > v2->mid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (v1->p1 < v2->p1) {
|
||||
return -1;
|
||||
}
|
||||
if (v1->p1 > v2->p1) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (v1->p2 < v2->p2) {
|
||||
return -1;
|
||||
}
|
||||
if (v1->p2 > v2->p2) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
double round_droprate(double r) {
|
||||
return std::round(r * 100000.0) / 100000.0;
|
||||
}
|
||||
|
||||
std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzoom, int basezoom, double basezoom_marker_width, sqlite3 *outdb, const char *outdir, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, json_object *filter, double droprate, int buffer, const char *tmpdir, double gamma, int read_parallel, int forcetable, const char *attribution, bool uses_gamma, long long *file_bbox, long long *file_bbox1, long long *file_bbox2, const char *prefilter, const char *postfilter, const char *description, bool guess_maxzoom, bool guess_cluster_maxzoom, std::unordered_map<std::string, int> const *attribute_types, const char *pgm, std::unordered_map<std::string, attribute_op> const *attribute_accum, std::map<std::string, std::string> const &attribute_descriptions, std::string const &commandline, int minimum_maxzoom) {
|
||||
int ret = EXIT_SUCCESS;
|
||||
|
||||
std::vector<struct reader> readers;
|
||||
@@ -1175,11 +1257,15 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
char treename[strlen(tmpdir) + strlen("/tree.XXXXXXXX") + 1];
|
||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
||||
char vertexname[strlen(tmpdir) + strlen("/vertex.XXXXXXXX") + 1];
|
||||
char nodename[strlen(tmpdir) + strlen("/node.XXXXXXXX") + 1];
|
||||
|
||||
sprintf(poolname, "%s%s", tmpdir, "/pool.XXXXXXXX");
|
||||
sprintf(treename, "%s%s", tmpdir, "/tree.XXXXXXXX");
|
||||
sprintf(geomname, "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
sprintf(indexname, "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
snprintf(poolname, sizeof(poolname), "%s%s", tmpdir, "/pool.XXXXXXXX");
|
||||
snprintf(treename, sizeof(treename), "%s%s", tmpdir, "/tree.XXXXXXXX");
|
||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
snprintf(vertexname, sizeof(vertexname), "%s%s", tmpdir, "/vertex.XXXXXXXX");
|
||||
snprintf(nodename, sizeof(nodename), "%s%s", tmpdir, "/node.XXXXXXXX");
|
||||
|
||||
r->poolfd = mkstemp_cloexec(poolname);
|
||||
if (r->poolfd < 0) {
|
||||
@@ -1201,6 +1287,16 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
perror(indexname);
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->vertexfd = mkstemp_cloexec(vertexname);
|
||||
if (r->vertexfd < 0) {
|
||||
perror(vertexname);
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->nodefd = mkstemp_cloexec(nodename);
|
||||
if (r->nodefd < 0) {
|
||||
perror(nodename);
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
r->poolfile = memfile_open(r->poolfd);
|
||||
if (r->poolfile == NULL) {
|
||||
@@ -1222,18 +1318,33 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
perror(indexname);
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->vertexfile = fopen_oflag(vertexname, "w+b", O_RDWR | O_CLOEXEC);
|
||||
if (r->vertexfile == NULL) {
|
||||
perror(("open vertexfile " + std::string(vertexname)).c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->nodefile = fopen_oflag(nodename, "w+b", O_RDWR | O_CLOEXEC);
|
||||
if (r->nodefile == NULL) {
|
||||
perror(nodename);
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->geompos = 0;
|
||||
r->indexpos = 0;
|
||||
r->vertexpos = 0;
|
||||
r->nodepos = 0;
|
||||
|
||||
unlink(poolname);
|
||||
unlink(treename);
|
||||
unlink(geomname);
|
||||
unlink(indexname);
|
||||
unlink(vertexname);
|
||||
unlink(nodename);
|
||||
|
||||
// To distinguish a null value
|
||||
{
|
||||
struct stringpool p;
|
||||
memfile_write(r->treefile, &p, sizeof(struct stringpool));
|
||||
bool in_memory;
|
||||
memfile_write(r->treefile, &p, sizeof(struct stringpool), in_memory);
|
||||
}
|
||||
|
||||
r->file_bbox[0] = r->file_bbox[1] = UINT_MAX;
|
||||
@@ -1638,7 +1749,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
// Serial reading of chunks that are then parsed in parallel
|
||||
|
||||
char readname[strlen(tmpdir) + strlen("/read.XXXXXXXX") + 1];
|
||||
sprintf(readname, "%s%s", tmpdir, "/read.XXXXXXXX");
|
||||
snprintf(readname, sizeof(readname), "%s%s", tmpdir, "/read.XXXXXXXX");
|
||||
int readfd = mkstemp_cloexec(readname);
|
||||
if (readfd < 0) {
|
||||
perror(readname);
|
||||
@@ -1691,7 +1802,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
checkdisk(&readers);
|
||||
ahead = 0;
|
||||
|
||||
sprintf(readname, "%s%s", tmpdir, "/read.XXXXXXXX");
|
||||
snprintf(readname, sizeof(readname), "%s%s", tmpdir, "/read.XXXXXXXX");
|
||||
readfd = mkstemp_cloexec(readname);
|
||||
if (readfd < 0) {
|
||||
perror(readname);
|
||||
@@ -1798,6 +1909,9 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
fflush(stderr);
|
||||
}
|
||||
|
||||
std::atomic<long long> vertexpos(0);
|
||||
std::atomic<long long> nodepos(0);
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
if (fclose(readers[i].geomfile) != 0) {
|
||||
perror("fclose geom");
|
||||
@@ -1813,6 +1927,13 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
perror("stat geom\n");
|
||||
exit(EXIT_STAT);
|
||||
}
|
||||
|
||||
vertexpos += readers[i].vertexpos;
|
||||
nodepos += readers[i].nodepos;
|
||||
}
|
||||
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "Merging string pool \r");
|
||||
}
|
||||
|
||||
// Create a combined string pool
|
||||
@@ -1826,7 +1947,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
|
||||
char poolname[strlen(tmpdir) + strlen("/pool.XXXXXXXX") + 1];
|
||||
sprintf(poolname, "%s%s", tmpdir, "/pool.XXXXXXXX");
|
||||
snprintf(poolname, sizeof(poolname), "%s%s", tmpdir, "/pool.XXXXXXXX");
|
||||
|
||||
int poolfd = mkstemp_cloexec(poolname);
|
||||
if (poolfd < 0) {
|
||||
@@ -1841,7 +1962,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
|
||||
unlink(poolname);
|
||||
|
||||
std::atomic<long long> poolpos(0);
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
@@ -1905,8 +2025,172 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
madvise(stringpool, poolpos, MADV_RANDOM);
|
||||
}
|
||||
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "Merging vertices \r");
|
||||
}
|
||||
|
||||
// Sort the vertices;
|
||||
// find nodes where the same central point is part of two different vertices
|
||||
{
|
||||
std::string tmpname = std::string(tmpdir) + "/vertex2.XXXXXX";
|
||||
int vertexfd = mkstemp((char *) tmpname.c_str());
|
||||
if (vertexfd < 0) {
|
||||
perror(("mkstemp vertexfile " + std::string(tmpname)).c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
unlink(tmpname.c_str());
|
||||
FILE *vertex_out = fdopen(vertexfd, "w+b");
|
||||
if (vertex_out == NULL) {
|
||||
perror(tmpname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
std::vector<FILE *> vertex_readers;
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
vertex_readers.push_back(readers[i].vertexfile);
|
||||
rewind(readers[i].vertexfile);
|
||||
}
|
||||
fqsort(vertex_readers, sizeof(vertex), vertexcmp, vertex_out, memsize / 20);
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
if (fclose(readers[i].vertexfile) != 0) {
|
||||
perror("fclose vertex");
|
||||
exit(EXIT_CLOSE);
|
||||
}
|
||||
}
|
||||
|
||||
rewind(vertex_out);
|
||||
|
||||
vertex prev(draw(VT_MOVETO, 0, 0), draw(VT_MOVETO, 0, 0), draw(VT_MOVETO, 0, 0));
|
||||
vertex v(draw(VT_MOVETO, 0, 0), draw(VT_MOVETO, 0, 0), draw(VT_MOVETO, 0, 0));
|
||||
while (fread((void *) &v, sizeof(vertex), 1, vertex_out)) {
|
||||
if (v.mid == prev.mid && (v.p1 != prev.p1 || v.p2 != prev.p2)) {
|
||||
long long x = v.mid.x * (1LL << geometry_scale);
|
||||
long long y = v.mid.y * (1LL << geometry_scale);
|
||||
|
||||
#if 0
|
||||
double lon, lat;
|
||||
tile2lonlat(x, y, 32, &lon, &lat);
|
||||
printf("{\"type\":\"Feature\", \"properties\":{}, \"geometry\":{\"type\":\"Point\", \"coordinates\":[%f,%f]}}\n", lon, lat);
|
||||
#endif
|
||||
|
||||
struct node n;
|
||||
n.index = encode_vertex((unsigned) x, (unsigned) y);
|
||||
|
||||
fwrite_check((char *) &n, sizeof(struct node), 1, readers[0].nodefile, &readers[0].nodepos, "vertices");
|
||||
}
|
||||
prev = v;
|
||||
}
|
||||
|
||||
fclose(vertex_out);
|
||||
}
|
||||
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "Merging nodes \r");
|
||||
}
|
||||
|
||||
std::string shared_nodes_bloom;
|
||||
shared_nodes_bloom.resize(34567891); // circa 34MB, size nowhere near a power of 2
|
||||
|
||||
// Sort nodes that can't be simplified away; scan the list to remove duplicates
|
||||
|
||||
FILE *shared_nodes;
|
||||
node *shared_nodes_map = NULL; // will be null if there are no shared nodes
|
||||
{
|
||||
// sort
|
||||
|
||||
std::string tmpname = std::string(tmpdir) + "/node2.XXXXXX";
|
||||
int nodefd = mkstemp((char *) tmpname.c_str());
|
||||
if (nodefd < 0) {
|
||||
perror(("mkstemp nodefile " + std::string(tmpname)).c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
unlink(tmpname.c_str());
|
||||
FILE *node_out;
|
||||
node_out = fdopen(nodefd, "w+b");
|
||||
if (node_out == NULL) {
|
||||
perror(tmpname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
std::vector<FILE *> node_readers;
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
node_readers.push_back(readers[i].nodefile);
|
||||
rewind(readers[i].nodefile);
|
||||
}
|
||||
|
||||
fqsort(node_readers, sizeof(node), nodecmp, node_out, memsize / 20);
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
if (fclose(readers[i].nodefile) != 0) {
|
||||
perror("fclose node");
|
||||
exit(EXIT_CLOSE);
|
||||
}
|
||||
}
|
||||
|
||||
rewind(node_out);
|
||||
|
||||
// scan
|
||||
|
||||
tmpname = std::string(tmpdir) + "/node3.XXXXXX";
|
||||
nodefd = mkstemp((char *) tmpname.c_str());
|
||||
if (nodefd < 0) {
|
||||
perror(("mkstemp nodefile " + std::string(tmpname)).c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
unlink(tmpname.c_str());
|
||||
shared_nodes = fdopen(nodefd, "w+b");
|
||||
if (shared_nodes == NULL) {
|
||||
perror(tmpname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
// `written` is to see if this node has already been preserved
|
||||
// and doesn't need to be preserved again
|
||||
struct node written;
|
||||
written.index = ULONG_MAX;
|
||||
|
||||
nodepos = 0;
|
||||
struct node here;
|
||||
while (fread((void *) &here, sizeof(here), 1, node_out)) {
|
||||
if (nodecmp((void *) &here, (void *) &written) != 0) {
|
||||
fwrite_check((void *) &here, sizeof(here), 1, shared_nodes, &nodepos, "shared nodes");
|
||||
written = here;
|
||||
|
||||
size_t bloom_ix = here.index % (shared_nodes_bloom.size() * 8);
|
||||
unsigned char bloom_mask = 1 << (bloom_ix & 7);
|
||||
bloom_ix >>= 3;
|
||||
shared_nodes_bloom[bloom_ix] |= bloom_mask;
|
||||
|
||||
#if 0
|
||||
unsigned wx, wy;
|
||||
decode_quadkey(here.index, &wx, &wy);
|
||||
double lon, lat;
|
||||
tile2lonlat(wx, wy, 32, &lon, &lat);
|
||||
printf("{\"type\":\"Feature\", \"properties\":{}, \"geometry\":{\"type\":\"Point\", \"coordinates\":[%f,%f]}}\n", lon, lat);
|
||||
#endif
|
||||
}
|
||||
}
|
||||
|
||||
fflush(shared_nodes);
|
||||
|
||||
if (nodepos > 0) {
|
||||
shared_nodes_map = (node *) mmap(NULL, nodepos, PROT_READ, MAP_PRIVATE, nodefd, 0);
|
||||
if (shared_nodes_map == (node *) MAP_FAILED) {
|
||||
perror("mmap nodes");
|
||||
exit(EXIT_MEMORY);
|
||||
}
|
||||
}
|
||||
|
||||
fclose(node_out);
|
||||
}
|
||||
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "Merging index \r");
|
||||
}
|
||||
|
||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
||||
sprintf(indexname, "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
|
||||
int indexfd = mkstemp_cloexec(indexname);
|
||||
if (indexfd < 0) {
|
||||
@@ -1922,7 +2206,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
unlink(indexname);
|
||||
|
||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
||||
sprintf(geomname, "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
|
||||
int geomfd = mkstemp_cloexec(geomname);
|
||||
if (geomfd < 0) {
|
||||
@@ -1937,7 +2221,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
unlink(geomname);
|
||||
|
||||
unsigned iz = 0, ix = 0, iy = 0;
|
||||
choose_first_zoom(file_bbox, readers, &iz, &ix, &iy, minzoom, buffer);
|
||||
choose_first_zoom(file_bbox, file_bbox1, file_bbox2, readers, &iz, &ix, &iy, minzoom, buffer);
|
||||
|
||||
if (justx >= 0) {
|
||||
iz = minzoom;
|
||||
@@ -1948,6 +2232,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
std::atomic<long long> geompos(0);
|
||||
|
||||
/* initial tile is normally 0/0/0 but can be iz/ix/iy if limited to one tile */
|
||||
long long estimated_complexity = 0; // to be replaced after writing the data
|
||||
fwrite_check(&estimated_complexity, sizeof(estimated_complexity), 1, geomfile, &geompos, fname);
|
||||
serialize_int(geomfile, iz, &geompos, fname);
|
||||
serialize_uint(geomfile, ix, &geompos, fname);
|
||||
serialize_uint(geomfile, iy, &geompos, fname);
|
||||
@@ -1957,6 +2243,13 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
/* end of tile */
|
||||
serialize_ulong_long(geomfile, 0, &geompos, fname); // EOF
|
||||
|
||||
estimated_complexity = geompos;
|
||||
fflush(geomfile);
|
||||
if (pwrite(fileno(geomfile), &estimated_complexity, sizeof(estimated_complexity), 0) != sizeof(estimated_complexity)) {
|
||||
perror("pwrite estimated complexity");
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
|
||||
if (fclose(geomfile) != 0) {
|
||||
perror("fclose geom");
|
||||
exit(EXIT_CLOSE);
|
||||
@@ -1979,7 +2272,9 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
long long s = progress_seq;
|
||||
long long geompos_print = geompos;
|
||||
long long poolpos_print = poolpos;
|
||||
fprintf(stderr, "%lld features, %lld bytes of geometry, %lld bytes of string pool\n", s, geompos_print, poolpos_print);
|
||||
long long vertexpos_print = vertexpos;
|
||||
long long nodepos_print = nodepos;
|
||||
fprintf(stderr, "%lld features, %lld bytes of geometry and attributes, %lld bytes of string pool, %lld bytes of vertices, %lld bytes of nodes\n", s, geompos_print, poolpos_print, vertexpos_print, nodepos_print);
|
||||
}
|
||||
|
||||
if (indexpos == 0) {
|
||||
@@ -2004,6 +2299,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
double mean = 0;
|
||||
size_t count = 0;
|
||||
double m2 = 0;
|
||||
size_t dupes = 0;
|
||||
|
||||
long long progress = -1;
|
||||
long long ip;
|
||||
@@ -2039,6 +2335,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
mean += delta / count;
|
||||
double delta2 = newValue - mean;
|
||||
m2 += delta * delta2;
|
||||
} else {
|
||||
dupes++;
|
||||
}
|
||||
|
||||
long long nprogress = 100 * ip / indices;
|
||||
@@ -2082,16 +2380,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
double want = nearby_ft / 2;
|
||||
|
||||
maxzoom = ceil(log(360 / (.00000274 * want)) / log(2) - full_detail);
|
||||
if (maxzoom < 0) {
|
||||
maxzoom = 0;
|
||||
}
|
||||
if (maxzoom > 32 - full_detail) {
|
||||
maxzoom = 32 - full_detail;
|
||||
}
|
||||
if (maxzoom > 33 - low_detail) { // that is, maxzoom - 1 > 32 - low_detail
|
||||
maxzoom = 33 - low_detail;
|
||||
}
|
||||
|
||||
if (!quiet) {
|
||||
fprintf(stderr,
|
||||
"Choosing a maxzoom of -z%d for features typically %d feet (%d meters) apart, ",
|
||||
@@ -2102,7 +2390,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
|
||||
bool changed = false;
|
||||
while (maxzoom < 32 - full_detail && maxzoom < 33 - low_detail && cluster_distance > 0) {
|
||||
while (maxzoom < 32 - full_detail && maxzoom < 33 - low_detail && maxzoom < cluster_maxzoom && cluster_distance > 0) {
|
||||
unsigned long long zoom_mingap = ((1LL << (32 - maxzoom)) / 256 * cluster_distance) * ((1LL << (32 - maxzoom)) / 256 * cluster_distance);
|
||||
if (avg > zoom_mingap) {
|
||||
break;
|
||||
@@ -2112,7 +2400,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
changed = true;
|
||||
}
|
||||
if (changed) {
|
||||
printf("Choosing a maxzoom of -z%d to keep most features distinct with cluster distance %d\n", maxzoom, cluster_distance);
|
||||
printf("Choosing a maxzoom of -z%d to keep most features distinct with cluster distance %d and cluster maxzoom %d\n", maxzoom, cluster_distance, cluster_maxzoom);
|
||||
}
|
||||
|
||||
if (droprate == -3) {
|
||||
@@ -2122,7 +2410,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
// features is small, the drop rate should be large because the features are evenly
|
||||
// spaced, and if the standard deviation is large, the drop rate can be small because
|
||||
// the features are in clumps.
|
||||
droprate = exp(-0.7681 * log(stddev) + 1.582);
|
||||
droprate = round_droprate(exp(-0.7681 * log(stddev) + 1.582));
|
||||
|
||||
if (droprate < 0) {
|
||||
droprate = 0;
|
||||
@@ -2131,6 +2419,13 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "Choosing a drop rate of %f\n", droprate);
|
||||
}
|
||||
|
||||
if (dupes != 0 && droprate != 0) {
|
||||
maxzoom += std::round(log((dupes + count) / count) / log(droprate));
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "Increasing maxzoom to %d to account for %zu duplicate feature locations\n", maxzoom, dupes);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2139,16 +2434,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
double want2 = exp(dist_sum / dist_count) / 8;
|
||||
int mz = ceil(log(360 / (.00000274 * want2)) / log(2) - full_detail);
|
||||
|
||||
if (mz < 0) {
|
||||
mz = 0;
|
||||
}
|
||||
if (mz > 32 - full_detail) {
|
||||
mz = 32 - full_detail;
|
||||
}
|
||||
if (mz > 33 - low_detail) { // that is, mz - 1 > 32 - low_detail
|
||||
mz = 33 - low_detail;
|
||||
}
|
||||
|
||||
if (mz > maxzoom || count <= 0) {
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "Choosing a maxzoom of -z%d for resolution of about %d feet (%d meters) within features\n", mz, (int) exp(dist_sum / dist_count), (int) (exp(dist_sum / dist_count) / 3.28084));
|
||||
@@ -2157,6 +2442,16 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
}
|
||||
|
||||
if (maxzoom < 0) {
|
||||
maxzoom = 0;
|
||||
}
|
||||
if (maxzoom > 32 - full_detail) {
|
||||
maxzoom = 32 - full_detail;
|
||||
}
|
||||
if (maxzoom > 33 - low_detail) { // that is, maxzoom - 1 > 32 - low_detail
|
||||
maxzoom = 33 - low_detail;
|
||||
}
|
||||
|
||||
double total_tile_count = 0;
|
||||
for (int i = 1; i <= maxzoom; i++) {
|
||||
double tile_count = ceil(area_sum / ((1LL << (32 - i)) * (1LL << (32 - i))));
|
||||
@@ -2203,6 +2498,11 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
}
|
||||
|
||||
if (cluster_maxzoom >= maxzoom && guess_cluster_maxzoom) {
|
||||
cluster_maxzoom = maxzoom - 1;
|
||||
fprintf(stderr, "Choosing a cluster maxzoom of -k%d to make all features visible at maximum zoom %d\n", cluster_maxzoom, maxzoom);
|
||||
}
|
||||
|
||||
if (basezoom < 0 || droprate < 0) {
|
||||
struct tile {
|
||||
unsigned x;
|
||||
@@ -2312,7 +2612,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
if (maxzoom == 0) {
|
||||
droprate = 2.5;
|
||||
} else {
|
||||
droprate = exp(log((double) max[0].count / max[maxzoom].count) / (maxzoom));
|
||||
droprate = round_droprate(exp(log((double) max[0].count / max[maxzoom].count) / (maxzoom)));
|
||||
if (!quiet) {
|
||||
fprintf(stderr, "Choosing a drop rate of -r%f to get from %lld to %lld in %d zooms\n", droprate, max[maxzoom].count, max[0].count, maxzoom);
|
||||
}
|
||||
@@ -2338,7 +2638,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
|
||||
if (max[z].count / interval >= max_features) {
|
||||
interval = (double) max[z].count / max_features;
|
||||
droprate = exp(log(interval) / (basezoom - z));
|
||||
droprate = round_droprate(exp(log(interval) / (basezoom - z)));
|
||||
interval = exp(log(droprate) * (basezoom - z));
|
||||
|
||||
if (!quiet) {
|
||||
@@ -2427,7 +2727,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
}
|
||||
|
||||
std::sort(ddv.begin(), ddv.end());
|
||||
std::stable_sort(ddv.begin(), ddv.end());
|
||||
|
||||
size_t i = 0;
|
||||
for (int z = 0; z <= basezoom; z++) {
|
||||
@@ -2483,7 +2783,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
std::atomic<unsigned> midx(0);
|
||||
std::atomic<unsigned> midy(0);
|
||||
std::vector<strategy> strategies;
|
||||
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz);
|
||||
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos, shared_nodes_bloom, basezoom, droprate, unidecode_data);
|
||||
|
||||
if (maxzoom != written) {
|
||||
if (written > minzoom) {
|
||||
@@ -2506,6 +2806,9 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
perror("close pool");
|
||||
}
|
||||
|
||||
fclose(shared_nodes);
|
||||
|
||||
// mbtiles-style bounding box and center
|
||||
double minlat = 0, minlon = 0, maxlat = 0, maxlon = 0, midlat = 0, midlon = 0;
|
||||
|
||||
tile2lonlat(midx, midy, maxzoom, &minlon, &maxlat);
|
||||
@@ -2530,6 +2833,17 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
midlon = maxlon;
|
||||
}
|
||||
|
||||
// antimeridian-aware bounding box
|
||||
double minlat2 = 0, minlon2 = 0, maxlat2 = 0, maxlon2 = 0;
|
||||
// choose whichever of the two calculated bboxes is narrower
|
||||
if (file_bbox2[2] - file_bbox2[0] < file_bbox1[2] - file_bbox1[0]) {
|
||||
tile2lonlat(file_bbox2[0], file_bbox2[1], 32, &minlon2, &maxlat2);
|
||||
tile2lonlat(file_bbox2[2], file_bbox2[3], 32, &maxlon2, &minlat2);
|
||||
} else {
|
||||
tile2lonlat(file_bbox1[0], file_bbox1[1], 32, &minlon2, &maxlat2);
|
||||
tile2lonlat(file_bbox1[2], file_bbox1[3], 32, &maxlon2, &minlat2);
|
||||
}
|
||||
|
||||
std::map<std::string, layermap_entry> merged_lm = merge_layermaps(layermaps);
|
||||
|
||||
for (auto ai = merged_lm.begin(); ai != merged_lm.end(); ++ai) {
|
||||
@@ -2537,7 +2851,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
ai->second.maxzoom = maxzoom;
|
||||
}
|
||||
|
||||
metadata m = make_metadata(fname, minzoom, maxzoom, minlat, minlon, maxlat, maxlon, midlat, midlon, attribution, merged_lm, true, description, !prevent[P_TILE_STATS], attribute_descriptions, "tippecanoe", commandline, strategies);
|
||||
metadata m = make_metadata(fname, minzoom, maxzoom, minlat, minlon, maxlat, maxlon, minlat2, minlon2, maxlat2, maxlon2, midlat, midlon, attribution, merged_lm, true, description, !prevent[P_TILE_STATS], attribute_descriptions, "tippecanoe", commandline, strategies, basezoom, droprate, retain_points_multiplier);
|
||||
if (outdb != NULL) {
|
||||
mbtiles_write_metadata(outdb, m, forcetable);
|
||||
} else {
|
||||
@@ -2557,7 +2871,7 @@ static bool has_name(struct option *long_options, int *pl) {
|
||||
return false;
|
||||
}
|
||||
|
||||
void set_attribute_type(std::map<std::string, int> &attribute_types, const char *arg) {
|
||||
void set_attribute_type(std::unordered_map<std::string, int> &attribute_types, const char *arg) {
|
||||
const char *s = strchr(arg, ':');
|
||||
if (s == NULL) {
|
||||
fprintf(stderr, "-T%s option must be in the form -Tname:type\n", arg);
|
||||
@@ -2584,37 +2898,57 @@ void set_attribute_type(std::map<std::string, int> &attribute_types, const char
|
||||
attribute_types.insert(std::pair<std::string, int>(name, t));
|
||||
}
|
||||
|
||||
void set_attribute_accum(std::map<std::string, attribute_op> &attribute_accum, const char *arg) {
|
||||
void set_attribute_value(const char *arg) {
|
||||
if (*arg == '{') {
|
||||
json_pull *jp = json_begin_string(arg);
|
||||
json_object *o = json_read_tree(jp);
|
||||
|
||||
if (o == NULL) {
|
||||
fprintf(stderr, "%s: --set-attribute %s: %s\n", *av, arg, jp->error);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
if (o->type != JSON_HASH) {
|
||||
fprintf(stderr, "%s: --set-attribute %s: not a JSON object\n", *av, arg);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < o->value.object.length; i++) {
|
||||
json_object *k = o->value.object.keys[i];
|
||||
json_object *v = o->value.object.values[i];
|
||||
|
||||
if (k->type != JSON_STRING) {
|
||||
fprintf(stderr, "%s: --set-attribute %s: key %zu not a string\n", *av, arg, i);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
serial_val val = stringify_value(v, "json", 1, o);
|
||||
set_attributes.emplace(k->value.string.string, val);
|
||||
}
|
||||
|
||||
json_free(o);
|
||||
json_end(jp);
|
||||
return;
|
||||
}
|
||||
|
||||
const char *s = strchr(arg, ':');
|
||||
if (s == NULL) {
|
||||
fprintf(stderr, "-E%s option must be in the form -Ename:method\n", arg);
|
||||
fprintf(stderr, "--set-attribute %s option must be in the form --set-attribute name:value\n", arg);
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
|
||||
std::string name = std::string(arg, s - arg);
|
||||
std::string type = std::string(s + 1);
|
||||
attribute_op t;
|
||||
std::string value = std::string(s + 1);
|
||||
|
||||
if (type == "sum") {
|
||||
t = op_sum;
|
||||
} else if (type == "product") {
|
||||
t = op_product;
|
||||
} else if (type == "mean") {
|
||||
t = op_mean;
|
||||
} else if (type == "max") {
|
||||
t = op_max;
|
||||
} else if (type == "min") {
|
||||
t = op_min;
|
||||
} else if (type == "concat") {
|
||||
t = op_concat;
|
||||
} else if (type == "comma") {
|
||||
t = op_comma;
|
||||
serial_val val;
|
||||
if (isdigit(value[0]) || value[0] == '-') {
|
||||
val.type = mvt_double;
|
||||
} else {
|
||||
fprintf(stderr, "Attribute method (%s) must be sum, product, mean, max, min, concat, or comma\n", type.c_str());
|
||||
exit(EXIT_ARGS);
|
||||
val.type = mvt_string;
|
||||
}
|
||||
|
||||
attribute_accum.insert(std::pair<std::string, attribute_op>(name, t));
|
||||
val.s = value;
|
||||
set_attributes.insert(std::pair<std::string, serial_val>(name, val));
|
||||
}
|
||||
|
||||
void parse_json_source(const char *arg, struct source &src) {
|
||||
@@ -2692,10 +3026,11 @@ int main(int argc, char **argv) {
|
||||
const char *postfilter = NULL;
|
||||
bool guess_maxzoom = false;
|
||||
int minimum_maxzoom = 0;
|
||||
bool guess_cluster_maxzoom = false;
|
||||
|
||||
std::set<std::string> exclude, include;
|
||||
std::map<std::string, int> attribute_types;
|
||||
std::map<std::string, attribute_op> attribute_accum;
|
||||
std::unordered_map<std::string, int> attribute_types;
|
||||
std::unordered_map<std::string, attribute_op> attribute_accum;
|
||||
std::map<std::string, std::string> attribute_descriptions;
|
||||
int exclude_all = 0;
|
||||
int read_parallel = 0;
|
||||
@@ -2736,6 +3071,8 @@ int main(int argc, char **argv) {
|
||||
{"minimum-zoom", required_argument, 0, 'Z'},
|
||||
{"smallest-maximum-zoom-guess", required_argument, 0, '~'},
|
||||
{"extend-zooms-if-still-dropping", no_argument, &additional[A_EXTEND_ZOOMS], 1},
|
||||
{"extend-zooms-if-still-dropping-maximum", required_argument, 0, '~'},
|
||||
{"generate-variable-depth-tile-pyramid", no_argument, &additional[A_VARIABLE_DEPTH_PYRAMID], 1},
|
||||
{"one-tile", required_argument, 0, 'R'},
|
||||
|
||||
{"Tile resolution", 0, 0, 0},
|
||||
@@ -2753,23 +3090,31 @@ int main(int argc, char **argv) {
|
||||
{"attribute-type", required_argument, 0, 'T'},
|
||||
{"attribute-description", required_argument, 0, 'Y'},
|
||||
{"accumulate-attribute", required_argument, 0, 'E'},
|
||||
{"accumulate-numeric-attributes", required_argument, 0, '~'},
|
||||
{"empty-csv-columns-are-null", no_argument, &prevent[P_EMPTY_CSV_COLUMNS], 1},
|
||||
{"convert-stringified-ids-to-numbers", no_argument, &additional[A_CONVERT_NUMERIC_IDS], 1},
|
||||
{"use-attribute-for-id", required_argument, 0, '~'},
|
||||
{"single-precision", no_argument, &prevent[P_SINGLE_PRECISION], 1},
|
||||
{"set-attribute", required_argument, 0, '~'},
|
||||
{"maximum-string-attribute-length", required_argument, 0, '~'},
|
||||
|
||||
{"Filtering features by attributes", 0, 0, 0},
|
||||
{"feature-filter-file", required_argument, 0, 'J'},
|
||||
{"feature-filter", required_argument, 0, 'j'},
|
||||
{"unidecode-data", required_argument, 0, '~'},
|
||||
|
||||
{"Dropping a fixed fraction of features by zoom level", 0, 0, 0},
|
||||
{"drop-rate", required_argument, 0, 'r'},
|
||||
{"retain-points-multiplier", required_argument, 0, '~'},
|
||||
{"base-zoom", required_argument, 0, 'B'},
|
||||
{"drop-denser", required_argument, 0, '~'},
|
||||
{"limit-base-zoom-to-maximum-zoom", no_argument, &prevent[P_BASEZOOM_ABOVE_MAXZOOM], 1},
|
||||
{"drop-lines", no_argument, &additional[A_LINE_DROP], 1},
|
||||
{"drop-polygons", no_argument, &additional[A_POLYGON_DROP], 1},
|
||||
{"cluster-distance", required_argument, 0, 'K'},
|
||||
{"cluster-maxzoom", required_argument, 0, 'k'},
|
||||
{"preserve-point-density-threshold", required_argument, 0, '~'},
|
||||
{"preserve-multiplier-density-threshold", required_argument, 0, '~'},
|
||||
|
||||
{"Dropping or merging a fraction of features to keep under tile size limits", 0, 0, 0},
|
||||
{"drop-densest-as-needed", no_argument, &additional[A_DROP_DENSEST_AS_NEEDED], 1},
|
||||
@@ -2791,6 +3136,7 @@ int main(int argc, char **argv) {
|
||||
{"simplify-only-low-zooms", no_argument, &prevent[P_SIMPLIFY_LOW], 1},
|
||||
{"simplification-at-maximum-zoom", required_argument, 0, '~'},
|
||||
{"no-tiny-polygon-reduction", no_argument, &prevent[P_TINY_POLYGON_REDUCTION], 1},
|
||||
{"no-tiny-polygon-reduction-at-maximum-zoom", no_argument, &prevent[P_TINY_POLYGON_REDUCTION_AT_MAXZOOM], 1},
|
||||
{"tiny-polygon-size", required_argument, 0, '~'},
|
||||
{"no-simplification-of-shared-nodes", no_argument, &prevent[P_SIMPLIFY_SHARED_NODES], 1},
|
||||
{"visvalingam", no_argument, &additional[A_VISVALINGAM], 1},
|
||||
@@ -2818,6 +3164,7 @@ int main(int argc, char **argv) {
|
||||
{"Adding calculated attributes", 0, 0, 0},
|
||||
{"calculate-feature-density", no_argument, &additional[A_CALCULATE_FEATURE_DENSITY], 1},
|
||||
{"generate-ids", no_argument, &additional[A_GENERATE_IDS], 1},
|
||||
{"calculate-feature-index", no_argument, &additional[A_CALCULATE_INDEX], 1},
|
||||
|
||||
{"Trying to correct bad source geometry", 0, 0, 0},
|
||||
{"detect-longitude-wraparound", no_argument, &additional[A_DETECT_WRAPAROUND], 1},
|
||||
@@ -2929,6 +3276,8 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
} else if (strcmp(opt, "use-attribute-for-id") == 0) {
|
||||
attribute_for_id = optarg;
|
||||
} else if (strcmp(opt, "set-attribute") == 0) {
|
||||
set_attribute_value(optarg);
|
||||
} else if (strcmp(opt, "smallest-maximum-zoom-guess") == 0) {
|
||||
maxzoom = MAX_ZOOM;
|
||||
guess_maxzoom = true;
|
||||
@@ -2975,6 +3324,20 @@ int main(int argc, char **argv) {
|
||||
fprintf(stderr, "%s: --drop-denser can be at most 100\n", argv[0]);
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
} else if (strcmp(opt, "preserve-point-density-threshold") == 0) {
|
||||
preserve_point_density_threshold = atoll_require(optarg, "Preserve point density threshold");
|
||||
} else if (strcmp(opt, "preserve-multiplier-density-threshold") == 0) {
|
||||
preserve_multiplier_density_threshold = atoll_require(optarg, "Preserve multiplier density threshold");
|
||||
} else if (strcmp(opt, "extend-zooms-if-still-dropping-maximum") == 0) {
|
||||
extend_zooms_max = atoll_require(optarg, "Maximum number by which to extend zooms");
|
||||
} else if (strcmp(opt, "retain-points-multiplier") == 0) {
|
||||
retain_points_multiplier = atoll_require(optarg, "Multiply the fraction of points retained by zoom level");
|
||||
} else if (strcmp(opt, "unidecode-data") == 0) {
|
||||
unidecode_data = read_unidecode(optarg);
|
||||
} else if (strcmp(opt, "maximum-string-attribute-length") == 0) {
|
||||
maximum_string_attribute_length = atoll_require(optarg, "Maximum string attribute length");
|
||||
} else if (strcmp(opt, "accumulate-numeric-attributes") == 0) {
|
||||
accumulate_numeric = optarg;
|
||||
} else {
|
||||
fprintf(stderr, "%s: Unrecognized option --%s\n", argv[0], opt);
|
||||
exit(EXIT_ARGS);
|
||||
@@ -3073,6 +3436,15 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
break;
|
||||
|
||||
case 'k':
|
||||
if (strcmp(optarg, "g") == 0) {
|
||||
cluster_maxzoom = MAX_ZOOM - 1;
|
||||
guess_cluster_maxzoom = true;
|
||||
} else {
|
||||
cluster_maxzoom = atoi_require(optarg, "Cluster maxzoom");
|
||||
}
|
||||
break;
|
||||
|
||||
case 'd':
|
||||
full_detail = atoi_require(optarg, "Full detail");
|
||||
if (full_detail > 30) {
|
||||
@@ -3248,7 +3620,7 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
|
||||
case 'v':
|
||||
fprintf(stderr, "tippecanoe %s\n", VERSION);
|
||||
fprintf(stderr, "tippecanoe %s\n", version_str().c_str());
|
||||
exit(EXIT_SUCCESS);
|
||||
|
||||
case 'P':
|
||||
@@ -3288,7 +3660,7 @@ int main(int argc, char **argv) {
|
||||
break;
|
||||
|
||||
case 'E':
|
||||
set_attribute_accum(attribute_accum, optarg);
|
||||
set_attribute_accum(attribute_accum, optarg, argv);
|
||||
break;
|
||||
|
||||
default: {
|
||||
@@ -3413,7 +3785,7 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
}
|
||||
|
||||
if (extra_detail >= 0) {
|
||||
if (extra_detail >= 0 || prevent[P_SIMPLIFY_SHARED_NODES] || additional[A_EXTEND_ZOOMS] || extend_zooms_max > 0) {
|
||||
geometry_scale = 0;
|
||||
} else {
|
||||
geometry_scale = 32 - (full_detail + maxzoom);
|
||||
@@ -3482,9 +3854,12 @@ int main(int argc, char **argv) {
|
||||
|
||||
long long file_bbox[4] = {UINT_MAX, UINT_MAX, 0, 0};
|
||||
|
||||
long long file_bbox1[4] = {0xFFFFFFFF, 0xFFFFFFFF, 0, 0}; // standard -180 to 180 world plane
|
||||
long long file_bbox2[4] = {0x1FFFFFFFF, 0xFFFFFFFF, 0x100000000, 0}; // 0 to 360 world plane
|
||||
|
||||
auto input_ret = read_input(sources, name ? name : out_mbtiles ? out_mbtiles
|
||||
: out_dir,
|
||||
maxzoom, minzoom, basezoom, basezoom_marker_width, outdb, out_dir, &exclude, &include, exclude_all, filter, droprate, buffer, tmpdir, gamma, read_parallel, forcetable, attribution, gamma != 0, file_bbox, prefilter, postfilter, description, guess_maxzoom, &attribute_types, argv[0], &attribute_accum, attribute_descriptions, commandline, minimum_maxzoom);
|
||||
maxzoom, minzoom, basezoom, basezoom_marker_width, outdb, out_dir, &exclude, &include, exclude_all, filter, droprate, buffer, tmpdir, gamma, read_parallel, forcetable, attribution, gamma != 0, file_bbox, file_bbox1, file_bbox2, prefilter, postfilter, description, guess_maxzoom, guess_cluster_maxzoom, &attribute_types, argv[0], &attribute_accum, attribute_descriptions, commandline, minimum_maxzoom);
|
||||
|
||||
ret = std::get<0>(input_ret);
|
||||
|
||||
|
||||
@@ -5,8 +5,10 @@
|
||||
#include <atomic>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <map>
|
||||
|
||||
#include "json_logger.hpp"
|
||||
#include "serial.hpp"
|
||||
|
||||
struct index {
|
||||
long long start = 0;
|
||||
@@ -53,10 +55,17 @@ extern size_t memsize;
|
||||
extern size_t max_tile_size;
|
||||
extern size_t max_tile_features;
|
||||
extern int cluster_distance;
|
||||
extern int cluster_maxzoom;
|
||||
extern std::string attribute_for_id;
|
||||
extern int tiny_polygon_size;
|
||||
extern size_t limit_tile_feature_count;
|
||||
extern size_t limit_tile_feature_count_at_maxzoom;
|
||||
extern std::map<std::string, serial_val> set_attributes;
|
||||
extern long long extend_zooms_max;
|
||||
extern int retain_points_multiplier;
|
||||
extern size_t maximum_string_attribute_length;
|
||||
extern std::string accumulate_numeric;
|
||||
extern unsigned long long preserve_multiplier_density_threshold;
|
||||
|
||||
struct order_field {
|
||||
std::string name;
|
||||
|
||||
+98
-11
@@ -5,6 +5,8 @@ like these \[la]MADE_WITH.md\[ra]\&.
|
||||
.PP
|
||||
This is the official home of Tippecanoe, developed and actively maintained by Erica Fischer \[la]https://github.com/e-n-f\[ra] at Felt \[la]https://felt.com\[ra]\&.
|
||||
.PP
|
||||
For a self\-hosted, API driven version of Tippecanoe, contact a technical sales engineer at \[la]sales@felt.com\[ra]\&. Felt produces highly performant, automatically projected versions of your data, and utilizes a rendering engine, built on top of MapLibre GL JS \[la]https://github.com/maplibre/maplibre-gl-js\[ra], to style vector and raster data.
|
||||
.PP
|
||||
Version 2.0.0 is equivalent to 1.36.0 \[la]https://github.com/mapbox/tippecanoe/tree/1.36.0\[ra] in the original repository. Thank you Mapbox for the many years of early support.
|
||||
.SH Intent
|
||||
.PP
|
||||
@@ -27,7 +29,15 @@ see the shape and relative popularity of every point of interest and every
|
||||
significant travel corridor.
|
||||
.SH Installation
|
||||
.PP
|
||||
For this fork you will need to build from the source repository:
|
||||
The easiest way to install tippecanoe on OSX is with Homebrew \[la]http://brew.sh/\[ra]:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
$ brew install tippecanoe
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
On Ubuntu it will usually be easiest to build from the source repository:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
@@ -50,7 +60,7 @@ $ tippecanoe \-o file.mbtiles [options] [file.json file.json.gz file.fgb ...]
|
||||
.RE
|
||||
.PP
|
||||
If no files are specified, it reads GeoJSON from the standard input.
|
||||
If multiple files are specified, each is placed in its own layer.
|
||||
If multiple files are specified, each is placed in its own layer \[la]#input-files-and-layer-names\[ra]\&.
|
||||
.PP
|
||||
The GeoJSON features need not be wrapped in a FeatureCollection.
|
||||
You can concatenate multiple GeoJSON features or files together,
|
||||
@@ -395,7 +405,7 @@ Parallel processing will also be automatic if the input file is in FlatGeobuf fo
|
||||
.IP \(bu 2
|
||||
\fB\fC\-zg\fR or \fB\fC\-\-maximum\-zoom=g\fR: Guess what is probably a reasonable maxzoom based on the spacing of features.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-smallest\-maximum\-zoom\-guess=\fR\fIzoom\fP: Guess what is probably a reasonable maxzoom based on the spacing of features, but using the specified \fIzoom\fP if a lower maxzoom is guessed.
|
||||
\fB\fC\-\-smallest\-maximum\-zoom\-guess=\fR\fIzoom\fP: Guess what is probably a reasonable maxzoom based on the spacing of features, but using the specified \fIzoom\fP if a lower maxzoom is guessed. If \fB\fC\-Bg\fR is also set, the base zoom will be set to the guessed maxzoom, with all the points carried forward into additional zooms through the one specified.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-Z\fR \fIzoom\fP or \fB\fC\-\-minimum\-zoom=\fR\fIzoom\fP: Minzoom: the lowest zoom level for which tiles are generated (default 0)
|
||||
.IP \(bu 2
|
||||
@@ -403,6 +413,11 @@ Parallel processing will also be automatic if the input file is in FlatGeobuf fo
|
||||
The detail and simplification options that ordinarily apply only to the maximum zoom level will apply both to the originally
|
||||
specified maximum zoom and to any levels added beyond that.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-extend\-zooms\-if\-still\-dropping\-maximum=\fR\fIcount\fP: Increase the maxzoom if features are still being dropped at that zoom level
|
||||
by up to \fIcount\fP zoom levels.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-at\fR or \fB\fC\-\-generate\-variable\-depth\-tile\-pyramid\fR: Don't produce child tiles for any tile that should be sufficient to be overzoomed to any higher zoom level. Such tiles will be produced with maximum detail and no simplification or polygon cleaning. Tiles with point features below the basezoom or where any features have to be dropped dynamically, or which contain too many features or bytes with full detail, will be written out with normal detail and split into child tiles. Tilesets generated with this option are suitable for use only with tile servers that will find the appropriate tile to overzoom from and will simplify and clean the geometries appropriately before serving the tile.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-R\fR \fIzoom\fP\fB\fC/\fR\fIx\fP\fB\fC/\fR\fIy\fP or \fB\fC\-\-one\-tile=\fR\fIzoom\fP\fB\fC/\fR\fIx\fP\fB\fC/\fR\fIy\fP: Set the minzoom and maxzoom to \fIzoom\fP and produce only
|
||||
the single specified tile at that zoom level.
|
||||
.RE
|
||||
@@ -460,8 +475,8 @@ zoom level precision (ft) precision (m) map scale
|
||||
\fB\fC\-z18\fR 1.5 in 4 cm 1:1250
|
||||
\fB\fC\-z19\fR 0.8 in 2 cm 1:600
|
||||
\fB\fC\-z20\fR 0.4 in 1 cm 1:300
|
||||
\fB\fC\-z21\fR 0.2 in 0.5 cm 1:150
|
||||
\fB\fC\-z22\fR 0.1 in 0.25 cm 1:75
|
||||
\fB\fC\-z21\fR 0.4 in 1 cm 1:300
|
||||
\fB\fC\-z22\fR 0.4 in 1 cm 1:300
|
||||
.TE
|
||||
.SS Tile resolution
|
||||
.RS
|
||||
@@ -503,6 +518,10 @@ If the type is \fB\fCint\fR and the original attribute was floating\-point, it i
|
||||
that are dropped, coalesced\-as\-needed, or clustered. The \fIoperation\fP may be
|
||||
\fB\fCsum\fR, \fB\fCproduct\fR, \fB\fCmean\fR, \fB\fCmax\fR, \fB\fCmin\fR, \fB\fCconcat\fR, or \fB\fCcomma\fR
|
||||
to specify how the named \fIattribute\fP is accumulated onto the attribute of the same name in a feature that does survive.
|
||||
The attributes and operations may also be specified as JSON keys and values: \fB\fC\-\-accumulate\-attribute='{"attr": "operation", "attr2", "operation2"}'\fR\&.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-set\-attribute\fR \fIattribute\fP\fB\fC:\fR\fIvalue\fP: Set the value of the specified \fIattribute\fP in each feature to the specified \fIvalue\fP\&. This is mostly useful to give an attribute in each feature an initial value for \fB\fC\-\-accumulate\-attribute\fR\&.
|
||||
The attributes and values may also be specified as JSON keys and values: \fB\fC\-\-set\-attribute='{"attr": value, "attr2", value}'\fR\&.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-pe\fR or \fB\fC\-\-empty\-csv\-columns\-are\-null\fR: Treat empty CSV columns as nulls rather than as empty strings.
|
||||
.IP \(bu 2
|
||||
@@ -511,11 +530,13 @@ to specify how the named \fIattribute\fP is accumulated onto the attribute of th
|
||||
\fB\fC\-\-use\-attribute\-for\-id=\fR\fIname\fP: Use the attribute with the specified \fIname\fP as if it were specified as the feature ID. (If this attribute is a stringified number, you must also use \fB\fC\-aI\fR to convert it to a number.)
|
||||
.IP \(bu 2
|
||||
\fB\fC\-pN\fR or \fB\fC\-\-single\-precision\fR: Write double\-precision numeric attribute values to tiles as single\-precision to reduce tile size.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-maximum\-string\-attribute\-length\fR=\fIlength\fP: Truncate string attributes that exceed the specified length in bytes.
|
||||
.RE
|
||||
.SS Filtering features by attributes
|
||||
.RS
|
||||
.IP \(bu 2
|
||||
\fB\fC\-j\fR \fIfilter\fP or \fB\fC\-\-feature\-filter\fR=\fIfilter\fP: Check features against a per\-layer filter (as defined in the Mapbox GL Style Specification \[la]https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter\[ra]) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer \fB\fC"*"\fR apply to all layers. The special variable \fB\fC$zoom\fR refers to the current zoom level.
|
||||
\fB\fC\-j\fR \fIfilter\fP or \fB\fC\-\-feature\-filter\fR=\fIfilter\fP: Check features against a per\-layer filter (as defined in the Mapbox GL Style Specification \[la]https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter\[ra] or in a Felt filter specification still to be finalized) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer \fB\fC"*"\fR apply to all layers. The special variable \fB\fC$zoom\fR refers to the current zoom level.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-J\fR \fIfilter\-file\fP or \fB\fC\-\-feature\-filter\-file\fR=\fIfilter\-file\fP: Like \fB\fC\-j\fR, but read the filter from a file.
|
||||
.RE
|
||||
@@ -563,6 +584,8 @@ If you use \fB\fC\-Bg\fR, it will guess a zoom level that will keep at most 50,0
|
||||
You can also specify a marker\-width with \fB\fC\-Bg\fR\fIwidth\fP to allow fewer features in the densest tile to
|
||||
compensate for the larger marker, or \fB\fC\-Bf\fR\fInumber\fP to allow at most \fInumber\fP features in the densest tile.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-retain\-points\-multiplier=\fR\fImultiple\fP: Retain the specified multiple of points instead of just the number of points that would ordinarily be retained by the drop rate. These can be thinned out later with the \fB\fC\-m\fR option to \fB\fCtippecanoe\-overzoom\fR\&. The start of each cluster is marked in the feature sequence by the \fB\fCtippecanoe:retain_points_multiplier_first\fR attribute. The \fB\fC\-\-tile\-size\-limit\fR will also be extended at low zoom levels to allow for the multiplied features.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-drop\-denser=\fR\fIpercentage\fP: When dropping dots at zoom levels below the base zoom, give the specified \fIpercentage\fP
|
||||
preference to retaining points in sparse areas and dropping points in dense areas.
|
||||
.IP \(bu 2
|
||||
@@ -573,6 +596,12 @@ preference to retaining points in sparse areas and dropping points in dense area
|
||||
\fB\fC\-ap\fR or \fB\fC\-\-drop\-polygons\fR: Let "dot" dropping at lower zooms apply to polygons too
|
||||
.IP \(bu 2
|
||||
\fB\fC\-K\fR \fIdistance\fP or \fB\fC\-\-cluster\-distance=\fR\fIdistance\fP: Cluster points (as with \fB\fC\-\-cluster\-densest\-as\-needed\fR, but without the experimental discovery process) that are approximately within \fIdistance\fP of each other. The units are tile coordinates within a nominally 256\-pixel tile, so the maximum value of 255 allows only one feature per tile. Values around 10 are probably appropriate for typical marker sizes. See \fB\fC\-\-cluster\-densest\-as\-needed\fR below for behavior.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-k\fR \fIzoom\fP or \fB\fC\-\-cluster\-maxzoom=\fR\fIzoom\fP: Max zoom on which to cluster points if clustering is enabled.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-kg\fR or \fB\fC\-\-cluster\-maxzoom=g\fR: Set \fB\fC\-\-cluster\-maxzoom=\fR to \fB\fCmaxzoom \- 1\fR so that all features are visible at the maximum zoom level.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-preserve\-point\-density\-threshold=\fR\fIlevel\fP: At the low zoom levels, do not reduce point density below the specified \fIlevel\fP, even if the specfied drop rate would normally call for it, so that low\-density areas of the map do not appear blank. The unit is the distance between preserved points, as a fraction of the size of a tile. Values of 32 or 64 are probably appropriate for typical marker sizes.
|
||||
.RE
|
||||
.SS Dropping a fraction of features to keep under tile size limits
|
||||
.RS
|
||||
@@ -612,10 +641,12 @@ the line or polygon within one tile unit of its proper location. You can probabl
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-simplification\-at\-maximum\-zoom=\fR\fIscale\fP: Use the specified \fIscale\fP at maxzoom instead of the standard simplification scale (which still applies at lower zooms)
|
||||
.IP \(bu 2
|
||||
\fB\fC\-pn\fR or \fB\fC\-\-no\-simplification\-of\-shared\-nodes\fR: Don't simplify away nodes that appear in more than one feature or are used multiple times within the same feature, so that the intersection node will not be lost from intersecting roads. (This will not be effective if you also use \fB\fC\-\-coalesce\fR or \fB\fC\-\-detect\-shared\-borders\fR\&.)
|
||||
\fB\fC\-pn\fR or \fB\fC\-\-no\-simplification\-of\-shared\-nodes\fR: Don't simplify away nodes at which LineStrings or Polygon rings converge, diverge, or cross. (This will not be effective if you also use \fB\fC\-\-coalesce\fR\&.) In between intersection nodes, LineString segments or polygon edges will be simplified identically in each feature if possible. Use this instead of \fB\fC\-\-detect\-shared\-borders\fR\&.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-pt\fR or \fB\fC\-\-no\-tiny\-polygon\-reduction\fR: Don't combine the area of very small polygons into small squares that represent their combined area.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-pT\fR or \fB\fC\-\-no\-tiny\-polygon\-reduction\-at\-maximum\-zoom\fR: Combine the area of very small polygons into small squares that represent their combined area only at zoom levels below the maximum.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-tiny\-polygon\-size=\fR\fIsize\fP: Use the specified \fIsize\fP for tiny polygons instead of the default 2. Anything above 6 or so will lead to visible artifacts with the default tile detail.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-av\fR or \fB\fC\-\-visvalingam\fR: Use Visvalingam's simplification algorithm rather than Douglas\-Peucker's.
|
||||
@@ -623,7 +654,7 @@ the line or polygon within one tile unit of its proper location. You can probabl
|
||||
.SS Attempts to improve shared polygon boundaries
|
||||
.RS
|
||||
.IP \(bu 2
|
||||
\fB\fC\-ab\fR or \fB\fC\-\-detect\-shared\-borders\fR: In the manner of TopoJSON \[la]https://github.com/mbostock/topojson/wiki/Introduction\[ra], detect borders that are shared between multiple polygons and simplify them identically in each polygon. This takes more time and memory than considering each polygon individually.
|
||||
\fB\fC\-ab\fR or \fB\fC\-\-detect\-shared\-borders\fR: DEPRECATED. In the manner of TopoJSON \[la]https://github.com/mbostock/topojson/wiki/Introduction\[ra], detect borders that are shared between multiple polygons and simplify them identically in each polygon. This takes more time and memory than considering each polygon individually. Use \fB\fCno\-simplification\-of\-shared\-nodes\fR instead, which is faster and more correct.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-aL\fR or \fB\fC\-\-grid\-low\-zooms\fR: At all zoom levels below \fImaxzoom\fP, snap all lines and polygons to a stairstep grid instead of allowing diagonals. You will also want to specify a tile resolution, probably \fB\fC\-D8\fR\&. This option provides a way to display continuous parcel, gridded, or binned data at low zooms without overwhelming the tiles with tiny polygons, since features will either get stretched out to the grid unit or lost entirely, depending on how they happened to be aligned in the original data. You probably don't want to use this.
|
||||
.RE
|
||||
@@ -663,6 +694,8 @@ the line or polygon within one tile unit of its proper location. You can probabl
|
||||
\fB\fC\-ag\fR or \fB\fC\-\-calculate\-feature\-density\fR: Add a new attribute, \fB\fCtippecanoe_feature_density\fR, to each feature, to record how densely features are spaced in that area of the tile. You can use this attribute in the style to produce a glowing effect where points are densely packed. It can range from 0 in the sparsest areas to 255 in the densest.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-ai\fR or \fB\fC\-\-generate\-ids\fR: Add an \fB\fCid\fR (a feature ID, not an attribute named \fB\fCid\fR) to each feature that does not already have one. There is currently no guarantee that the \fB\fCid\fR added will be stable between runs or that it will not conflict with manually\-assigned feature IDs. Future versions of Tippecanoe may change the mechanism for allocating IDs.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-aX\fR or \fB\fC\-\-calculate\-feature\-index\fR: Add a \fB\fCtippecanoe:index\fR field to each feature, giving its index in the quadkey or hilbert sequence.
|
||||
.RE
|
||||
.SS Trying to correct bad source geometry
|
||||
.RS
|
||||
@@ -859,7 +892,7 @@ this minimum size and others will not be drawn at all, preserving the total area
|
||||
all of them should have had together.
|
||||
.PP
|
||||
Features in the same tile that share the same type and attributes are coalesced
|
||||
together into a single geometry if you use \fB\fC\-\-coalesce\fR\&. You are strongly encouraged to use \-x to exclude
|
||||
together into a single geometry if you use \fB\fC\-\-coalesce\fR\&. You are strongly encouraged to use \fB\fC\-x\fR to exclude
|
||||
any unnecessary attributes to reduce wasted file size.
|
||||
.PP
|
||||
If a tile is larger than 500K, it will try encoding that tile at progressively
|
||||
@@ -873,7 +906,7 @@ Linux:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
sudo apt\-get install build\-essential libsqlite3\-dev zlib1g\-dev
|
||||
sudo apt\-get install gcc g++ make libsqlite3\-dev zlib1g\-dev
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
@@ -952,6 +985,15 @@ The options are:
|
||||
\fB\fC\-e\fR \fIdirectory\fP or \fB\fC\-\-output\-to\-directory=\fR\fIdirectory\fP: Write the new tiles to the specified directory instead of to an mbtiles file.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-f\fR or \fB\fC\-\-force\fR: Remove \fIout.mbtiles\fP if it already exists.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-r\fR or \fB\fC\-\-read\-from\fR: list of input mbtiles to read from.
|
||||
.RE
|
||||
.SS Overzooming
|
||||
.RS
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-overzoom\fR: If one of the source tilesets has a larger maxzoom than the others, scale up tiles from the tilesets with the lower maxzooms so they will all have the same maxzoom in the output tileset.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-buffer=\fR\fIpixels\fP or \fB\fC\-b\fR \fIpixels\fP: Set the size of the tile buffer in the overzoomed tiles.
|
||||
.RE
|
||||
.SS Tileset description and attribution
|
||||
.RS
|
||||
@@ -986,10 +1028,12 @@ The options are:
|
||||
.SS Filtering features and feature attributes
|
||||
.RS
|
||||
.IP \(bu 2
|
||||
\fB\fC\-x\fR \fIkey\fP or \fB\fC\-\-exclude=\fR\fIkey\fP: Remove attributes of type \fIkey\fP from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want.
|
||||
\fB\fC\-x\fR \fIkey\fP or \fB\fC\-\-exclude=\fR\fIkey\fP: Remove attributes named \fIkey\fP from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want. You can use multiple \fB\fC\-x\fR options to remove multiple attributes.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-X\fR or \fB\fC\-\-exclude\-all\fR: Remove all attributes from the output.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-y\fR \fIkey\fP or \fB\fC\-\-include=\fR\fIkey\fP: Remove all attributes except for those named \fIkey\fP from the output. You can use multiple \fB\fC\-y\fR options to retain multiple attributes.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-i\fR or \fB\fC\-\-if\-matched\fR: Only include features that matched the CSV.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-j\fR \fIfilter\fP or \fB\fC\-\-feature\-filter\fR=\fIfilter\fP: Check features against a per\-layer filter (as defined in the Mapbox GL Style Specification \[la]https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter\[ra]) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer \fB\fC"*"\fR apply to all layers.
|
||||
@@ -1205,3 +1249,46 @@ Join block geometries to employment attributes:
|
||||
$ tippecanoe\-json\-tool \-c in_wac_S000_JT00_2015.csv tl_2010_18157_tabblock10.sort.json > blocks\-wac.json
|
||||
.fi
|
||||
.RE
|
||||
.SH tippecanoe\-overzoom
|
||||
.PP
|
||||
The \fB\fCtippecanoe\-overzoom\fR utility creates a vector tile from one of its parent tiles,
|
||||
clipping and scaling the geometry from the parent tile and excluding features that
|
||||
are clipped away. The idea is that if you create very high resolution tiles
|
||||
(using \fB\fC\-\-extra\-detail\fR) at a moderate zoom level, you can use \fB\fCtippecanoe\-overzoom\fR
|
||||
to turn those into moderate detail tiles at high zoom levels, for the benefit of
|
||||
renderers that cannot internally overzoom high\-resolution tiles without losing
|
||||
some of the precision. Running:
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
tippecanoe\-overzoom \-o out.mvt.gz in.mvt.gz inz/inx/iny outz/outx/outy
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
reads tile \fB\fCinz/inx/iny\fR of \fB\fCin.mvt.gz\fR and produces tile \fB\fCoutz/outx/outy\fR of \fB\fCout.mvt.gz\fR\&.
|
||||
.PP
|
||||
.RS
|
||||
.nf
|
||||
tippecanoe\-overzoom \-o out.mvt.gz \-t outz/outx/outy in.mvt.gz inz/inx/iny in2.mvt.gz in2z/in2x/in2y in3.mvt.gz in3z/in3x/in3y
|
||||
.fi
|
||||
.RE
|
||||
.PP
|
||||
reads tile \fB\fCinz/inx/iny\fR of \fB\fCin.mvt.gz\fR, tile \fB\fCin2z/in2x/in2y\fR of \fB\fCin2.mvt.gz\fR, and tile \fB\fCin3z/in3x/in3y\fR of \fB\fCin3.mvt.gz\fR,
|
||||
and produces tile \fB\fCoutz/outx/outy\fR of \fB\fCout.mvt.gz\fR from them.
|
||||
.SS Options
|
||||
.RS
|
||||
.IP \(bu 2
|
||||
\fB\fC\-b\fR \fIbuffer\fP: Set the tile buffer in the output tile (default 5)
|
||||
.IP \(bu 2
|
||||
\fB\fC\-d\fR \fIdetail\fP: Set the detail of the output tile (default 12)
|
||||
.IP \(bu 2
|
||||
\fB\fC\-y\fR \fIattribute\fP: Retain the specified \fIattribute\fP in the output features. All attributes that are not named in a \fB\fC\-y\fR option will be removed.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-j\fR \fIfilter\fP: Filter features using the same expression syntax as in tippecanoe.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-m\fR: If a tile was created with the \fB\fC\-\-retain\-points\-multiplier\fR option, thin the tile back down to its normal feature count during overzooming. The first feature from each cluster will be retained, unless \fB\fC\-j\fR is used to specify a filter, in which case the first matching filter from each cluster will be retained instead.
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-preserve\-input\-order\fR: Restore a set of filtered features to its original input order
|
||||
.IP \(bu 2
|
||||
\fB\fC\-\-accumulate\-attribute\fR: Behaves as in \fB\fCtippecanoe\fR to sum attributes from the features of a multiplier cluster that are not included in the final output. The attributes from features that are filtered away with \fB\fC\-j\fR are \fInot\fP accumulated onto the output feature.
|
||||
.RE
|
||||
|
||||
+127
-68
@@ -7,6 +7,8 @@
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sqlite3.h>
|
||||
#include <cmath>
|
||||
#include <climits>
|
||||
#include <vector>
|
||||
#include <string>
|
||||
#include <set>
|
||||
@@ -100,16 +102,7 @@ sqlite3 *mbtiles_open(char *dbname, char **argv, int forcetable) {
|
||||
}
|
||||
|
||||
void mbtiles_write_tile(sqlite3 *outdb, int z, int tx, int ty, const char *data, int size) {
|
||||
// Store tiles by a hash of their contents (fnv1a 64-bit)
|
||||
// http://www.isthe.com/chongo/tech/comp/fnv/
|
||||
const unsigned long long fnv_offset_basis = 14695981039346656037u;
|
||||
const unsigned long long fnv_prime = 1099511628211u;
|
||||
unsigned long long h = fnv_offset_basis;
|
||||
for (int i = 0; i < size; i++) {
|
||||
h ^= (unsigned char) data[i];
|
||||
h *= fnv_prime;
|
||||
}
|
||||
std::string hash = std::to_string(h);
|
||||
std::string hash = std::to_string(fnv1a(std::string(data, size)));
|
||||
|
||||
// following https://github.com/mapbox/node-mbtiles/blob/master/lib/mbtiles.js
|
||||
|
||||
@@ -190,21 +183,21 @@ void mbtiles_erase_zoom(sqlite3 *outdb, int z) {
|
||||
}
|
||||
}
|
||||
|
||||
bool type_and_string::operator<(const type_and_string &o) const {
|
||||
if (string < o.string) {
|
||||
bool serial_val::operator<(const serial_val &o) const {
|
||||
if (s < o.s) {
|
||||
return true;
|
||||
}
|
||||
if (string == o.string && type < o.type) {
|
||||
if (s == o.s && type < o.type) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool type_and_string::operator!=(const type_and_string &o) const {
|
||||
bool serial_val::operator!=(const serial_val &o) const {
|
||||
if (type != o.type) {
|
||||
return true;
|
||||
}
|
||||
if (string != o.string) {
|
||||
if (s != o.s) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
@@ -255,7 +248,7 @@ void tilestats(std::map<std::string, layermap_entry> const &layermap1, size_t el
|
||||
state.nospace = true;
|
||||
state.json_write_string(geomtype);
|
||||
|
||||
size_t attrib_count = layer.second.file_keys.size();
|
||||
size_t attrib_count = layer.second.tilestats.size();
|
||||
if (attrib_count > max_tilestats_attributes) {
|
||||
attrib_count = max_tilestats_attributes;
|
||||
}
|
||||
@@ -271,7 +264,7 @@ void tilestats(std::map<std::string, layermap_entry> const &layermap1, size_t el
|
||||
state.json_write_array();
|
||||
|
||||
size_t attrs = 0;
|
||||
for (auto attribute : layer.second.file_keys) {
|
||||
for (auto attribute : layer.second.tilestats) {
|
||||
if (attrs == elements) {
|
||||
break;
|
||||
}
|
||||
@@ -332,15 +325,15 @@ void tilestats(std::map<std::string, layermap_entry> const &layermap1, size_t el
|
||||
vals++;
|
||||
|
||||
state.nospace = true;
|
||||
state.json_write_stringified(value.string);
|
||||
state.json_write_stringified(value.s);
|
||||
} else {
|
||||
std::string trunc = truncate16(value.string, 256);
|
||||
std::string trunc = truncate16(value.s, 256);
|
||||
|
||||
if (trunc.size() == value.string.size()) {
|
||||
if (trunc.size() == value.s.size()) {
|
||||
vals++;
|
||||
|
||||
state.nospace = true;
|
||||
state.json_write_string(value.string);
|
||||
state.json_write_string(value.s);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -451,6 +444,14 @@ std::string stringify_strategies(std::vector<strategy> const &strategies) {
|
||||
any = true;
|
||||
}
|
||||
|
||||
if (strategies[i].truncated_zooms > 0) {
|
||||
state.nospace = true;
|
||||
state.json_write_string("truncated_zooms");
|
||||
state.nospace = true;
|
||||
state.json_write_number(strategies[i].truncated_zooms);
|
||||
any = true;
|
||||
}
|
||||
|
||||
state.nospace = true;
|
||||
state.json_end_hash();
|
||||
}
|
||||
@@ -530,6 +531,15 @@ void mbtiles_write_metadata(sqlite3 *db, const metadata &m, bool forcetable) {
|
||||
}
|
||||
sqlite3_free(sql);
|
||||
|
||||
sql = sqlite3_mprintf("INSERT INTO metadata (name, value) VALUES ('antimeridian_adjusted_bounds', '%f,%f,%f,%f');", m.minlon2, m.minlat2, m.maxlon2, m.maxlat2);
|
||||
if (sqlite3_exec(db, sql, NULL, NULL, &err) != SQLITE_OK) {
|
||||
fprintf(stderr, "set bounds: %s\n", err);
|
||||
if (!forcetable) {
|
||||
exit(EXIT_SQLITE);
|
||||
}
|
||||
}
|
||||
sqlite3_free(sql);
|
||||
|
||||
sql = sqlite3_mprintf("INSERT INTO metadata (name, value) VALUES ('type', %Q);", m.type.c_str());
|
||||
if (sqlite3_exec(db, sql, NULL, NULL, &err) != SQLITE_OK) {
|
||||
fprintf(stderr, "set type: %s\n", err);
|
||||
@@ -588,6 +598,17 @@ void mbtiles_write_metadata(sqlite3 *db, const metadata &m, bool forcetable) {
|
||||
sqlite3_free(sql);
|
||||
}
|
||||
|
||||
if (m.decisions_json.size() > 0) {
|
||||
sql = sqlite3_mprintf("INSERT INTO metadata (name, value) VALUES ('tippecanoe_decisions', %Q);", m.decisions_json.c_str());
|
||||
if (sqlite3_exec(db, sql, NULL, NULL, &err) != SQLITE_OK) {
|
||||
fprintf(stderr, "set decisions: %s\n", err);
|
||||
if (!forcetable) {
|
||||
exit(EXIT_SQLITE);
|
||||
}
|
||||
}
|
||||
sqlite3_free(sql);
|
||||
}
|
||||
|
||||
if (m.vector_layers_json.size() > 0 || m.tilestats_json.size() > 0) {
|
||||
std::string json;
|
||||
json_writer state(&json);
|
||||
@@ -629,7 +650,22 @@ void mbtiles_write_metadata(sqlite3 *db, const metadata &m, bool forcetable) {
|
||||
}
|
||||
}
|
||||
|
||||
metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minlat, double minlon, double maxlat, double maxlon, double midlat, double midlon, const char *attribution, std::map<std::string, layermap_entry> const &layermap, bool vector, const char *description, bool do_tilestats, std::map<std::string, std::string> const &attribute_descriptions, std::string const &program, std::string const &commandline, std::vector<strategy> const &strategies) {
|
||||
static double sixdig(double val) {
|
||||
return std::round(val * 1e6) / 1e6;
|
||||
}
|
||||
|
||||
#define str(x) #x
|
||||
#define xstr(x) str(x)
|
||||
std::string version_str() {
|
||||
std::string s = VERSION;
|
||||
std::string build_info = xstr(BUILD_INFO);
|
||||
if (build_info.size() > 0) {
|
||||
s += " " + build_info;
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minlat, double minlon, double maxlat, double maxlon, double minlat2, double minlon2, double maxlat2, double maxlon2, double midlat, double midlon, const char *attribution, std::map<std::string, layermap_entry> const &layermap, bool vector, const char *description, bool do_tilestats, std::map<std::string, std::string> const &attribute_descriptions, std::string const &program, std::string const &commandline, std::vector<strategy> const &strategies, int basezoom, double droprate, int retain_points_multiplier) {
|
||||
metadata m;
|
||||
|
||||
m.name = fname;
|
||||
@@ -641,24 +677,40 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
|
||||
m.minzoom = minzoom;
|
||||
m.maxzoom = maxzoom;
|
||||
|
||||
m.minlat = minlat;
|
||||
m.minlon = minlon;
|
||||
m.maxlat = maxlat;
|
||||
m.maxlon = maxlon;
|
||||
m.minlat = sixdig(minlat);
|
||||
m.minlon = sixdig(minlon);
|
||||
m.maxlat = sixdig(maxlat);
|
||||
m.maxlon = sixdig(maxlon);
|
||||
|
||||
m.center_lat = midlat;
|
||||
m.center_lon = midlon;
|
||||
m.minlat2 = sixdig(minlat2);
|
||||
m.minlon2 = sixdig(minlon2);
|
||||
m.maxlat2 = sixdig(maxlat2);
|
||||
m.maxlon2 = sixdig(maxlon2);
|
||||
|
||||
m.center_lat = sixdig(midlat);
|
||||
m.center_lon = sixdig(midlon);
|
||||
m.center_z = maxzoom;
|
||||
|
||||
if (attribution != NULL) {
|
||||
m.attribution = attribution;
|
||||
}
|
||||
|
||||
m.generator = program + " " + VERSION;
|
||||
m.generator = program + " " + version_str();
|
||||
m.generator_options = commandline;
|
||||
|
||||
m.strategies_json = stringify_strategies(strategies);
|
||||
|
||||
if (isinf(droprate)) {
|
||||
droprate = LLONG_MAX;
|
||||
}
|
||||
if (basezoom != maxzoom || droprate != 2.5 || retain_points_multiplier != 1) {
|
||||
m.decisions_json = std::string("{") +
|
||||
"\"basezoom\":" + milo::dtoa_milo(basezoom) + "," +
|
||||
"\"droprate\":" + milo::dtoa_milo(droprate) + "," +
|
||||
"\"retain_points_multiplier\":" + std::to_string(retain_points_multiplier) +
|
||||
std::string("}");
|
||||
}
|
||||
|
||||
if (vector) {
|
||||
{
|
||||
json_writer state(&m.vector_layers_json);
|
||||
@@ -672,7 +724,7 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < lnames.size(); i++) {
|
||||
auto fk = layermap.find(lnames[i]);
|
||||
auto ts = layermap.find(lnames[i]);
|
||||
state.nospace = true;
|
||||
state.json_write_hash();
|
||||
|
||||
@@ -684,17 +736,17 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
|
||||
state.nospace = true;
|
||||
state.json_write_string("description");
|
||||
state.nospace = true;
|
||||
state.json_write_string(fk->second.description);
|
||||
state.json_write_string(ts->second.description);
|
||||
|
||||
state.nospace = true;
|
||||
state.json_write_string("minzoom");
|
||||
state.nospace = true;
|
||||
state.json_write_signed(fk->second.minzoom);
|
||||
state.json_write_signed(ts->second.minzoom);
|
||||
|
||||
state.nospace = true;
|
||||
state.json_write_string("maxzoom");
|
||||
state.nospace = true;
|
||||
state.json_write_signed(fk->second.maxzoom);
|
||||
state.json_write_signed(ts->second.maxzoom);
|
||||
|
||||
state.nospace = true;
|
||||
state.json_write_string("fields");
|
||||
@@ -703,7 +755,7 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
|
||||
|
||||
bool first = true;
|
||||
size_t attribute_count = 0;
|
||||
for (auto j = fk->second.file_keys.begin(); j != fk->second.file_keys.end(); ++j) {
|
||||
for (auto j = ts->second.tilestats.begin(); j != ts->second.tilestats.end(); ++j) {
|
||||
if (first) {
|
||||
first = false;
|
||||
}
|
||||
@@ -810,35 +862,35 @@ std::map<std::string, layermap_entry> merge_layermaps(std::vector<std::map<std::
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
for (auto fk = map->second.file_keys.begin(); fk != map->second.file_keys.end(); ++fk) {
|
||||
std::string attribname = fk->first;
|
||||
for (auto ts = map->second.tilestats.begin(); ts != map->second.tilestats.end(); ++ts) {
|
||||
std::string attribname = ts->first;
|
||||
if (trunc) {
|
||||
attribname = truncate16(attribname, 256);
|
||||
}
|
||||
|
||||
auto fk2 = out_entry->second.file_keys.find(attribname);
|
||||
auto ts2 = out_entry->second.tilestats.find(attribname);
|
||||
|
||||
if (fk2 == out_entry->second.file_keys.end()) {
|
||||
out_entry->second.file_keys.insert(std::pair<std::string, type_and_string_stats>(attribname, fk->second));
|
||||
if (ts2 == out_entry->second.tilestats.end()) {
|
||||
out_entry->second.tilestats.insert(std::pair<std::string, tilestat>(attribname, ts->second));
|
||||
} else {
|
||||
for (auto val : fk->second.sample_values) {
|
||||
auto pt = std::lower_bound(fk2->second.sample_values.begin(), fk2->second.sample_values.end(), val);
|
||||
if (pt == fk2->second.sample_values.end() || *pt != val) { // not found
|
||||
fk2->second.sample_values.insert(pt, val);
|
||||
for (auto val : ts->second.sample_values) {
|
||||
auto pt = std::lower_bound(ts2->second.sample_values.begin(), ts2->second.sample_values.end(), val);
|
||||
if (pt == ts2->second.sample_values.end() || *pt != val) { // not found
|
||||
ts2->second.sample_values.insert(pt, val);
|
||||
|
||||
if (fk2->second.sample_values.size() > max_tilestats_sample_values) {
|
||||
fk2->second.sample_values.pop_back();
|
||||
if (ts2->second.sample_values.size() > max_tilestats_sample_values) {
|
||||
ts2->second.sample_values.pop_back();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fk2->second.type |= fk->second.type;
|
||||
ts2->second.type |= ts->second.type;
|
||||
|
||||
if (fk->second.min < fk2->second.min) {
|
||||
fk2->second.min = fk->second.min;
|
||||
if (ts->second.min < ts2->second.min) {
|
||||
ts2->second.min = ts->second.min;
|
||||
}
|
||||
if (fk->second.max > fk2->second.max) {
|
||||
fk2->second.max = fk->second.max;
|
||||
if (ts->second.max > ts2->second.max) {
|
||||
ts2->second.max = ts->second.max;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -859,41 +911,48 @@ std::map<std::string, layermap_entry> merge_layermaps(std::vector<std::map<std::
|
||||
return out;
|
||||
}
|
||||
|
||||
void add_to_file_keys(std::map<std::string, type_and_string_stats> &file_keys, std::string const &attrib, type_and_string const &val) {
|
||||
void add_to_tilestats(std::map<std::string, tilestat> &tilestats, std::string const &attrib, serial_val const &val) {
|
||||
if (val.type == mvt_null) {
|
||||
return;
|
||||
}
|
||||
|
||||
auto fka = file_keys.find(attrib);
|
||||
if (fka == file_keys.end()) {
|
||||
file_keys.insert(std::pair<std::string, type_and_string_stats>(attrib, type_and_string_stats()));
|
||||
fka = file_keys.find(attrib);
|
||||
auto tsa = tilestats.find(attrib);
|
||||
if (tsa == tilestats.end()) {
|
||||
tilestats.insert(std::pair<std::string, tilestat>(attrib, tilestat()));
|
||||
tsa = tilestats.find(attrib);
|
||||
}
|
||||
|
||||
if (fka == file_keys.end()) {
|
||||
if (tsa == tilestats.end()) {
|
||||
fprintf(stderr, "Can't happen (tilestats)\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
if (val.type == mvt_double) {
|
||||
double d = atof(val.string.c_str());
|
||||
double d = atof(val.s.c_str());
|
||||
|
||||
if (d < fka->second.min) {
|
||||
fka->second.min = d;
|
||||
if (d < tsa->second.min) {
|
||||
tsa->second.min = d;
|
||||
}
|
||||
if (d > fka->second.max) {
|
||||
fka->second.max = d;
|
||||
if (d > tsa->second.max) {
|
||||
tsa->second.max = d;
|
||||
}
|
||||
}
|
||||
|
||||
auto pt = std::lower_bound(fka->second.sample_values.begin(), fka->second.sample_values.end(), val);
|
||||
if (pt == fka->second.sample_values.end() || *pt != val) { // not found
|
||||
fka->second.sample_values.insert(pt, val);
|
||||
|
||||
if (fka->second.sample_values.size() > max_tilestats_sample_values) {
|
||||
fka->second.sample_values.pop_back();
|
||||
auto pt = std::lower_bound(tsa->second.sample_values.begin(), tsa->second.sample_values.end(), val);
|
||||
if (pt == tsa->second.sample_values.end() || *pt != val) { // not found
|
||||
if (tsa->second.sample_values.size() >= max_tilestats_sample_values) {
|
||||
if (pt == tsa->second.sample_values.end()) {
|
||||
// insertion point would be at the end,
|
||||
// and the list is already full, so do nothing
|
||||
} else {
|
||||
// bump the former last value, insert this one
|
||||
tsa->second.sample_values.insert(pt, val);
|
||||
tsa->second.sample_values.pop_back();
|
||||
}
|
||||
} else {
|
||||
tsa->second.sample_values.insert(pt, val);
|
||||
}
|
||||
}
|
||||
|
||||
fka->second.type |= (1 << val.type);
|
||||
tsa->second.type |= (1 << val.type);
|
||||
}
|
||||
|
||||
+8
-13
@@ -9,16 +9,8 @@ extern size_t max_tilestats_attributes;
|
||||
extern size_t max_tilestats_sample_values;
|
||||
extern size_t max_tilestats_values;
|
||||
|
||||
struct type_and_string {
|
||||
int type = 0;
|
||||
std::string string = "";
|
||||
|
||||
bool operator<(const type_and_string &o) const;
|
||||
bool operator!=(const type_and_string &o) const;
|
||||
};
|
||||
|
||||
struct type_and_string_stats {
|
||||
std::vector<type_and_string> sample_values = std::vector<type_and_string>(); // sorted
|
||||
struct tilestat {
|
||||
std::vector<serial_val> sample_values = std::vector<serial_val>(); // sorted
|
||||
double min = INFINITY;
|
||||
double max = -INFINITY;
|
||||
int type = 0;
|
||||
@@ -26,7 +18,7 @@ struct type_and_string_stats {
|
||||
|
||||
struct layermap_entry {
|
||||
size_t id = 0;
|
||||
std::map<std::string, type_and_string_stats> file_keys{};
|
||||
std::map<std::string, tilestat> tilestats{};
|
||||
int minzoom = 0;
|
||||
int maxzoom = 0;
|
||||
std::string description = "";
|
||||
@@ -52,6 +44,7 @@ struct metadata {
|
||||
int maxzoom;
|
||||
|
||||
double minlat, minlon, maxlat, maxlon;
|
||||
double minlat2, minlon2, maxlat2, maxlon2; // antimeridian-aware
|
||||
|
||||
double center_lon, center_lat;
|
||||
int center_z;
|
||||
@@ -65,6 +58,7 @@ struct metadata {
|
||||
|
||||
std::string vector_layers_json;
|
||||
std::string tilestats_json;
|
||||
std::string decisions_json;
|
||||
};
|
||||
|
||||
#include "tile.hpp"
|
||||
@@ -74,7 +68,7 @@ sqlite3 *mbtiles_open(char *dbname, char **argv, int forcetable);
|
||||
void mbtiles_write_tile(sqlite3 *outdb, int z, int tx, int ty, const char *data, int size);
|
||||
void mbtiles_erase_zoom(sqlite3 *outdb, int z);
|
||||
|
||||
metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minlat, double minlon, double maxlat, double maxlon, double midlat, double midlon, const char *attribution, std::map<std::string, layermap_entry> const &layermap, bool vector, const char *description, bool do_tilestats, std::map<std::string, std::string> const &attribute_descriptions, std::string const &program, std::string const &commandline, std::vector<strategy> const &strategies);
|
||||
metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minlat, double minlon, double maxlat, double maxlon, double minlat2, double minlon2, double maxlat2, double maxlon2, double midlat, double midlon, const char *attribution, std::map<std::string, layermap_entry> const &layermap, bool vector, const char *description, bool do_tilestats, std::map<std::string, std::string> const &attribute_descriptions, std::string const &program, std::string const &commandline, std::vector<strategy> const &strategies, int basezoom, double droprate, int retain_points_multiplier);
|
||||
void mbtiles_write_metadata(sqlite3 *db, const metadata &m, bool forcetable);
|
||||
|
||||
void mbtiles_close(sqlite3 *outdb, const char *pgm);
|
||||
@@ -82,6 +76,7 @@ void mbtiles_close(sqlite3 *outdb, const char *pgm);
|
||||
std::map<std::string, layermap_entry> merge_layermaps(std::vector<std::map<std::string, layermap_entry> > const &maps);
|
||||
std::map<std::string, layermap_entry> merge_layermaps(std::vector<std::map<std::string, layermap_entry> > const &maps, bool trunc);
|
||||
|
||||
void add_to_file_keys(std::map<std::string, type_and_string_stats> &file_keys, std::string const &layername, type_and_string const &val);
|
||||
void add_to_tilestats(std::map<std::string, tilestat> &tilestats, std::string const &layername, serial_val const &val);
|
||||
std::string version_str();
|
||||
|
||||
#endif
|
||||
|
||||
+4
-3
@@ -2,6 +2,7 @@
|
||||
#include <string.h>
|
||||
#include <unistd.h>
|
||||
#include <sys/mman.h>
|
||||
#include <errno.h>
|
||||
#include "memfile.hpp"
|
||||
|
||||
#define INCREMENT 131072
|
||||
@@ -48,10 +49,11 @@ int memfile_close(struct memfile *file) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
int memfile_write(struct memfile *file, void *s, long long len) {
|
||||
int memfile_write(struct memfile *file, void *s, long long len, bool &in_memory) {
|
||||
// If it is full, append to the file.
|
||||
// If it is not full yet, append to the string in memory.
|
||||
|
||||
in_memory = false;
|
||||
if (file->fp != NULL) {
|
||||
if (fwrite(s, sizeof(char), len, file->fp) != (size_t) len) {
|
||||
return 0;
|
||||
@@ -60,6 +62,7 @@ int memfile_write(struct memfile *file, void *s, long long len) {
|
||||
} else {
|
||||
file->map.append(std::string((char *) s, len));
|
||||
file->off += len;
|
||||
in_memory = true;
|
||||
}
|
||||
|
||||
return len;
|
||||
@@ -75,8 +78,6 @@ void memfile_full(struct memfile *file) {
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
fprintf(stderr, "Tippecanoe string pool is full (%zu); switching to file\n", file->off);
|
||||
|
||||
file->fp = fdopen(file->fd, "wb");
|
||||
if (file->fp == NULL) {
|
||||
fprintf(stderr, "fdopen memfile: %s\n", strerror(errno));
|
||||
|
||||
+1
-1
@@ -15,7 +15,7 @@ struct memfile {
|
||||
|
||||
struct memfile *memfile_open(int fd);
|
||||
int memfile_close(struct memfile *file);
|
||||
int memfile_write(struct memfile *file, void *s, long long len);
|
||||
int memfile_write(struct memfile *file, void *s, long long len, bool &in_memory);
|
||||
void memfile_full(struct memfile *file);
|
||||
|
||||
#endif
|
||||
|
||||
@@ -14,6 +14,8 @@
|
||||
#include "protozero/pbf_writer.hpp"
|
||||
#include "milo/dtoa_milo.h"
|
||||
#include "errors.hpp"
|
||||
#include "serial.hpp"
|
||||
#include "text.hpp"
|
||||
|
||||
mvt_geometry::mvt_geometry(int nop, long long nx, long long ny) {
|
||||
this->op = nop;
|
||||
@@ -89,7 +91,7 @@ int compress(std::string const &input, std::string &output, bool gz) {
|
||||
deflate_s.opaque = Z_NULL;
|
||||
deflate_s.avail_in = 0;
|
||||
deflate_s.next_in = Z_NULL;
|
||||
deflateInit2(&deflate_s, Z_BEST_COMPRESSION, Z_DEFLATED, gz ? 31 : 15, 8, Z_DEFAULT_STRATEGY);
|
||||
deflateInit2(&deflate_s, Z_DEFAULT_COMPRESSION, Z_DEFLATED, gz ? 31 : 15, 8, Z_DEFAULT_STRATEGY);
|
||||
deflate_s.next_in = (Bytef *) input.data();
|
||||
deflate_s.avail_in = input.size();
|
||||
size_t length = 0;
|
||||
@@ -109,7 +111,7 @@ int compress(std::string const &input, std::string &output, bool gz) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
bool mvt_tile::decode(const std::string &message, bool &was_compressed) {
|
||||
layers.clear();
|
||||
std::string src;
|
||||
|
||||
@@ -126,6 +128,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
}
|
||||
|
||||
protozero::pbf_reader reader(src);
|
||||
std::shared_ptr<std::string> string_pool = std::make_shared<std::string>();
|
||||
|
||||
while (reader.next()) {
|
||||
switch (reader.tag()) {
|
||||
@@ -156,7 +159,12 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
switch (value_reader.tag()) {
|
||||
case 1: /* string */
|
||||
value.type = mvt_string;
|
||||
value.string_value = value_reader.get_string();
|
||||
value.s = string_pool;
|
||||
{
|
||||
auto v = value_reader.get_view();
|
||||
std::string_view sv(v.data(), v.size());
|
||||
value.set_string_value(sv);
|
||||
}
|
||||
break;
|
||||
|
||||
case 2: /* float */
|
||||
@@ -195,7 +203,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
}
|
||||
}
|
||||
|
||||
layer.values.push_back(value);
|
||||
layer.values.push_back(std::move(value));
|
||||
break;
|
||||
}
|
||||
|
||||
@@ -223,6 +231,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
case 2: /* tag */
|
||||
{
|
||||
auto pi = feature_reader.get_packed_uint32();
|
||||
feature.tags.reserve(std::distance(pi.first, pi.second));
|
||||
for (auto it = pi.first; it != pi.second; ++it) {
|
||||
feature.tags.push_back(*it);
|
||||
}
|
||||
@@ -236,6 +245,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
case 4: /* geometry */
|
||||
{
|
||||
auto pi = feature_reader.get_packed_uint32();
|
||||
geoms.reserve(std::distance(pi.first, pi.second));
|
||||
for (auto it = pi.first; it != pi.second; ++it) {
|
||||
geoms.push_back(*it);
|
||||
}
|
||||
@@ -249,6 +259,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
}
|
||||
|
||||
long long px = 0, py = 0;
|
||||
feature.geometry.reserve(geoms.size()); // probably not quite right, but still plausible
|
||||
for (size_t g = 0; g < geoms.size(); g++) {
|
||||
uint32_t geom = geoms[g];
|
||||
uint32_t op = geom & 7;
|
||||
@@ -260,14 +271,14 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
py += protozero::decode_zigzag32(geoms[g + 2]);
|
||||
g += 2;
|
||||
|
||||
feature.geometry.push_back(mvt_geometry(op, px, py));
|
||||
feature.geometry.emplace_back(op, px, py);
|
||||
}
|
||||
} else {
|
||||
feature.geometry.push_back(mvt_geometry(op, 0, 0));
|
||||
feature.geometry.emplace_back(op, 0, 0);
|
||||
}
|
||||
}
|
||||
|
||||
layer.features.push_back(feature);
|
||||
layer.features.push_back(std::move(feature));
|
||||
break;
|
||||
}
|
||||
|
||||
@@ -277,14 +288,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
|
||||
}
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < layer.keys.size(); i++) {
|
||||
layer.key_map.insert(std::pair<std::string, size_t>(layer.keys[i], i));
|
||||
}
|
||||
for (size_t i = 0; i < layer.values.size(); i++) {
|
||||
layer.value_map.insert(std::pair<mvt_value, size_t>(layer.values[i], i));
|
||||
}
|
||||
|
||||
layers.push_back(layer);
|
||||
layers.push_back(std::move(layer));
|
||||
break;
|
||||
}
|
||||
|
||||
@@ -304,15 +308,18 @@ struct sorted_value {
|
||||
bool operator<(const sorted_value &sv) const {
|
||||
if (val < sv.val) {
|
||||
return true;
|
||||
} else {
|
||||
return false;
|
||||
} else if (val == sv.val) {
|
||||
if (orig < sv.orig) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
std::string mvt_tile::encode() {
|
||||
std::string data;
|
||||
|
||||
protozero::pbf_writer writer(data);
|
||||
|
||||
for (size_t i = 0; i < layers.size(); i++) {
|
||||
@@ -334,41 +341,58 @@ std::string mvt_tile::encode() {
|
||||
protozero::pbf_writer value_writer(value_string);
|
||||
mvt_value &pbv = layers[i].values[v];
|
||||
|
||||
if (pbv.type == mvt_string) {
|
||||
value_writer.add_string(1, pbv.string_value);
|
||||
} else if (pbv.type == mvt_float) {
|
||||
switch (pbv.type) {
|
||||
case mvt_string:
|
||||
value_writer.add_string(1, pbv.get_string_value());
|
||||
break;
|
||||
case mvt_float:
|
||||
value_writer.add_float(2, pbv.numeric_value.float_value);
|
||||
} else if (pbv.type == mvt_double) {
|
||||
break;
|
||||
case mvt_double:
|
||||
value_writer.add_double(3, pbv.numeric_value.double_value);
|
||||
} else if (pbv.type == mvt_int) {
|
||||
break;
|
||||
case mvt_int:
|
||||
value_writer.add_int64(4, pbv.numeric_value.int_value);
|
||||
} else if (pbv.type == mvt_uint) {
|
||||
break;
|
||||
case mvt_uint:
|
||||
value_writer.add_uint64(5, pbv.numeric_value.uint_value);
|
||||
} else if (pbv.type == mvt_sint) {
|
||||
break;
|
||||
case mvt_sint:
|
||||
value_writer.add_sint64(6, pbv.numeric_value.sint_value);
|
||||
} else if (pbv.type == mvt_bool) {
|
||||
break;
|
||||
case mvt_bool:
|
||||
value_writer.add_bool(7, pbv.numeric_value.bool_value);
|
||||
} else if (pbv.type == mvt_null) {
|
||||
break;
|
||||
case mvt_null:
|
||||
fprintf(stderr, "Internal error: trying to write null attribute to tile\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
} else {
|
||||
default:
|
||||
fprintf(stderr, "Internal error: trying to write undefined attribute type to tile\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
sorted_value sv;
|
||||
sv.val = value_string;
|
||||
sv.val = std::move(value_string);
|
||||
sv.orig = v;
|
||||
sorted_values.push_back(sv);
|
||||
sorted_values.push_back(std::move(sv));
|
||||
}
|
||||
|
||||
std::sort(sorted_values.begin(), sorted_values.end());
|
||||
std::stable_sort(sorted_values.begin(), sorted_values.end());
|
||||
std::vector<size_t> mapping;
|
||||
mapping.resize(sorted_values.size());
|
||||
|
||||
size_t value_index = 0;
|
||||
for (size_t v = 0; v < sorted_values.size(); v++) {
|
||||
mapping[sorted_values[v].orig] = v;
|
||||
mapping[sorted_values[v].orig] = value_index;
|
||||
layer_writer.add_message(4, sorted_values[v].val);
|
||||
|
||||
// crunch out duplicates that were missed by the hashing
|
||||
while (v + 1 < sorted_values.size() && sorted_values[v].val == sorted_values[v + 1].val) {
|
||||
mapping[sorted_values[v + 1].orig] = value_index;
|
||||
v++;
|
||||
}
|
||||
|
||||
value_index++;
|
||||
}
|
||||
|
||||
for (size_t f = 0; f < layers[i].features.size(); f++) {
|
||||
@@ -455,15 +479,70 @@ bool mvt_value::operator<(const mvt_value &o) const {
|
||||
return true;
|
||||
}
|
||||
if (type == o.type) {
|
||||
if ((type == mvt_string && string_value < o.string_value) ||
|
||||
(type == mvt_float && numeric_value.float_value < o.numeric_value.float_value) ||
|
||||
(type == mvt_double && numeric_value.double_value < o.numeric_value.double_value) ||
|
||||
(type == mvt_int && numeric_value.int_value < o.numeric_value.int_value) ||
|
||||
(type == mvt_uint && numeric_value.uint_value < o.numeric_value.uint_value) ||
|
||||
(type == mvt_sint && numeric_value.sint_value < o.numeric_value.sint_value) ||
|
||||
(type == mvt_bool && numeric_value.bool_value < o.numeric_value.bool_value) ||
|
||||
(type == mvt_null && numeric_value.null_value < o.numeric_value.null_value)) {
|
||||
return true;
|
||||
switch (type) {
|
||||
case mvt_string:
|
||||
return get_string_view() < o.get_string_view();
|
||||
|
||||
case mvt_float:
|
||||
return numeric_value.float_value < o.numeric_value.float_value;
|
||||
|
||||
case mvt_double:
|
||||
return numeric_value.double_value < o.numeric_value.double_value;
|
||||
|
||||
case mvt_int:
|
||||
return numeric_value.int_value < o.numeric_value.int_value;
|
||||
|
||||
case mvt_uint:
|
||||
return numeric_value.uint_value < o.numeric_value.uint_value;
|
||||
|
||||
case mvt_sint:
|
||||
return numeric_value.sint_value < o.numeric_value.sint_value;
|
||||
|
||||
case mvt_bool:
|
||||
return numeric_value.bool_value < o.numeric_value.bool_value;
|
||||
|
||||
case mvt_null:
|
||||
return numeric_value.null_value < o.numeric_value.null_value;
|
||||
|
||||
default:
|
||||
fprintf(stderr, "mvt_value::operator<<: can't happen\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
bool mvt_value::operator==(const mvt_value &o) const {
|
||||
if (type == o.type) {
|
||||
switch (type) {
|
||||
case mvt_string:
|
||||
return get_string_view() == o.get_string_view();
|
||||
|
||||
case mvt_float:
|
||||
return numeric_value.float_value == o.numeric_value.float_value;
|
||||
|
||||
case mvt_double:
|
||||
return numeric_value.double_value == o.numeric_value.double_value;
|
||||
|
||||
case mvt_int:
|
||||
return numeric_value.int_value == o.numeric_value.int_value;
|
||||
|
||||
case mvt_uint:
|
||||
return numeric_value.uint_value == o.numeric_value.uint_value;
|
||||
|
||||
case mvt_sint:
|
||||
return numeric_value.sint_value == o.numeric_value.sint_value;
|
||||
|
||||
case mvt_bool:
|
||||
return numeric_value.bool_value == o.numeric_value.bool_value;
|
||||
|
||||
case mvt_null:
|
||||
return numeric_value.null_value == o.numeric_value.null_value;
|
||||
|
||||
default:
|
||||
fprintf(stderr, "mvt_value::operator==: can't happen\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -481,7 +560,7 @@ static std::string quote(std::string const &s) {
|
||||
buf.push_back(ch);
|
||||
} else if (ch < ' ') {
|
||||
char tmp[7];
|
||||
sprintf(tmp, "\\u%04x", ch);
|
||||
snprintf(tmp, sizeof(tmp), "\\u%04x", ch);
|
||||
buf.append(std::string(tmp));
|
||||
} else {
|
||||
buf.push_back(ch);
|
||||
@@ -492,61 +571,58 @@ static std::string quote(std::string const &s) {
|
||||
}
|
||||
|
||||
std::string mvt_value::toString() const {
|
||||
if (type == mvt_string) {
|
||||
return quote(string_value);
|
||||
} else if (type == mvt_int) {
|
||||
switch (type) {
|
||||
case mvt_string:
|
||||
return quote(get_string_value());
|
||||
case mvt_int:
|
||||
return std::to_string(numeric_value.int_value);
|
||||
} else if (type == mvt_double) {
|
||||
case mvt_double: {
|
||||
double v = numeric_value.double_value;
|
||||
if (v == (long long) v) {
|
||||
return std::to_string((long long) v);
|
||||
} else {
|
||||
return milo::dtoa_milo(v);
|
||||
}
|
||||
} else if (type == mvt_float) {
|
||||
}
|
||||
case mvt_float: {
|
||||
double v = numeric_value.float_value;
|
||||
if (v == (long long) v) {
|
||||
return std::to_string((long long) v);
|
||||
} else {
|
||||
return milo::dtoa_milo(v);
|
||||
}
|
||||
} else if (type == mvt_sint) {
|
||||
}
|
||||
case mvt_sint:
|
||||
return std::to_string(numeric_value.sint_value);
|
||||
} else if (type == mvt_uint) {
|
||||
case mvt_uint:
|
||||
return std::to_string(numeric_value.uint_value);
|
||||
} else if (type == mvt_bool) {
|
||||
case mvt_bool:
|
||||
return numeric_value.bool_value ? "true" : "false";
|
||||
} else if (type == mvt_null) {
|
||||
case mvt_null:
|
||||
return "null";
|
||||
} else {
|
||||
return "unknown";
|
||||
default:
|
||||
return "unknown " + std::to_string(type);
|
||||
}
|
||||
}
|
||||
|
||||
void mvt_layer::tag(mvt_feature &feature, std::string key, mvt_value value) {
|
||||
size_t ko, vo;
|
||||
|
||||
std::map<std::string, size_t>::iterator ki = key_map.find(key);
|
||||
std::map<mvt_value, size_t>::iterator vi = value_map.find(value);
|
||||
|
||||
if (ki == key_map.end()) {
|
||||
ko = keys.size();
|
||||
void mvt_layer::tag(mvt_feature &feature, std::string const &key, mvt_value const &value) {
|
||||
size_t key_hash = fnv1a(key) % key_dedup.size();
|
||||
if (key_dedup[key_hash] >= 0 &&
|
||||
keys[key_dedup[key_hash]] == key) {
|
||||
} else {
|
||||
key_dedup[key_hash] = keys.size();
|
||||
keys.push_back(key);
|
||||
key_map.insert(std::pair<std::string, size_t>(key, ko));
|
||||
} else {
|
||||
ko = ki->second;
|
||||
}
|
||||
feature.tags.push_back(key_dedup[key_hash]);
|
||||
|
||||
if (vi == value_map.end()) {
|
||||
vo = values.size();
|
||||
size_t value_hash = std::hash<mvt_value>()(value) % value_dedup.size();
|
||||
if (value_dedup[value_hash] >= 0 &&
|
||||
values[value_dedup[value_hash]] == value) {
|
||||
} else {
|
||||
value_dedup[value_hash] = values.size();
|
||||
values.push_back(value);
|
||||
value_map.insert(std::pair<mvt_value, size_t>(value, vo));
|
||||
} else {
|
||||
vo = vi->second;
|
||||
}
|
||||
|
||||
feature.tags.push_back(ko);
|
||||
feature.tags.push_back(vo);
|
||||
feature.tags.push_back(value_dedup[value_hash]);
|
||||
}
|
||||
|
||||
bool is_integer(const char *s, long long *v) {
|
||||
@@ -620,10 +696,17 @@ bool is_unsigned_integer(const char *s, unsigned long long *v) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
mvt_value stringified_to_mvt_value(int type, const char *s) {
|
||||
// This converts a serial_val-style attribute value to an mvt_value
|
||||
// to store in a tile. If the value is numeric, it tries to choose
|
||||
// the type (int, uint, sint, float, or double) that will give the
|
||||
// smallest representation in the tile without losing precision,
|
||||
// regardless of how the value was represented in the original source.
|
||||
mvt_value stringified_to_mvt_value(int type, const char *s, std::shared_ptr<std::string> const &tile_stringpool) {
|
||||
mvt_value tv;
|
||||
tv.s = tile_stringpool;
|
||||
|
||||
if (type == mvt_double) {
|
||||
switch (type) {
|
||||
case mvt_double: {
|
||||
long long v;
|
||||
unsigned long long uv;
|
||||
if (is_unsigned_integer(s, &uv)) {
|
||||
@@ -662,16 +745,149 @@ mvt_value stringified_to_mvt_value(int type, const char *s) {
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if (type == mvt_bool) {
|
||||
} break;
|
||||
case mvt_bool:
|
||||
tv.type = mvt_bool;
|
||||
tv.numeric_value.bool_value = (s[0] == 't');
|
||||
} else if (type == mvt_null) {
|
||||
break;
|
||||
case mvt_null:
|
||||
tv.type = mvt_null;
|
||||
tv.numeric_value.null_value = 0;
|
||||
} else {
|
||||
break;
|
||||
default:
|
||||
tv.type = mvt_string;
|
||||
tv.string_value = s;
|
||||
tv.set_string_value(s);
|
||||
}
|
||||
|
||||
return tv;
|
||||
}
|
||||
|
||||
// This converts a mvt_value attribute value from a tile back into
|
||||
// a serial_val for more convenient parsing and comparison without
|
||||
// having to handle all of the vector tile numeric types separately.
|
||||
// All numeric types are given the type mvt_double in the serial_val
|
||||
// whether the actual value is integer or floating point.
|
||||
serial_val mvt_value_to_serial_val(mvt_value const &v) {
|
||||
serial_val sv;
|
||||
|
||||
switch (v.type) {
|
||||
case mvt_string:
|
||||
sv.type = mvt_string;
|
||||
sv.s = v.get_string_value();
|
||||
break;
|
||||
case mvt_float:
|
||||
sv.type = mvt_double;
|
||||
sv.s = milo::dtoa_milo(v.numeric_value.float_value);
|
||||
break;
|
||||
case mvt_double:
|
||||
sv.type = mvt_double;
|
||||
sv.s = milo::dtoa_milo(v.numeric_value.double_value);
|
||||
break;
|
||||
case mvt_int:
|
||||
sv.type = mvt_double;
|
||||
sv.s = std::to_string(v.numeric_value.int_value);
|
||||
break;
|
||||
case mvt_uint:
|
||||
sv.type = mvt_double;
|
||||
sv.s = std::to_string(v.numeric_value.uint_value);
|
||||
break;
|
||||
case mvt_sint:
|
||||
sv.type = mvt_double;
|
||||
sv.s = std::to_string(v.numeric_value.sint_value);
|
||||
break;
|
||||
case mvt_bool:
|
||||
sv.type = mvt_bool;
|
||||
sv.s = v.numeric_value.bool_value ? "true" : "false";
|
||||
break;
|
||||
case mvt_null:
|
||||
sv.type = mvt_null;
|
||||
sv.s = "null";
|
||||
break;
|
||||
default:
|
||||
fprintf(stderr, "unhandled mvt_type %d\n", v.type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
return sv;
|
||||
}
|
||||
|
||||
// This extracts an integer value from an mvt_value
|
||||
long long mvt_value_to_long_long(mvt_value const &v) {
|
||||
switch (v.type) {
|
||||
case mvt_string:
|
||||
return atoll(v.c_str());
|
||||
case mvt_float:
|
||||
return v.numeric_value.float_value;
|
||||
case mvt_double:
|
||||
return v.numeric_value.double_value;
|
||||
case mvt_int:
|
||||
return v.numeric_value.int_value;
|
||||
case mvt_uint:
|
||||
return v.numeric_value.uint_value;
|
||||
case mvt_sint:
|
||||
return v.numeric_value.sint_value;
|
||||
case mvt_bool:
|
||||
return v.numeric_value.bool_value;
|
||||
case mvt_null:
|
||||
return 0;
|
||||
default:
|
||||
fprintf(stderr, "unhandled mvt_type %d\n", v.type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
|
||||
// This extracts a double value from an mvt_value
|
||||
double mvt_value_to_double(mvt_value const &v) {
|
||||
switch (v.type) {
|
||||
case mvt_string:
|
||||
return atof(v.c_str());
|
||||
case mvt_float:
|
||||
return v.numeric_value.float_value;
|
||||
case mvt_double:
|
||||
return v.numeric_value.double_value;
|
||||
case mvt_int:
|
||||
return v.numeric_value.int_value;
|
||||
case mvt_uint:
|
||||
return v.numeric_value.uint_value;
|
||||
case mvt_sint:
|
||||
return v.numeric_value.sint_value;
|
||||
case mvt_bool:
|
||||
return v.numeric_value.bool_value;
|
||||
case mvt_null:
|
||||
return 0;
|
||||
default:
|
||||
fprintf(stderr, "unhandled mvt_type %d\n", v.type);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
|
||||
void get_bbox(std::vector<mvt_geometry> const &geom,
|
||||
long long *xmin, long long *ymin, long long *xmax, long long *ymax,
|
||||
int z, int tx, int ty, int detail) {
|
||||
*xmin = LLONG_MAX;
|
||||
*ymin = LLONG_MAX;
|
||||
*xmax = 0;
|
||||
*ymax = 0;
|
||||
|
||||
for (auto const &g : geom) {
|
||||
if (g.op == mvt_moveto || g.op == mvt_lineto) {
|
||||
long long x = g.x;
|
||||
long long y = g.y;
|
||||
|
||||
// to world scale
|
||||
x = x * (1LL << (32 - z - detail));
|
||||
y = y * (1LL << (32 - z - detail));
|
||||
|
||||
// to world origin
|
||||
if (z > 0) {
|
||||
x += (1LL << (32 - z)) * tx;
|
||||
y += (1LL << (32 - z)) * ty;
|
||||
}
|
||||
|
||||
*xmin = std::min(*xmin, x);
|
||||
*ymin = std::min(*ymin, y);
|
||||
*xmax = std::max(*xmax, x);
|
||||
*ymax = std::max(*ymax, y);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,9 +3,16 @@
|
||||
|
||||
#include <sqlite3.h>
|
||||
#include <string>
|
||||
#include <map>
|
||||
#include <string_view>
|
||||
#include <unordered_map>
|
||||
#include <set>
|
||||
#include <vector>
|
||||
#include <optional>
|
||||
#include <memory>
|
||||
#include <cmath>
|
||||
|
||||
#include "errors.hpp"
|
||||
#include "text.hpp"
|
||||
|
||||
struct mvt_value;
|
||||
struct mvt_layer;
|
||||
@@ -48,7 +55,8 @@ struct mvt_feature {
|
||||
int /* mvt_geometry_type */ type = 0;
|
||||
unsigned long long id = 0;
|
||||
bool has_id = false;
|
||||
bool dropped = false;
|
||||
int dropped = 0;
|
||||
size_t seq = 0; // used for ordering in overzoom
|
||||
|
||||
mvt_feature() {
|
||||
has_id = false;
|
||||
@@ -65,11 +73,17 @@ enum mvt_value_type {
|
||||
mvt_sint,
|
||||
mvt_bool,
|
||||
mvt_null,
|
||||
|
||||
mvt_no_such_key,
|
||||
};
|
||||
|
||||
struct mvt_value;
|
||||
double mvt_value_to_double(mvt_value const &v);
|
||||
|
||||
struct mvt_value {
|
||||
mvt_value_type type;
|
||||
std::string string_value;
|
||||
std::shared_ptr<std::string> s;
|
||||
|
||||
union {
|
||||
float float_value;
|
||||
double double_value;
|
||||
@@ -78,16 +92,100 @@ struct mvt_value {
|
||||
long long sint_value;
|
||||
bool bool_value;
|
||||
int null_value;
|
||||
struct {
|
||||
size_t off;
|
||||
size_t len;
|
||||
} string_value;
|
||||
} numeric_value;
|
||||
|
||||
std::string get_string_value() const {
|
||||
if (type == mvt_string) {
|
||||
return std::string(*s, numeric_value.string_value.off, numeric_value.string_value.len);
|
||||
} else {
|
||||
return toString();
|
||||
}
|
||||
}
|
||||
|
||||
std::string_view get_string_view() const {
|
||||
return std::string_view(s->c_str() + numeric_value.string_value.off, numeric_value.string_value.len);
|
||||
}
|
||||
|
||||
const char *c_str() const {
|
||||
return s->c_str() + numeric_value.string_value.off;
|
||||
}
|
||||
|
||||
void set_string_value(const std::string_view &val) {
|
||||
if (s == nullptr) {
|
||||
s = std::make_shared<std::string>();
|
||||
}
|
||||
|
||||
type = mvt_string;
|
||||
numeric_value.string_value.off = s->size();
|
||||
numeric_value.string_value.len = val.size();
|
||||
s->append(val);
|
||||
s->push_back('\0');
|
||||
}
|
||||
|
||||
bool is_numeric() const {
|
||||
return type == mvt_float ||
|
||||
type == mvt_double ||
|
||||
type == mvt_int ||
|
||||
type == mvt_uint ||
|
||||
type == mvt_sint;
|
||||
}
|
||||
|
||||
double to_double() const {
|
||||
return mvt_value_to_double(*this);
|
||||
}
|
||||
|
||||
bool operator<(const mvt_value &o) const;
|
||||
bool operator==(const mvt_value &o) const;
|
||||
std::string toString() const;
|
||||
|
||||
mvt_value() {
|
||||
this->type = mvt_double;
|
||||
this->string_value = "";
|
||||
this->numeric_value.double_value = 0;
|
||||
}
|
||||
|
||||
mvt_value(double v) {
|
||||
this->type = mvt_double;
|
||||
this->numeric_value.double_value = v;
|
||||
}
|
||||
};
|
||||
|
||||
template <>
|
||||
struct std::hash<mvt_value> {
|
||||
std::size_t operator()(const mvt_value &k) const {
|
||||
switch (k.type) {
|
||||
case mvt_string:
|
||||
return fnv1a(k.c_str(), 0);
|
||||
|
||||
case mvt_float:
|
||||
return fnv1a(sizeof(float), (void *) &k.numeric_value.float_value);
|
||||
|
||||
case mvt_double:
|
||||
return fnv1a(sizeof(double), (void *) &k.numeric_value.double_value);
|
||||
|
||||
case mvt_int:
|
||||
return fnv1a(sizeof(long long), (void *) &k.numeric_value.int_value);
|
||||
|
||||
case mvt_uint:
|
||||
return fnv1a(sizeof(unsigned long long), (void *) &k.numeric_value.uint_value);
|
||||
|
||||
case mvt_sint:
|
||||
return fnv1a(sizeof(long long), (void *) &k.numeric_value.sint_value);
|
||||
|
||||
case mvt_bool:
|
||||
return fnv1a(sizeof(bool), (void *) &k.numeric_value.bool_value);
|
||||
|
||||
case mvt_null:
|
||||
return fnv1a(sizeof(int), (void *) &k.numeric_value.null_value);
|
||||
|
||||
default:
|
||||
fprintf(stderr, "mvt_value hash can't happen\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
struct mvt_layer {
|
||||
@@ -99,18 +197,22 @@ struct mvt_layer {
|
||||
long long extent = 0;
|
||||
|
||||
// Add a key-value pair to a feature, using this layer's constant pool
|
||||
void tag(mvt_feature &feature, std::string key, mvt_value value);
|
||||
void tag(mvt_feature &feature, std::string const &key, mvt_value const &value);
|
||||
|
||||
// For tracking the key-value constants already used in this layer
|
||||
std::map<std::string, size_t> key_map{};
|
||||
std::map<mvt_value, size_t> value_map{};
|
||||
std::vector<ssize_t> key_dedup = std::vector<ssize_t>(65536, -1);
|
||||
std::vector<ssize_t> value_dedup = std::vector<ssize_t>(65536, -1);
|
||||
|
||||
int detail() const {
|
||||
return std::round(std::log(extent) / std::log(2));
|
||||
}
|
||||
};
|
||||
|
||||
struct mvt_tile {
|
||||
std::vector<mvt_layer> layers{};
|
||||
|
||||
std::string encode();
|
||||
bool decode(std::string &message, bool &was_compressed);
|
||||
bool decode(const std::string &message, bool &was_compressed);
|
||||
};
|
||||
|
||||
bool is_compressed(std::string const &data);
|
||||
@@ -118,8 +220,16 @@ int decompress(std::string const &input, std::string &output);
|
||||
int compress(std::string const &input, std::string &output, bool gz);
|
||||
int dezig(unsigned n);
|
||||
|
||||
mvt_value stringified_to_mvt_value(int type, const char *s);
|
||||
mvt_value stringified_to_mvt_value(int type, const char *s, std::shared_ptr<std::string> const &tile_stringpool);
|
||||
long long mvt_value_to_long_long(mvt_value const &v);
|
||||
|
||||
bool is_integer(const char *s, long long *v);
|
||||
bool is_unsigned_integer(const char *s, unsigned long long *v);
|
||||
|
||||
struct serial_val;
|
||||
serial_val mvt_value_to_serial_val(mvt_value const &v);
|
||||
|
||||
void get_bbox(std::vector<mvt_geometry> const &geom,
|
||||
long long *xmin, long long *ymin, long long *xmax, long long *ymax,
|
||||
int z, int tx, int ty, int detail);
|
||||
#endif
|
||||
|
||||
+39
-36
@@ -1,50 +1,53 @@
|
||||
#ifndef OPTIONS_HPP
|
||||
#define OPTIONS_HPP
|
||||
|
||||
#define A_COALESCE ((int) 'c')
|
||||
#define A_REVERSE ((int) 'r')
|
||||
#define A_REORDER ((int) 'o')
|
||||
#define A_LINE_DROP ((int) 'l')
|
||||
#define A_DEBUG_POLYGON ((int) '@')
|
||||
#define A_POLYGON_DROP ((int) 'p')
|
||||
#define A_DETECT_SHARED_BORDERS ((int) 'b')
|
||||
#define A_PREFER_RADIX_SORT ((int) 'R')
|
||||
#define A_CALCULATE_FEATURE_DENSITY ((int) 'g')
|
||||
#define A_INCREASE_GAMMA_AS_NEEDED ((int) 'G')
|
||||
#define A_DROP_DENSEST_AS_NEEDED ((int) 's')
|
||||
#define A_DROP_FRACTION_AS_NEEDED ((int) 'd')
|
||||
#define A_DROP_SMALLEST_AS_NEEDED ((int) 'n')
|
||||
#define A_COALESCE_DENSEST_AS_NEEDED ((int) 'S')
|
||||
#define A_COALESCE_SMALLEST_AS_NEEDED ((int) 'N')
|
||||
#define A_COALESCE_FRACTION_AS_NEEDED ((int) 'D')
|
||||
#define A_GRID_LOW_ZOOMS ((int) 'L')
|
||||
#define A_DETECT_WRAPAROUND ((int) 'w')
|
||||
#define A_EXTEND_ZOOMS ((int) 'e')
|
||||
#define A_CLUSTER_DENSEST_AS_NEEDED ((int) 'C')
|
||||
#define A_GENERATE_IDS ((int) 'i')
|
||||
#define A_COALESCE_FRACTION_AS_NEEDED ((int) 'D')
|
||||
#define A_INCREASE_GAMMA_AS_NEEDED ((int) 'G')
|
||||
#define A_CONVERT_NUMERIC_IDS ((int) 'I')
|
||||
#define A_HILBERT ((int) 'h')
|
||||
#define A_VISVALINGAM ((int) 'v')
|
||||
#define A_GRID_LOW_ZOOMS ((int) 'L')
|
||||
#define A_COALESCE_SMALLEST_AS_NEEDED ((int) 'N')
|
||||
#define A_GENERATE_POLYGON_LABEL_POINTS ((int) 'P')
|
||||
#define A_PREFER_RADIX_SORT ((int) 'R')
|
||||
#define A_COALESCE_DENSEST_AS_NEEDED ((int) 'S')
|
||||
#define A_CALCULATE_INDEX ((int) 'X')
|
||||
#define A_DETECT_SHARED_BORDERS ((int) 'b')
|
||||
#define A_COALESCE ((int) 'c')
|
||||
#define A_DROP_FRACTION_AS_NEEDED ((int) 'd')
|
||||
#define A_EXTEND_ZOOMS ((int) 'e')
|
||||
#define A_CALCULATE_FEATURE_DENSITY ((int) 'g')
|
||||
#define A_HILBERT ((int) 'h')
|
||||
#define A_GENERATE_IDS ((int) 'i')
|
||||
#define A_LINE_DROP ((int) 'l')
|
||||
#define A_DROP_SMALLEST_AS_NEEDED ((int) 'n')
|
||||
#define A_REORDER ((int) 'o')
|
||||
#define A_POLYGON_DROP ((int) 'p')
|
||||
#define A_REVERSE ((int) 'r')
|
||||
#define A_DROP_DENSEST_AS_NEEDED ((int) 's')
|
||||
#define A_VARIABLE_DEPTH_PYRAMID ((int) 't')
|
||||
#define A_VISVALINGAM ((int) 'v')
|
||||
#define A_DETECT_WRAPAROUND ((int) 'w')
|
||||
|
||||
#define P_SIMPLIFY ((int) 's')
|
||||
#define P_SIMPLIFY_LOW ((int) 'S')
|
||||
#define P_SIMPLIFY_SHARED_NODES ((int) 'n')
|
||||
#define P_SINGLE_PRECISION ((int) 'N')
|
||||
#define P_FEATURE_LIMIT ((int) 'f')
|
||||
#define P_KILOBYTE_LIMIT ((int) 'k')
|
||||
#define P_DYNAMIC_DROP ((int) 'd')
|
||||
#define P_INPUT_ORDER ((int) 'i')
|
||||
#define P_POLYGON_SPLIT ((int) 'p')
|
||||
#define P_CLIPPING ((int) 'c')
|
||||
#define P_DUPLICATION ((int) 'D')
|
||||
#define P_TINY_POLYGON_REDUCTION ((int) 't')
|
||||
#define P_TILE_COMPRESSION ((int) 'C')
|
||||
#define P_TILE_STATS ((int) 'g')
|
||||
#define P_USE_SOURCE_POLYGON_WINDING ((int) 'w')
|
||||
#define P_DUPLICATION ((int) 'D')
|
||||
#define P_SINGLE_PRECISION ((int) 'N')
|
||||
#define P_SIMPLIFY_LOW ((int) 'S')
|
||||
#define P_TINY_POLYGON_REDUCTION_AT_MAXZOOM ((int) 'T')
|
||||
#define P_REVERSE_SOURCE_POLYGON_WINDING ((int) 'W')
|
||||
#define P_EMPTY_CSV_COLUMNS ((int) 'e')
|
||||
#define P_BASEZOOM_ABOVE_MAXZOOM ((int) 'b')
|
||||
#define P_CLIPPING ((int) 'c')
|
||||
#define P_DYNAMIC_DROP ((int) 'd')
|
||||
#define P_EMPTY_CSV_COLUMNS ((int) 'e')
|
||||
#define P_FEATURE_LIMIT ((int) 'f')
|
||||
#define P_TILE_STATS ((int) 'g')
|
||||
#define P_INPUT_ORDER ((int) 'i')
|
||||
#define P_KILOBYTE_LIMIT ((int) 'k')
|
||||
#define P_SIMPLIFY_SHARED_NODES ((int) 'n')
|
||||
#define P_POLYGON_SPLIT ((int) 'p')
|
||||
#define P_SIMPLIFY ((int) 's')
|
||||
#define P_TINY_POLYGON_REDUCTION ((int) 't')
|
||||
#define P_USE_SOURCE_POLYGON_WINDING ((int) 'w')
|
||||
|
||||
extern int prevent[256];
|
||||
extern int additional[256];
|
||||
|
||||
+277
@@ -0,0 +1,277 @@
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <getopt.h>
|
||||
#include <string>
|
||||
#include <set>
|
||||
#include "errors.hpp"
|
||||
#include "mvt.hpp"
|
||||
#include "geometry.hpp"
|
||||
#include "evaluator.hpp"
|
||||
#include "attribute.hpp"
|
||||
#include "text.hpp"
|
||||
#include "read_json.hpp"
|
||||
|
||||
extern char *optarg;
|
||||
extern int optind;
|
||||
|
||||
int detail = 12; // tippecanoe-style: mvt extent == 1 << detail
|
||||
int buffer = 5; // tippecanoe-style: mvt buffer == extent * buffer / 256;
|
||||
bool demultiply = false;
|
||||
bool do_compress = true;
|
||||
|
||||
std::string filter;
|
||||
bool preserve_input_order = false;
|
||||
std::unordered_map<std::string, attribute_op> attribute_accum;
|
||||
std::vector<std::string> unidecode_data;
|
||||
std::vector<mvt_layer> bins;
|
||||
std::string accumulate_numeric;
|
||||
|
||||
std::set<std::string> keep;
|
||||
std::set<std::string> exclude;
|
||||
std::vector<std::string> exclude_prefix;
|
||||
|
||||
void usage(char **argv) {
|
||||
fprintf(stderr, "Usage: %s -o newtile.pbf.gz tile.pbf.gz oz/ox/oy nz/nx/ny\n", argv[0]);
|
||||
fprintf(stderr, "to create tile nz/nx/ny from tile oz/ox/oy\n");
|
||||
fprintf(stderr, "Usage: %s -o newtile.pbf.gz -t nz/nx/ny tile.pbf.gz oz/ox/oy tile2.pbf.gz oz2/ox2/oy2\n", argv[0]);
|
||||
fprintf(stderr, "to create tile nz/nx/ny from tiles oz/ox/oy and oz2/ox2/oy2\n");
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
int i;
|
||||
const char *outtile = NULL;
|
||||
const char *outfile = NULL;
|
||||
double simplification = 0;
|
||||
double tiny_polygon_size = 0;
|
||||
std::string assign_to_bins;
|
||||
std::string bin_by_id_list;
|
||||
|
||||
std::vector<input_tile> sources;
|
||||
|
||||
struct option long_options[] = {
|
||||
{"include", required_argument, 0, 'y'},
|
||||
{"exclude", required_argument, 0, 'x'},
|
||||
{"exclude-prefix", required_argument, 0, 'x' & 0x1F},
|
||||
{"full-detail", required_argument, 0, 'd'},
|
||||
{"buffer", required_argument, 0, 'b'},
|
||||
{"output", required_argument, 0, 'o'},
|
||||
{"filter-points-multiplier", no_argument, 0, 'm'},
|
||||
{"feature-filter", required_argument, 0, 'j'},
|
||||
{"preserve-input-order", no_argument, 0, 'o' & 0x1F},
|
||||
{"accumulate-attribute", required_argument, 0, 'E'},
|
||||
{"unidecode-data", required_argument, 0, 'u' & 0x1F},
|
||||
{"line-simplification", required_argument, 0, 'S'},
|
||||
{"tiny-polygon-size", required_argument, 0, 's' & 0x1F},
|
||||
{"source-tile", required_argument, 0, 't'},
|
||||
{"assign-to-bins", required_argument, 0, 'b' & 0x1F},
|
||||
{"bin-by-id-list", required_argument, 0, 'c' & 0x1F},
|
||||
{"accumulate-numeric-attributes", required_argument, 0, 'a' & 0x1F},
|
||||
{"no-tile-compression", no_argument, 0, 'd' & 0x1F},
|
||||
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
std::string getopt_str;
|
||||
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
|
||||
if (long_options[lo].val > ' ') {
|
||||
getopt_str.push_back(long_options[lo].val);
|
||||
|
||||
if (long_options[lo].has_arg == required_argument) {
|
||||
getopt_str.push_back(':');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
int option_index = 0;
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), long_options, &option_index)) != -1) {
|
||||
switch (i) {
|
||||
case 'y':
|
||||
keep.insert(optarg);
|
||||
break;
|
||||
|
||||
case 'x':
|
||||
exclude.insert(optarg);
|
||||
break;
|
||||
|
||||
case 'x' & 0x1F:
|
||||
exclude_prefix.push_back(optarg);
|
||||
break;
|
||||
|
||||
case 'o':
|
||||
outfile = optarg;
|
||||
break;
|
||||
|
||||
case 'd':
|
||||
detail = atoi(optarg);
|
||||
break;
|
||||
|
||||
case 'b':
|
||||
buffer = atoi(optarg);
|
||||
break;
|
||||
|
||||
case 'm':
|
||||
demultiply = true;
|
||||
break;
|
||||
|
||||
case 'j':
|
||||
filter = optarg;
|
||||
break;
|
||||
|
||||
case 'o' & 0x1F:
|
||||
preserve_input_order = true;
|
||||
break;
|
||||
|
||||
case 'E':
|
||||
set_attribute_accum(attribute_accum, optarg, argv);
|
||||
break;
|
||||
|
||||
case 'u' & 0x1F:
|
||||
unidecode_data = read_unidecode(optarg);
|
||||
break;
|
||||
|
||||
case 't':
|
||||
outtile = optarg;
|
||||
break;
|
||||
|
||||
case 's' & 0x1F:
|
||||
tiny_polygon_size = atof(optarg);
|
||||
break;
|
||||
|
||||
case 'S':
|
||||
simplification = atof(optarg);
|
||||
break;
|
||||
|
||||
case 'b' & 0x1F:
|
||||
assign_to_bins = optarg;
|
||||
break;
|
||||
|
||||
case 'c' & 0x1F:
|
||||
bin_by_id_list = optarg;
|
||||
break;
|
||||
|
||||
case 'a' & 0x1F:
|
||||
accumulate_numeric = optarg;
|
||||
break;
|
||||
|
||||
case 'd' & 0x1F:
|
||||
do_compress = false;
|
||||
break;
|
||||
|
||||
default:
|
||||
fprintf(stderr, "Unrecognized flag -%c\n", i);
|
||||
usage(argv);
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<input_tile> its;
|
||||
int nz, nx, ny;
|
||||
|
||||
if (outtile == NULL) { // single input
|
||||
if (argc - optind != 3) {
|
||||
fprintf(stderr, "Wrong number of arguments\n");
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
const char *infile = argv[optind + 0];
|
||||
|
||||
int oz, ox, oy;
|
||||
if (sscanf(argv[optind + 1], "%d/%d/%d", &oz, &ox, &oy) != 3) {
|
||||
fprintf(stderr, "%s: not in z/x/y form\n", argv[optind + 1]);
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
if (sscanf(argv[optind + 2], "%d/%d/%d", &nz, &nx, &ny) != 3) {
|
||||
fprintf(stderr, "%s: not in z/x/y form\n", argv[optind + 2]);
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
input_tile s;
|
||||
s.tile = infile;
|
||||
s.z = oz;
|
||||
s.x = ox;
|
||||
s.y = oy;
|
||||
|
||||
sources.push_back(s);
|
||||
} else { // multiple inputs
|
||||
if ((argc - optind) % 2 != 0) {
|
||||
fprintf(stderr, "Unpaired arguments\n");
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
if (sscanf(outtile, "%d/%d/%d", &nz, &nx, &ny) != 3) {
|
||||
fprintf(stderr, "%s: not in z/x/y form\n", outtile);
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
for (i = optind; i + 1 < argc; i += 2) {
|
||||
int oz, ox, oy;
|
||||
if (sscanf(argv[i + 1], "%d/%d/%d", &oz, &ox, &oy) != 3) {
|
||||
fprintf(stderr, "%s: not in z/x/y form\n", argv[i + 1]);
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
input_tile s;
|
||||
s.tile = argv[i];
|
||||
s.z = oz;
|
||||
s.x = ox;
|
||||
s.y = oy;
|
||||
|
||||
sources.push_back(s);
|
||||
}
|
||||
}
|
||||
|
||||
if (assign_to_bins.size() != 0) {
|
||||
FILE *f = fopen(assign_to_bins.c_str(), "r");
|
||||
if (f == NULL) {
|
||||
perror(assign_to_bins.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
int det = detail;
|
||||
if (det < 0) {
|
||||
det = 12;
|
||||
}
|
||||
bins = parse_layers(f, nz, nx, ny, 1LL << det, true);
|
||||
fclose(f);
|
||||
}
|
||||
|
||||
json_object *json_filter = NULL;
|
||||
if (filter.size() > 0) {
|
||||
json_filter = parse_filter(filter.c_str());
|
||||
}
|
||||
|
||||
for (auto const &s : sources) {
|
||||
std::string tile;
|
||||
char buf[1000];
|
||||
int len;
|
||||
|
||||
FILE *f = fopen(s.tile.c_str(), "rb");
|
||||
if (f == NULL) {
|
||||
perror(s.tile.c_str());
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
while ((len = fread(buf, sizeof(char), 1000, f)) > 0) {
|
||||
tile.append(std::string(buf, len));
|
||||
}
|
||||
fclose(f);
|
||||
|
||||
input_tile t = s;
|
||||
t.tile = std::move(tile);
|
||||
its.push_back(std::move(t));
|
||||
}
|
||||
|
||||
std::string out = overzoom(its, nz, nx, ny, detail, buffer, keep, exclude, exclude_prefix, do_compress, NULL, demultiply, json_filter, preserve_input_order, attribute_accum, unidecode_data, simplification, tiny_polygon_size, bins, bin_by_id_list, accumulate_numeric, SIZE_MAX);
|
||||
|
||||
FILE *f = fopen(outfile, "wb");
|
||||
if (f == NULL) {
|
||||
perror(outfile);
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
fwrite(out.c_str(), sizeof(char), out.size(), f);
|
||||
fclose(f);
|
||||
|
||||
return 0;
|
||||
}
|
||||
+75
-235
@@ -25,6 +25,7 @@
|
||||
#include "geometry.hpp"
|
||||
#include "serial.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "thread.hpp"
|
||||
|
||||
extern "C" {
|
||||
#include "jsonpull/jsonpull.h"
|
||||
@@ -54,7 +55,7 @@ void *run_writer(void *a) {
|
||||
|
||||
json_writer state(fp);
|
||||
for (size_t i = 0; i < wa->layers->size(); i++) {
|
||||
layer_to_geojson((*(wa->layers))[i], wa->z, wa->x, wa->y, false, true, false, true, 0, 0, 0, true, state, 0);
|
||||
layer_to_geojson((*(wa->layers))[i], wa->z, wa->x, wa->y, false, true, false, true, 0, 0, 0, true, state, 0, std::set<std::string>());
|
||||
}
|
||||
|
||||
if (fclose(fp) != 0) {
|
||||
@@ -73,235 +74,78 @@ void *run_writer(void *a) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
// XXX deduplicate
|
||||
static std::vector<mvt_geometry> to_feature(drawvec &geom) {
|
||||
std::vector<mvt_geometry> out;
|
||||
|
||||
for (size_t i = 0; i < geom.size(); i++) {
|
||||
out.push_back(mvt_geometry(geom[i].op, geom[i].x, geom[i].y));
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
// Reads from the postfilter
|
||||
std::vector<mvt_layer> parse_layers(int fd, int z, unsigned x, unsigned y, std::vector<std::map<std::string, layermap_entry>> *layermaps, size_t tiling_seg, std::vector<std::vector<std::string>> *layer_unmaps, int extent) {
|
||||
std::map<std::string, mvt_layer> ret;
|
||||
|
||||
FILE *f = fdopen(fd, "r");
|
||||
if (f == NULL) {
|
||||
perror("fdopen filter output");
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
json_pull *jp = json_begin_file(f);
|
||||
|
||||
while (1) {
|
||||
json_object *j = json_read(jp);
|
||||
if (j == NULL) {
|
||||
if (jp->error != NULL) {
|
||||
fprintf(stderr, "Filter output:%d: %s: ", jp->line, jp->error);
|
||||
if (jp->root != NULL) {
|
||||
json_context(jp->root);
|
||||
} else {
|
||||
fprintf(stderr, "\n");
|
||||
}
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
std::vector<mvt_layer> out = parse_layers(f, z, x, y, extent, false);
|
||||
|
||||
json_free(jp->root);
|
||||
break;
|
||||
}
|
||||
|
||||
json_object *type = json_hash_get(j, "type");
|
||||
if (type == NULL || type->type != JSON_STRING) {
|
||||
continue;
|
||||
}
|
||||
if (strcmp(type->value.string.string, "Feature") != 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
json_object *geometry = json_hash_get(j, "geometry");
|
||||
if (geometry == NULL) {
|
||||
fprintf(stderr, "Filter output:%d: filtered feature with no geometry: ", jp->line);
|
||||
json_context(j);
|
||||
json_free(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
json_object *properties = json_hash_get(j, "properties");
|
||||
if (properties == NULL || (properties->type != JSON_HASH && properties->type != JSON_NULL)) {
|
||||
fprintf(stderr, "Filter output:%d: feature without properties hash: ", jp->line);
|
||||
json_context(j);
|
||||
json_free(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
json_object *geometry_type = json_hash_get(geometry, "type");
|
||||
if (geometry_type == NULL) {
|
||||
fprintf(stderr, "Filter output:%d: null geometry (additional not reported): ", jp->line);
|
||||
json_context(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
if (geometry_type->type != JSON_STRING) {
|
||||
fprintf(stderr, "Filter output:%d: geometry type is not a string: ", jp->line);
|
||||
json_context(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
json_object *coordinates = json_hash_get(geometry, "coordinates");
|
||||
if (coordinates == NULL || coordinates->type != JSON_ARRAY) {
|
||||
fprintf(stderr, "Filter output:%d: feature without coordinates array: ", jp->line);
|
||||
json_context(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
int t;
|
||||
for (t = 0; t < GEOM_TYPES; t++) {
|
||||
if (strcmp(geometry_type->value.string.string, geometry_names[t]) == 0) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (t >= GEOM_TYPES) {
|
||||
fprintf(stderr, "Filter output:%d: Can't handle geometry type %s: ", jp->line, geometry_type->value.string.string);
|
||||
json_context(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
std::string layername = "unknown";
|
||||
json_object *tippecanoe = json_hash_get(j, "tippecanoe");
|
||||
json_object *layer = NULL;
|
||||
if (tippecanoe != NULL) {
|
||||
layer = json_hash_get(tippecanoe, "layer");
|
||||
if (layer != NULL && layer->type == JSON_STRING) {
|
||||
layername = std::string(layer->value.string.string);
|
||||
}
|
||||
}
|
||||
|
||||
if (ret.count(layername) == 0) {
|
||||
mvt_layer l;
|
||||
l.name = layername;
|
||||
l.version = 2;
|
||||
l.extent = extent;
|
||||
|
||||
ret.insert(std::pair<std::string, mvt_layer>(layername, l));
|
||||
}
|
||||
auto l = ret.find(layername);
|
||||
|
||||
drawvec dv;
|
||||
parse_geometry(t, coordinates, dv, VT_MOVETO, "Filter output", jp->line, j);
|
||||
if (mb_geometry[t] == VT_POLYGON) {
|
||||
dv = fix_polygon(dv);
|
||||
}
|
||||
|
||||
// Scale and offset geometry from global to tile
|
||||
for (size_t i = 0; i < dv.size(); i++) {
|
||||
long long scale = 1LL << (32 - z);
|
||||
dv[i].x = std::round((dv[i].x - scale * x) * extent / (double) scale);
|
||||
dv[i].y = std::round((dv[i].y - scale * y) * extent / (double) scale);
|
||||
}
|
||||
|
||||
if (mb_geometry[t] == VT_POLYGON) {
|
||||
dv = clean_or_clip_poly(dv, 0, 0, false);
|
||||
if (dv.size() < 3) {
|
||||
dv.clear();
|
||||
}
|
||||
}
|
||||
dv = remove_noop(dv, mb_geometry[t], 0);
|
||||
if (mb_geometry[t] == VT_POLYGON) {
|
||||
dv = close_poly(dv);
|
||||
}
|
||||
|
||||
if (dv.size() > 0) {
|
||||
mvt_feature feature;
|
||||
feature.type = mb_geometry[t];
|
||||
feature.geometry = to_feature(dv);
|
||||
|
||||
json_object *id = json_hash_get(j, "id");
|
||||
if (id != NULL && id->type == JSON_NUMBER) {
|
||||
feature.id = id->value.number.number;
|
||||
if (id->value.number.large_unsigned > 0) {
|
||||
feature.id = id->value.number.large_unsigned;
|
||||
}
|
||||
feature.has_id = true;
|
||||
}
|
||||
|
||||
std::map<std::string, layermap_entry> &layermap = (*layermaps)[tiling_seg];
|
||||
if (layermap.count(layername) == 0) {
|
||||
layermap_entry lme = layermap_entry(layermap.size());
|
||||
lme.minzoom = z;
|
||||
lme.maxzoom = z;
|
||||
|
||||
layermap.insert(std::pair<std::string, layermap_entry>(layername, lme));
|
||||
|
||||
if (lme.id >= (*layer_unmaps)[tiling_seg].size()) {
|
||||
(*layer_unmaps)[tiling_seg].resize(lme.id + 1);
|
||||
(*layer_unmaps)[tiling_seg][lme.id] = layername;
|
||||
}
|
||||
}
|
||||
|
||||
auto fk = layermap.find(layername);
|
||||
if (fk == layermap.end()) {
|
||||
fprintf(stderr, "Internal error: layer %s not found\n", layername.c_str());
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
if (z < fk->second.minzoom) {
|
||||
fk->second.minzoom = z;
|
||||
}
|
||||
if (z > fk->second.maxzoom) {
|
||||
fk->second.maxzoom = z;
|
||||
}
|
||||
|
||||
if (feature.type == mvt_point) {
|
||||
fk->second.points++;
|
||||
} else if (feature.type == mvt_linestring) {
|
||||
fk->second.lines++;
|
||||
} else if (feature.type == mvt_polygon) {
|
||||
fk->second.polygons++;
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < properties->value.object.length; i++) {
|
||||
int tp = -1;
|
||||
std::string s;
|
||||
|
||||
stringify_value(properties->value.object.values[i], tp, s, "Filter output", jp->line, j);
|
||||
|
||||
// Nulls can be excluded here because this is the postfilter
|
||||
// and it is nearly time to create the vector representation
|
||||
|
||||
if (tp >= 0 && tp != mvt_null) {
|
||||
mvt_value v = stringified_to_mvt_value(tp, s.c_str());
|
||||
l->second.tag(feature, std::string(properties->value.object.keys[i]->value.string.string), v);
|
||||
|
||||
type_and_string attrib;
|
||||
attrib.type = tp;
|
||||
attrib.string = s;
|
||||
|
||||
add_to_file_keys(fk->second.file_keys, std::string(properties->value.object.keys[i]->value.string.string), attrib);
|
||||
}
|
||||
}
|
||||
|
||||
l->second.features.push_back(feature);
|
||||
}
|
||||
|
||||
json_free(j);
|
||||
}
|
||||
|
||||
json_end(jp);
|
||||
if (fclose(f) != 0) {
|
||||
perror("fclose postfilter output");
|
||||
exit(EXIT_CLOSE);
|
||||
}
|
||||
|
||||
std::vector<mvt_layer> final;
|
||||
for (auto a : ret) {
|
||||
final.push_back(a.second);
|
||||
for (auto const &layer : out) {
|
||||
std::string layername = layer.name;
|
||||
|
||||
std::map<std::string, layermap_entry> &layermap = (*layermaps)[tiling_seg];
|
||||
if (layermap.count(layername) == 0) {
|
||||
layermap_entry lme = layermap_entry(layermap.size());
|
||||
lme.minzoom = z;
|
||||
lme.maxzoom = z;
|
||||
|
||||
layermap.insert(std::pair<std::string, layermap_entry>(layername, lme));
|
||||
|
||||
if (lme.id >= (*layer_unmaps)[tiling_seg].size()) {
|
||||
(*layer_unmaps)[tiling_seg].resize(lme.id + 1);
|
||||
(*layer_unmaps)[tiling_seg][lme.id] = layername;
|
||||
}
|
||||
}
|
||||
|
||||
auto ts = layermap.find(layername);
|
||||
if (ts == layermap.end()) {
|
||||
fprintf(stderr, "Internal error: layer %s not found\n", layername.c_str());
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
if (z < ts->second.minzoom) {
|
||||
ts->second.minzoom = z;
|
||||
}
|
||||
if (z > ts->second.maxzoom) {
|
||||
ts->second.maxzoom = z;
|
||||
}
|
||||
|
||||
for (auto const &feature : layer.features) {
|
||||
if (feature.type == mvt_point) {
|
||||
ts->second.points++;
|
||||
} else if (feature.type == mvt_linestring) {
|
||||
ts->second.lines++;
|
||||
} else if (feature.type == mvt_polygon) {
|
||||
ts->second.polygons++;
|
||||
}
|
||||
|
||||
for (size_t i = 0; i + 1 < feature.tags.size(); i += 2) {
|
||||
const std::string &key = layer.keys[feature.tags[i]];
|
||||
const mvt_value &val = layer.values[feature.tags[i + 1]];
|
||||
|
||||
// Nulls can be excluded here because this is the postfilter
|
||||
// and it is nearly time to create the vector representation
|
||||
|
||||
if (val.type != mvt_null) {
|
||||
add_to_tilestats(ts->second.tilestats, key, mvt_value_to_serial_val(val));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return final;
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
// Reads from the prefilter
|
||||
serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::vector<std::map<std::string, layermap_entry>> *layermaps, size_t tiling_seg, std::vector<std::vector<std::string>> *layer_unmaps, bool postfilter) {
|
||||
serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::vector<std::map<std::string, layermap_entry>> *layermaps, size_t tiling_seg, std::vector<std::vector<std::string>> *layer_unmaps, bool postfilter, key_pool &key_pool) {
|
||||
serial_feature sf;
|
||||
|
||||
while (1) {
|
||||
@@ -381,7 +225,7 @@ serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::
|
||||
drawvec dv;
|
||||
parse_geometry(t, coordinates, dv, VT_MOVETO, "Filter output", jp->line, j);
|
||||
if (mb_geometry[t] == VT_POLYGON) {
|
||||
dv = fix_polygon(dv);
|
||||
dv = fix_polygon(dv, false, false);
|
||||
}
|
||||
|
||||
// Scale and offset geometry from global to tile
|
||||
@@ -432,7 +276,9 @@ serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::
|
||||
|
||||
json_object *dropped = json_hash_get(tippecanoe, "dropped");
|
||||
if (dropped != NULL && dropped->type == JSON_TRUE) {
|
||||
sf.dropped = true;
|
||||
sf.dropped = FEATURE_DROPPED; // dropped
|
||||
} else {
|
||||
sf.dropped = FEATURE_KEPT; // kept
|
||||
}
|
||||
}
|
||||
|
||||
@@ -477,49 +323,42 @@ serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::
|
||||
}
|
||||
}
|
||||
|
||||
auto fk = layermap.find(layername);
|
||||
if (fk == layermap.end()) {
|
||||
auto ts = layermap.find(layername);
|
||||
if (ts == layermap.end()) {
|
||||
fprintf(stderr, "Internal error: layer %s not found\n", layername.c_str());
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
sf.layer = fk->second.id;
|
||||
sf.layer = ts->second.id;
|
||||
|
||||
if (z < fk->second.minzoom) {
|
||||
fk->second.minzoom = z;
|
||||
if (z < ts->second.minzoom) {
|
||||
ts->second.minzoom = z;
|
||||
}
|
||||
if (z > fk->second.maxzoom) {
|
||||
fk->second.maxzoom = z;
|
||||
if (z > ts->second.maxzoom) {
|
||||
ts->second.maxzoom = z;
|
||||
}
|
||||
|
||||
if (!postfilter) {
|
||||
if (sf.t == mvt_point) {
|
||||
fk->second.points++;
|
||||
ts->second.points++;
|
||||
} else if (sf.t == mvt_linestring) {
|
||||
fk->second.lines++;
|
||||
ts->second.lines++;
|
||||
} else if (sf.t == mvt_polygon) {
|
||||
fk->second.polygons++;
|
||||
ts->second.polygons++;
|
||||
}
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < properties->value.object.length; i++) {
|
||||
serial_val v;
|
||||
v.type = -1;
|
||||
|
||||
stringify_value(properties->value.object.values[i], v.type, v.s, "Filter output", jp->line, j);
|
||||
serial_val v = stringify_value(properties->value.object.values[i], "Filter output", jp->line, j);
|
||||
|
||||
// Nulls can be excluded here because the expression evaluation filter
|
||||
// would have already run before prefiltering
|
||||
|
||||
if (v.type >= 0 && v.type != mvt_null) {
|
||||
sf.full_keys.push_back(std::string(properties->value.object.keys[i]->value.string.string));
|
||||
if (v.type != mvt_null) {
|
||||
sf.full_keys.push_back(key_pool.pool(std::string(properties->value.object.keys[i]->value.string.string)));
|
||||
sf.full_values.push_back(v);
|
||||
|
||||
type_and_string attrib;
|
||||
attrib.string = v.s;
|
||||
attrib.type = v.type;
|
||||
|
||||
if (!postfilter) {
|
||||
add_to_file_keys(fk->second.file_keys, std::string(properties->value.object.keys[i]->value.string.string), attrib);
|
||||
add_to_tilestats(ts->second.tilestats, std::string(properties->value.object.keys[i]->value.string.string), v);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -642,6 +481,7 @@ std::vector<mvt_layer> filter_layers(const char *filter, std::vector<mvt_layer>
|
||||
wa.extent = extent;
|
||||
|
||||
pthread_t writer;
|
||||
// this does need to be a real thread, so we can pipe both to and from it
|
||||
if (pthread_create(&writer, NULL, run_writer, &wa) != 0) {
|
||||
perror("pthread_create (filter writer)");
|
||||
exit(EXIT_PTHREAD);
|
||||
|
||||
+2
-1
@@ -1,3 +1,4 @@
|
||||
struct key_pool;
|
||||
std::vector<mvt_layer> filter_layers(const char *filter, std::vector<mvt_layer> &layer, unsigned z, unsigned x, unsigned y, std::vector<std::map<std::string, layermap_entry>> *layermaps, size_t tiling_seg, std::vector<std::vector<std::string>> *layer_unmaps, int extent);
|
||||
void setup_filter(const char *filter, int *write_to, int *read_from, pid_t *pid, unsigned z, unsigned x, unsigned y);
|
||||
serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::vector<std::map<std::string, layermap_entry>> *layermaps, size_t tiling_seg, std::vector<std::vector<std::string>> *layer_unmaps, bool filters);
|
||||
serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::vector<std::map<std::string, layermap_entry>> *layermaps, size_t tiling_seg, std::vector<std::vector<std::string>> *layer_unmaps, bool filters, key_pool &key_pool);
|
||||
|
||||
+18
-24
@@ -1,6 +1,7 @@
|
||||
#ifndef PMTILES_HPP
|
||||
#define PMTILES_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <sstream>
|
||||
#include <vector>
|
||||
@@ -165,11 +166,11 @@ struct entryv3 {
|
||||
}
|
||||
};
|
||||
|
||||
struct {
|
||||
struct entryv3_cmp {
|
||||
bool operator()(entryv3 a, entryv3 b) const {
|
||||
return a.tile_id < b.tile_id;
|
||||
}
|
||||
} entryv3_cmp;
|
||||
};
|
||||
|
||||
struct entry_zxy {
|
||||
uint8_t z;
|
||||
@@ -183,18 +184,6 @@ struct entry_zxy {
|
||||
}
|
||||
};
|
||||
|
||||
struct {
|
||||
bool operator()(entry_zxy a, entry_zxy b) const {
|
||||
if (a.z != b.z) {
|
||||
return a.z < b.z;
|
||||
}
|
||||
if (a.x != b.x) {
|
||||
return a.x < b.x;
|
||||
}
|
||||
return a.y < b.y;
|
||||
}
|
||||
} entry_zxy_cmp;
|
||||
|
||||
struct varint_too_long_exception : std::exception {
|
||||
const char *what() const noexcept override {
|
||||
return "varint too long exception";
|
||||
@@ -311,14 +300,14 @@ void rotate(int64_t n, int64_t &x, int64_t &y, int64_t rx, int64_t ry) {
|
||||
}
|
||||
|
||||
zxy t_on_level(uint8_t z, uint64_t pos) {
|
||||
int64_t n = 1 << z;
|
||||
int64_t n = 1LL << z;
|
||||
int64_t rx, ry, s, t = pos;
|
||||
int64_t tx = 0;
|
||||
int64_t ty = 0;
|
||||
|
||||
for (s = 1; s < n; s *= 2) {
|
||||
rx = 1 & (t / 2);
|
||||
ry = 1 & (t ^ rx);
|
||||
rx = 1LL & (t / 2);
|
||||
ry = 1LL & (t ^ rx);
|
||||
rotate(s, tx, ty, rx, ry);
|
||||
tx += s * rx;
|
||||
ty += s * ry;
|
||||
@@ -385,28 +374,33 @@ entryv3 find_tile(const std::vector<entryv3> &entries, uint64_t tile_id) {
|
||||
|
||||
inline zxy tileid_to_zxy(uint64_t tileid) {
|
||||
uint64_t acc = 0;
|
||||
uint8_t t_z = 0;
|
||||
while (true) {
|
||||
for (uint8_t t_z = 0; t_z < 32; t_z++) {
|
||||
uint64_t num_tiles = (1LL << t_z) * (1LL << t_z);
|
||||
if (acc + num_tiles > tileid) {
|
||||
return t_on_level(t_z, tileid - acc);
|
||||
}
|
||||
acc += num_tiles;
|
||||
t_z++;
|
||||
}
|
||||
throw std::overflow_error("tile zoom exceeds 64-bit limit");
|
||||
}
|
||||
|
||||
inline uint64_t zxy_to_tileid(uint8_t z, uint32_t x, uint32_t y) {
|
||||
if (z > 31) {
|
||||
throw std::overflow_error("tile zoom exceeds 64-bit limit");
|
||||
}
|
||||
if (x > (1U << z) - 1 || y > (1U << z) - 1) {
|
||||
throw std::overflow_error("tile x/y outside zoom level bounds");
|
||||
}
|
||||
uint64_t acc = 0;
|
||||
for (uint8_t t_z = 0; t_z < z; t_z++) acc += (1LL << t_z) * (1LL << t_z);
|
||||
int64_t n = 1 << z;
|
||||
int64_t n = 1LL << z;
|
||||
int64_t rx, ry, s, d = 0;
|
||||
int64_t tx = x;
|
||||
int64_t ty = y;
|
||||
for (s = n / 2; s > 0; s /= 2) {
|
||||
rx = (tx & s) > 0;
|
||||
ry = (ty & s) > 0;
|
||||
d += s * s * ((3 * rx) ^ ry);
|
||||
d += s * s * ((3LL * rx) ^ ry);
|
||||
rotate(s, tx, ty, rx, ry);
|
||||
}
|
||||
return acc + d;
|
||||
@@ -491,7 +485,7 @@ inline std::tuple<std::string, std::string, int> build_root_leaves(const std::fu
|
||||
std::vector<pmtiles::entryv3> root_entries;
|
||||
std::string leaves_bytes;
|
||||
int num_leaves = 0;
|
||||
for (size_t i = 0; i <= entries.size(); i += leaf_size) {
|
||||
for (size_t i = 0; i < entries.size(); i += leaf_size) {
|
||||
num_leaves++;
|
||||
int end = i + leaf_size;
|
||||
if (i + leaf_size > entries.size()) {
|
||||
@@ -585,4 +579,4 @@ inline std::pair<uint64_t, uint32_t> get_tile(const std::function<std::string(co
|
||||
}
|
||||
|
||||
} // namespace pmtiles
|
||||
#endif
|
||||
#endif
|
||||
|
||||
+23
-4
@@ -10,6 +10,7 @@
|
||||
#include "pmtiles_file.hpp"
|
||||
#include "mvt.hpp"
|
||||
#include "write_json.hpp"
|
||||
#include "main.hpp"
|
||||
|
||||
bool pmtiles_has_suffix(const char *filename) {
|
||||
if (filename == nullptr) {
|
||||
@@ -102,9 +103,19 @@ std::string metadata_to_pmtiles_json(metadata m) {
|
||||
state.json_write_string("strategies");
|
||||
state.json_write_json(m.strategies_json);
|
||||
}
|
||||
|
||||
if (m.decisions_json.size() > 0) {
|
||||
state.json_comma_newline();
|
||||
state.json_write_string("tippecanoe_decisions");
|
||||
state.json_write_json(m.decisions_json);
|
||||
}
|
||||
|
||||
out(state, "generator", m.generator);
|
||||
out(state, "generator_options", m.generator_options);
|
||||
|
||||
std::string bounds2 = std::to_string(m.minlon2) + "," + std::to_string(m.minlat2) + "," + std::to_string(m.maxlon2) + "," + std::to_string(m.maxlat2);
|
||||
out(state, "antimeridian_adjusted_bounds", bounds2);
|
||||
|
||||
if (m.vector_layers_json.size() > 0) {
|
||||
state.json_comma_newline();
|
||||
state.json_write_string("vector_layers");
|
||||
@@ -125,7 +136,7 @@ std::string metadata_to_pmtiles_json(metadata m) {
|
||||
return compressed;
|
||||
}
|
||||
|
||||
void mbtiles_map_image_to_pmtiles(char *fname, metadata m, bool tile_compression, bool quiet, bool quiet_progress) {
|
||||
void mbtiles_map_image_to_pmtiles(char *fname, metadata m, bool tile_compression, bool fquiet, bool fquiet_progress) {
|
||||
sqlite3 *db;
|
||||
|
||||
if (sqlite3_open(fname, &db) != SQLITE_OK) {
|
||||
@@ -163,7 +174,7 @@ void mbtiles_map_image_to_pmtiles(char *fname, metadata m, bool tile_compression
|
||||
sqlite3_finalize(stmt);
|
||||
}
|
||||
|
||||
std::sort(tile_ids.begin(), tile_ids.end());
|
||||
std::stable_sort(tile_ids.begin(), tile_ids.end());
|
||||
|
||||
std::unordered_map<std::string, std::pair<unsigned long long, unsigned long>> hash_to_offset_len;
|
||||
std::vector<pmtiles::entryv3> entries;
|
||||
@@ -194,12 +205,14 @@ void mbtiles_map_image_to_pmtiles(char *fname, metadata m, bool tile_compression
|
||||
tmp_ostream.open(tmpname.c_str(), std::ios::out | std::ios::binary);
|
||||
|
||||
int idx = 0;
|
||||
int progress_reported = -1;
|
||||
for (auto const &tile_id : tile_ids) {
|
||||
idx = idx + 1;
|
||||
double progress = ((double) idx / tile_ids.size()) * 100;
|
||||
pmtiles::zxy zxy = pmtiles::tileid_to_zxy(tile_id);
|
||||
if (!quiet && !quiet_progress) {
|
||||
if (!fquiet && !fquiet_progress && progress_time() && (int) progress != progress_reported) {
|
||||
fprintf(stderr, " %3.1f%% %d/%u/%u \r", progress, zxy.z, zxy.x, zxy.y);
|
||||
progress_reported = (int) progress;
|
||||
fflush(stderr);
|
||||
}
|
||||
sqlite3_bind_int(map_stmt, 1, zxy.z);
|
||||
@@ -250,7 +263,7 @@ void mbtiles_map_image_to_pmtiles(char *fname, metadata m, bool tile_compression
|
||||
|
||||
// finalize PMTiles archive.
|
||||
{
|
||||
std::sort(entries.begin(), entries.end(), pmtiles::entryv3_cmp);
|
||||
std::stable_sort(entries.begin(), entries.end(), pmtiles::entryv3_cmp());
|
||||
|
||||
std::string root_bytes;
|
||||
std::string leaves_bytes;
|
||||
@@ -422,6 +435,12 @@ sqlite3 *pmtilesmeta2tmp(const char *fname, const char *pmtiles_map) {
|
||||
fprintf(stderr, "set %s in metadata: %s\n", key, err);
|
||||
}
|
||||
sqlite3_free(sql);
|
||||
} else if (strcmp(key, "tippecanoe_decisions") == 0 && o->value.object.values[i]->type == JSON_HASH) {
|
||||
sql = sqlite3_mprintf("INSERT INTO metadata (name, value) VALUES ('tippecanoe_decisions', %Q);", json_stringify(o->value.object.values[i]));
|
||||
if (sqlite3_exec(db, sql, NULL, NULL, &err) != SQLITE_OK) {
|
||||
fprintf(stderr, "set %s in metadata: %s\n", key, err);
|
||||
}
|
||||
sqlite3_free(sql);
|
||||
} else if (o->value.object.keys[i]->type != JSON_STRING || o->value.object.values[i]->type != JSON_STRING) {
|
||||
fprintf(stderr, "%s\n", key);
|
||||
fprintf(stderr, "%s: non-string in metadata\n", fname);
|
||||
|
||||
@@ -7,32 +7,33 @@
|
||||
#include "memfile.hpp"
|
||||
#include "pool.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "text.hpp"
|
||||
|
||||
int swizzlecmp(const char *a, const char *b) {
|
||||
ssize_t alen = strlen(a);
|
||||
ssize_t blen = strlen(b);
|
||||
|
||||
if (strcmp(a, b) == 0) {
|
||||
return 0;
|
||||
inline long long swizzlecmp(const char *a, int atype, unsigned long long ahash, const char *b, int btype, unsigned long long bhash) {
|
||||
if ((long long) ahash == (long long) bhash) {
|
||||
if (atype == btype) {
|
||||
return strcmp(a, b);
|
||||
} else {
|
||||
return atype - btype;
|
||||
}
|
||||
} else {
|
||||
return (long long) (ahash - bhash);
|
||||
}
|
||||
|
||||
long long hash1 = 0, hash2 = 0;
|
||||
for (ssize_t i = alen - 1; i >= 0; i--) {
|
||||
hash1 = (hash1 * 37 + a[i]) & INT_MAX;
|
||||
}
|
||||
for (ssize_t i = blen - 1; i >= 0; i--) {
|
||||
hash2 = (hash2 * 37 + b[i]) & INT_MAX;
|
||||
}
|
||||
|
||||
int h1 = hash1, h2 = hash2;
|
||||
if (h1 == h2) {
|
||||
return strcmp(a, b);
|
||||
}
|
||||
|
||||
return h1 - h2;
|
||||
}
|
||||
|
||||
long long addpool(struct memfile *poolfile, struct memfile *treefile, const char *s, char type) {
|
||||
long long addpool(struct memfile *poolfile, struct memfile *treefile, const char *s, char type, std::vector<ssize_t> &dedup) {
|
||||
unsigned long long hash = fnv1a(s, type);
|
||||
size_t hash_off = hash % dedup.size();
|
||||
|
||||
if (dedup[hash_off] >= 0 &&
|
||||
poolfile->map[dedup[hash_off]] == type &&
|
||||
strcmp(poolfile->map.c_str() + dedup[hash_off] + 1, s) == 0) {
|
||||
// printf("hit for %s\n", s);
|
||||
return dedup[hash_off];
|
||||
} else {
|
||||
// printf("miss for %s\n", s);
|
||||
}
|
||||
|
||||
unsigned long *sp = &treefile->tree;
|
||||
size_t depth = 0;
|
||||
|
||||
@@ -43,17 +44,17 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
|
||||
}
|
||||
|
||||
while (*sp != 0) {
|
||||
int cmp = swizzlecmp(s, poolfile->map.c_str() + ((struct stringpool *) (treefile->map.c_str() + *sp))->off + 1);
|
||||
|
||||
if (cmp == 0) {
|
||||
cmp = type - (poolfile->map.c_str() + ((struct stringpool *) (treefile->map.c_str() + *sp))->off)[0];
|
||||
}
|
||||
long long cmp = swizzlecmp(s, type, hash,
|
||||
poolfile->map.c_str() + ((struct stringpool *) (treefile->map.c_str() + *sp))->off + 1,
|
||||
(poolfile->map.c_str() + ((struct stringpool *) (treefile->map.c_str() + *sp))->off)[0],
|
||||
((struct stringpool *) (treefile->map.c_str() + *sp))->hash);
|
||||
|
||||
if (cmp < 0) {
|
||||
sp = &(((struct stringpool *) (treefile->map.c_str() + *sp))->left);
|
||||
} else if (cmp > 0) {
|
||||
sp = &(((struct stringpool *) (treefile->map.c_str() + *sp))->right);
|
||||
} else {
|
||||
dedup[hash_off] = ((struct stringpool *) (treefile->map.c_str() + *sp))->off;
|
||||
return ((struct stringpool *) (treefile->map.c_str() + *sp))->off;
|
||||
}
|
||||
|
||||
@@ -65,14 +66,20 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
|
||||
// the pool is full yet.
|
||||
|
||||
long long off = poolfile->off;
|
||||
if (memfile_write(poolfile, &type, 1) < 0) {
|
||||
bool in_memory = false;
|
||||
|
||||
if (memfile_write(poolfile, &type, 1, in_memory) < 0) {
|
||||
perror("memfile write");
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
if (memfile_write(poolfile, (void *) s, strlen(s) + 1) < 0) {
|
||||
if (memfile_write(poolfile, (void *) s, strlen(s) + 1, in_memory) < 0) {
|
||||
perror("memfile write");
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
|
||||
if (in_memory) {
|
||||
dedup[hash_off] = off;
|
||||
}
|
||||
return off;
|
||||
}
|
||||
}
|
||||
@@ -95,11 +102,12 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
|
||||
// to the newly-added strings.
|
||||
|
||||
long long off = poolfile->off;
|
||||
if (memfile_write(poolfile, &type, 1) < 0) {
|
||||
bool in_memory;
|
||||
if (memfile_write(poolfile, &type, 1, in_memory) < 0) {
|
||||
perror("memfile write");
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
if (memfile_write(poolfile, (void *) s, strlen(s) + 1) < 0) {
|
||||
if (memfile_write(poolfile, (void *) s, strlen(s) + 1, in_memory) < 0) {
|
||||
perror("memfile write");
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
@@ -115,14 +123,18 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
|
||||
}
|
||||
|
||||
long long off = poolfile->off;
|
||||
if (memfile_write(poolfile, &type, 1) < 0) {
|
||||
bool in_memory = false;
|
||||
if (memfile_write(poolfile, &type, 1, in_memory) < 0) {
|
||||
perror("memfile write");
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
if (memfile_write(poolfile, (void *) s, strlen(s) + 1) < 0) {
|
||||
if (memfile_write(poolfile, (void *) s, strlen(s) + 1, in_memory) < 0) {
|
||||
perror("memfile write");
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
if (in_memory) {
|
||||
dedup[hash_off] = off;
|
||||
}
|
||||
|
||||
if (off >= LONG_MAX || treefile->off >= LONG_MAX) {
|
||||
// Tree or pool is bigger than 2GB
|
||||
@@ -138,9 +150,10 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
|
||||
tsp.left = 0;
|
||||
tsp.right = 0;
|
||||
tsp.off = off;
|
||||
tsp.hash = hash;
|
||||
|
||||
long long p = treefile->off;
|
||||
if (memfile_write(treefile, &tsp, sizeof(struct stringpool)) < 0) {
|
||||
if (memfile_write(treefile, &tsp, sizeof(struct stringpool), in_memory) < 0) {
|
||||
perror("memfile write");
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
|
||||
@@ -4,9 +4,11 @@
|
||||
struct stringpool {
|
||||
unsigned long left = 0;
|
||||
unsigned long right = 0;
|
||||
|
||||
unsigned long off = 0;
|
||||
size_t hash; // hash of the string at this node
|
||||
};
|
||||
|
||||
long long addpool(struct memfile *poolfile, struct memfile *treefile, const char *s, char type);
|
||||
long long addpool(struct memfile *poolfile, struct memfile *treefile, const char *s, char type, std::vector<ssize_t> &dedup);
|
||||
|
||||
#endif
|
||||
|
||||
+12
-8
@@ -22,8 +22,8 @@ struct projection *projection = &projections[0];
|
||||
void lonlat2tile(double lon, double lat, int zoom, long long *x, long long *y) {
|
||||
// Place infinite and NaN coordinates off the edge of the Mercator plane
|
||||
|
||||
int lat_class = fpclassify(lat);
|
||||
int lon_class = fpclassify(lon);
|
||||
int lat_class = std::fpclassify(lat);
|
||||
int lon_class = std::fpclassify(lon);
|
||||
bool bad_lon = false;
|
||||
|
||||
if (lat_class == FP_INFINITE || lat_class == FP_NAN) {
|
||||
@@ -57,8 +57,8 @@ void lonlat2tile(double lon, double lat, int zoom, long long *x, long long *y) {
|
||||
double lat_rad = lat * M_PI / 180;
|
||||
unsigned long long n = 1LL << zoom;
|
||||
|
||||
long long llx = n * ((lon + 180) / 360);
|
||||
long long lly = n * (1 - (log(tan(lat_rad) + 1 / cos(lat_rad)) / M_PI)) / 2;
|
||||
long long llx = std::round(n * ((lon + 180) / 360));
|
||||
long long lly = std::round(n * (1 - (log(tan(lat_rad) + 1 / cos(lat_rad)) / M_PI)) / 2);
|
||||
|
||||
*x = llx;
|
||||
*y = lly;
|
||||
@@ -74,8 +74,8 @@ void tile2lonlat(long long x, long long y, int zoom, double *lon, double *lat) {
|
||||
void epsg3857totile(double ix, double iy, int zoom, long long *x, long long *y) {
|
||||
// Place infinite and NaN coordinates off the edge of the Mercator plane
|
||||
|
||||
int iy_class = fpclassify(iy);
|
||||
int ix_class = fpclassify(ix);
|
||||
int iy_class = std::fpclassify(iy);
|
||||
int ix_class = std::fpclassify(ix);
|
||||
|
||||
if (iy_class == FP_INFINITE || iy_class == FP_NAN) {
|
||||
iy = 40000000.0;
|
||||
@@ -84,8 +84,8 @@ void epsg3857totile(double ix, double iy, int zoom, long long *x, long long *y)
|
||||
ix = 40000000.0;
|
||||
}
|
||||
|
||||
*x = ix * (1LL << 31) / 6378137.0 / M_PI + (1LL << 31);
|
||||
*y = ((1LL << 32) - 1) - (iy * (1LL << 31) / 6378137.0 / M_PI + (1LL << 31));
|
||||
*x = std::round(ix * (1LL << 31) / 6378137.0 / M_PI + (1LL << 31));
|
||||
*y = std::round(((1LL << 32) - 1) - (iy * (1LL << 31) / 6378137.0 / M_PI + (1LL << 31)));
|
||||
|
||||
if (zoom != 0) {
|
||||
*x = std::round((double) *x / (1LL << (32 - zoom)));
|
||||
@@ -217,3 +217,7 @@ void set_projection_or_exit(const char *optarg) {
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
}
|
||||
|
||||
unsigned long long encode_vertex(unsigned int wx, unsigned int wy) {
|
||||
return (((unsigned long long) wx) << 32) | wy;
|
||||
}
|
||||
|
||||
@@ -26,4 +26,6 @@ void decode_quadkey(unsigned long long index, unsigned *wx, unsigned *wy);
|
||||
unsigned long long encode_hilbert(unsigned int wx, unsigned int wy);
|
||||
void decode_hilbert(unsigned long long index, unsigned *wx, unsigned *wy);
|
||||
|
||||
unsigned long long encode_vertex(unsigned int wx, unsigned int wy);
|
||||
|
||||
#endif
|
||||
|
||||
+270
-37
@@ -13,6 +13,7 @@
|
||||
#include "mvt.hpp"
|
||||
#include "milo/dtoa_milo.h"
|
||||
#include "errors.hpp"
|
||||
#include "serial.hpp"
|
||||
|
||||
const char *geometry_names[GEOM_TYPES] = {
|
||||
"Point",
|
||||
@@ -45,7 +46,7 @@ void json_context(json_object *j) {
|
||||
char *s = json_stringify(j);
|
||||
|
||||
if (strlen(s) >= 500) {
|
||||
sprintf(s + 497, "...");
|
||||
snprintf(s + 497, strlen(s) + 1 - 497, "...");
|
||||
}
|
||||
|
||||
fprintf(stderr, "in JSON object %s\n", s);
|
||||
@@ -116,54 +117,286 @@ void parse_geometry(int t, json_object *j, drawvec &out, int op, const char *fna
|
||||
}
|
||||
}
|
||||
|
||||
void stringify_value(json_object *value, int &type, std::string &stringified, const char *reading, int line, json_object *feature) {
|
||||
// This is used to convert a JSON attribute value into a serial_val-style
|
||||
// type and stringified value. All numeric values, even if they are integers,
|
||||
// even integers that are too large to fit in a double but will still be
|
||||
// stringified with their original precision, are recorded here as mvt_double.
|
||||
serial_val stringify_value(json_object *value, const char *reading, int line, json_object *feature) {
|
||||
serial_val sv;
|
||||
|
||||
if (value != NULL) {
|
||||
int vt = value->type;
|
||||
std::string val;
|
||||
|
||||
if (vt == JSON_STRING) {
|
||||
val = value->value.string.string;
|
||||
} else if (vt == JSON_NUMBER) {
|
||||
if (value->value.number.large_unsigned != 0) {
|
||||
val = std::to_string(value->value.number.large_unsigned);
|
||||
} else if (value->value.number.large_signed != 0) {
|
||||
val = std::to_string(value->value.number.large_signed);
|
||||
} else {
|
||||
val = milo::dtoa_milo(value->value.number.number);
|
||||
}
|
||||
} else if (vt == JSON_TRUE) {
|
||||
val = "true";
|
||||
} else if (vt == JSON_FALSE) {
|
||||
val = "false";
|
||||
} else if (vt == JSON_NULL) {
|
||||
val = "null";
|
||||
} else {
|
||||
const char *v = json_stringify(value);
|
||||
val = std::string(v);
|
||||
free((void *) v); // stringify
|
||||
}
|
||||
sv.type = mvt_string;
|
||||
sv.s = value->value.string.string;
|
||||
|
||||
if (vt == JSON_STRING) {
|
||||
type = mvt_string;
|
||||
stringified = val;
|
||||
std::string err = check_utf8(val);
|
||||
if (err != "") {
|
||||
std::string err = check_utf8(sv.s);
|
||||
if (err.size() > 0) {
|
||||
fprintf(stderr, "%s:%d: %s: ", reading, line, err.c_str());
|
||||
json_context(feature);
|
||||
exit(EXIT_UTF8);
|
||||
}
|
||||
} else if (vt == JSON_NUMBER) {
|
||||
type = mvt_double;
|
||||
stringified = val;
|
||||
} else if (vt == JSON_TRUE || vt == JSON_FALSE) {
|
||||
type = mvt_bool;
|
||||
stringified = val;
|
||||
sv.type = mvt_double;
|
||||
|
||||
if (value->value.number.large_unsigned != 0) {
|
||||
sv.s = std::to_string(value->value.number.large_unsigned);
|
||||
} else if (value->value.number.large_signed != 0) {
|
||||
sv.s = std::to_string(value->value.number.large_signed);
|
||||
} else {
|
||||
sv.s = milo::dtoa_milo(value->value.number.number);
|
||||
}
|
||||
} else if (vt == JSON_TRUE) {
|
||||
sv.type = mvt_bool;
|
||||
sv.s = "true";
|
||||
} else if (vt == JSON_FALSE) {
|
||||
sv.type = mvt_bool;
|
||||
sv.s = "false";
|
||||
} else if (vt == JSON_NULL) {
|
||||
type = mvt_null;
|
||||
stringified = "null";
|
||||
sv.type = mvt_null;
|
||||
sv.s = "null";
|
||||
} else {
|
||||
type = mvt_string;
|
||||
stringified = val;
|
||||
sv.type = mvt_string;
|
||||
const char *v = json_stringify(value);
|
||||
sv.s = std::string(v);
|
||||
free((void *) v); // stringify
|
||||
}
|
||||
}
|
||||
|
||||
return sv;
|
||||
}
|
||||
|
||||
// XXX deduplicate
|
||||
static std::vector<mvt_geometry> to_feature(drawvec &geom) {
|
||||
std::vector<mvt_geometry> out;
|
||||
|
||||
for (size_t i = 0; i < geom.size(); i++) {
|
||||
out.push_back(mvt_geometry(geom[i].op, geom[i].x, geom[i].y));
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<mvt_layer> parse_layers(FILE *fp, int z, unsigned x, unsigned y, int extent, bool fix_longitudes) {
|
||||
std::map<std::string, mvt_layer> ret;
|
||||
std::shared_ptr<std::string> tile_stringpool = std::make_shared<std::string>();
|
||||
|
||||
json_pull *jp = json_begin_file(fp);
|
||||
while (1) {
|
||||
json_object *j = json_read(jp);
|
||||
if (j == NULL) {
|
||||
if (jp->error != NULL) {
|
||||
fprintf(stderr, "Filter output:%d: %s: ", jp->line, jp->error);
|
||||
if (jp->root != NULL) {
|
||||
json_context(jp->root);
|
||||
} else {
|
||||
fprintf(stderr, "\n");
|
||||
}
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
json_free(jp->root);
|
||||
break;
|
||||
}
|
||||
|
||||
json_object *type = json_hash_get(j, "type");
|
||||
if (type == NULL || type->type != JSON_STRING) {
|
||||
continue;
|
||||
}
|
||||
if (strcmp(type->value.string.string, "Feature") != 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
json_object *geometry = json_hash_get(j, "geometry");
|
||||
if (geometry == NULL) {
|
||||
fprintf(stderr, "Filter output:%d: filtered feature with no geometry: ", jp->line);
|
||||
json_context(j);
|
||||
json_free(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
json_object *properties = json_hash_get(j, "properties");
|
||||
if (properties == NULL || (properties->type != JSON_HASH && properties->type != JSON_NULL)) {
|
||||
fprintf(stderr, "Filter output:%d: feature without properties hash: ", jp->line);
|
||||
json_context(j);
|
||||
json_free(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
json_object *geometry_type = json_hash_get(geometry, "type");
|
||||
if (geometry_type == NULL) {
|
||||
fprintf(stderr, "Filter output:%d: null geometry (additional not reported): ", jp->line);
|
||||
json_context(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
if (geometry_type->type != JSON_STRING) {
|
||||
fprintf(stderr, "Filter output:%d: geometry type is not a string: ", jp->line);
|
||||
json_context(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
json_object *coordinates = json_hash_get(geometry, "coordinates");
|
||||
if (coordinates == NULL || coordinates->type != JSON_ARRAY) {
|
||||
fprintf(stderr, "Filter output:%d: feature without coordinates array: ", jp->line);
|
||||
json_context(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
int t;
|
||||
for (t = 0; t < GEOM_TYPES; t++) {
|
||||
if (strcmp(geometry_type->value.string.string, geometry_names[t]) == 0) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (t >= GEOM_TYPES) {
|
||||
fprintf(stderr, "Filter output:%d: Can't handle geometry type %s: ", jp->line, geometry_type->value.string.string);
|
||||
json_context(j);
|
||||
exit(EXIT_JSON);
|
||||
}
|
||||
|
||||
std::string layername = "unknown";
|
||||
json_object *tippecanoe = json_hash_get(j, "tippecanoe");
|
||||
json_object *layer = NULL;
|
||||
if (tippecanoe != NULL) {
|
||||
layer = json_hash_get(tippecanoe, "layer");
|
||||
if (layer != NULL && layer->type == JSON_STRING) {
|
||||
layername = std::string(layer->value.string.string);
|
||||
}
|
||||
}
|
||||
|
||||
if (ret.count(layername) == 0) {
|
||||
mvt_layer l;
|
||||
l.name = layername;
|
||||
l.version = 2;
|
||||
l.extent = extent;
|
||||
|
||||
ret.insert(std::pair<std::string, mvt_layer>(layername, l));
|
||||
}
|
||||
auto l = ret.find(layername);
|
||||
|
||||
drawvec dv;
|
||||
parse_geometry(t, coordinates, dv, VT_MOVETO, "Filter output", jp->line, j);
|
||||
|
||||
// handle longitude wraparound
|
||||
//
|
||||
// this is supposed to be data for a single tile,
|
||||
// so any jump from the left hand side edge of the world
|
||||
// to the right edge, or vice versa, is unexpected,
|
||||
// so move it to the other side.
|
||||
|
||||
if (fix_longitudes && mb_geometry[t] == VT_POLYGON) {
|
||||
const long long quarter_world = 1LL << 30;
|
||||
const long long world = 1LL << 32;
|
||||
|
||||
bool copy_to_left = false;
|
||||
bool copy_to_right = false;
|
||||
|
||||
for (size_t i = 0; i < dv.size(); i++) {
|
||||
// is this vertex on a different side of the world
|
||||
// than the first vertex? then shift this one to match
|
||||
if (i > 0) {
|
||||
if ((dv[0].x < quarter_world) && (dv[i].x > 3 * quarter_world)) {
|
||||
dv[i].x -= world;
|
||||
}
|
||||
if ((dv[0].x > 3 * quarter_world) && (dv[i].x < quarter_world)) {
|
||||
dv[i].x += world;
|
||||
}
|
||||
}
|
||||
|
||||
// does it stick off the edge of the world?
|
||||
// then we need another copy on the other side of the world
|
||||
if (dv[i].x < 0) {
|
||||
copy_to_right = true;
|
||||
}
|
||||
if (dv[i].x > world) {
|
||||
copy_to_left = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (copy_to_left) {
|
||||
size_t n = dv.size();
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
dv.emplace_back(dv[i].op, dv[i].x - world, (long long) dv[i].y);
|
||||
}
|
||||
}
|
||||
if (copy_to_right) {
|
||||
size_t n = dv.size();
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
dv.emplace_back(dv[i].op, dv[i].x + world, (long long) dv[i].y);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (mb_geometry[t] == VT_POLYGON) {
|
||||
dv = fix_polygon(dv, false, false);
|
||||
}
|
||||
|
||||
// Offset and scale geometry from global to tile
|
||||
for (size_t i = 0; i < dv.size(); i++) {
|
||||
long long scale = 1LL << (32 - z);
|
||||
|
||||
// offset to tile
|
||||
dv[i].x -= scale * x;
|
||||
dv[i].y -= scale * y;
|
||||
|
||||
// scale to tile
|
||||
dv[i].x = std::round(dv[i].x * extent / (double) scale);
|
||||
dv[i].y = std::round(dv[i].y * extent / (double) scale);
|
||||
}
|
||||
|
||||
if (mb_geometry[t] == VT_POLYGON) {
|
||||
// don't try scaling up because we may have coordinates
|
||||
// on the other side of the world
|
||||
dv = clean_or_clip_poly(dv, z, 256, true, false);
|
||||
if (dv.size() < 3) {
|
||||
dv.clear();
|
||||
}
|
||||
}
|
||||
dv = remove_noop(dv, mb_geometry[t], 0);
|
||||
if (mb_geometry[t] == VT_POLYGON) {
|
||||
dv = close_poly(dv);
|
||||
}
|
||||
|
||||
if (dv.size() > 0) {
|
||||
mvt_feature feature;
|
||||
feature.type = mb_geometry[t];
|
||||
feature.geometry = to_feature(dv);
|
||||
|
||||
json_object *id = json_hash_get(j, "id");
|
||||
if (id != NULL && id->type == JSON_NUMBER) {
|
||||
feature.id = id->value.number.number;
|
||||
if (id->value.number.large_unsigned > 0) {
|
||||
feature.id = id->value.number.large_unsigned;
|
||||
}
|
||||
feature.has_id = true;
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < properties->value.object.length; i++) {
|
||||
serial_val sv = stringify_value(properties->value.object.values[i], "Filter output", jp->line, j);
|
||||
|
||||
// Nulls can be excluded here because this is the postfilter
|
||||
// and it is nearly time to create the vector representation
|
||||
|
||||
if (sv.type != mvt_null) {
|
||||
mvt_value v = stringified_to_mvt_value(sv.type, sv.s.c_str(), tile_stringpool);
|
||||
l->second.tag(feature, std::string(properties->value.object.keys[i]->value.string.string), v);
|
||||
}
|
||||
}
|
||||
|
||||
l->second.features.push_back(feature);
|
||||
}
|
||||
|
||||
json_free(j);
|
||||
}
|
||||
|
||||
json_end(jp);
|
||||
|
||||
std::vector<mvt_layer> final;
|
||||
for (auto a : ret) {
|
||||
final.push_back(a.second);
|
||||
}
|
||||
|
||||
return final;
|
||||
}
|
||||
|
||||
+2
-1
@@ -12,5 +12,6 @@ extern int mb_geometry[GEOM_TYPES];
|
||||
|
||||
void json_context(json_object *j);
|
||||
void parse_geometry(int t, json_object *j, drawvec &out, int op, const char *fname, int line, json_object *feature);
|
||||
std::vector<mvt_layer> parse_layers(FILE *fp, int z, unsigned x, unsigned y, int extent, bool fix_longitudes);
|
||||
|
||||
void stringify_value(json_object *value, int &type, std::string &stringified, const char *reading, int line, json_object *feature);
|
||||
serial_val stringify_value(json_object *value, const char *reading, int line, json_object *feature);
|
||||
|
||||
+280
-144
@@ -23,10 +23,11 @@
|
||||
#include "evaluator.hpp"
|
||||
#include "milo/dtoa_milo.h"
|
||||
#include "errors.hpp"
|
||||
#include "text.hpp"
|
||||
|
||||
// Offset coordinates to keep them positive
|
||||
#define COORD_OFFSET (4LL << 32)
|
||||
#define SHIFT_RIGHT(a) ((long long) std::round((double)(a) / (1LL << geometry_scale)))
|
||||
#define SHIFT_RIGHT(a) ((long long) std::round((double) (a) / (1LL << geometry_scale)))
|
||||
#define SHIFT_LEFT(a) ((((a) + (COORD_OFFSET >> geometry_scale)) << geometry_scale) - COORD_OFFSET)
|
||||
|
||||
// write to file
|
||||
@@ -52,25 +53,16 @@ void serialize_long_long(FILE *out, long long n, std::atomic<long long> *fpos, c
|
||||
}
|
||||
|
||||
void serialize_ulong_long(FILE *out, unsigned long long zigzag, std::atomic<long long> *fpos, const char *fname) {
|
||||
while (1) {
|
||||
unsigned char b = zigzag & 0x7F;
|
||||
if ((zigzag >> 7) != 0) {
|
||||
b |= 0x80;
|
||||
if (putc(b, out) == EOF) {
|
||||
fprintf(stderr, "%s: Write to temporary file failed: %s\n", fname, strerror(errno));
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
*fpos += 1;
|
||||
zigzag >>= 7;
|
||||
} else {
|
||||
if (putc(b, out) == EOF) {
|
||||
fprintf(stderr, "%s: Write to temporary file failed: %s\n", fname, strerror(errno));
|
||||
exit(EXIT_WRITE);
|
||||
}
|
||||
*fpos += 1;
|
||||
break;
|
||||
}
|
||||
char buf[10]; // ceil(64 / 7)
|
||||
char *s = buf;
|
||||
|
||||
while (zigzag >= 0x80) {
|
||||
*s++ = (zigzag & 0x7F) | 0x80;
|
||||
zigzag >>= 7;
|
||||
}
|
||||
|
||||
*s++ = zigzag;
|
||||
fwrite_check(buf, sizeof(char), s - buf, out, fpos, fname);
|
||||
}
|
||||
|
||||
void serialize_byte(FILE *out, signed char n, std::atomic<long long> *fpos, const char *fname) {
|
||||
@@ -89,17 +81,16 @@ size_t fwrite_check(const void *ptr, size_t size, size_t nitems, std::string &st
|
||||
}
|
||||
|
||||
void serialize_ulong_long(std::string &out, unsigned long long zigzag) {
|
||||
while (1) {
|
||||
unsigned char b = zigzag & 0x7F;
|
||||
if ((zigzag >> 7) != 0) {
|
||||
b |= 0x80;
|
||||
out += b;
|
||||
zigzag >>= 7;
|
||||
} else {
|
||||
out += b;
|
||||
break;
|
||||
}
|
||||
char buf[10]; // ceil(64 / 7)
|
||||
char *s = buf;
|
||||
|
||||
while (zigzag >= 0x80) {
|
||||
*s++ = (zigzag & 0x7F) | 0x80;
|
||||
zigzag >>= 7;
|
||||
}
|
||||
|
||||
*s++ = zigzag;
|
||||
out.append(buf, s - buf);
|
||||
}
|
||||
|
||||
void serialize_long_long(std::string &out, long long n) {
|
||||
@@ -122,19 +113,19 @@ void serialize_uint(std::string &out, unsigned n) {
|
||||
|
||||
// read from memory
|
||||
|
||||
void deserialize_int(char **f, int *n) {
|
||||
void deserialize_int(const char **f, int *n) {
|
||||
long long ll;
|
||||
deserialize_long_long(f, &ll);
|
||||
*n = ll;
|
||||
}
|
||||
|
||||
void deserialize_long_long(char **f, long long *n) {
|
||||
void deserialize_long_long(const char **f, long long *n) {
|
||||
unsigned long long zigzag = 0;
|
||||
deserialize_ulong_long(f, &zigzag);
|
||||
*n = protozero::decode_zigzag64(zigzag);
|
||||
}
|
||||
|
||||
void deserialize_ulong_long(char **f, unsigned long long *zigzag) {
|
||||
void deserialize_ulong_long(const char **f, unsigned long long *zigzag) {
|
||||
*zigzag = 0;
|
||||
int shift = 0;
|
||||
|
||||
@@ -152,74 +143,17 @@ void deserialize_ulong_long(char **f, unsigned long long *zigzag) {
|
||||
}
|
||||
}
|
||||
|
||||
void deserialize_uint(char **f, unsigned *n) {
|
||||
void deserialize_uint(const char **f, unsigned *n) {
|
||||
unsigned long long v;
|
||||
deserialize_ulong_long(f, &v);
|
||||
*n = v;
|
||||
}
|
||||
|
||||
void deserialize_byte(char **f, signed char *n) {
|
||||
void deserialize_byte(const char **f, signed char *n) {
|
||||
memcpy(n, *f, sizeof(signed char));
|
||||
*f += sizeof(signed char);
|
||||
}
|
||||
|
||||
// read from file
|
||||
|
||||
int deserialize_long_long_io(FILE *f, long long *n, std::atomic<long long> *geompos) {
|
||||
unsigned long long zigzag = 0;
|
||||
int ret = deserialize_ulong_long_io(f, &zigzag, geompos);
|
||||
*n = protozero::decode_zigzag64(zigzag);
|
||||
return ret;
|
||||
}
|
||||
|
||||
int deserialize_ulong_long_io(FILE *f, unsigned long long *zigzag, std::atomic<long long> *geompos) {
|
||||
*zigzag = 0;
|
||||
int shift = 0;
|
||||
|
||||
while (1) {
|
||||
int c = getc(f);
|
||||
if (c == EOF) {
|
||||
return 0;
|
||||
}
|
||||
(*geompos)++;
|
||||
|
||||
if ((c & 0x80) == 0) {
|
||||
*zigzag |= ((unsigned long long) c) << shift;
|
||||
shift += 7;
|
||||
break;
|
||||
} else {
|
||||
*zigzag |= ((unsigned long long) (c & 0x7F)) << shift;
|
||||
shift += 7;
|
||||
}
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
int deserialize_int_io(FILE *f, int *n, std::atomic<long long> *geompos) {
|
||||
long long ll = 0;
|
||||
int ret = deserialize_long_long_io(f, &ll, geompos);
|
||||
*n = ll;
|
||||
return ret;
|
||||
}
|
||||
|
||||
int deserialize_uint_io(FILE *f, unsigned *n, std::atomic<long long> *geompos) {
|
||||
unsigned long long v;
|
||||
deserialize_ulong_long_io(f, &v, geompos);
|
||||
*n = v;
|
||||
return 1;
|
||||
}
|
||||
|
||||
int deserialize_byte_io(FILE *f, signed char *n, std::atomic<long long> *geompos) {
|
||||
int c = getc(f);
|
||||
if (c == EOF) {
|
||||
return 0;
|
||||
}
|
||||
*n = c;
|
||||
(*geompos)++;
|
||||
return 1;
|
||||
}
|
||||
|
||||
static void write_geometry(drawvec const &dv, std::string &out, long long wx, long long wy) {
|
||||
for (size_t i = 0; i < dv.size(); i++) {
|
||||
if (dv[i].op == VT_MOVETO || dv[i].op == VT_LINETO) {
|
||||
@@ -244,7 +178,6 @@ std::string serialize_feature(serial_feature *sf, long long wx, long long wy) {
|
||||
#define FLAG_LAYER 7
|
||||
|
||||
#define FLAG_LABEL_POINT 6
|
||||
#define FLAG_SEQ 5
|
||||
#define FLAG_INDEX 4
|
||||
#define FLAG_EXTENT 3
|
||||
#define FLAG_ID 2
|
||||
@@ -254,21 +187,18 @@ std::string serialize_feature(serial_feature *sf, long long wx, long long wy) {
|
||||
long long layer = 0;
|
||||
layer |= sf->layer << FLAG_LAYER;
|
||||
layer |= (sf->label_point != 0) << FLAG_LABEL_POINT;
|
||||
layer |= (sf->seq != 0) << FLAG_SEQ;
|
||||
layer |= (sf->index != 0) << FLAG_INDEX;
|
||||
layer |= (sf->extent != 0) << FLAG_EXTENT;
|
||||
layer |= sf->has_id << FLAG_ID;
|
||||
layer |= sf->has_tippecanoe_minzoom << FLAG_MINZOOM;
|
||||
layer |= sf->has_tippecanoe_maxzoom << FLAG_MAXZOOM;
|
||||
layer |= (sf->tippecanoe_minzoom != -1) << FLAG_MINZOOM;
|
||||
layer |= (sf->tippecanoe_maxzoom != -1) << FLAG_MAXZOOM;
|
||||
|
||||
serialize_long_long(s, layer);
|
||||
if (sf->seq != 0) {
|
||||
serialize_long_long(s, sf->seq);
|
||||
}
|
||||
if (sf->has_tippecanoe_minzoom) {
|
||||
serialize_long_long(s, sf->seq);
|
||||
if (sf->tippecanoe_minzoom != -1) {
|
||||
serialize_int(s, sf->tippecanoe_minzoom);
|
||||
}
|
||||
if (sf->has_tippecanoe_maxzoom) {
|
||||
if (sf->tippecanoe_maxzoom != -1) {
|
||||
serialize_int(s, sf->tippecanoe_maxzoom);
|
||||
}
|
||||
if (sf->has_id) {
|
||||
@@ -281,6 +211,7 @@ std::string serialize_feature(serial_feature *sf, long long wx, long long wy) {
|
||||
|
||||
if (sf->index != 0) {
|
||||
serialize_ulong_long(s, sf->index);
|
||||
serialize_ulong_long(s, sf->gap);
|
||||
}
|
||||
if (sf->label_point != 0) {
|
||||
serialize_ulong_long(s, sf->label_point);
|
||||
@@ -301,17 +232,15 @@ std::string serialize_feature(serial_feature *sf, long long wx, long long wy) {
|
||||
return s;
|
||||
}
|
||||
|
||||
serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y) {
|
||||
serial_feature deserialize_feature(std::string const &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y) {
|
||||
serial_feature sf;
|
||||
char *cp = (char *) geoms.c_str();
|
||||
const char *cp = geoms.c_str();
|
||||
|
||||
deserialize_byte(&cp, &sf.t);
|
||||
deserialize_long_long(&cp, &sf.layer);
|
||||
|
||||
sf.seq = 0;
|
||||
if (sf.layer & (1 << FLAG_SEQ)) {
|
||||
deserialize_long_long(&cp, &sf.seq);
|
||||
}
|
||||
deserialize_long_long(&cp, &sf.seq);
|
||||
|
||||
sf.tippecanoe_minzoom = -1;
|
||||
sf.tippecanoe_maxzoom = -1;
|
||||
@@ -331,6 +260,7 @@ serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx,
|
||||
deserialize_int(&cp, &sf.segment);
|
||||
|
||||
sf.index = 0;
|
||||
sf.gap = 0;
|
||||
sf.label_point = 0;
|
||||
sf.extent = 0;
|
||||
|
||||
@@ -338,6 +268,7 @@ serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx,
|
||||
|
||||
if (sf.layer & (1 << FLAG_INDEX)) {
|
||||
deserialize_ulong_long(&cp, &sf.index);
|
||||
deserialize_ulong_long(&cp, &sf.gap);
|
||||
}
|
||||
if (sf.layer & (1 << FLAG_LABEL_POINT)) {
|
||||
deserialize_ulong_long(&cp, &sf.label_point);
|
||||
@@ -382,19 +313,27 @@ static long long scale_geometry(struct serialization_state *sst, long long *bbox
|
||||
long long y = geom[i].y;
|
||||
|
||||
if (additional[A_DETECT_WRAPAROUND]) {
|
||||
x += offset;
|
||||
if (has_prev) {
|
||||
if (x - prev > (1LL << 31)) {
|
||||
offset -= 1LL << 32;
|
||||
x -= 1LL << 32;
|
||||
} else if (prev - x > (1LL << 31)) {
|
||||
offset += 1LL << 32;
|
||||
x += 1LL << 32;
|
||||
if (geom[i].op == VT_LINETO) {
|
||||
x += offset;
|
||||
if (has_prev) {
|
||||
// jumps at least 180° but not exactly 360°,
|
||||
// which in some data sets is an intentional
|
||||
// line across the world
|
||||
if (x - prev > (1LL << 31) && x - prev != (1LL << 32)) {
|
||||
offset -= 1LL << 32;
|
||||
x -= 1LL << 32;
|
||||
} else if (prev - x > (1LL << 31) && prev - x != (1LL << 32)) {
|
||||
offset += 1LL << 32;
|
||||
x += 1LL << 32;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
has_prev = true;
|
||||
prev = x;
|
||||
has_prev = true;
|
||||
prev = x;
|
||||
} else {
|
||||
offset = 0;
|
||||
prev = x;
|
||||
}
|
||||
}
|
||||
|
||||
if (x < bbox[0]) {
|
||||
@@ -450,15 +389,70 @@ static std::string strip_zeroes(std::string s) {
|
||||
return s;
|
||||
}
|
||||
|
||||
int nodecmp(const void *void1, const void *void2) {
|
||||
node *n1 = (node *) void1;
|
||||
node *n2 = (node *) void2;
|
||||
|
||||
if (n1->index < n2->index) {
|
||||
return -1;
|
||||
} else if (n1->index > n2->index) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void add_scaled_node(struct reader *r, serialization_state *sst, draw g) {
|
||||
long long x = SHIFT_LEFT(g.x);
|
||||
long long y = SHIFT_LEFT(g.y);
|
||||
|
||||
struct node n;
|
||||
n.index = encode_vertex((unsigned) x, (unsigned) y);
|
||||
|
||||
fwrite_check((char *) &n, sizeof(struct node), 1, r->nodefile, &r->nodepos, sst->fname);
|
||||
}
|
||||
|
||||
// called from frontends
|
||||
int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
int serialize_feature(struct serialization_state *sst, serial_feature &sf, std::string const &layername) {
|
||||
struct reader *r = &(*sst->readers)[sst->segment];
|
||||
key_pool key_pool;
|
||||
|
||||
sf.bbox[0] = LLONG_MAX;
|
||||
sf.bbox[1] = LLONG_MAX;
|
||||
sf.bbox[2] = LLONG_MIN;
|
||||
sf.bbox[3] = LLONG_MIN;
|
||||
|
||||
for (size_t i = 0; i < sf.geometry.size(); i++) {
|
||||
if (sf.geometry[i].op == VT_MOVETO || sf.geometry[i].op == VT_LINETO) {
|
||||
// standard -180 to 180 world plane
|
||||
//
|
||||
// mask X, since it wraps around
|
||||
// pin Y, since it doesn't
|
||||
|
||||
long long x = sf.geometry[i].x & 0xFFFFFFFF;
|
||||
long long y = std::max(std::min(sf.geometry[i].y, 0xFFFFFFFFLL), 0LL);
|
||||
|
||||
{
|
||||
r->file_bbox1[0] = std::min(r->file_bbox1[0], x);
|
||||
r->file_bbox1[1] = std::min(r->file_bbox1[1], y);
|
||||
r->file_bbox1[2] = std::max(r->file_bbox1[2], x);
|
||||
r->file_bbox1[3] = std::max(r->file_bbox1[3], y);
|
||||
|
||||
// printf("%llx,%llx %llx,%llx %llx,%llx ", x, y, r->file_bbox1[0], r->file_bbox1[1], r->file_bbox1[2], r->file_bbox1[3]);
|
||||
|
||||
// shift the western hemisphere 360 degrees to the east
|
||||
if (x < 0x80000000) { // prime meridian
|
||||
x += 0x100000000;
|
||||
}
|
||||
|
||||
r->file_bbox2[0] = std::min(r->file_bbox2[0], x);
|
||||
r->file_bbox2[1] = std::min(r->file_bbox2[1], y);
|
||||
r->file_bbox2[2] = std::max(r->file_bbox2[2], x);
|
||||
r->file_bbox2[3] = std::max(r->file_bbox2[3], y);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// try to remind myself that the geometry in this function is in SCALED COORDINATES
|
||||
drawvec scaled_geometry = sf.geometry;
|
||||
sf.geometry.clear();
|
||||
@@ -467,19 +461,20 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
// This has to happen after scaling so that the wraparound detection has happened first.
|
||||
// Otherwise the inner/outer calculation will be confused by bad geometries.
|
||||
if (sf.t == VT_POLYGON) {
|
||||
scaled_geometry = fix_polygon(scaled_geometry);
|
||||
scaled_geometry = fix_polygon(scaled_geometry, prevent[P_USE_SOURCE_POLYGON_WINDING], prevent[P_REVERSE_SOURCE_POLYGON_WINDING]);
|
||||
}
|
||||
|
||||
for (auto &c : clipbboxes) {
|
||||
if (sf.t == VT_POLYGON) {
|
||||
scaled_geometry = simple_clip_poly(scaled_geometry, SHIFT_RIGHT(c.minx), SHIFT_RIGHT(c.miny), SHIFT_RIGHT(c.maxx), SHIFT_RIGHT(c.maxy));
|
||||
scaled_geometry = simple_clip_poly(scaled_geometry, SHIFT_RIGHT(c.minx), SHIFT_RIGHT(c.miny), SHIFT_RIGHT(c.maxx), SHIFT_RIGHT(c.maxy), prevent[P_SIMPLIFY_SHARED_NODES]);
|
||||
} else if (sf.t == VT_LINE) {
|
||||
scaled_geometry = clip_lines(scaled_geometry, SHIFT_RIGHT(c.minx), SHIFT_RIGHT(c.miny), SHIFT_RIGHT(c.maxx), SHIFT_RIGHT(c.maxy));
|
||||
scaled_geometry = remove_noop(scaled_geometry, sf.t, 0);
|
||||
} else if (sf.t == VT_POINT) {
|
||||
scaled_geometry = clip_point(scaled_geometry, SHIFT_RIGHT(c.minx), SHIFT_RIGHT(c.miny), SHIFT_RIGHT(c.maxx), SHIFT_RIGHT(c.maxy));
|
||||
}
|
||||
|
||||
scaled_geometry = remove_noop(scaled_geometry, sf.t, 0);
|
||||
|
||||
sf.bbox[0] = LLONG_MAX;
|
||||
sf.bbox[1] = LLONG_MAX;
|
||||
sf.bbox[2] = LLONG_MIN;
|
||||
@@ -509,6 +504,100 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (prevent[P_SIMPLIFY_SHARED_NODES]) {
|
||||
scaled_geometry = remove_noop(scaled_geometry, sf.t, 0);
|
||||
|
||||
if (sf.t == VT_POLYGON || sf.t == VT_LINE) {
|
||||
for (size_t i = 0; i < scaled_geometry.size(); i++) {
|
||||
if (scaled_geometry[i].op == VT_MOVETO) {
|
||||
size_t j;
|
||||
|
||||
for (j = i + 1; j < scaled_geometry.size(); j++) {
|
||||
if (scaled_geometry[j].op != VT_LINETO) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (sf.t == VT_POLYGON && j - i >= 4) {
|
||||
for (size_t k = i; k < j - 1; k++) {
|
||||
// % (j - i - 1) because we don't want the duplicate last point
|
||||
|
||||
struct vertex v(
|
||||
scaled_geometry[(k - i + 0) % (j - i - 1) + i],
|
||||
scaled_geometry[(k - i + 1) % (j - i - 1) + i],
|
||||
scaled_geometry[(k - i + 2) % (j - i - 1) + i]);
|
||||
|
||||
fwrite_check((char *) &v, sizeof(struct vertex), 1, r->vertexfile, &r->vertexpos, sst->fname);
|
||||
}
|
||||
} else if (sf.t == VT_LINE && j - i >= 2) {
|
||||
for (size_t k = i; k + 2 < j; k++) {
|
||||
struct vertex v(
|
||||
scaled_geometry[k + 0],
|
||||
scaled_geometry[k + 1],
|
||||
scaled_geometry[k + 2]);
|
||||
|
||||
fwrite_check((char *) &v, sizeof(struct vertex), 1, r->vertexfile, &r->vertexpos, sst->fname);
|
||||
}
|
||||
}
|
||||
|
||||
// since the starting point is never simplified away,
|
||||
// don't let it be simplified away in any other polygons either.
|
||||
// Needs to appear twice here so that the check below will see
|
||||
// it as appearing in multiple features.
|
||||
add_scaled_node(r, sst, scaled_geometry[i]);
|
||||
|
||||
if (sf.t == VT_LINE && j - i >= 2) {
|
||||
// linestrings also need to preserve the last point
|
||||
|
||||
add_scaled_node(r, sst, scaled_geometry[j - 1]);
|
||||
} else if (sf.t == VT_POLYGON && j - i >= 4) {
|
||||
// To avoid letting polygons get simplified away to nothing,
|
||||
// also keep the furthest-away point from the initial point
|
||||
// (which Douglas-Peucker simplification would keep anyway,
|
||||
// if its search weren't being split up by polygon side).
|
||||
|
||||
double far = 0;
|
||||
size_t which = i;
|
||||
for (size_t k = i + 1; k < j - 1; k++) {
|
||||
double xd = scaled_geometry[k].x - scaled_geometry[i].x;
|
||||
double yd = scaled_geometry[k].y - scaled_geometry[i].y;
|
||||
double d = xd * xd + yd * yd;
|
||||
if (d > far ||
|
||||
((d == far) && (scaled_geometry[k] < scaled_geometry[which]))) {
|
||||
far = d;
|
||||
which = k;
|
||||
}
|
||||
}
|
||||
|
||||
add_scaled_node(r, sst, scaled_geometry[which]);
|
||||
|
||||
// And, likewise, the point most distant from those two points,
|
||||
// which probably would also be the one that Douglas-Peucker
|
||||
// would keep next.
|
||||
|
||||
far = 0;
|
||||
size_t which2 = i;
|
||||
|
||||
for (size_t k = i + 1; k < j - 1; k++) {
|
||||
double d = distance_from_line(scaled_geometry[k].x, scaled_geometry[k].y,
|
||||
scaled_geometry[i].x, scaled_geometry[i].y,
|
||||
scaled_geometry[which].x, scaled_geometry[which].y);
|
||||
if ((d > far) ||
|
||||
((d == far) && (scaled_geometry[k] < scaled_geometry[which2]))) {
|
||||
far = d;
|
||||
which2 = k;
|
||||
}
|
||||
}
|
||||
|
||||
add_scaled_node(r, sst, scaled_geometry[which2]);
|
||||
}
|
||||
|
||||
i = j - 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!sf.has_id) {
|
||||
if (additional[A_GENERATE_IDS]) {
|
||||
sf.has_id = true;
|
||||
@@ -523,7 +612,7 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
locs.push_back(encode_index(SHIFT_LEFT(scaled_geometry[i].x), SHIFT_LEFT(scaled_geometry[i].y)));
|
||||
}
|
||||
}
|
||||
std::sort(locs.begin(), locs.end());
|
||||
std::stable_sort(locs.begin(), locs.end());
|
||||
size_t n = 0;
|
||||
double sum = 0;
|
||||
for (size_t i = 1; i < locs.size(); i++) {
|
||||
@@ -586,10 +675,6 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
*(sst->area_sum) += extent;
|
||||
}
|
||||
|
||||
if (!prevent[P_INPUT_ORDER]) {
|
||||
sf.seq = 0;
|
||||
}
|
||||
|
||||
unsigned long long bbox_index;
|
||||
long long midx, midy;
|
||||
|
||||
@@ -597,6 +682,13 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
// keep old behavior, which loses one bit of precision at the bottom
|
||||
midx = (sf.bbox[0] / 2 + sf.bbox[2] / 2) & ((1LL << 32) - 1);
|
||||
midy = (sf.bbox[1] / 2 + sf.bbox[3] / 2) & ((1LL << 32) - 1);
|
||||
} else if ((additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED]) && sf.t == VT_POLYGON) {
|
||||
// This probably should really apply to all polygons,
|
||||
// but I hate to change the feature sequence in all the
|
||||
// test fixtures again
|
||||
draw scaled_center = center_of_mass_mp(scaled_geometry);
|
||||
midx = SHIFT_LEFT(scaled_center.x) & ((1LL << 32) - 1);
|
||||
midy = SHIFT_LEFT(scaled_center.y) & ((1LL << 32) - 1);
|
||||
} else {
|
||||
// To reduce the chances of giving multiple polygons or linestrings
|
||||
// the same index, use an arbitrary but predictable point from the
|
||||
@@ -612,7 +704,9 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
for (size_t i = 0; i < scaled_geometry.size(); i++) {
|
||||
ix += scaled_geometry[i].x + scaled_geometry[i].y;
|
||||
}
|
||||
ix = ix % scaled_geometry.size();
|
||||
if (scaled_geometry.size() != 0) {
|
||||
ix = ix % scaled_geometry.size();
|
||||
}
|
||||
|
||||
// If off the edge of the plane, mask to bring it back into the addressable area
|
||||
midx = SHIFT_LEFT(scaled_geometry[ix].x) & ((1LL << 32) - 1);
|
||||
@@ -620,6 +714,14 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
}
|
||||
|
||||
bbox_index = encode_index(midx, midy);
|
||||
if (additional[A_CALCULATE_INDEX]) {
|
||||
sf.full_keys.push_back(key_pool.pool("tippecanoe:index"));
|
||||
|
||||
serial_val sv;
|
||||
sv.type = mvt_double;
|
||||
sv.s = std::to_string(bbox_index);
|
||||
sf.full_values.push_back(sv);
|
||||
}
|
||||
|
||||
if (sf.t == VT_POLYGON && additional[A_GENERATE_POLYGON_LABEL_POINTS]) {
|
||||
drawvec dv = polygon_to_anchor(scaled_geometry);
|
||||
@@ -630,17 +732,31 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
}
|
||||
}
|
||||
|
||||
if (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED] || additional[A_CALCULATE_FEATURE_DENSITY] || additional[A_DROP_SMALLEST_AS_NEEDED] || additional[A_COALESCE_SMALLEST_AS_NEEDED] || additional[A_INCREASE_GAMMA_AS_NEEDED] || additional[A_GENERATE_POLYGON_LABEL_POINTS] || sst->uses_gamma || cluster_distance != 0) {
|
||||
if (additional[A_DROP_DENSEST_AS_NEEDED] ||
|
||||
additional[A_COALESCE_DENSEST_AS_NEEDED] ||
|
||||
additional[A_CLUSTER_DENSEST_AS_NEEDED] ||
|
||||
additional[A_CALCULATE_FEATURE_DENSITY] ||
|
||||
additional[A_DROP_SMALLEST_AS_NEEDED] ||
|
||||
additional[A_COALESCE_SMALLEST_AS_NEEDED] ||
|
||||
additional[A_DROP_FRACTION_AS_NEEDED] ||
|
||||
additional[A_COALESCE_FRACTION_AS_NEEDED] ||
|
||||
prevent[P_DYNAMIC_DROP] ||
|
||||
additional[A_INCREASE_GAMMA_AS_NEEDED] ||
|
||||
additional[A_GENERATE_POLYGON_LABEL_POINTS] ||
|
||||
sst->uses_gamma ||
|
||||
retain_points_multiplier > 1 ||
|
||||
preserve_multiplier_density_threshold > 0 ||
|
||||
cluster_distance != 0) {
|
||||
sf.index = bbox_index;
|
||||
} else {
|
||||
sf.index = 0;
|
||||
}
|
||||
|
||||
if (sst->layermap->count(sf.layername) == 0) {
|
||||
sst->layermap->insert(std::pair<std::string, layermap_entry>(sf.layername, layermap_entry(sst->layermap->size())));
|
||||
if (sst->layermap->count(layername) == 0) {
|
||||
sst->layermap->emplace(layername, layermap_entry(sst->layermap->size()));
|
||||
}
|
||||
|
||||
auto ai = sst->layermap->find(sf.layername);
|
||||
auto ai = sst->layermap->find(layername);
|
||||
if (ai != sst->layermap->end()) {
|
||||
sf.layer = ai->second.id;
|
||||
|
||||
@@ -654,12 +770,28 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
}
|
||||
}
|
||||
} else {
|
||||
fprintf(stderr, "Internal error: can't find layer name %s\n", sf.layername.c_str());
|
||||
fprintf(stderr, "Internal error: can't find layer name %s\n", layername.c_str());
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
for (auto &kv : set_attributes) {
|
||||
bool found = false;
|
||||
for (size_t i = 0; i < sf.full_keys.size(); i++) {
|
||||
if (*sf.full_keys[i] == kv.first) {
|
||||
sf.full_values[i] = kv.second;
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!found) {
|
||||
sf.full_keys.push_back(key_pool.pool(kv.first));
|
||||
sf.full_values.push_back(kv.second);
|
||||
}
|
||||
}
|
||||
|
||||
for (ssize_t i = (ssize_t) sf.full_keys.size() - 1; i >= 0; i--) {
|
||||
coerce_value(sf.full_keys[i], sf.full_values[i].type, sf.full_values[i].s, sst->attribute_types);
|
||||
coerce_value(*sf.full_keys[i], sf.full_values[i].type, sf.full_values[i].s, sst->attribute_types);
|
||||
|
||||
if (prevent[P_SINGLE_PRECISION]) {
|
||||
if (sf.full_values[i].type == mvt_double) {
|
||||
@@ -670,12 +802,12 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
}
|
||||
}
|
||||
|
||||
if (sf.full_keys[i] == attribute_for_id) {
|
||||
if (*sf.full_keys[i] == attribute_for_id) {
|
||||
if (sf.full_values[i].type != mvt_double && !additional[A_CONVERT_NUMERIC_IDS]) {
|
||||
static bool warned = false;
|
||||
|
||||
if (!warned) {
|
||||
fprintf(stderr, "Warning: Attribute \"%s\"=\"%s\" as feature ID is not a number\n", sf.full_keys[i].c_str(), sf.full_values[i].s.c_str());
|
||||
fprintf(stderr, "Warning: Attribute \"%s\"=\"%s\" as feature ID is not a number\n", sf.full_keys[i]->c_str(), sf.full_values[i].s.c_str());
|
||||
warned = true;
|
||||
}
|
||||
} else {
|
||||
@@ -708,12 +840,12 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
}
|
||||
|
||||
if (sst->exclude_all) {
|
||||
if (sst->include->count(sf.full_keys[i]) == 0) {
|
||||
if (sst->include->count(*sf.full_keys[i]) == 0) {
|
||||
sf.full_keys.erase(sf.full_keys.begin() + i);
|
||||
sf.full_values.erase(sf.full_values.begin() + i);
|
||||
continue;
|
||||
}
|
||||
} else if (sst->exclude->count(sf.full_keys[i]) != 0) {
|
||||
} else if (sst->exclude->count(*sf.full_keys[i]) != 0) {
|
||||
sf.full_keys.erase(sf.full_keys.begin() + i);
|
||||
sf.full_values.erase(sf.full_values.begin() + i);
|
||||
continue;
|
||||
@@ -722,18 +854,22 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
|
||||
if (!sst->filters) {
|
||||
for (size_t i = 0; i < sf.full_keys.size(); i++) {
|
||||
type_and_string attrib;
|
||||
attrib.type = sf.full_values[i].type;
|
||||
attrib.string = sf.full_values[i].s;
|
||||
auto ts = sst->layermap->find(layername);
|
||||
add_to_tilestats(ts->second.tilestats, *sf.full_keys[i], sf.full_values[i]);
|
||||
}
|
||||
}
|
||||
|
||||
auto fk = sst->layermap->find(sf.layername);
|
||||
add_to_file_keys(fk->second.file_keys, sf.full_keys[i], attrib);
|
||||
if (maximum_string_attribute_length > 0) {
|
||||
for (size_t i = 0; i < sf.full_keys.size(); i++) {
|
||||
if (sf.full_values[i].type == mvt_string && sf.full_values[i].s.size() > maximum_string_attribute_length) {
|
||||
sf.full_values[i].s = truncate_string(sf.full_values[i].s, maximum_string_attribute_length);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < sf.full_keys.size(); i++) {
|
||||
sf.keys.push_back(addpool(r->poolfile, r->treefile, sf.full_keys[i].c_str(), mvt_string));
|
||||
sf.values.push_back(addpool(r->poolfile, r->treefile, sf.full_values[i].s.c_str(), sf.full_values[i].type));
|
||||
sf.keys.push_back(addpool(r->poolfile, r->treefile, sf.full_keys[i]->c_str(), mvt_string, r->key_dedup));
|
||||
sf.values.push_back(addpool(r->poolfile, r->treefile, sf.full_values[i].s.c_str(), sf.full_values[i].type, r->value_dedup));
|
||||
}
|
||||
|
||||
long long geomstart = r->geompos;
|
||||
@@ -777,7 +913,7 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
void coerce_value(std::string const &key, int &vt, std::string &val, std::map<std::string, int> const *attribute_types) {
|
||||
void coerce_value(std::string const &key, int &vt, std::string &val, std::unordered_map<std::string, int> const *attribute_types) {
|
||||
auto a = (*attribute_types).find(key);
|
||||
if (a != attribute_types->end()) {
|
||||
if (a->second == mvt_string) {
|
||||
|
||||
+179
-29
@@ -6,6 +6,7 @@
|
||||
#include <string.h>
|
||||
#include <vector>
|
||||
#include <atomic>
|
||||
#include <memory>
|
||||
#include <sys/stat.h>
|
||||
#include "geometry.hpp"
|
||||
#include "mbtiles.hpp"
|
||||
@@ -25,21 +26,66 @@ void serialize_ulong_long(std::string &out, unsigned long long n);
|
||||
void serialize_byte(std::string &out, signed char n);
|
||||
void serialize_uint(std::string &out, unsigned n);
|
||||
|
||||
void deserialize_int(char **f, int *n);
|
||||
void deserialize_long_long(char **f, long long *n);
|
||||
void deserialize_ulong_long(char **f, unsigned long long *n);
|
||||
void deserialize_uint(char **f, unsigned *n);
|
||||
void deserialize_byte(char **f, signed char *n);
|
||||
|
||||
int deserialize_int_io(FILE *f, int *n, std::atomic<long long> *geompos);
|
||||
int deserialize_long_long_io(FILE *f, long long *n, std::atomic<long long> *geompos);
|
||||
int deserialize_ulong_long_io(FILE *f, unsigned long long *n, std::atomic<long long> *geompos);
|
||||
int deserialize_uint_io(FILE *f, unsigned *n, std::atomic<long long> *geompos);
|
||||
int deserialize_byte_io(FILE *f, signed char *n, std::atomic<long long> *geompos);
|
||||
void deserialize_int(const char **f, int *n);
|
||||
void deserialize_long_long(const char **f, long long *n);
|
||||
void deserialize_ulong_long(const char **f, unsigned long long *n);
|
||||
void deserialize_uint(const char **f, unsigned *n);
|
||||
void deserialize_byte(const char **f, signed char *n);
|
||||
|
||||
// This is the main representation of attribute values in memory and
|
||||
// in the string pool. The type is one of the mvt_value type (mvt_string,
|
||||
// mvt_double, mvt_bool, or mvt_null). Note that all numeric values,
|
||||
// whether integer or floating point, use mvt_double here.
|
||||
struct serial_val {
|
||||
int type = 0;
|
||||
std::string s = "";
|
||||
int type;
|
||||
std::string s;
|
||||
|
||||
bool operator<(const serial_val &o) const;
|
||||
bool operator!=(const serial_val &o) const;
|
||||
|
||||
serial_val() {
|
||||
type = 0;
|
||||
}
|
||||
|
||||
serial_val(int t, const std::string &val)
|
||||
: type(t), s(val) {
|
||||
}
|
||||
|
||||
// These functions for interface compatibility with mvt_value:
|
||||
|
||||
serial_val(double val) {
|
||||
type = mvt_double;
|
||||
s = milo::dtoa_milo(val);
|
||||
}
|
||||
|
||||
double to_double() const {
|
||||
return atof(s.c_str());
|
||||
}
|
||||
|
||||
std::string get_string_value() const {
|
||||
return s;
|
||||
}
|
||||
|
||||
void set_string_value(std::string const &val) {
|
||||
type = mvt_string;
|
||||
s = val;
|
||||
}
|
||||
};
|
||||
|
||||
struct key_pool {
|
||||
std::unordered_map<std::string, std::shared_ptr<std::string>> mapping;
|
||||
|
||||
std::shared_ptr<std::string> pool(std::string const &s) {
|
||||
auto f = mapping.find(s);
|
||||
if (f != mapping.end()) {
|
||||
return f->second;
|
||||
}
|
||||
|
||||
std::shared_ptr<std::string> p = std::make_shared<std::string>();
|
||||
*p = s;
|
||||
mapping.emplace(s, p);
|
||||
return p;
|
||||
}
|
||||
};
|
||||
|
||||
struct serial_feature {
|
||||
@@ -53,53 +99,101 @@ struct serial_feature {
|
||||
bool has_id = false;
|
||||
unsigned long long id = 0;
|
||||
|
||||
bool has_tippecanoe_minzoom = false;
|
||||
int tippecanoe_minzoom = 0;
|
||||
|
||||
bool has_tippecanoe_maxzoom = false;
|
||||
int tippecanoe_maxzoom = 0;
|
||||
int tippecanoe_minzoom = -1;
|
||||
int tippecanoe_maxzoom = -1;
|
||||
|
||||
drawvec geometry = drawvec();
|
||||
unsigned long long index = 0;
|
||||
unsigned long long gap = 0; // filled in during z0. square of planar distance
|
||||
unsigned long long label_point = 0;
|
||||
long long extent = 0;
|
||||
|
||||
// These fields are not directly serialized, but are used
|
||||
// to create the keys and values references into the string pool
|
||||
// during initial serialization
|
||||
|
||||
std::vector<std::shared_ptr<std::string>> full_keys{};
|
||||
std::vector<serial_val> full_values{};
|
||||
|
||||
// These fields are generated from full_keys and full_values
|
||||
// during initial serialization and then replace the string
|
||||
// representations:
|
||||
|
||||
std::vector<long long> keys{};
|
||||
std::vector<long long> values{};
|
||||
|
||||
// XXX This isn't serialized. Should it be here?
|
||||
// These fields are used during tiling,
|
||||
// but are not serialized and are not expected
|
||||
// to be provided by frontends:
|
||||
|
||||
long long bbox[4] = {0, 0, 0, 0};
|
||||
std::vector<std::string> full_keys{};
|
||||
std::vector<serial_val> full_values{};
|
||||
std::string layername = "";
|
||||
bool dropped = false;
|
||||
drawvec edge_nodes; // what nodes at the tile edge were added during clipping?
|
||||
|
||||
#define FEATURE_DROPPED -1
|
||||
#define FEATURE_KEPT 0
|
||||
#define FEATURE_ADDED_FOR_MULTIPLIER_DENSITY INT_MAX
|
||||
// <0: dropped
|
||||
// 0: kept
|
||||
// >0: sequence number of additional feature kept by retain-points-multiplier
|
||||
// INT_MAX: additional feature kept by preserve-multiplier-density-threshold
|
||||
int dropped = FEATURE_DROPPED; // was this feature dropped by rate?
|
||||
|
||||
// unsigned long long drop_by; // dot-dropping priority
|
||||
bool reduced; // is polygon dust
|
||||
bool coalesced; // was coalesced from multiple features
|
||||
int line_detail; // current tile resolution being used for simplification
|
||||
int extra_detail; // extra tile resolution to retain in output
|
||||
int maxzoom;
|
||||
double spacing; // feature spacing for --calculate-feature-density
|
||||
double simplification; // simplification level at this zoom level
|
||||
std::vector<ssize_t> arc_polygon; // used in --detect-shared-borders
|
||||
ssize_t renamed; // used in --detect-shared-borders logic
|
||||
long long clustered; // does this feature need the clustered/point_count attributes?
|
||||
const char *stringpool; // string pool for keys/values lookup
|
||||
std::shared_ptr<std::string> tile_stringpool; // string pool for mvt_value construction
|
||||
std::set<std::string> need_tilestats;
|
||||
std::unordered_map<std::string, accum_state> attribute_accum_state;
|
||||
|
||||
int z; // tile being produced
|
||||
int tx;
|
||||
int ty;
|
||||
};
|
||||
|
||||
std::string serialize_feature(serial_feature *sf, long long wx, long long wy);
|
||||
serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y);
|
||||
serial_feature deserialize_feature(std::string const &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y);
|
||||
|
||||
struct reader {
|
||||
int poolfd = -1;
|
||||
int treefd = -1;
|
||||
int geomfd = -1;
|
||||
int indexfd = -1;
|
||||
int vertexfd = -1;
|
||||
int nodefd = -1;
|
||||
|
||||
struct memfile *poolfile = NULL;
|
||||
struct memfile *treefile = NULL;
|
||||
FILE *geomfile = NULL;
|
||||
FILE *indexfile = NULL;
|
||||
FILE *vertexfile = NULL;
|
||||
FILE *nodefile = NULL;
|
||||
|
||||
std::atomic<long long> geompos;
|
||||
std::atomic<long long> indexpos;
|
||||
std::atomic<long long> vertexpos;
|
||||
std::atomic<long long> nodepos;
|
||||
|
||||
long long file_bbox[4] = {0, 0, 0, 0};
|
||||
long long file_bbox1[4] = {0xFFFFFFFF, 0xFFFFFFFF, 0, 0}; // standard -180 to 180 world plane
|
||||
long long file_bbox2[4] = {0x1FFFFFFFF, 0xFFFFFFFF, 0x100000000, 0}; // 0 to 360 world plane
|
||||
|
||||
struct stat geomst {};
|
||||
|
||||
char *geom_map = NULL;
|
||||
|
||||
std::vector<ssize_t> key_dedup = std::vector<ssize_t>(655536, -1);
|
||||
std::vector<ssize_t> value_dedup = std::vector<ssize_t>(655536, -1);
|
||||
|
||||
reader()
|
||||
: geompos(0), indexpos(0) {
|
||||
: geompos(0), indexpos(0), vertexpos(0), nodepos(0) {
|
||||
}
|
||||
|
||||
reader(reader const &r) {
|
||||
@@ -107,11 +201,15 @@ struct reader {
|
||||
treefd = r.treefd;
|
||||
geomfd = r.geomfd;
|
||||
indexfd = r.indexfd;
|
||||
vertexfd = r.vertexfd;
|
||||
nodefd = r.nodefd;
|
||||
|
||||
poolfile = r.poolfile;
|
||||
treefile = r.treefile;
|
||||
geomfile = r.geomfile;
|
||||
indexfile = r.indexfile;
|
||||
vertexfile = r.vertexfile;
|
||||
nodefile = r.nodefile;
|
||||
|
||||
long long p = r.geompos;
|
||||
geompos = p;
|
||||
@@ -119,6 +217,12 @@ struct reader {
|
||||
p = r.indexpos;
|
||||
indexpos = p;
|
||||
|
||||
p = r.vertexpos;
|
||||
vertexpos = p;
|
||||
|
||||
p = r.nodepos;
|
||||
nodepos = p;
|
||||
|
||||
memcpy(file_bbox, r.file_bbox, sizeof(file_bbox));
|
||||
|
||||
geomst = r.geomst;
|
||||
@@ -154,13 +258,59 @@ struct serialization_state {
|
||||
|
||||
std::map<std::string, layermap_entry> *layermap = NULL;
|
||||
|
||||
std::map<std::string, int> const *attribute_types = NULL;
|
||||
std::unordered_map<std::string, int> const *attribute_types = NULL;
|
||||
std::set<std::string> *exclude = NULL;
|
||||
std::set<std::string> *include = NULL;
|
||||
int exclude_all = 0;
|
||||
};
|
||||
|
||||
int serialize_feature(struct serialization_state *sst, serial_feature &sf);
|
||||
void coerce_value(std::string const &key, int &vt, std::string &val, std::map<std::string, int> const *attribute_types);
|
||||
struct vertex {
|
||||
// these are scaled geometry,
|
||||
// but because scaling is disabled if P_SHARED_NODES is set,
|
||||
// they are effectively also world coordinates
|
||||
draw p1;
|
||||
draw mid;
|
||||
draw p2;
|
||||
|
||||
vertex(draw one, draw joint, draw two) {
|
||||
if (one < two) {
|
||||
p1 = one;
|
||||
p2 = two;
|
||||
} else {
|
||||
p1 = two;
|
||||
p2 = one;
|
||||
}
|
||||
|
||||
mid = joint;
|
||||
}
|
||||
|
||||
bool operator<(const vertex &v) const {
|
||||
if (mid < v.mid) {
|
||||
return true;
|
||||
} else if (mid == v.mid) {
|
||||
if (p1 < v.p1) {
|
||||
return true;
|
||||
} else if (p1 == v.p1) {
|
||||
if (p2 < v.p2) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
struct node {
|
||||
// this is in quadkey coordinates so that the nodes for each tile
|
||||
// will be adjacent in memory, reducing potential thrashing during
|
||||
// the binary search.
|
||||
unsigned long long index;
|
||||
};
|
||||
|
||||
int nodecmp(const void *void1, const void *void2);
|
||||
|
||||
int serialize_feature(struct serialization_state *sst, serial_feature &sf, std::string const &layername);
|
||||
void coerce_value(std::string const &key, int &vt, std::string &val, std::unordered_map<std::string, int> const *attribute_types);
|
||||
|
||||
#endif
|
||||
|
||||
@@ -0,0 +1,575 @@
|
||||
#include <stdio.h>
|
||||
#include <algorithm>
|
||||
#include "geometry.hpp"
|
||||
#include "serial.hpp"
|
||||
#include "options.hpp"
|
||||
|
||||
// This should all be removed and replaced with --no-simplification-of-shared-nodes
|
||||
|
||||
// Does not fix up moveto/lineto
|
||||
static drawvec reverse_subring(drawvec const &dv) {
|
||||
drawvec out;
|
||||
|
||||
for (size_t i = dv.size(); i > 0; i--) {
|
||||
out.push_back(dv[i - 1]);
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
struct edge {
|
||||
unsigned x1 = 0;
|
||||
unsigned y1 = 0;
|
||||
unsigned x2 = 0;
|
||||
unsigned y2 = 0;
|
||||
unsigned ring = 0;
|
||||
|
||||
edge(unsigned _x1, unsigned _y1, unsigned _x2, unsigned _y2, unsigned _ring) {
|
||||
x1 = _x1;
|
||||
y1 = _y1;
|
||||
x2 = _x2;
|
||||
y2 = _y2;
|
||||
ring = _ring;
|
||||
}
|
||||
|
||||
bool operator<(const edge &s) const {
|
||||
long long cmp = (long long) y1 - s.y1;
|
||||
if (cmp == 0) {
|
||||
cmp = (long long) x1 - s.x1;
|
||||
}
|
||||
if (cmp == 0) {
|
||||
cmp = (long long) y2 - s.y2;
|
||||
}
|
||||
if (cmp == 0) {
|
||||
cmp = (long long) x2 - s.x2;
|
||||
}
|
||||
return cmp < 0;
|
||||
}
|
||||
};
|
||||
|
||||
struct edgecmp_ring {
|
||||
bool operator()(const edge &a, const edge &b) {
|
||||
long long cmp = (long long) a.y1 - b.y1;
|
||||
if (cmp == 0) {
|
||||
cmp = (long long) a.x1 - b.x1;
|
||||
}
|
||||
if (cmp == 0) {
|
||||
cmp = (long long) a.y2 - b.y2;
|
||||
}
|
||||
if (cmp == 0) {
|
||||
cmp = (long long) a.x2 - b.x2;
|
||||
}
|
||||
if (cmp == 0) {
|
||||
cmp = (long long) a.ring - b.ring;
|
||||
}
|
||||
return cmp < 0;
|
||||
}
|
||||
} edgecmp_ring;
|
||||
|
||||
bool edges_same(std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e1, std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e2) {
|
||||
if ((e2.second - e2.first) != (e1.second - e1.first)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
while (e1.first != e1.second) {
|
||||
if (e1.first->ring != e2.first->ring) {
|
||||
return false;
|
||||
}
|
||||
|
||||
++e1.first;
|
||||
++e2.first;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
bool find_common_edges(std::vector<serial_feature> &features, int z, int line_detail, double simplification, int maxzoom, double merge_fraction) {
|
||||
size_t merge_count = ceil((1 - merge_fraction) * features.size());
|
||||
|
||||
for (size_t i = 0; i < features.size(); i++) {
|
||||
if (features[i].t == VT_POLYGON) {
|
||||
{
|
||||
drawvec &g = features[i].geometry;
|
||||
drawvec out;
|
||||
|
||||
for (size_t k = 0; k < g.size(); k++) {
|
||||
if (g[k].op == VT_LINETO && k > 0 && g[k - 1] == g[k]) {
|
||||
;
|
||||
} else {
|
||||
out.push_back(g[k]);
|
||||
}
|
||||
}
|
||||
|
||||
features[i].geometry = out;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Construct a mapping from all polygon edges to the set of rings
|
||||
// that each edge appears in. (The ring number is across all polygons;
|
||||
// we don't need to look it back up, just to tell where it changes.)
|
||||
|
||||
std::vector<edge> edges;
|
||||
size_t ring = 0;
|
||||
for (size_t i = 0; i < features.size(); i++) {
|
||||
if (features[i].t == VT_POLYGON) {
|
||||
{
|
||||
for (size_t k = 0; k + 1 < features[i].geometry.size(); k++) {
|
||||
if (features[i].geometry[k].op == VT_MOVETO) {
|
||||
ring++;
|
||||
}
|
||||
|
||||
if (features[i].geometry[k + 1].op == VT_LINETO) {
|
||||
drawvec dv;
|
||||
if (features[i].geometry[k] < features[i].geometry[k + 1]) {
|
||||
dv.push_back(features[i].geometry[k]);
|
||||
dv.push_back(features[i].geometry[k + 1]);
|
||||
} else {
|
||||
dv.push_back(features[i].geometry[k + 1]);
|
||||
dv.push_back(features[i].geometry[k]);
|
||||
}
|
||||
|
||||
edges.push_back(edge(dv[0].x, dv[0].y, dv[1].x, dv[1].y, ring));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
std::stable_sort(edges.begin(), edges.end(), edgecmp_ring);
|
||||
std::set<draw> necessaries;
|
||||
|
||||
// Now mark all the points where the set of rings using the edge on one side
|
||||
// is not the same as the set of rings using the edge on the other side.
|
||||
|
||||
for (size_t i = 0; i < features.size(); i++) {
|
||||
if (features[i].t == VT_POLYGON) {
|
||||
{
|
||||
drawvec &g = features[i].geometry;
|
||||
|
||||
for (size_t k = 0; k < g.size(); k++) {
|
||||
g[k].necessary = 0;
|
||||
}
|
||||
|
||||
for (size_t a = 0; a < g.size(); a++) {
|
||||
if (g[a].op == VT_MOVETO) {
|
||||
size_t b;
|
||||
|
||||
for (b = a + 1; b < g.size(); b++) {
|
||||
if (g[b].op != VT_LINETO) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// -1 because of duplication at the end
|
||||
size_t s = b - a - 1;
|
||||
|
||||
if (s > 0) {
|
||||
drawvec left;
|
||||
if (g[a + (s - 1) % s] < g[a]) {
|
||||
left.push_back(g[a + (s - 1) % s]);
|
||||
left.push_back(g[a]);
|
||||
} else {
|
||||
left.push_back(g[a]);
|
||||
left.push_back(g[a + (s - 1) % s]);
|
||||
}
|
||||
if (left[1] < left[0]) {
|
||||
fprintf(stderr, "left misordered\n");
|
||||
}
|
||||
std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e1 = std::equal_range(edges.begin(), edges.end(), edge(left[0].x, left[0].y, left[1].x, left[1].y, 0));
|
||||
|
||||
for (size_t k = 0; k < s; k++) {
|
||||
drawvec right;
|
||||
|
||||
if (g[a + k] < g[a + k + 1]) {
|
||||
right.push_back(g[a + k]);
|
||||
right.push_back(g[a + k + 1]);
|
||||
} else {
|
||||
right.push_back(g[a + k + 1]);
|
||||
right.push_back(g[a + k]);
|
||||
}
|
||||
|
||||
std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e2 = std::equal_range(edges.begin(), edges.end(), edge(right[0].x, right[0].y, right[1].x, right[1].y, 0));
|
||||
|
||||
if (right[1] < right[0]) {
|
||||
fprintf(stderr, "left misordered\n");
|
||||
}
|
||||
|
||||
if (e1.first == e1.second || e2.first == e2.second) {
|
||||
fprintf(stderr, "Internal error: polygon edge lookup failed for %lld,%lld to %lld,%lld or %lld,%lld to %lld,%lld\n", (long long) left[0].x, (long long) left[0].y, (long long) left[1].x, (long long) left[1].y, (long long) right[0].x, (long long) right[0].y, (long long) right[1].x, (long long) right[1].y);
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
if (!edges_same(e1, e2)) {
|
||||
g[a + k].necessary = 1;
|
||||
necessaries.insert(g[a + k]);
|
||||
}
|
||||
|
||||
e1 = e2;
|
||||
}
|
||||
}
|
||||
|
||||
a = b - 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
edges.clear();
|
||||
std::map<drawvec, size_t> arcs;
|
||||
std::multimap<ssize_t, size_t> merge_candidates; // from arc to serial_feature
|
||||
|
||||
// Roll rings that include a necessary point around so they start at one
|
||||
|
||||
for (size_t i = 0; i < features.size(); i++) {
|
||||
if (features[i].t == VT_POLYGON) {
|
||||
{
|
||||
drawvec &g = features[i].geometry;
|
||||
|
||||
for (size_t k = 0; k < g.size(); k++) {
|
||||
if (necessaries.count(g[k]) != 0) {
|
||||
g[k].necessary = 1;
|
||||
}
|
||||
}
|
||||
|
||||
for (size_t k = 0; k < g.size(); k++) {
|
||||
if (g[k].op == VT_MOVETO) {
|
||||
ssize_t necessary = -1;
|
||||
ssize_t lowest = k;
|
||||
size_t l;
|
||||
for (l = k + 1; l < g.size(); l++) {
|
||||
if (g[l].op != VT_LINETO) {
|
||||
break;
|
||||
}
|
||||
|
||||
if (g[l].necessary) {
|
||||
necessary = l;
|
||||
}
|
||||
if (g[l] < g[lowest]) {
|
||||
lowest = l;
|
||||
}
|
||||
}
|
||||
|
||||
if (necessary < 0) {
|
||||
necessary = lowest;
|
||||
// Add a necessary marker if there was none in the ring,
|
||||
// so the arc code below can find it.
|
||||
g[lowest].necessary = 1;
|
||||
}
|
||||
|
||||
{
|
||||
drawvec tmp;
|
||||
|
||||
// l - 1 because the endpoint is duplicated
|
||||
for (size_t m = necessary; m < l - 1; m++) {
|
||||
tmp.push_back(g[m]);
|
||||
}
|
||||
for (ssize_t m = k; m < necessary; m++) {
|
||||
tmp.push_back(g[m]);
|
||||
}
|
||||
|
||||
// replace the endpoint
|
||||
tmp.push_back(g[necessary]);
|
||||
|
||||
if (tmp.size() != l - k) {
|
||||
fprintf(stderr, "internal error shifting ring\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
for (size_t m = 0; m < tmp.size(); m++) {
|
||||
if (m == 0) {
|
||||
tmp[m].op = VT_MOVETO;
|
||||
} else {
|
||||
tmp[m].op = VT_LINETO;
|
||||
}
|
||||
|
||||
g[k + m] = tmp[m];
|
||||
}
|
||||
}
|
||||
|
||||
// Now peel off each set of segments from one necessary point to the next
|
||||
// into an "arc" as in TopoJSON
|
||||
|
||||
for (size_t m = k; m < l; m++) {
|
||||
if (!g[m].necessary) {
|
||||
fprintf(stderr, "internal error in arc building\n");
|
||||
exit(EXIT_IMPOSSIBLE);
|
||||
}
|
||||
|
||||
drawvec arc;
|
||||
size_t n;
|
||||
for (n = m; n < l; n++) {
|
||||
arc.push_back(g[n]);
|
||||
if (n > m && g[n].necessary) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
auto f = arcs.find(arc);
|
||||
if (f == arcs.end()) {
|
||||
drawvec arc2 = reverse_subring(arc);
|
||||
|
||||
auto f2 = arcs.find(arc2);
|
||||
if (f2 == arcs.end()) {
|
||||
// Add new arc
|
||||
size_t added = arcs.size() + 1;
|
||||
arcs.insert(std::pair<drawvec, size_t>(arc, added));
|
||||
features[i].arc_polygon.push_back(added);
|
||||
merge_candidates.insert(std::pair<ssize_t, size_t>(added, i));
|
||||
} else {
|
||||
features[i].arc_polygon.push_back(-(ssize_t) f2->second);
|
||||
merge_candidates.insert(std::pair<ssize_t, size_t>(-(ssize_t) f2->second, i));
|
||||
}
|
||||
} else {
|
||||
features[i].arc_polygon.push_back(f->second);
|
||||
merge_candidates.insert(std::pair<ssize_t, size_t>(f->second, i));
|
||||
}
|
||||
|
||||
m = n - 1;
|
||||
}
|
||||
|
||||
features[i].arc_polygon.push_back(0);
|
||||
|
||||
k = l - 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Simplify each arc
|
||||
|
||||
std::vector<drawvec> simplified_arcs;
|
||||
|
||||
for (auto ai = arcs.begin(); ai != arcs.end(); ++ai) {
|
||||
if (simplified_arcs.size() < ai->second + 1) {
|
||||
simplified_arcs.resize(ai->second + 1);
|
||||
}
|
||||
|
||||
drawvec dv = ai->first;
|
||||
for (size_t i = 0; i < dv.size(); i++) {
|
||||
if (i == 0) {
|
||||
dv[i].op = VT_MOVETO;
|
||||
} else {
|
||||
dv[i].op = VT_LINETO;
|
||||
}
|
||||
}
|
||||
if (!(prevent[P_SIMPLIFY] || (z == maxzoom && prevent[P_SIMPLIFY_LOW]) || (z < maxzoom && additional[A_GRID_LOW_ZOOMS]))) {
|
||||
// tx and ty are 0 here because we aren't trying to do anything with the shared_nodes_map
|
||||
simplified_arcs[ai->second] = simplify_lines(dv, z, 0, 0, line_detail, !(prevent[P_CLIPPING] || prevent[P_DUPLICATION]), simplification, 4, drawvec(), NULL, 0, "");
|
||||
} else {
|
||||
simplified_arcs[ai->second] = dv;
|
||||
}
|
||||
}
|
||||
|
||||
// If necessary, merge some adjacent polygons into some other polygons
|
||||
|
||||
struct merge_order {
|
||||
ssize_t edge = 0;
|
||||
unsigned long long gap = 0;
|
||||
size_t p1 = 0;
|
||||
size_t p2 = 0;
|
||||
|
||||
bool operator<(const merge_order &m) const {
|
||||
return gap < m.gap;
|
||||
}
|
||||
};
|
||||
std::vector<merge_order> order;
|
||||
|
||||
for (ssize_t i = 0; i < (ssize_t) simplified_arcs.size(); i++) {
|
||||
auto r1 = merge_candidates.equal_range(i);
|
||||
for (auto r1i = r1.first; r1i != r1.second; ++r1i) {
|
||||
auto r2 = merge_candidates.equal_range(-i);
|
||||
for (auto r2i = r2.first; r2i != r2.second; ++r2i) {
|
||||
if (r1i->second != r2i->second) {
|
||||
merge_order mo;
|
||||
mo.edge = i;
|
||||
if (features[r1i->second].index > features[r2i->second].index) {
|
||||
mo.gap = features[r1i->second].index - features[r2i->second].index;
|
||||
} else {
|
||||
mo.gap = features[r2i->second].index - features[r1i->second].index;
|
||||
}
|
||||
mo.p1 = r1i->second;
|
||||
mo.p2 = r2i->second;
|
||||
order.push_back(mo);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
std::stable_sort(order.begin(), order.end());
|
||||
|
||||
size_t merged = 0;
|
||||
for (size_t o = 0; o < order.size(); o++) {
|
||||
if (merged >= merge_count) {
|
||||
break;
|
||||
}
|
||||
|
||||
size_t i = order[o].p1;
|
||||
while (features[i].renamed >= 0) {
|
||||
i = features[i].renamed;
|
||||
}
|
||||
size_t i2 = order[o].p2;
|
||||
while (features[i2].renamed >= 0) {
|
||||
i2 = features[i2].renamed;
|
||||
}
|
||||
|
||||
for (size_t j = 0; j < features[i].arc_polygon.size() && merged < merge_count; j++) {
|
||||
if (features[i].arc_polygon[j] == order[o].edge) {
|
||||
{
|
||||
// XXX snap links
|
||||
if (features[order[o].p2].arc_polygon.size() > 0) {
|
||||
// This has to merge the ring that contains the anti-arc to this arc
|
||||
// into the current ring, and then add whatever other rings were in
|
||||
// that feature on to the end.
|
||||
//
|
||||
// This can't be good for keeping parent-child relationships among
|
||||
// the rings in order, but Wagyu should sort that out later
|
||||
|
||||
std::vector<ssize_t> additions;
|
||||
std::vector<ssize_t> &here = features[i].arc_polygon;
|
||||
std::vector<ssize_t> &other = features[i2].arc_polygon;
|
||||
|
||||
#if 0
|
||||
printf("seeking %zd\n", features[i].arc_polygon[j]);
|
||||
printf("before: ");
|
||||
for (size_t k = 0; k < here.size(); k++) {
|
||||
printf("%zd ", here[k]);
|
||||
}
|
||||
printf("\n");
|
||||
printf("other: ");
|
||||
for (size_t k = 0; k < other.size(); k++) {
|
||||
printf("%zd ", other[k]);
|
||||
}
|
||||
printf("\n");
|
||||
#endif
|
||||
|
||||
for (size_t k = 0; k < other.size(); k++) {
|
||||
size_t l;
|
||||
for (l = k; l < other.size(); l++) {
|
||||
if (other[l] == 0) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (l >= other.size()) {
|
||||
l--;
|
||||
}
|
||||
|
||||
#if 0
|
||||
for (size_t m = k; m <= l; m++) {
|
||||
printf("%zd ", other[m]);
|
||||
}
|
||||
printf("\n");
|
||||
#endif
|
||||
|
||||
size_t m;
|
||||
for (m = k; m <= l; m++) {
|
||||
if (other[m] == -features[i].arc_polygon[j]) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (m <= l) {
|
||||
// Found the shared arc
|
||||
|
||||
here.erase(here.begin() + j);
|
||||
|
||||
size_t off = 0;
|
||||
for (size_t n = m + 1; n < l; n++) {
|
||||
here.insert(here.begin() + j + off, other[n]);
|
||||
off++;
|
||||
}
|
||||
for (size_t n = k; n < m; n++) {
|
||||
here.insert(here.begin() + j + off, other[n]);
|
||||
off++;
|
||||
}
|
||||
} else {
|
||||
// Looking at some other ring
|
||||
|
||||
for (size_t n = k; n <= l; n++) {
|
||||
additions.push_back(other[n]);
|
||||
}
|
||||
}
|
||||
|
||||
k = l;
|
||||
}
|
||||
|
||||
features[i2].arc_polygon.clear();
|
||||
features[i2].renamed = i;
|
||||
merged++;
|
||||
|
||||
for (size_t k = 0; k < additions.size(); k++) {
|
||||
features[i].arc_polygon.push_back(additions[k]);
|
||||
}
|
||||
|
||||
#if 0
|
||||
printf("after: ");
|
||||
for (size_t k = 0; k < here.size(); k++) {
|
||||
printf("%zd ", here[k]);
|
||||
}
|
||||
printf("\n");
|
||||
#endif
|
||||
|
||||
#if 0
|
||||
for (size_t k = 0; k + 1 < here.size(); k++) {
|
||||
if (here[k] != 0 && here[k + 1] != 0) {
|
||||
if (simplified_arcs[here[k + 1]][0] != simplified_arcs[here[k]][simplified_arcs[here[k]].size() - 1]) {
|
||||
printf("error from %zd to %zd\n", here[k], here[k + 1]);
|
||||
}
|
||||
}
|
||||
}
|
||||
#endif
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Turn the arc representations of the polygons back into standard polygon geometries
|
||||
|
||||
for (size_t i = 0; i < features.size(); i++) {
|
||||
if (features[i].t == VT_POLYGON) {
|
||||
features[i].geometry.clear();
|
||||
bool at_start = true;
|
||||
draw first(-1, 0, 0);
|
||||
|
||||
for (size_t j = 0; j < features[i].arc_polygon.size(); j++) {
|
||||
ssize_t p = features[i].arc_polygon[j];
|
||||
|
||||
if (p == 0) {
|
||||
if (first.op >= 0) {
|
||||
features[i].geometry.push_back(first);
|
||||
first = draw(-1, 0, 0);
|
||||
}
|
||||
at_start = true;
|
||||
} else if (p > 0) {
|
||||
for (size_t k = 0; k + 1 < simplified_arcs[p].size(); k++) {
|
||||
if (at_start) {
|
||||
features[i].geometry.push_back(draw(VT_MOVETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y));
|
||||
first = draw(VT_LINETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y);
|
||||
} else {
|
||||
features[i].geometry.push_back(draw(VT_LINETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y));
|
||||
}
|
||||
at_start = 0;
|
||||
}
|
||||
} else { /* p < 0 */
|
||||
for (ssize_t k = simplified_arcs[-p].size() - 1; k > 0; k--) {
|
||||
if (at_start) {
|
||||
features[i].geometry.push_back(draw(VT_MOVETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y));
|
||||
first = draw(VT_LINETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y);
|
||||
} else {
|
||||
features[i].geometry.push_back(draw(VT_LINETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y));
|
||||
}
|
||||
at_start = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (merged >= merge_count) {
|
||||
return true;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
bool find_common_edges(std::vector<serial_feature> &features, int z, int line_detail, double simplification, int maxzoom, double merge_fraction);
|
||||
@@ -0,0 +1,127 @@
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <unistd.h>
|
||||
#include <vector>
|
||||
#include <string>
|
||||
|
||||
#define MAX_MEMORY (1024 * 1024 * 1024) // 1 GB
|
||||
|
||||
void fqsort(std::vector<FILE *> &inputs, size_t width, int (*cmp)(const void *, const void *), FILE *out, size_t mem) {
|
||||
std::string pivot;
|
||||
FILE *fp1, *fp2;
|
||||
|
||||
if (mem > MAX_MEMORY) {
|
||||
mem = MAX_MEMORY;
|
||||
}
|
||||
|
||||
{
|
||||
// read some elements into memory to choose a pivot from
|
||||
//
|
||||
// this is in its own scope so `buf` can go out of scope
|
||||
// before trying to do any sub-sorts.
|
||||
|
||||
std::string buf;
|
||||
|
||||
bool read_everything = false;
|
||||
for (size_t i = 0; i < inputs.size(); i++) {
|
||||
if (buf.size() > mem) {
|
||||
break;
|
||||
}
|
||||
|
||||
while (true) {
|
||||
std::string element;
|
||||
element.resize(width);
|
||||
|
||||
size_t n = fread((void *) element.c_str(), width, 1, inputs[i]);
|
||||
if (n == 0) {
|
||||
if (i + 1 == inputs.size()) {
|
||||
read_everything = true;
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
buf.append(element);
|
||||
|
||||
if (buf.size() > mem) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
qsort((void *) buf.c_str(), buf.size() / width, width, cmp);
|
||||
|
||||
// If that was everything we have to sort, we are done.
|
||||
|
||||
if (read_everything) {
|
||||
fwrite((void *) buf.c_str(), buf.size() / width, width, out);
|
||||
return;
|
||||
}
|
||||
|
||||
// Otherwise, choose a pivot from it, make some temporary files,
|
||||
// write what we have to those files, and then partition the rest
|
||||
// of the input into them.
|
||||
|
||||
// This would be unstable if the pivot is one of several elements
|
||||
// that compare equal. Does it matter?
|
||||
|
||||
size_t pivot_off = width * (buf.size() / width / 2);
|
||||
pivot = std::string(buf, pivot_off, width);
|
||||
|
||||
std::string t1 = "/tmp/sort1.XXXXXX";
|
||||
std::string t2 = "/tmp/sort2.XXXXXX";
|
||||
|
||||
int fd1 = mkstemp((char *) t1.c_str());
|
||||
unlink(t1.c_str());
|
||||
int fd2 = mkstemp((char *) t2.c_str());
|
||||
unlink(t2.c_str());
|
||||
|
||||
fp1 = fdopen(fd1, "w+b");
|
||||
if (fp1 == NULL) {
|
||||
perror(t1.c_str());
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
fp2 = fdopen(fd2, "w+b");
|
||||
if (fp2 == NULL) {
|
||||
perror(t2.c_str());
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
fwrite((void *) buf.c_str(), sizeof(char), pivot_off, fp1);
|
||||
fwrite((void *) ((char *) buf.c_str() + pivot_off), sizeof(char), buf.size() - pivot_off, fp2);
|
||||
}
|
||||
|
||||
// read the remaining input into the temporary files
|
||||
|
||||
for (size_t i = 0; i < inputs.size(); i++) {
|
||||
while (true) {
|
||||
std::string element;
|
||||
element.resize(width);
|
||||
|
||||
size_t n = fread((void *) element.c_str(), width, 1, inputs[i]);
|
||||
if (n == 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
if (cmp((void *) element.c_str(), (void *) pivot.c_str()) < 0) {
|
||||
fwrite((void *) element.c_str(), width, 1, fp1);
|
||||
} else {
|
||||
fwrite((void *) element.c_str(), width, 1, fp2);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Now sort the sub-ranges into the output.
|
||||
|
||||
rewind(fp1);
|
||||
rewind(fp2);
|
||||
|
||||
std::vector<FILE *> v1;
|
||||
v1.emplace_back(fp1);
|
||||
fqsort(v1, width, cmp, out, mem);
|
||||
fclose(fp1);
|
||||
|
||||
std::vector<FILE *> v2;
|
||||
v2.emplace_back(fp2);
|
||||
fqsort(v2, width, cmp, out, mem);
|
||||
fclose(fp2);
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
#ifndef SORT_HPP
|
||||
#define SORT_HPP
|
||||
|
||||
void fqsort(std::vector<FILE *> &inputs, size_t width, int (*cmp)(const void *, const void *), FILE *out, size_t mem);
|
||||
|
||||
#endif
|
||||
@@ -0,0 +1,565 @@
|
||||
ޢx
|
||||
parsed(�
felt:h3_index SCALERANKNATSCALE LABELRANK
|
||||
FEATURECLANAME NAMEASCIIADM0CAP WORLDCITYMEGACITYSOV0NAMESOV_A3ADM0NAMEADM0_A3ADM1NAMEISO_A2LATITUDE LONGITUDEPOP_MAXPOP_MIN POP_OTHERRANK_MAXRANK_MINLS_NAME MAX_POP10 MAX_POP20 MAX_POP50
|
||||
MAX_POP300
|
||||
MAX_POP310
|
||||
MAX_NATSCA
|
||||
MIN_AREAKM
|
||||
MAX_AREAKM
|
||||
MIN_AREAMI
|
||||
MAX_AREAMI MIN_PERKM MAX_PERKM MIN_PERMI MAX_PERMI
|
||||
MIN_BBXMIN
|
||||
MAX_BBXMIN
|
||||
MIN_BBXMAX
|
||||
MAX_BBXMAX
|
||||
MIN_BBYMIN
|
||||
MAX_BBYMIN
|
||||
MIN_BBYMAX
|
||||
MAX_BBYMAX MEAN_BBXC MEAN_BBYCUN_FIDPOP1950POP1955POP1960POP1965POP1970POP1975POP1980POP1985POP1990POP1995POP2000POP2005POP2010POP2015POP2020POP2025POP2050MIN_ZOOM
|
||||
WIKIDATAIDWOF_IDCAPALTNAME_ENNAME_DENAME_ESNAME_FRNAME_PTNAME_RUNAME_ZHNAME_ARNAME_BNNAME_ELNAME_HINAME_HUNAME_IDNAME_ITNAME_JANAME_KONAME_NLNAME_PLNAME_SVNAME_TRNAME_VINE_IDNAME_FANAME_HENAME_UKNAME_URNAME_ZHT
|
||||
GEONAMESIDfelt:cluster_size,tippecanoe:retain_points_multiplier_sequence)tippecanoe:retain_points_multiplier_firstPOP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990NAMEALTPOP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINTIMEZONEPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990POP_MINPOP1990NAMEPARPOP_MINPOP1990POP_MINPOP1990POP_MINPOP1990"
|
||||
CA"
|
||||
RU"
|
||||
US"
|
||||
CAN"
|
||||
RUS"
|
||||
USA"
|
||||
Kobuk"
|
||||
Alaska"
|
||||
Ambler"
|
||||
Barrow"
|
||||
Canada"
|
||||
Holman"
|
||||
Inuvik"
|
||||
Q79333"
|
||||
Q79350"
|
||||
Q79395"
|
||||
Q79408"
|
||||
Q79714"
|
||||
Q79717"
|
||||
Q79809"
|
||||
Q79810"
|
||||
Q79813"
|
||||
Q80033"
|
||||
Q80133"
|
||||
Russia"
|
||||
巴罗"
|
||||
巴羅"
|
||||
배로"
|
||||
코북"
|
||||
홀만"
|
||||
Atqasuk"
|
||||
Nunavut"
|
||||
Q338686"
|
||||
Q465235"
|
||||
Q602422"
|
||||
Q606622"
|
||||
Q718822"
|
||||
Q719547"
|
||||
Q719585"
|
||||
Q719605"
|
||||
Q719611"
|
||||
Q719707"
|
||||
Q924763"
|
||||
Selawik"
|
||||
Wiseman"
|
||||
İnuvik"
|
||||
|
||||
Enurmino"
|
||||
|
||||
Kaktovik"
|
||||
|
||||
Kivalina"
|
||||
|
||||
Kotzebue"
|
||||
|
||||
Paulatuk"
|
||||
|
||||
Q1013046"
|
||||
|
||||
Q1014088"
|
||||
|
||||
Q1014119"
|
||||
|
||||
Q1020259"
|
||||
|
||||
Q1938137"
|
||||
|
||||
Q2305684"
|
||||
|
||||
Resolute"
|
||||
|
||||
Taloyoak"
|
||||
|
||||
Zvezdnyi"
|
||||
|
||||
Zvezdnyj"
|
||||
|
||||
Zvezdnyy"
|
||||
|
||||
Zvjozdny"
|
||||
|
||||
בארו"
|
||||
|
||||
بارو"
|
||||
|
||||
بیرو"
|
||||
|
||||
کوبک"
|
||||
Allakaket"
|
||||
Enoermino"
|
||||
Enourmino"
|
||||
Kugluktuk"
|
||||
Utqiagvik"
|
||||
Zvëzdnyj"
|
||||
コブク"
|
||||
バロー"
|
||||
剑桥湾"
|
||||
劍橋灣"
|
||||
安布勒"
|
||||
懷斯曼"
|
||||
温赖特"
|
||||
科伯克"
|
||||
科策布"
|
||||
約阿港"
|
||||
育空堡"
|
||||
셀라윅"
|
||||
앰블러"
|
||||
앳쿼석"
|
||||
이누빅"
|
||||
코체부"
|
||||
|
||||
Fort Yukon"
|
||||
|
||||
Gjoa Haven"
|
||||
|
||||
Point Hope"
|
||||
|
||||
Ulukhaktok"
|
||||
|
||||
Utqiaġvik"
|
||||
|
||||
Wainwright"
|
||||
|
||||
Кобук"
|
||||
|
||||
אלקקט"
|
||||
|
||||
אמבלר"
|
||||
|
||||
חולמן"
|
||||
|
||||
קובוק"
|
||||
|
||||
امبلر"
|
||||
|
||||
انوئک"
|
||||
|
||||
رزلوت"
|
||||
|
||||
كوبوك"
|
||||
|
||||
کوبوک"
|
||||
Fort Yukón"
|
||||
Mys Chmidta"
|
||||
Mys Shmidta"
|
||||
Mys Sjmidta"
|
||||
Mys Szmidta"
|
||||
Mys Šmidta"
|
||||
Prudhoe Bay"
|
||||
Tuktoyaktuk"
|
||||
Ikaluktutiak"
|
||||
Mys Schmidta"
|
||||
Resolute Bay"
|
||||
Tsiigehtchic"
|
||||
Άμπλερ"
|
||||
Χόλμαν"
|
||||
Інувік"
|
||||
Амблер"
|
||||
Барроу"
|
||||
Инувик"
|
||||
Коцебу"
|
||||
וויזמן"
|
||||
טלויאק"
|
||||
קוצביו"
|
||||
רזולוט"
|
||||
أمبلير"
|
||||
إنوفيك"
|
||||
ایمبلر"
|
||||
سلاویک"
|
||||
سیلاوک"
|
||||
سیگیچک"
|
||||
هولمان"
|
||||
وایزمن"
|
||||
پالتاک"
|
||||
پولاٹک"
|
||||
ہولمان"
|
||||
努勒維特"
|
||||
因纽维克"
|
||||
基瓦利纳"
|
||||
塔洛約克"
|
||||
塞拉威克"
|
||||
施密特岬"
|
||||
波拉圖克"
|
||||
雷索卢特"
|
||||
齊格奇克"
|
||||
뚝토약툭"
|
||||
리솔루트"
|
||||
알라카켓"
|
||||
와이즈맨"
|
||||
치이게치"
|
||||
키발리나"
|
||||
탈로요크"
|
||||
파울라툭"
|
||||
Cambridge Bay"
|
||||
United States"
|
||||
포트 유콘"
|
||||
America/Inuvik"
|
||||
America/Juneau"
|
||||
Campbridge Bay"
|
||||
Fort McPherson"
|
||||
Prudhoe-öböl"
|
||||
Ινούβικ"
|
||||
Κομπούκ"
|
||||
Μπάροου"
|
||||
Σελαβίκ"
|
||||
Аткасук"
|
||||
Вайзмен"
|
||||
Полатук"
|
||||
Селавик"
|
||||
Селавік"
|
||||
Уайзмен"
|
||||
Цігечик"
|
||||
אטקאסוק"
|
||||
אינוויק"
|
||||
סלאוויק"
|
||||
קקטוביק"
|
||||
آتکاسوک"
|
||||
آلاکاکت"
|
||||
اینوویک"
|
||||
ایٹقاسک"
|
||||
بولاتوك"
|
||||
تالویوک"
|
||||
تسیگچیک"
|
||||
زویزڈنج"
|
||||
سيلاويك"
|
||||
كوتزبيو"
|
||||
نورمینو"
|
||||
وائزمین"
|
||||
وينرايت"
|
||||
وینرائٹ"
|
||||
وینرایت"
|
||||
ٹالویوک"
|
||||
کگ لک تک"
|
||||
کیکٹووک"
|
||||
Populated place"
|
||||
جوا ہیون"
|
||||
وايز مان"
|
||||
پرودو بی"
|
||||
कोबुक"
|
||||
কসেভো"
|
||||
কোবুক"
|
||||
アンブラー"
|
||||
エヌルミノ"
|
||||
カクトビク"
|
||||
キバリーナ"
|
||||
セラウィク"
|
||||
タロヨーク"
|
||||
ワイズマン"
|
||||
卡克托维克"
|
||||
普拉德霍湾"
|
||||
普拉德霍灣"
|
||||
波因特霍普"
|
||||
烏路卡托克"
|
||||
艾奴爾米諾"
|
||||
艾特跨萨克"
|
||||
茲韋茲德尼"
|
||||
迈斯施密达"
|
||||
阿拉卡基特"
|
||||
麥克弗森堡"
|
||||
麦克弗森堡"
|
||||
에누르미노"
|
||||
웨인라이트"
|
||||
카크토비크"
|
||||
쿠글룩투크"
|
||||
America/Resolute"
|
||||
Vịnh Cambridge"
|
||||
Αλακάκετ"
|
||||
Ατκασούκ"
|
||||
Ζβέζντιγ"
|
||||
Κάκτοβικ"
|
||||
Κιβαλίνα"
|
||||
Ρεσολούτ"
|
||||
Вейнрайт"
|
||||
Енурміно"
|
||||
Звёздный"
|
||||
Кактовик"
|
||||
Кактовік"
|
||||
Кивалина"
|
||||
Паулатук"
|
||||
Резольют"
|
||||
Талойоак"
|
||||
Уэйнрайт"
|
||||
Цигетчик"
|
||||
Энурмино"
|
||||
אנורמינו"
|
||||
ויינרייט"
|
||||
פאולאטוק"
|
||||
ציג'טצ'יק"
|
||||
קיוולינה"
|
||||
أتكيوسوك"
|
||||
ألاكاكيت"
|
||||
إنورمينو"
|
||||
انورمینو"
|
||||
تالويواك"
|
||||
ریسولوتی"
|
||||
زووزدنیج"
|
||||
زيفازدني"
|
||||
كاكتوفيك"
|
||||
كيفالينا"
|
||||
کاکتوویک"
|
||||
کوتزابیو"
|
||||
کوتزهبیو"
|
||||
کیوالینا"
|
||||
미스 슈미타"
|
||||
조아 헤이븐"
|
||||
포인트 호프"
|
||||
포트 맥퍼슨"
|
||||
프루도 베이"
|
||||
Форт-Юкон"
|
||||
פוינט הופ"
|
||||
פרודו ביי"
|
||||
بوينت هوب"
|
||||
پروڈہو بے"
|
||||
پوینت هوپ"
|
||||
کمبریج بی"
|
||||
کیمبرج بے"
|
||||
Γουάιζμαν"
|
||||
Ενουρμίνο"
|
||||
Ταλογιοάκ"
|
||||
Аллакакет"
|
||||
Звьоздний"
|
||||
Куглуктук"
|
||||
זביוזדניי"
|
||||
ایلاکاکیٹ"
|
||||
كوغلوكتوك"
|
||||
مایسشمیتا"
|
||||
ٹکٹویاکٹک"
|
||||
کوگلاکتوک"
|
||||
इनुविक"
|
||||
होलमैन"
|
||||
ইনুভিক"
|
||||
アラカキット"
|
||||
イヌヴィック"
|
||||
コッツビュー"
|
||||
シュミット岬"
|
||||
ズヴョズニィ"
|
||||
パウラトゥク"
|
||||
プルードー湾"
|
||||
レゾリュート"
|
||||
伊尔库茨克州"
|
||||
图克托亚图克"
|
||||
圖克托亞圖克"
|
||||
库格鲁克图克"
|
||||
庫格魯克圖克"
|
||||
즈베즈드니이"
|
||||
Μις Σμίντα"
|
||||
Йоа-Гейвен"
|
||||
Йоа-Хейвен"
|
||||
Мис Барроу"
|
||||
Мис Шмідта"
|
||||
Мыс Шмидта"
|
||||
Пойнт-Гоуп"
|
||||
Пойнт-Хоуп"
|
||||
Прадхо-Бей"
|
||||
מיס שמידטה"
|
||||
פורט יוקון"
|
||||
جیجوی هیون"
|
||||
فورت يوكون"
|
||||
فورت یوکان"
|
||||
فورٹ یوکون"
|
||||
ميس شميدتا"
|
||||
پوائنٹ ہوپ"
|
||||
캠브리지 베이"
|
||||
Γουέινραϊτ"
|
||||
Κοτζέμπουε"
|
||||
Παουλατούκ"
|
||||
Тактояктук"
|
||||
Туктояктук"
|
||||
Улукхакток"
|
||||
גג'ואה האבן"
|
||||
טוקטויאטוק"
|
||||
קוגלולקטוק"
|
||||
اولاکاکچوک"
|
||||
تسيجيهتشيك"
|
||||
Northwest Territories"
|
||||
Γκόα Χέιβεν"
|
||||
Πόιντ Χόουπ"
|
||||
خليج برودهو"
|
||||
فورٹ مکفرسن"
|
||||
مئیس شمیدتا"
|
||||
ऐम्ब्लर"
|
||||
पॉलातुक"
|
||||
रेसोलूट"
|
||||
वेनराइट"
|
||||
सेलाविक"
|
||||
ওয়েনরিট"
|
||||
তালোয়াক"
|
||||
সেলাউইক"
|
||||
アットカサック"
|
||||
ウェインライト"
|
||||
クグルクトゥク"
|
||||
ポイントホープ"
|
||||
توكتوياكتوك"
|
||||
توکتویاکتوک"
|
||||
Προύντο Μπέι"
|
||||
Φορτ Γιούκον"
|
||||
פורט מקפרסון"
|
||||
خليج كامبردج"
|
||||
قرية ريزولوت"
|
||||
Chukchi Autonomous Okrug"
|
||||
United States of America"
|
||||
Τσιγκέχτσιτς"
|
||||
فورت مک فرسون"
|
||||
قرية غوا هافن"
|
||||
अत्कासुक"
|
||||
अलाकाकेट"
|
||||
इनर्मिनो"
|
||||
किवालिना"
|
||||
टालोयोआक"
|
||||
वाइज़मैन"
|
||||
কাকতোভিক"
|
||||
কিভালিনা"
|
||||
পাউলাটুক"
|
||||
রেজোলিউট"
|
||||
সিগেটছিক"
|
||||
হোলম্যান"
|
||||
ウルクハクトーク"
|
||||
ジョア・ヘイヴン"
|
||||
トサイイジチック"
|
||||
フォートユーコン"
|
||||
Кеймбридж-Бей"
|
||||
Кеймбрідж-Бей"
|
||||
فورت ماكفرسون"
|
||||
पॉइंट होप"
|
||||
Κουγκλουκτούκ"
|
||||
קיימברדיג' ביי"
|
||||
Κέιμπριτζ Μπέι"
|
||||
Φορτ ΜακΦέρσον"
|
||||
Форт-Макферсон"
|
||||
काक्तोविक"
|
||||
এনিউরমিনো"
|
||||
ケンブリッジ・ベイ"
|
||||
トゥクトヤクトゥク"
|
||||
Τουκτογιάκτουκ"
|
||||
ज़ोआ हेवेन"
|
||||
प्रुदहो बे"
|
||||
मिस श्मिता"
|
||||
পয়েন্ট হোপ"
|
||||
ফোর্ট ইউকন"
|
||||
उत्कीग्विक"
|
||||
कुग्लुकटूक"
|
||||
कोटज़ेबूये"
|
||||
অ্যাম্বলার"
|
||||
আল্লাকাকেত"
|
||||
উৎকিয়াভিক"
|
||||
ওয়াইজম্যান"
|
||||
কুগ্লুকটুক"!
|
||||
গজোয়া হাভেন"!
|
||||
প্রুদোহে বে"#
|
||||
!त्सीगेत्चिक"#
|
||||
!フォートマクファーソン"$
|
||||
"कैम्ब्रिज बे"$
|
||||
"फ़ोर्ट यूकॉन"$
|
||||
"কেম্ব্রিজ বে"$
|
||||
"মাইস শ্মিদতা"&
|
||||
$অ্যাটকোয়াসুক"&
|
||||
$জেভেজদিনিয়াজ"&
|
||||
$তুকতোইয়াকতুক")
|
||||
'ज़्वेज़्दनिज़")
|
||||
'तुक्तोयाक्तुक"0
|
||||
.फ़ोर्ट मैकफ़र्सन"0
|
||||
.ফোর্ট ম্যাকফারসন" | ||||