mirror of
https://github.com/felt/tippecanoe.git
synced 2026-10-02 08:25:40 +02:00
Merge main into the MLT branch
main moved each tool's option table to file scope so that usage.cpp can generate the usage message from it (#409), which conflicted with the tables the MLT options had been added to. The three MLT options move to the file-scope tables, under the same headings they were listed with before: "Setting or disabling tile size limits" in tippecanoe and tile-join, and "Output tile" in tippecanoe-overzoom. Also combine the new radix-sort-test with the MLT tests in the test target, add usage.o to the tools that gained MLT objects, keep both the MLT=0 and the new docs CI jobs, and regenerate the man page for the README changes, which the docs job now checks.
This commit is contained in:
Claude
@@ -30,3 +30,20 @@ jobs:
|
||||
- run: sudo apt-get install libsqlite3-dev
|
||||
- run: uname -a; make MLT=0
|
||||
- run: make MLT=0 test
|
||||
|
||||
docs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
# Pinned, because different go-md2man versions produce different roff
|
||||
# for the same input, which would make this check fail spuriously.
|
||||
- name: Install go-md2man
|
||||
run: go install github.com/cpuguy83/go-md2man/v2@v2.0.7
|
||||
- name: Regenerate the man page
|
||||
run: PATH="$PATH:$(go env GOPATH)/bin" make -B docs
|
||||
- name: Check that the man page is up to date with README.md
|
||||
run: |
|
||||
git diff --exit-code man/tippecanoe.1 || {
|
||||
echo "::error::man/tippecanoe.1 is out of date. Run 'make docs' and commit the result."
|
||||
exit 1
|
||||
}
|
||||
|
||||
@@ -38,6 +38,9 @@ tippecanoe-json-tool
|
||||
tippecanoe-overzoom
|
||||
unit
|
||||
|
||||
# Left behind if man page generation fails
|
||||
man/tippecanoe.1.tmp
|
||||
|
||||
# Tests
|
||||
tests/**/*.mbtiles
|
||||
tests/**/*.check
|
||||
|
||||
@@ -1,3 +1,71 @@
|
||||
# 2.81.0
|
||||
|
||||
* Add `--drop-by-attribute-as-needed=`*attribute* to drop the features with
|
||||
the lowest values of a numeric attribute from oversized tiles, and
|
||||
`--drop-by-attribute-order=desc` to drop the highest values instead.
|
||||
Features exactly at the threshold are kept rather than dropped. (#384, #385)
|
||||
* Add `--exclude-all-tile-geometries` to tile-join, to produce tiles that
|
||||
carry only attributes. (#382)
|
||||
* Generate each tool's usage message from the same option table that
|
||||
`getopt_long()` reads, so the hand-written lists in tile-join,
|
||||
tippecanoe-overzoom, tippecanoe-json-tool, tippecanoe-decode, and
|
||||
tippecanoe-enumerate can no longer fall behind the options actually
|
||||
accepted. Options previously reachable only by their short names are now
|
||||
listed. tippecanoe-overzoom reports a missing `-o` instead of passing a
|
||||
null pointer to `fopen()`, and tippecanoe prints its usage when run with
|
||||
no arguments. (#409)
|
||||
* Fix the radix sort used by `--prefer-radix-sort`. A bucket written out
|
||||
directly rather than through the merge was written one byte longer than
|
||||
its length prefix claimed, desynchronizing everything read from the
|
||||
geometry after it. Subdividing could also recurse forever once it ran out
|
||||
of files to split with, shifting by the full width of the index and
|
||||
writing past the end of the arrays of buckets. Radix-sorted output is now
|
||||
checked against the in-memory sort rather than against a stored copy. (#404)
|
||||
* Read FlatGeobuf integer and float properties as numbers. They were tagged
|
||||
with types that the rest of tippecanoe does not treat as numeric, so they
|
||||
were reported in tilestats as "mixed", with quoted values and no min or
|
||||
max, and warned when used as a feature ID. ULong properties are now also
|
||||
read as unsigned rather than signed. (#395)
|
||||
* Respect the `-t` temporary directory option in sorting operations, which
|
||||
previously always used the system temporary directory. (#368)
|
||||
* Keep `--generate-variable-depth-tile-pyramid` from silently dropping
|
||||
features whose explicit per-feature `minzoom` is deeper than the zoom at
|
||||
which their region becomes a leaf. Such a feature was excluded from the
|
||||
leaf tile while its children were never generated, so it appeared at no
|
||||
zoom at all. (#397, #399)
|
||||
* Drop a polygon hole that no remaining ring can parent, instead of failing
|
||||
the whole run. Degenerate input could abort tiling over a single
|
||||
unrepresentable sliver. (#401)
|
||||
* Clamp the feature extent to the `long long` range before converting it,
|
||||
at both ends. The previous `extent <= LLONG_MAX` guard was doubly wrong:
|
||||
`LLONG_MAX` is not representable as a double and rounds up, so an extent
|
||||
at the very top of the range overflowed the conversion and came out as the
|
||||
most negative value rather than the largest, and the guard admitted
|
||||
everything below `LLONG_MIN` as well, which overflowed the other way. Areas
|
||||
are signed, so holes that outweigh their rings can reach the low end. (#406)
|
||||
* Initialize the full width of the `mvt_value` numeric union, which left the
|
||||
bytes of the wider unused member indeterminate even though the implicit
|
||||
copy constructor copies the union as a whole. (#406)
|
||||
* Replace all variable-length arrays with `std::vector` and `std::string`,
|
||||
and build with `-Wvla`. VLAs are a compiler extension rather than standard
|
||||
C++, and clang warns about every one of them by default. (#406)
|
||||
* Remove the unused Dockerfiles, Travis configuration, and lambda
|
||||
directory. (#365)
|
||||
* Correct README statements that disagreed with the code. Among them, `-aD`
|
||||
and `-aS` were documented the wrong way round,
|
||||
`--limit-base-zoom-to-maximum-zoom` was given as `-Pb` rather than `-pb`,
|
||||
and the dot-dropping description had both the fraction and the zoom
|
||||
direction backwards: tippecanoe keeps 1/2.5 of the dots at zooms below the
|
||||
base zoom, rather than dropping that share above it. (#410)
|
||||
* Generate `man/tippecanoe.1` with go-md2man rather than md2man-roff, which
|
||||
is packaged only as a Ruby gem and so had let the page drift out of date.
|
||||
The page now has a proper header and a NAME section, so `man -k` and
|
||||
`whatis` can find it, and no longer silently drops or mangles text the old
|
||||
converter mishandled. CI checks it against README.md. (#408)
|
||||
* Documentation fixes: correct three misspellings in the README and man
|
||||
page, repair the dead All Streets link, and tag more README code blocks
|
||||
with their language. (#375, #391, #400)
|
||||
|
||||
# 2.80.0
|
||||
|
||||
* Remove undocumented command-line options
|
||||
|
||||
@@ -35,7 +35,7 @@ CXX := $(CXX)
|
||||
CFLAGS := $(CFLAGS) -fPIE -DBUILD_INFO=$(BUILD_INFO)
|
||||
CXXFLAGS := $(CXXFLAGS) -std=c++17 -fPIE -DBUILD_INFO=$(BUILD_INFO) $(MLT_FLAGS)
|
||||
LDFLAGS := $(LDFLAGS)
|
||||
WARNING_FLAGS := -Wall -Wshadow -Wsign-compare -Wextra -Wunreachable-code -Wuninitialized -Wshadow
|
||||
WARNING_FLAGS := -Wall -Wshadow -Wsign-compare -Wextra -Wunreachable-code -Wuninitialized -Wshadow -Wvla
|
||||
RELEASE_FLAGS := -O3 -DNDEBUG
|
||||
DEBUG_FLAGS := -O0 -DDEBUG -fno-inline-functions -fno-omit-frame-pointer
|
||||
|
||||
@@ -72,8 +72,44 @@ install: tippecanoe tippecanoe-enumerate tippecanoe-decode tile-join tippecanoe-
|
||||
uninstall:
|
||||
rm $(PREFIX)/bin/tippecanoe $(PREFIX)/bin/tippecanoe-enumerate $(PREFIX)/bin/tippecanoe-decode $(PREFIX)/bin/tile-join $(MANDIR)/tippecanoe.1 $(PREFIX)/bin/tippecanoe-json-tool
|
||||
|
||||
# The man page is generated from README.md by go-md2man, which is packaged for
|
||||
# most systems (`brew install go-md2man`, `apt-get install go-md2man`) or can be
|
||||
# built with `go install github.com/cpuguy83/go-md2man/v2@v2.0.7`. CI checks that
|
||||
# the committed man page matches the README, so you don't have to regenerate it
|
||||
# yourself if you don't have go-md2man installed.
|
||||
#
|
||||
# README.md has no .TH or NAME section of its own, since neither would make sense
|
||||
# on GitHub, so prepend them here. go-md2man reads the leading "%" line as the
|
||||
# man page's title, section, date, and source. The version deliberately doesn't
|
||||
# appear there: it would make this page a build product of version.hpp, so every
|
||||
# release would have to regenerate it just to rewrite that one line, and the docs
|
||||
# CI job would fail on any version bump that forgot to.
|
||||
#
|
||||
# Two fixups on the way out:
|
||||
#
|
||||
# - README.md's own title heading becomes the second .SH, right below the NAME
|
||||
# section added above, which reads as a stray "tippecanoe" section. Rename it
|
||||
# to DESCRIPTION, where the text under it belongs anyway.
|
||||
# - go-md2man separates paragraphs with a blank line in addition to the .PP
|
||||
# macro, and a blank line is itself a break in roff, so the two together
|
||||
# double-space the whole page. Drop them, except within .EX and .TS blocks,
|
||||
# where a blank line is part of the example or table rather than spacing.
|
||||
man/tippecanoe.1: README.md
|
||||
md2man-roff README.md > man/tippecanoe.1
|
||||
{ \
|
||||
echo '% TIPPECANOE 1 "" "tippecanoe"'; \
|
||||
echo; \
|
||||
echo '# NAME'; \
|
||||
echo; \
|
||||
echo 'tippecanoe - build vector tilesets from GeoJSON, FlatGeobuf, or CSV features'; \
|
||||
echo; \
|
||||
cat README.md; \
|
||||
} | go-md2man \
|
||||
| awk ' \
|
||||
/^\.SH / && ++sh == 2 { print ".SH DESCRIPTION"; next } \
|
||||
/^\.(EX|TS)$$/ { lit = 1 } \
|
||||
/^\.(EE|TE)$$/ { lit = 0 } \
|
||||
lit || !/^$$/ \
|
||||
' > $@.tmp && mv $@.tmp $@
|
||||
|
||||
PG=
|
||||
|
||||
@@ -98,25 +134,25 @@ ifneq ($(MLT),0)
|
||||
$(MLT_LIBS): $(MLT_BUILD_STAMP)
|
||||
endif
|
||||
|
||||
tippecanoe: geojson.o jsonpull/jsonpull.o tile.o pool.o mbtiles.o geometry.o projection.o memfile.o mvt.o mlt.o mlt_decode.o serial.o main.o platform.o text.o dirtiles.o pmtiles_file.o plugin.o read_json.o write_json.o geobuf.o flatgeobuf.o evaluator.o geocsv.o csv.o geojson-loop.o json_logger.o visvalingam.o compression.o clip.o sort.o attribute.o thread.o shared_borders.o clipper2/src/clipper.engine.o $(MLT_LIBS)
|
||||
tippecanoe: geojson.o jsonpull/jsonpull.o tile.o pool.o mbtiles.o geometry.o projection.o memfile.o mvt.o mlt.o mlt_decode.o serial.o main.o platform.o text.o dirtiles.o pmtiles_file.o plugin.o read_json.o write_json.o geobuf.o flatgeobuf.o evaluator.o geocsv.o csv.o geojson-loop.o json_logger.o visvalingam.o compression.o clip.o sort.o attribute.o thread.o shared_borders.o usage.o clipper2/src/clipper.engine.o $(MLT_LIBS)
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
tippecanoe-enumerate: enumerate.o
|
||||
tippecanoe-enumerate: enumerate.o usage.o
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lsqlite3
|
||||
|
||||
tippecanoe-decode: decode.o projection.o mvt.o mlt_decode.o write_json.o text.o jsonpull/jsonpull.o dirtiles.o pmtiles_file.o $(MLT_DECODE_LIBS)
|
||||
tippecanoe-decode: decode.o projection.o mvt.o mlt_decode.o write_json.o text.o jsonpull/jsonpull.o dirtiles.o pmtiles_file.o usage.o $(MLT_DECODE_LIBS)
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3
|
||||
|
||||
tile-join: tile-join.o platform.o projection.o mbtiles.o mvt.o mlt.o mlt_decode.o memfile.o dirtiles.o jsonpull/jsonpull.o text.o evaluator.o csv.o write_json.o pmtiles_file.o clip.o attribute.o thread.o read_json.o clipper2/src/clipper.engine.o $(MLT_LIBS)
|
||||
tile-join: tile-join.o platform.o projection.o mbtiles.o mvt.o mlt.o mlt_decode.o memfile.o dirtiles.o jsonpull/jsonpull.o text.o evaluator.o csv.o write_json.o pmtiles_file.o clip.o attribute.o thread.o read_json.o usage.o clipper2/src/clipper.engine.o $(MLT_LIBS)
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
tippecanoe-json-tool: jsontool.o jsonpull/jsonpull.o csv.o text.o geojson-loop.o
|
||||
tippecanoe-json-tool: jsontool.o jsonpull/jsonpull.o csv.o text.o geojson-loop.o usage.o
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
unit: unit.o text.o sort.o mvt.o mlt.o mlt_decode.o projection.o clip.o attribute.o jsonpull/jsonpull.o evaluator.o read_json.o clipper2/src/clipper.engine.o $(MLT_LIBS)
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
tippecanoe-overzoom: overzoom.o mvt.o mlt.o mlt_decode.o clip.o evaluator.o jsonpull/jsonpull.o text.o attribute.o read_json.o projection.o read_json.o clipper2/src/clipper.engine.o $(MLT_LIBS)
|
||||
tippecanoe-overzoom: overzoom.o mvt.o mlt.o mlt_decode.o clip.o evaluator.o jsonpull/jsonpull.o text.o attribute.o read_json.o projection.o read_json.o usage.o clipper2/src/clipper.engine.o $(MLT_LIBS)
|
||||
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
|
||||
|
||||
-include $(wildcard *.d)
|
||||
@@ -139,7 +175,7 @@ indent:
|
||||
TESTS = $(wildcard tests/*/out/*.json)
|
||||
SPACE = $(NULL) $(NULL)
|
||||
|
||||
test: tippecanoe tippecanoe-decode $(addsuffix .check,$(TESTS)) raw-tiles-test parallel-test pbf-test join-test enumerate-test decode-test join-filter-test unit json-tool-test allow-existing-test csv-test layer-json-test pmtiles-test decode-pmtiles-test overzoom-test flatgeobuf-test $(MLT_TESTS)
|
||||
test: tippecanoe tippecanoe-decode $(addsuffix .check,$(TESTS)) raw-tiles-test parallel-test radix-sort-test pbf-test join-test enumerate-test decode-test join-filter-test unit json-tool-test allow-existing-test csv-test layer-json-test pmtiles-test decode-pmtiles-test overzoom-test flatgeobuf-test $(MLT_TESTS)
|
||||
./unit
|
||||
|
||||
suffixes = json json.gz
|
||||
@@ -174,7 +210,7 @@ nogeobuf = tests/overflow/out/-z0.json $(wildcard tests/stringid/out/*.json)
|
||||
geobuf-test: tippecanoe-json-tool $(addsuffix .checkbuf,$(filter-out $(nogeobuf),$(TESTS)))
|
||||
|
||||
# For quicker address sanitizer build, hope that regular JSON parsing is tested enough by parallel and join tests
|
||||
fewer-tests: tippecanoe tippecanoe-decode geobuf-test raw-tiles-test parallel-test pbf-test join-test enumerate-test decode-test join-filter-test unit
|
||||
fewer-tests: tippecanoe tippecanoe-decode geobuf-test raw-tiles-test parallel-test radix-sort-test pbf-test join-test enumerate-test decode-test join-filter-test unit
|
||||
|
||||
# XXX Use proper makefile rules instead of a for loop
|
||||
%.json.checkbuf:
|
||||
@@ -185,6 +221,42 @@ fewer-tests: tippecanoe tippecanoe-decode geobuf-test raw-tiles-test parallel-te
|
||||
cmp $@.out $(patsubst %.checkbuf,%,$@)
|
||||
rm $@.out $@.mbtiles
|
||||
|
||||
# The result of the sort must not depend on how the sort was performed, so instead of
|
||||
# checking the sorted output against an expected copy of it, check that sorting by radix
|
||||
# produces the same tiles as sorting in memory. --prefer-radix-sort lowers the memory
|
||||
# limit to 8K, which radix() then halves again, so the radix subdivision has to recurse
|
||||
# until each bucket is under 4K. How deeply that recurses depends on how many files the
|
||||
# machine will let us open at once, but the sorted result is the same either way, so
|
||||
# this comparison doesn't depend on that.
|
||||
#
|
||||
# What sends the sort down its rarely-taken paths is the shape of the input rather than
|
||||
# the size of it: a feature whose geometry alone is bigger than the memory limit is
|
||||
# sorted as a bucket of its own, and features that share a long run of leading index
|
||||
# bits have to be subdivided until there are no bits left. The first two inputs are
|
||||
# each one of those on purpose -- several separated features too big to sort in memory,
|
||||
# and many features at one location -- and the rest are for breadth.
|
||||
radix-sort-test: tippecanoe tippecanoe-decode
|
||||
mkdir -p tests/radix-sort
|
||||
perl -e 'for ($$f = 0; $$f < 8; $$f++) { print "{ \"type\": \"Feature\", \"properties\": { \"f\": $$f }, \"geometry\": { \"type\": \"LineString\", \"coordinates\": ["; for ($$i = 0; $$i < 2000; $$i++) { print "," unless $$i == 0; printf "[%f,%f]", $$f * 40 - 175 + $$i * 0.001, $$i % 2 * 0.5 - 20; } print "] } }\n"; }' > tests/radix-sort/bigfeatures.json
|
||||
perl -e 'for ($$i = 0; $$i < 500; $$i++) { print "{ \"type\": \"Feature\", \"properties\": { \"i\": $$i }, \"geometry\": { \"type\": \"Point\", \"coordinates\": [ 17, 42 ] } }\n"; }' > tests/radix-sort/onelocation.json
|
||||
$(MAKE) radix-sort-compare RADIXIN="tests/radix-sort/bigfeatures.json" RADIXARGS="-z4"
|
||||
$(MAKE) radix-sort-compare RADIXIN="tests/radix-sort/onelocation.json" RADIXARGS="-z4"
|
||||
$(MAKE) radix-sort-compare RADIXIN="tests/feature-filter/in.json" RADIXARGS="-z4"
|
||||
$(MAKE) radix-sort-compare RADIXIN="tests/ne_110m_ocean/in.json" RADIXARGS="-z4"
|
||||
$(MAKE) radix-sort-compare RADIXIN="tests/border/in.json" RADIXARGS="-z4"
|
||||
$(MAKE) radix-sort-compare RADIXIN="tests/loop/in.json" RADIXARGS="-z4"
|
||||
$(MAKE) radix-sort-compare RADIXIN="tests/tl_2022_11_tract/in.json.gz" RADIXARGS="-z4"
|
||||
$(MAKE) radix-sort-compare RADIXIN="tests/epsg-3857/in.json" RADIXARGS="-z4 -sEPSG:3857"
|
||||
rm -r tests/radix-sort
|
||||
|
||||
radix-sort-compare:
|
||||
./tippecanoe -q -f $(RADIXARGS) -o tests/radix-sort/memory.mbtiles $(RADIXIN)
|
||||
./tippecanoe -q -f $(RADIXARGS) -aR -o tests/radix-sort/radix.mbtiles $(RADIXIN)
|
||||
./tippecanoe-decode -x generator -x generator_options -x name -x description tests/radix-sort/memory.mbtiles > tests/radix-sort/memory.json
|
||||
./tippecanoe-decode -x generator -x generator_options -x name -x description tests/radix-sort/radix.mbtiles > tests/radix-sort/radix.json
|
||||
cmp tests/radix-sort/memory.json tests/radix-sort/radix.json
|
||||
rm tests/radix-sort/memory.mbtiles tests/radix-sort/radix.mbtiles tests/radix-sort/memory.json tests/radix-sort/radix.json
|
||||
|
||||
parallel-test: $(eval SHELL:=$(ADVSHELL))
|
||||
mkdir -p tests/parallel
|
||||
perl -e 'for ($$i = 0; $$i < 20; $$i++) { $$lon = rand(360) - 180; $$lat = rand(180) - 90; $$k = rand(1); $$v = rand(1); print "{ \"type\": \"Feature\", \"properties\": { \"yes\": \"no\", \"who\": 1, \"$$k\": \"$$v\" }, \"geometry\": { \"type\": \"Point\", \"coordinates\": [ $$lon, $$lat ] } }\n"; }' > tests/parallel/in1.json
|
||||
|
||||
@@ -64,7 +64,7 @@ Usage
|
||||
-----
|
||||
|
||||
```
|
||||
$ tippecanoe -o file.mbtiles [options] [file.json file.json.gz file.fgb ...]
|
||||
$ tippecanoe -o file.mbtiles [options] [file.json file.json.gz file.fgb file.csv ...]
|
||||
```
|
||||
|
||||
If no files are specified, it reads GeoJSON from the standard input.
|
||||
@@ -297,6 +297,11 @@ If your features have a lot of attributes, use `-y` to keep only the ones you re
|
||||
|
||||
If your input is formatted as newline-delimited GeoJSON, use `-P` to make input parsing a lot faster.
|
||||
|
||||
Many of the options below have a short form beginning with `-a` or `-p`. These are the
|
||||
`-a`_letters_ (`--additional=`_letters_) and `-p`_letters_ (`--prevent=`_letters_) options, and each
|
||||
accepts several letters at once, so for example `-ansd` is the same as `-an -as -ad`
|
||||
and `-pkC` is the same as `-pk -pC`.
|
||||
|
||||
### Output tileset
|
||||
|
||||
* `-o` _file_`.mbtiles`, _file_`.pmtiles` or `--output=`_file_`.mbtiles`: Name the output file.
|
||||
@@ -307,9 +312,9 @@ If your input is formatted as newline-delimited GeoJSON, use `-P` to make input
|
||||
|
||||
### Tileset description and attribution
|
||||
|
||||
* `-n` _name_ or `--name=`_name_: Human-readable name for the tileset (default file.json)
|
||||
* `-n` _name_ or `--name=`_name_: Human-readable name for the tileset (default: the name of the output file or directory)
|
||||
* `-A` _text_ or `--attribution=`_text_: Attribution (HTML) to be shown with maps that use data from this tileset.
|
||||
* `-N` _description_ or `--description=`_description_: Description for the tileset (default file.mbtiles)
|
||||
* `-N` _description_ or `--description=`_description_: Description for the tileset (default: the name of the output file or directory)
|
||||
|
||||
### Input files and layer names
|
||||
|
||||
@@ -326,7 +331,7 @@ If your input is formatted as newline-delimited GeoJSON, use `-P` to make input
|
||||
tippecanoe -z5 -o world.mbtiles -L'{"file":"ne_10m_admin_0_countries.json", "layer":"countries", "description":"Natural Earth countries"}'
|
||||
```
|
||||
|
||||
CSV input files currently support only Point geometries, from columns named `latitude`, `longitude`, `lat`, `lon`, `long`, `lng`, `x`, or `y`.
|
||||
CSV input files currently support only Point geometries, from columns named `lat`, `lon`, `long`, `lng`, `x`, or `y`, or from any column whose name contains `latitude` or `longitude`. Column names are matched without regard to case.
|
||||
|
||||
### Parallel processing of input
|
||||
|
||||
@@ -421,8 +426,9 @@ be reduced to the maximum that can be used with the specified _maxzoom_.
|
||||
* `-Y`_attribute_`:`_description_ or `--attribute-description=`_attribute_`:`_description_: Set the `description` for the specified attribute in the tileset metadata to _description_ instead of the usual `String`, `Number`, or `Boolean`.
|
||||
* `-E`_attribute_`:`_operation_ or `--accumulate-attribute=`_attribute_`:`_operation_: Preserve the named _attribute_ from features
|
||||
that are dropped, coalesced-as-needed, or clustered. The _operation_ may be
|
||||
`sum`, `product`, `mean`, `max`, `min`, `concat`, or `comma`
|
||||
`sum`, `product`, `mean`, `max`, `min`, `concat`, `comma`, or `count`
|
||||
to specify how the named _attribute_ is accumulated onto the attribute of the same name in a feature that does survive.
|
||||
(The `count` operation replaces the attribute with the number of features, including the survivor, that carried it.)
|
||||
The attributes and operations may also be specified as JSON keys and values: `--accumulate-attribute='{"attr": "operation", "attr2": "operation2"}'`.
|
||||
* `--set-attribute` _attribute_`:`_value_: Set the value of the specified _attribute_ in each feature to the specified _value_. This is mostly useful to give an attribute in each feature an initial value for `--accumulate-attribute`.
|
||||
The attributes and values may also be specified as JSON keys and values: `--set-attribute='{"attr": value, "attr2": value}'`.
|
||||
@@ -472,16 +478,17 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
|
||||
If you use `-Bg`, it will guess a zoom level that will keep at most 50,000 features in the densest tile.
|
||||
You can also specify a marker-width with `-Bg`*width* to allow fewer features in the densest tile to
|
||||
compensate for the larger marker, or `-Bf`*number* to allow at most *number* features in the densest tile.
|
||||
* `--retain-points-multiplier=`_multiple_: Retain the specified multiple of points instead of just the number of points that would ordinarily be retained by the drop rate. These can be thinned out later with the `-m` option to `tippecanoe-overzoom`. The start of each cluster is marked in the feature sequence by the `tippecanoe:retain_points_multiplier_first` attribute. The `--tile-size-limit` will also be extended at low zoom levels to allow for the multiplied features.
|
||||
* `--retain-points-multiplier=`_multiple_: Retain the specified multiple of points instead of just the number of points that would ordinarily be retained by the drop rate. These can be thinned out later with the `-m` option to `tippecanoe-overzoom`. The start of each cluster is marked in the feature sequence by the `tippecanoe:retain_points_multiplier_first` attribute. The maximum tile size (`--maximum-tile-bytes`) will also be extended at low zoom levels to allow for the multiplied features.
|
||||
* `--drop-denser=`_percentage_: When dropping dots at zoom levels below the base zoom, give the specified _percentage_
|
||||
preference to retaining points in sparse areas and dropping points in dense areas.
|
||||
* `--limit-base-zoom-to-maximum-zoom` or `-Pb`: Limit the guessed base zoom not to exceed the maxzoom, even if this would put more than the requested number of features in a base zoom tile.
|
||||
* `--limit-base-zoom-to-maximum-zoom` or `-pb`: Limit the guessed base zoom not to exceed the maxzoom, even if this would put more than the requested number of features in a base zoom tile.
|
||||
* `-al` or `--drop-lines`: Let "dot" dropping at lower zooms apply to lines too
|
||||
* `-ap` or `--drop-polygons`: Let "dot" dropping at lower zooms apply to polygons too
|
||||
* `-K` _distance_ or `--cluster-distance=`_distance_: Cluster points (as with `--cluster-densest-as-needed`, but without the experimental discovery process) that are approximately within _distance_ of each other. The units are tile coordinates within a nominally 256-pixel tile, so the maximum value of 255 allows only one feature per tile. Values around 10 are probably appropriate for typical marker sizes. See `--cluster-densest-as-needed` below for behavior.
|
||||
* `-k` _zoom_ or `--cluster-maxzoom=`_zoom_: Max zoom on which to cluster points if clustering is enabled.
|
||||
* `-kg` or `--cluster-maxzoom=g`: Set `--cluster-maxzoom=` to `maxzoom - 1` so that all features are visible at the maximum zoom level.
|
||||
* `--preserve-point-density-threshold=`_level_: At the low zoom levels, do not reduce point density below the specified _level_, even if the specified drop rate would normally call for it, so that low-density areas of the map do not appear blank. The unit is the distance between preserved points, as a fraction of the size of a tile. Values of 32 or 64 are probably appropriate for typical marker sizes.
|
||||
* `--preserve-multiplier-density-threshold=`_level_: As with `--preserve-point-density-threshold`, but for the additional features retained by `--retain-points-multiplier`: features that would otherwise be dropped are instead added to the multiplier cluster if they are farther than the specified _level_ from the previous retained feature, so that sparse areas still have features available to be thinned to. The unit is the same as for `--preserve-point-density-threshold`.
|
||||
|
||||
### Dropping a fraction of features to keep under tile size limits
|
||||
|
||||
@@ -490,10 +497,11 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
|
||||
* `-an` or `--drop-smallest-as-needed`: Dynamically drop the smallest features (physically smallest: the shortest lines or the smallest polygons) from each zoom level to keep large tiles under the 500K size limit.
|
||||
* `--drop-by-attribute-as-needed=`_attribute_: Dynamically drop features with the lowest values of the specified numeric _attribute_ from each zoom level to keep large tiles under the 500K size limit. Use `--drop-by-attribute-order=desc` to instead drop features with the highest values.
|
||||
* `-aN` or `--coalesce-smallest-as-needed`: Dynamically combine the smallest features (physically smallest: the shortest lines or the smallest polygons or the densest points) from each zoom level into other nearby features to keep large tiles under the 500K size limit. This option will probably not help very much with LineStrings. It is mostly intended for polygons, to maintain the full original area covered by polygons while still reducing the feature count somehow. The attributes of the small polygons are *not* preserved into the combined features (except through `--accumulate-attribute`), only their geometry. Furthermore, the polygons to which nested polygons are coalesced may not necessarily be the immediately enclosing features.
|
||||
* `-aD` or `--coalesce-densest-as-needed`: Dynamically combine the densest features from each zoom level into other nearby features to keep large tiles under the 500K size limit. (Again, mostly useful for polygons.)
|
||||
* `-aS` or `--coalesce-fraction-as-needed`: Dynamically combine a fraction of features from each zoom level into other nearby features to keep large tiles under the 500K size limit. (Again, mostly useful for polygons.)
|
||||
* `-aS` or `--coalesce-densest-as-needed`: Dynamically combine the densest features from each zoom level into other nearby features to keep large tiles under the 500K size limit. (Again, mostly useful for polygons.)
|
||||
* `-aD` or `--coalesce-fraction-as-needed`: Dynamically combine a fraction of features from each zoom level into other nearby features to keep large tiles under the 500K size limit. (Again, mostly useful for polygons.)
|
||||
* `-pd` or `--force-feature-limit`: Dynamically drop some fraction of features from large tiles to keep them under the 500K size limit. It will probably look ugly at the tile boundaries. (This is like `-ad` but applies to each tile individually, not to the entire zoom level.) You probably don't want to use this.
|
||||
* `-aC` or `--cluster-densest-as-needed`: If a tile is too large, try to reduce its size by increasing the minimum spacing between features, and leaving one placeholder feature from each group. The remaining feature will be given a `"clustered": true` attribute to indicate that it represents a cluster, a `"point_count"` attribute to indicate the number of features that were clustered into it, and a `"sqrt_point_count"` attribute to indicate the relative width of a feature to represent the cluster. If the features being clustered are points, the representative feature will be located at the average of the original points' locations; otherwise, one of the original features will be left as the representative.
|
||||
* `-aC` or `--cluster-densest-as-needed`: If a tile is too large, try to reduce its size by increasing the minimum spacing between features, and leaving one placeholder feature from each group. The remaining feature will be given a `"clustered": true` attribute to indicate that it represents a cluster, a `"point_count"` attribute to indicate the number of features that were clustered into it, a `"point_count_abbreviated"` attribute containing that count abbreviated for display (for example `1.2k` or `15k`), and a `"sqrt_point_count"` attribute to indicate the relative width of a feature to represent the cluster. If the features being clustered are points, the representative feature will be located at the average of the original points' locations (unless you use `--keep-point-cluster-position`); otherwise, one of the original features will be left as the representative.
|
||||
* `-aa` or `--keep-point-cluster-position`: When clustering points, leave the representative feature at the location of the first point of the cluster instead of moving it to the average of the clustered points' locations.
|
||||
|
||||
### Dropping tightly overlapping features
|
||||
|
||||
@@ -515,7 +523,7 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
|
||||
|
||||
### Attempts to improve shared polygon boundaries
|
||||
|
||||
* `-ab` or `--detect-shared-borders`: DEPRECATED. In the manner of [TopoJSON](https://github.com/mbostock/topojson/wiki/Introduction), detect borders that are shared between multiple polygons and simplify them identically in each polygon. This takes more time and memory than considering each polygon individually. Use `no-simplification-of-shared-nodes` instead, which is faster and more correct.
|
||||
* `-ab` or `--detect-shared-borders`: DEPRECATED. In the manner of [TopoJSON](https://github.com/mbostock/topojson/wiki/Introduction), detect borders that are shared between multiple polygons and simplify them identically in each polygon. This takes more time and memory than considering each polygon individually. Use `--no-simplification-of-shared-nodes` instead, which is faster and more correct.
|
||||
* `-aL` or `--grid-low-zooms`: At all zoom levels below _maxzoom_, snap all lines and polygons to a stairstep grid instead of allowing diagonals. You will also want to specify a tile resolution, probably `-D8`. This option provides a way to display continuous parcel, gridded, or binned data at low zooms without overwhelming the tiles with tiny polygons, since features will either get stretched out to the grid unit or lost entirely, depending on how they happened to be aligned in the original data. You probably don't want to use this.
|
||||
|
||||
### Controlling clipping to tile boundaries
|
||||
@@ -577,9 +585,16 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
|
||||
* `-q` or `--quiet`: Work quietly instead of reporting progress or warning messages
|
||||
* `-Q` or `--no-progress-indicator`: Don't report progress, but still give warnings
|
||||
* `-U` _seconds_ or `--progress-interval=`_seconds_: Don't report progress more often than the specified number of _seconds_.
|
||||
* `-u` or `--json-progress`: like `-quiet` but logs progress as a JSON object. Use in combination with `-U`.
|
||||
* `-u` or `--json-progress`: like `--quiet` but logs progress as a JSON object. Use in combination with `-U`.
|
||||
|
||||
### Version
|
||||
|
||||
* `-v` or `--version`: Report Tippecanoe's version number
|
||||
|
||||
### Help
|
||||
|
||||
* `-H` or `--help`: List the available options and exit
|
||||
|
||||
### Filters
|
||||
|
||||
* `-C` _command_ or `--prefilter=`_command_: Specify a shell filter command to be run at the start of assembling each tile
|
||||
@@ -683,11 +698,11 @@ Geometric simplifications
|
||||
At every zoom level, line and polygon features are subjected to Douglas-Peucker
|
||||
simplification to the resolution of the tile.
|
||||
|
||||
For point features, it drops 1/2.5 of the dots for each zoom level above the
|
||||
For point features, it keeps only 1/2.5 of the dots for each zoom level below the
|
||||
point base zoom (which is normally the same as the `-z` max zoom, but can be
|
||||
a different zoom specified with `-B` if you have precise but sparse data).
|
||||
I don't know why 2.5 is the appropriate number, but the densities of many different
|
||||
data sets fall off at about this same rate. You can use -r to specify a different rate.
|
||||
data sets fall off at about this same rate. You can use `-r` to specify a different rate.
|
||||
|
||||
You can use the gamma option to thin out especially dense clusters of points.
|
||||
For any area where dots are closer than one pixel together (at whatever zoom level),
|
||||
@@ -713,8 +728,13 @@ lower resolutions before failing if it still doesn't fit.
|
||||
Development
|
||||
-----------
|
||||
|
||||
Requires sqlite3 and zlib (should already be installed on MacOS). Rebuilding the manpage
|
||||
uses md2man (`gem install md2man`).
|
||||
Requires sqlite3 and zlib (should already be installed on MacOS).
|
||||
|
||||
The manpage is generated from this README by `make docs`, which uses
|
||||
[go-md2man](https://github.com/cpuguy83/go-md2man) (`brew install go-md2man` or
|
||||
`apt-get install go-md2man`). You don't have to run it yourself: CI regenerates the
|
||||
manpage and fails if the committed copy doesn't match, so it will tell you if an
|
||||
edit here needs `make docs` run against it.
|
||||
|
||||
Linux:
|
||||
|
||||
@@ -728,15 +748,16 @@ and perhaps
|
||||
|
||||
make install
|
||||
|
||||
Tippecanoe now requires features from the 2011 C++ standard. If your compiler is older than
|
||||
that, you will need to install a newer one. On MacOS, updating to the latest XCode should
|
||||
get you a new enough version of `clang++`. On Linux, you should be able to upgrade `g++` with
|
||||
Tippecanoe requires features from the 2017 C++ standard (it is built with `-std=c++17`).
|
||||
If your compiler is older than that, you will need to install a newer one. On MacOS, updating
|
||||
to the latest XCode should get you a new enough version of `clang++`. On Linux, you should be
|
||||
able to upgrade `g++` with
|
||||
|
||||
```sh
|
||||
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
|
||||
sudo apt-get update -y
|
||||
sudo apt-get install -y g++-5
|
||||
export CXX=g++-5
|
||||
sudo apt-get install -y g++-9
|
||||
export CXX=g++-9
|
||||
```
|
||||
|
||||
Docker Image
|
||||
@@ -793,11 +814,14 @@ The options are:
|
||||
* `-o` *out.mbtiles*, *out.pmtiles* or `--output=`*out.mbtiles*: Write the new tiles to the specified .mbtiles file.
|
||||
* `-e` *directory* or `--output-to-directory=`*directory*: Write the new tiles to the specified directory instead of to an mbtiles file.
|
||||
* `-f` or `--force`: Remove *out.mbtiles* if it already exists.
|
||||
* `-r` or `--read-from`: list of input mbtiles to read from.
|
||||
|
||||
### Input tilesets
|
||||
|
||||
* `-r` *file* or `--read-from=`*file*: Read the list of input tilesets from the named *file*, one filename per line, instead of naming them all on the command line.
|
||||
|
||||
### Overzooming
|
||||
|
||||
* `--overzoom`: If one of the source tilesets has a larger maxzoom than the others, scale up tiles from the tilesets with the lower maxzooms so they will all have the same maxzoom in the output tileset.
|
||||
* `-O` or `--overzoom`: If one of the source tilesets has a larger maxzoom than the others, scale up tiles from the tilesets with the lower maxzooms so they will all have the same maxzoom in the output tileset.
|
||||
* `--buffer=`_pixels_ or `-b` _pixels_: Set the size of the tile buffer in the overzoomed tiles.
|
||||
|
||||
### Tileset description and attribution
|
||||
@@ -826,6 +850,8 @@ The options are:
|
||||
* `-x` *key* or `--exclude=`*key*: Remove attributes named *key* from the output. You can use this to remove the field you are matching against if you no longer need it after joining, or to remove any other attributes you don't want. You can use multiple `-x` options to remove multiple attributes.
|
||||
* `-X` or `--exclude-all`: Remove all attributes from the output.
|
||||
* `-y` *key* or `--include=`*key*: Remove all attributes except for those named *key* from the output. You can use multiple `-y` options to retain multiple attributes.
|
||||
* `--exclude-all-tile-attributes`: Remove the attributes that were already present in the source tiles, keeping only any that are joined from a CSV with `-c`.
|
||||
* `--exclude-all-tile-geometries`: Copy the features' attributes but not their geometries, so the output tiles describe the features without locating them.
|
||||
* `-i` or `--if-matched`: Only include features that matched the CSV.
|
||||
* `-j` *filter* or `--feature-filter`=*filter*: Check features against a per-layer filter (as defined in the [Mapbox GL Style Specification](https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter)) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer `"*"` apply to all layers.
|
||||
* `-J` *filter-file* or `--feature-filter-file`=*filter-file*: Like `-j`, but read the filter from a file.
|
||||
@@ -843,6 +869,10 @@ The options are:
|
||||
* `--tile-stats-sample-values-limit=`*count*: Calculate `tilestats` attribute statistics based on *count* values instead of the default 1000.
|
||||
* `--tile-stats-values-limit=`*count*: Report *count* unique attribute values in `tilestats` instead of the default 100.
|
||||
|
||||
### Progress indicator
|
||||
|
||||
* `-q` or `--quiet`: Work quietly instead of reporting progress
|
||||
|
||||
Because tile-join just copies the geometries to the new .mbtiles without processing them
|
||||
(except to rescale the extents if necessary),
|
||||
it doesn't have any of tippecanoe's recourses if the new tiles are bigger than the 500K tile limit.
|
||||
@@ -937,8 +967,10 @@ resolutions.
|
||||
* `-c` or `--tag-layer-and-zoom`: Include each feature's layer and zoom level as part of its `tippecanoe` object rather than as a FeatureCollection wrapper
|
||||
* `-S` or `--stats`: Just report statistics about each tile's size and the number of features in it, as a JSON structure.
|
||||
* `-f` or `--force`: Decode tiles even if polygon ring order or closure problems are detected
|
||||
* `-I` or `--integer`: Report coordinates in integer tile coordinates
|
||||
* `-F` or `--fraction`: Report coordinates as a fraction of the tile extent
|
||||
* `-I` or `--integer-coordinates`: Report coordinates in integer tile coordinates
|
||||
* `-F` or `--fractional-coordinates`: Report coordinates as a fraction of the tile extent
|
||||
* `-y` _attribute_ or `--include=`*attribute*: Include only the named attributes in the decoded features, excluding all those not explicitly named. (Multiple `-y` options can be specified.)
|
||||
* `-x` _name_ or `--exclude-metadata-row=`*name*: Omit the named row from the tileset metadata in the output. (Multiple `-x` options can be specified.)
|
||||
|
||||
tippecanoe-json-tool
|
||||
====================
|
||||
@@ -1029,13 +1061,22 @@ is detected from the tile data. The output tile is a Mapbox Vector Tile unless
|
||||
|
||||
### Options
|
||||
|
||||
* `-b` *buffer*: Set the tile buffer in the output tile (default 5)
|
||||
* `-d` *detail*: Set the detail of the output tile (default 12)
|
||||
* `-y` *attribute*: Retain the specified *attribute* in the output features. All attributes that are not named in a `-y` option will be removed.
|
||||
* `-j` *filter*: Filter features using the same expression syntax as in tippecanoe.
|
||||
* `-m`: If a tile was created with the `--retain-points-multiplier` option, thin the tile back down to its normal feature count during overzooming. The first feature from each cluster will be retained, unless `-j` is used to specify a filter, in which case the first matching filter from each cluster will be retained instead.
|
||||
* `-o` *file* or `--output=`*file*: Write the output tile to the named *file*.
|
||||
* `-t` _zoom_`/`_x_`/`_y_ or `--source-tile=`_zoom_`/`_x_`/`_y_: Specify the coordinates of the tile to produce, so that several input tiles, each followed by its own _zoom_`/`_x_`/`_y_, can be combined into it, as in the second example above.
|
||||
* `-b` *buffer* or `--buffer=`*buffer*: Set the tile buffer in the output tile (default 5)
|
||||
* `-d` *detail* or `--full-detail=`*detail*: Set the detail of the output tile (default 12)
|
||||
* `-y` *attribute* or `--include=`*attribute*: Retain the specified *attribute* in the output features. All attributes that are not named in a `-y` option will be removed.
|
||||
* `-x` *attribute* or `--exclude=`*attribute*: Remove the specified *attribute* from the output features.
|
||||
* `--exclude-prefix=`*prefix*: Remove any attribute whose name begins with the specified *prefix* from the output features.
|
||||
* `-j` *filter* or `--feature-filter=`*filter*: Filter features using the same expression syntax as in tippecanoe.
|
||||
* `-J` *filter-file* or `--feature-filter-file=`*filter-file*: Like `-j`, but read the filter from a file.
|
||||
* `-m` or `--filter-points-multiplier`: If a tile was created with the `--retain-points-multiplier` option, thin the tile back down to its normal feature count during overzooming. The first feature from each cluster will be retained, unless `-j` is used to specify a filter, in which case the first matching filter from each cluster will be retained instead.
|
||||
* `-S` *scale* or `--line-simplification=`*scale*: Simplify lines and polygons in the output tile, multiplying the standard tolerance by *scale*. The default of 0 means not to simplify at all.
|
||||
* `--tiny-polygon-size=`*size*: Combine the area of very small polygons into small squares of the specified *size* that represent their combined area, as `tippecanoe` does. The default of 0 means not to do tiny polygon reduction at all.
|
||||
* `--deduplicate-by-id`: When several input tiles are combined, include only the first feature with any given feature ID within each layer, so that features that appear in more than one input tile are not duplicated in the output.
|
||||
* `--preserve-input-order`: Restore a set of filtered features to its original input order
|
||||
* `--accumulate-attribute`: Behaves as in `tippecanoe` to sum attributes from the features of a multiplier cluster that are not included in the final output. The attributes from features that are filtered away with `-j` are *not* accumulated onto the output feature.
|
||||
* `-E` *attribute*`:`*operation* or `--accumulate-attribute=`*attribute*`:`*operation*: Behaves as in `tippecanoe` to sum attributes from the features of a multiplier cluster that are not included in the final output. The attributes from features that are filtered away with `-j` are *not* accumulated onto the output feature.
|
||||
* `--no-tile-compression`: Don't compress the vector tile data in the output tile.
|
||||
* `--output-format=`*format*: Set the tile encoding format. Supported values: `mvt` (default, Mapbox Vector Tiles) or `mlt` ([MapLibre Tiles](https://github.com/maplibre/maplibre-tile-spec)). Applies regardless of what the input tiles are encoded in.
|
||||
* `--pretessellate`: When using `--output-format=mlt`, pre-triangulate polygon geometries. Only applies to layers where all features are polygons.
|
||||
* `--no-mlt-feature-sort`: When using `--output-format=mlt`, disable within-tile spatial sorting of features by Hilbert curve index. Sorting is on by default.
|
||||
|
||||
+39
-27
@@ -24,6 +24,7 @@
|
||||
#include "dirtiles.hpp"
|
||||
#include "pmtiles_file.hpp"
|
||||
#include "errors.hpp"
|
||||
#include "usage.hpp"
|
||||
|
||||
int minzoom = 0;
|
||||
int maxzoom = 32;
|
||||
@@ -549,8 +550,42 @@ void decode(char *fname, int z, unsigned x, unsigned y, std::set<std::string> co
|
||||
}
|
||||
}
|
||||
|
||||
static const struct option long_options[] = {
|
||||
{"Tiles to decode", 0, 0, 0},
|
||||
{"minimum-zoom", required_argument, 0, 'Z'},
|
||||
{"maximum-zoom", required_argument, 0, 'z'},
|
||||
{"layer", required_argument, 0, 'l'},
|
||||
|
||||
{"Output format", 0, 0, 0},
|
||||
{"projection", required_argument, 0, 's'},
|
||||
{"fractional-coordinates", no_argument, 0, 'F'},
|
||||
{"integer-coordinates", no_argument, 0, 'I'},
|
||||
{"tag-layer-and-zoom", no_argument, 0, 'c'},
|
||||
{"stats", no_argument, 0, 'S'},
|
||||
|
||||
{"Filtering the output", 0, 0, 0},
|
||||
{"include", required_argument, 0, 'y'},
|
||||
{"exclude-metadata-row", required_argument, 0, 'x'},
|
||||
|
||||
{"Ignoring errors in the input", 0, 0, 0},
|
||||
{"force", no_argument, 0, 'f'},
|
||||
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
// the options above, with the usage message headings removed
|
||||
static struct option real_long_options[sizeof(long_options) / sizeof(long_options[0])];
|
||||
|
||||
void usage(char **argv) {
|
||||
fprintf(stderr, "Usage: %s [-s projection] [-Z minzoom] [-z maxzoom] [-l layer ...] file.mbtiles [zoom x y]\n", argv[0]);
|
||||
static const char *const forms[] = {
|
||||
"[options] tileset",
|
||||
"[options] tileset zoom x y",
|
||||
NULL,
|
||||
};
|
||||
|
||||
print_usage(stderr, argv[0], forms, long_options, NULL);
|
||||
fprintf(stderr, "\nThe tileset can be an .mbtiles or .pmtiles file or a directory of tiles,\n");
|
||||
fprintf(stderr, "or, if zoom/x/y is specified, a single .pbf tile.\n");
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
|
||||
@@ -564,33 +599,10 @@ int main(int argc, char **argv) {
|
||||
std::set<std::string> exclude_meta;
|
||||
int coordinate_mode = 0;
|
||||
|
||||
struct option long_options[] = {
|
||||
{"projection", required_argument, 0, 's'},
|
||||
{"fractional-coordinates", no_argument, 0, 'F'},
|
||||
{"integer-coordinates", no_argument, 0, 'I'},
|
||||
{"maximum-zoom", required_argument, 0, 'z'},
|
||||
{"minimum-zoom", required_argument, 0, 'Z'},
|
||||
{"layer", required_argument, 0, 'l'},
|
||||
{"tag-layer-and-zoom", no_argument, 0, 'c'},
|
||||
{"stats", no_argument, 0, 'S'},
|
||||
{"force", no_argument, 0, 'f'},
|
||||
{"exclude-metadata-row", required_argument, 0, 'x'},
|
||||
{"include", required_argument, 0, 'y'},
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
strip_usage_headings(long_options, real_long_options);
|
||||
std::string getopt_str = getopt_string(real_long_options);
|
||||
|
||||
std::string getopt_str;
|
||||
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
|
||||
if (long_options[lo].val > ' ') {
|
||||
getopt_str.push_back(long_options[lo].val);
|
||||
|
||||
if (long_options[lo].has_arg == required_argument) {
|
||||
getopt_str.push_back(':');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), long_options, NULL)) != -1) {
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), real_long_options, NULL)) != -1) {
|
||||
switch (i) {
|
||||
case 0:
|
||||
break;
|
||||
|
||||
+19
-2
@@ -1,8 +1,11 @@
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <unistd.h>
|
||||
#include <getopt.h>
|
||||
#include <string>
|
||||
#include <sqlite3.h>
|
||||
#include "errors.hpp"
|
||||
#include "usage.hpp"
|
||||
|
||||
void enumerate(char *fname) {
|
||||
sqlite3 *db;
|
||||
@@ -48,8 +51,20 @@ void enumerate(char *fname) {
|
||||
}
|
||||
}
|
||||
|
||||
// there are no options, but the table is still what the usage message
|
||||
// and the getopt string are derived from, so that they will keep up
|
||||
// with any options that are added later
|
||||
static const struct option long_options[] = {
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
void usage(char **argv) {
|
||||
fprintf(stderr, "Usage: %s file.mbtiles ...\n", argv[0]);
|
||||
static const char *const forms[] = {
|
||||
"file.mbtiles ...",
|
||||
NULL,
|
||||
};
|
||||
|
||||
print_usage(stderr, argv[0], forms, long_options, NULL);
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
|
||||
@@ -58,7 +73,9 @@ int main(int argc, char **argv) {
|
||||
// extern char *optarg;
|
||||
int i;
|
||||
|
||||
while ((i = getopt(argc, argv, "")) != -1) {
|
||||
std::string getopt_str = getopt_string(long_options);
|
||||
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), long_options, NULL)) != -1) {
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
|
||||
+26
-15
@@ -13,6 +13,7 @@
|
||||
#include "geojson-loop.hpp"
|
||||
#include "milo/dtoa_milo.h"
|
||||
#include "errors.hpp"
|
||||
#include "usage.hpp"
|
||||
|
||||
int fail = EXIT_SUCCESS;
|
||||
bool wrap = false;
|
||||
@@ -407,34 +408,45 @@ void process(FILE *fp, const char *fname) {
|
||||
json_end(jp);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
const char *csv = NULL;
|
||||
|
||||
struct option long_options[] = {
|
||||
static const struct option long_options[] = {
|
||||
{"Wrapping the output", 0, 0, 0},
|
||||
{"wrap", no_argument, 0, 'w'},
|
||||
|
||||
{"Sorting and joining", 0, 0, 0},
|
||||
{"extract", required_argument, 0, 'e'},
|
||||
{"csv", required_argument, 0, 'c'},
|
||||
{"empty-csv-columns-are-null", no_argument, &pe, 1},
|
||||
|
||||
{"", 0, 0, 0},
|
||||
{"prevent", required_argument, 0, 'p'},
|
||||
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
std::string getopt_str;
|
||||
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
|
||||
if (long_options[lo].val > ' ') {
|
||||
getopt_str.push_back(long_options[lo].val);
|
||||
// the options above, with the usage message headings removed
|
||||
static struct option real_long_options[sizeof(long_options) / sizeof(long_options[0])];
|
||||
|
||||
if (long_options[lo].has_arg == required_argument) {
|
||||
getopt_str.push_back(':');
|
||||
}
|
||||
}
|
||||
void usage(char **argv) {
|
||||
static const char *const forms[] = {
|
||||
"[options] [file.json ...]",
|
||||
NULL,
|
||||
};
|
||||
|
||||
print_usage(stderr, argv[0], forms, long_options, NULL);
|
||||
fprintf(stderr, "\nIf no files are named, the JSON is read from the standard input.\n");
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
const char *csv = NULL;
|
||||
|
||||
strip_usage_headings(long_options, real_long_options);
|
||||
std::string getopt_str = getopt_string(real_long_options);
|
||||
|
||||
extern int optind;
|
||||
int i;
|
||||
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), long_options, NULL)) != -1) {
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), real_long_options, NULL)) != -1) {
|
||||
switch (i) {
|
||||
case 0:
|
||||
break;
|
||||
@@ -461,8 +473,7 @@ int main(int argc, char **argv) {
|
||||
break;
|
||||
|
||||
default:
|
||||
fprintf(stderr, "Unexpected option -%c\n", i);
|
||||
exit(EXIT_ARGS);
|
||||
usage(argv);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -58,6 +58,7 @@
|
||||
#include "geometry.hpp"
|
||||
#include "serial.hpp"
|
||||
#include "options.hpp"
|
||||
#include "usage.hpp"
|
||||
#include "mvt.hpp"
|
||||
#include "dirtiles.hpp"
|
||||
#include "evaluator.hpp"
|
||||
@@ -215,7 +216,7 @@ void init_cpus() {
|
||||
// MacOS can run out of system file descriptors
|
||||
// even if we stay under the rlimit, so try to
|
||||
// find out the real limit.
|
||||
long long fds[MAX_FILES];
|
||||
std::vector<long long> fds(MAX_FILES);
|
||||
long long i;
|
||||
for (i = 0; i < MAX_FILES; i++) {
|
||||
fds[i] = open(get_null_device(), O_RDONLY | O_CLOEXEC);
|
||||
@@ -449,7 +450,7 @@ void *run_sort(void *v) {
|
||||
}
|
||||
|
||||
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::unordered_map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
|
||||
long long segs[CPUS + 1];
|
||||
std::vector<long long> segs(CPUS + 1);
|
||||
segs[0] = 0;
|
||||
segs[CPUS] = len;
|
||||
|
||||
@@ -461,11 +462,11 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
|
||||
}
|
||||
}
|
||||
|
||||
double dist_sums[CPUS];
|
||||
size_t dist_counts[CPUS];
|
||||
double area_sums[CPUS];
|
||||
std::vector<double> dist_sums(CPUS);
|
||||
std::vector<size_t> dist_counts(CPUS);
|
||||
std::vector<double> area_sums(CPUS);
|
||||
|
||||
std::atomic<long long> layer_seq[CPUS];
|
||||
std::vector<std::atomic<long long> > layer_seq(CPUS);
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
// To preserve feature ordering, unique id for each segment
|
||||
// begins with that segment's offset into the input
|
||||
@@ -479,7 +480,7 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
|
||||
std::vector<serialization_state> sst;
|
||||
sst.resize(CPUS);
|
||||
|
||||
pthread_t pthreads[CPUS];
|
||||
std::vector<pthread_t> pthreads(CPUS);
|
||||
std::vector<std::set<serial_val> > file_subkeys;
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
@@ -743,51 +744,61 @@ void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::ato
|
||||
}
|
||||
|
||||
void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int splits, long long mem, const char *tmpdir, long long *availfiles, FILE *geomfile, FILE *indexfile, std::atomic<long long> *geompos_out, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, int basezoom, double droprate, double gamma, struct drop_state *ds) {
|
||||
// Arranged as bits to facilitate subdividing again if a subdivided file is still huge
|
||||
int splitbits = log(splits) / log(2);
|
||||
// Arranged as bits to facilitate subdividing again if a subdivided file is still huge.
|
||||
//
|
||||
// There must be at least two buckets. With only one, each subdivision would
|
||||
// consume no bits of the index, so it would never reach the maximum prefix
|
||||
// that stops the recursion, and the shift that chooses a feature's bucket
|
||||
// below would be by the full width of the index. That shift is undefined,
|
||||
// and what it does in practice is to mask the shift count down to zero, so
|
||||
// the bucket number comes out as the whole shifted index instead of as 0
|
||||
// and the writes are made through whatever is found beyond the end of the
|
||||
// arrays of buckets.
|
||||
int splitbits = 1;
|
||||
if (splits > 1) {
|
||||
splitbits = log(splits) / log(2);
|
||||
}
|
||||
splits = 1 << splitbits;
|
||||
|
||||
FILE *geomfiles[splits];
|
||||
FILE *indexfiles[splits];
|
||||
int geomfds[splits];
|
||||
int indexfds[splits];
|
||||
std::atomic<long long> sub_geompos[splits];
|
||||
std::vector<FILE *> geomfiles(splits);
|
||||
std::vector<FILE *> indexfiles(splits);
|
||||
std::vector<int> geomfds(splits);
|
||||
std::vector<int> indexfds(splits);
|
||||
std::vector<std::atomic<long long> > sub_geompos(splits);
|
||||
|
||||
int i;
|
||||
for (i = 0; i < splits; i++) {
|
||||
sub_geompos[i] = 0;
|
||||
|
||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
std::string geomname = std::string(tmpdir) + "/geom.XXXXXXXX";
|
||||
std::string indexname = std::string(tmpdir) + "/index.XXXXXXXX";
|
||||
|
||||
geomfds[i] = mkstemp_cloexec(geomname);
|
||||
geomfds[i] = mkstemp_cloexec(&geomname[0]);
|
||||
if (geomfds[i] < 0) {
|
||||
perror(geomname);
|
||||
perror(geomname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
indexfds[i] = mkstemp_cloexec(indexname);
|
||||
indexfds[i] = mkstemp_cloexec(&indexname[0]);
|
||||
if (indexfds[i] < 0) {
|
||||
perror(indexname);
|
||||
perror(indexname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
geomfiles[i] = fopen_oflag(geomname, "wb", O_WRONLY | O_CLOEXEC);
|
||||
geomfiles[i] = fopen_oflag(geomname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||
if (geomfiles[i] == NULL) {
|
||||
perror(geomname);
|
||||
perror(geomname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
indexfiles[i] = fopen_oflag(indexname, "wb", O_WRONLY | O_CLOEXEC);
|
||||
indexfiles[i] = fopen_oflag(indexname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||
if (indexfiles[i] == NULL) {
|
||||
perror(indexname);
|
||||
perror(indexname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
*availfiles -= 4;
|
||||
|
||||
unlink(geomname);
|
||||
unlink(indexname);
|
||||
unlink(geomname.c_str());
|
||||
unlink(indexname.c_str());
|
||||
}
|
||||
|
||||
for (i = 0; i < inputs; i++) {
|
||||
@@ -891,7 +902,14 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
||||
}
|
||||
|
||||
if (indexst.st_size > 0) {
|
||||
if (indexst.st_size + geomst.st_size < mem) {
|
||||
// Subdividing again would only make progress if the next level could
|
||||
// split into at least two buckets, which it can't if there are no longer
|
||||
// enough files left to do it with. In that case, sort in memory instead,
|
||||
// even though this is more memory than we wanted to use at once, since
|
||||
// it is the only way left to get this bucket sorted.
|
||||
bool can_subdivide = *availfiles / 4 > 1;
|
||||
|
||||
if (indexst.st_size + geomst.st_size < mem || !can_subdivide) {
|
||||
std::atomic<long long> indexpos(indexst.st_size);
|
||||
int bytes = sizeof(struct index);
|
||||
|
||||
@@ -909,13 +927,13 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
||||
}
|
||||
|
||||
size_t nmerges = (indexpos + unit - 1) / unit;
|
||||
struct mergelist merges[nmerges];
|
||||
std::vector<struct mergelist> merges(nmerges);
|
||||
|
||||
for (size_t a = 0; a < nmerges; a++) {
|
||||
merges[a].start = merges[a].end = 0;
|
||||
}
|
||||
|
||||
pthread_t pthreads[CPUS];
|
||||
std::vector<pthread_t> pthreads(CPUS);
|
||||
std::vector<sort_arg> args;
|
||||
|
||||
for (size_t a = 0; a < CPUS; a++) {
|
||||
@@ -923,7 +941,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
||||
a,
|
||||
CPUS,
|
||||
indexpos,
|
||||
merges,
|
||||
merges.data(),
|
||||
indexfds[i],
|
||||
nmerges,
|
||||
unit,
|
||||
@@ -961,7 +979,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
||||
madvise(geommap, geomst.st_size, MADV_RANDOM);
|
||||
madvise(geommap, geomst.st_size, MADV_WILLNEED);
|
||||
|
||||
merge(merges, nmerges, (unsigned char *) indexmap, indexfile, bytes, geommap, geomfile, geompos_out, progress, progress_max, progress_reported, maxzoom, gamma, ds);
|
||||
merge(merges.data(), nmerges, (unsigned char *) indexmap, indexfile, bytes, geommap, geomfile, geompos_out, progress, progress_max, progress_reported, maxzoom, gamma, ds);
|
||||
|
||||
madvise(indexmap, indexst.st_size, MADV_DONTNEED);
|
||||
if (munmap(indexmap, indexst.st_size) < 0) {
|
||||
@@ -994,7 +1012,11 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
||||
struct index ix = indexmap[a];
|
||||
long long pos = *geompos_out;
|
||||
|
||||
fwrite_check(geommap + ix.start, ix.end - ix.start, 1, geomfile, geompos_out, "geom");
|
||||
// MAGIC: This knows that the feature minzoom is the last byte of the serialized feature
|
||||
// and is writing one byte less and then adding the byte for the minzoom,
|
||||
// the same as merge() does.
|
||||
|
||||
fwrite_check(geommap + ix.start, 1, ix.end - ix.start - 1, geomfile, geompos_out, "geom");
|
||||
int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom, gamma);
|
||||
serialize_byte(geomfile, feature_minzoom, geompos_out, "merge geometry");
|
||||
|
||||
@@ -1096,8 +1118,8 @@ void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FI
|
||||
mem /= 2;
|
||||
|
||||
long long geom_total = 0;
|
||||
int geomfds[nreaders];
|
||||
int indexfds[nreaders];
|
||||
std::vector<int> geomfds(nreaders);
|
||||
std::vector<int> indexfds(nreaders);
|
||||
for (int i = 0; i < nreaders; i++) {
|
||||
geomfds[i] = readers[i].geomfd;
|
||||
indexfds[i] = readers[i].indexfd;
|
||||
@@ -1110,12 +1132,12 @@ void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FI
|
||||
geom_total += geomst.st_size;
|
||||
}
|
||||
|
||||
struct drop_state ds[maxzoom + 1];
|
||||
prep_drop_states(ds, maxzoom, basezoom, droprate);
|
||||
std::vector<struct drop_state> ds(maxzoom + 1);
|
||||
prep_drop_states(ds.data(), maxzoom, basezoom, droprate);
|
||||
|
||||
long long progress = 0, progress_max = geom_total, progress_reported = -1;
|
||||
long long availfiles_before = availfiles;
|
||||
radix1(geomfds, indexfds, nreaders, 0, splits, mem, tmpdir, &availfiles, geomfile, indexfile, geompos, &progress, &progress_max, &progress_reported, maxzoom, basezoom, droprate, gamma, ds);
|
||||
radix1(geomfds.data(), indexfds.data(), nreaders, 0, splits, mem, tmpdir, &availfiles, geomfile, indexfile, geompos, &progress, &progress_max, &progress_reported, maxzoom, basezoom, droprate, gamma, ds.data());
|
||||
|
||||
if (availfiles - 2 * nreaders != availfiles_before) {
|
||||
fprintf(stderr, "Internal error: miscounted available file descriptors: %lld vs %lld\n", availfiles - 2 * nreaders, availfiles);
|
||||
@@ -1224,79 +1246,72 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
struct reader *r = &readers[i];
|
||||
|
||||
char poolname[strlen(tmpdir) + strlen("/pool.XXXXXXXX") + 1];
|
||||
char treename[strlen(tmpdir) + strlen("/tree.XXXXXXXX") + 1];
|
||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
||||
char vertexname[strlen(tmpdir) + strlen("/vertex.XXXXXXXX") + 1];
|
||||
char nodename[strlen(tmpdir) + strlen("/node.XXXXXXXX") + 1];
|
||||
std::string poolname = std::string(tmpdir) + "/pool.XXXXXXXX";
|
||||
std::string treename = std::string(tmpdir) + "/tree.XXXXXXXX";
|
||||
std::string geomname = std::string(tmpdir) + "/geom.XXXXXXXX";
|
||||
std::string indexname = std::string(tmpdir) + "/index.XXXXXXXX";
|
||||
std::string vertexname = std::string(tmpdir) + "/vertex.XXXXXXXX";
|
||||
std::string nodename = std::string(tmpdir) + "/node.XXXXXXXX";
|
||||
|
||||
snprintf(poolname, sizeof(poolname), "%s%s", tmpdir, "/pool.XXXXXXXX");
|
||||
snprintf(treename, sizeof(treename), "%s%s", tmpdir, "/tree.XXXXXXXX");
|
||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
snprintf(vertexname, sizeof(vertexname), "%s%s", tmpdir, "/vertex.XXXXXXXX");
|
||||
snprintf(nodename, sizeof(nodename), "%s%s", tmpdir, "/node.XXXXXXXX");
|
||||
|
||||
r->poolfd = mkstemp_cloexec(poolname);
|
||||
r->poolfd = mkstemp_cloexec(&poolname[0]);
|
||||
if (r->poolfd < 0) {
|
||||
perror(poolname);
|
||||
perror(poolname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->treefd = mkstemp_cloexec(treename);
|
||||
r->treefd = mkstemp_cloexec(&treename[0]);
|
||||
if (r->treefd < 0) {
|
||||
perror(treename);
|
||||
perror(treename.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->geomfd = mkstemp_cloexec(geomname);
|
||||
r->geomfd = mkstemp_cloexec(&geomname[0]);
|
||||
if (r->geomfd < 0) {
|
||||
perror(geomname);
|
||||
perror(geomname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->indexfd = mkstemp_cloexec(indexname);
|
||||
r->indexfd = mkstemp_cloexec(&indexname[0]);
|
||||
if (r->indexfd < 0) {
|
||||
perror(indexname);
|
||||
perror(indexname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->vertexfd = mkstemp_cloexec(vertexname);
|
||||
r->vertexfd = mkstemp_cloexec(&vertexname[0]);
|
||||
if (r->vertexfd < 0) {
|
||||
perror(vertexname);
|
||||
perror(vertexname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->nodefd = mkstemp_cloexec(nodename);
|
||||
r->nodefd = mkstemp_cloexec(&nodename[0]);
|
||||
if (r->nodefd < 0) {
|
||||
perror(nodename);
|
||||
perror(nodename.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
r->poolfile = memfile_open(r->poolfd);
|
||||
if (r->poolfile == NULL) {
|
||||
perror(poolname);
|
||||
perror(poolname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->treefile = memfile_open(r->treefd);
|
||||
if (r->treefile == NULL) {
|
||||
perror(treename);
|
||||
perror(treename.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->geomfile = fopen_oflag(geomname, "wb", O_WRONLY | O_CLOEXEC);
|
||||
r->geomfile = fopen_oflag(geomname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||
if (r->geomfile == NULL) {
|
||||
perror(geomname);
|
||||
perror(geomname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->indexfile = fopen_oflag(indexname, "wb", O_WRONLY | O_CLOEXEC);
|
||||
r->indexfile = fopen_oflag(indexname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||
if (r->indexfile == NULL) {
|
||||
perror(indexname);
|
||||
perror(indexname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->vertexfile = fopen_oflag(vertexname, "w+b", O_RDWR | O_CLOEXEC);
|
||||
r->vertexfile = fopen_oflag(vertexname.c_str(), "w+b", O_RDWR | O_CLOEXEC);
|
||||
if (r->vertexfile == NULL) {
|
||||
perror(("open vertexfile " + std::string(vertexname)).c_str());
|
||||
perror(("open vertexfile " + vertexname).c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->nodefile = fopen_oflag(nodename, "w+b", O_RDWR | O_CLOEXEC);
|
||||
r->nodefile = fopen_oflag(nodename.c_str(), "w+b", O_RDWR | O_CLOEXEC);
|
||||
if (r->nodefile == NULL) {
|
||||
perror(nodename);
|
||||
perror(nodename.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
r->geompos = 0;
|
||||
@@ -1304,12 +1319,12 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
r->vertexpos = 0;
|
||||
r->nodepos = 0;
|
||||
|
||||
unlink(poolname);
|
||||
unlink(treename);
|
||||
unlink(geomname);
|
||||
unlink(indexname);
|
||||
unlink(vertexname);
|
||||
unlink(nodename);
|
||||
unlink(poolname.c_str());
|
||||
unlink(treename.c_str());
|
||||
unlink(geomname.c_str());
|
||||
unlink(indexname.c_str());
|
||||
unlink(vertexname.c_str());
|
||||
unlink(nodename.c_str());
|
||||
|
||||
// To distinguish a null value
|
||||
{
|
||||
@@ -1334,8 +1349,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
std::atomic<long long> progress_seq(0);
|
||||
|
||||
// 2 * CPUS: One per reader thread, one per tiling thread
|
||||
int initialized[2 * CPUS];
|
||||
unsigned initial_x[2 * CPUS], initial_y[2 * CPUS];
|
||||
std::vector<int> initialized(2 * CPUS);
|
||||
std::vector<unsigned> initial_x(2 * CPUS), initial_y(2 * CPUS);
|
||||
for (size_t i = 0; i < 2 * CPUS; i++) {
|
||||
initialized[i] = initial_x[i] = initial_y[i] = 0;
|
||||
}
|
||||
@@ -1466,10 +1481,10 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
exit(EXIT_MEMORY);
|
||||
}
|
||||
|
||||
std::atomic<long long> layer_seq[CPUS];
|
||||
double dist_sums[CPUS];
|
||||
size_t dist_counts[CPUS];
|
||||
double area_sums[CPUS];
|
||||
std::vector<std::atomic<long long> > layer_seq(CPUS);
|
||||
std::vector<double> dist_sums(CPUS);
|
||||
std::vector<size_t> dist_counts(CPUS);
|
||||
std::vector<double> area_sums(CPUS);
|
||||
std::vector<struct serialization_state> sst;
|
||||
sst.resize(CPUS);
|
||||
|
||||
@@ -1539,10 +1554,10 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
exit(EXIT_MEMORY);
|
||||
}
|
||||
|
||||
std::atomic<long long> layer_seq[CPUS];
|
||||
double dist_sums[CPUS];
|
||||
size_t dist_counts[CPUS];
|
||||
double area_sums[CPUS];
|
||||
std::vector<std::atomic<long long> > layer_seq(CPUS);
|
||||
std::vector<double> dist_sums(CPUS);
|
||||
std::vector<size_t> dist_counts(CPUS);
|
||||
std::vector<double> area_sums(CPUS);
|
||||
std::vector<struct serialization_state> sst;
|
||||
sst.resize(CPUS);
|
||||
|
||||
@@ -1599,10 +1614,10 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
|
||||
if (sources[source].format == "csv" || (sources[source].file.size() > 4 && sources[source].file.substr(sources[source].file.size() - 4) == std::string(".csv"))) {
|
||||
std::atomic<long long> layer_seq[CPUS];
|
||||
double dist_sums[CPUS];
|
||||
size_t dist_counts[CPUS];
|
||||
double area_sums[CPUS];
|
||||
std::vector<std::atomic<long long> > layer_seq(CPUS);
|
||||
std::vector<double> dist_sums(CPUS);
|
||||
std::vector<size_t> dist_counts(CPUS);
|
||||
std::vector<double> area_sums(CPUS);
|
||||
|
||||
std::vector<struct serialization_state> sst;
|
||||
sst.resize(CPUS);
|
||||
@@ -1687,7 +1702,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
|
||||
if (map != NULL && map != MAP_FAILED && read_parallel_this) {
|
||||
do_read_parallel(map, st.st_size - off, overall_offset, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, &layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, uses_gamma, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||
do_read_parallel(map, st.st_size - off, overall_offset, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, &layermaps, initialized.data(), initial_x.data(), initial_y.data(), maxzoom, sources[layer].layer, uses_gamma, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||
overall_offset += st.st_size - off;
|
||||
checkdisk(&readers);
|
||||
|
||||
@@ -1719,19 +1734,18 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
if (read_parallel_this) {
|
||||
// Serial reading of chunks that are then parsed in parallel
|
||||
|
||||
char readname[strlen(tmpdir) + strlen("/read.XXXXXXXX") + 1];
|
||||
snprintf(readname, sizeof(readname), "%s%s", tmpdir, "/read.XXXXXXXX");
|
||||
int readfd = mkstemp_cloexec(readname);
|
||||
std::string readname = std::string(tmpdir) + "/read.XXXXXXXX";
|
||||
int readfd = mkstemp_cloexec(&readname[0]);
|
||||
if (readfd < 0) {
|
||||
perror(readname);
|
||||
perror(readname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
FILE *readfp = fdopen(readfd, "w");
|
||||
if (readfp == NULL) {
|
||||
perror(readname);
|
||||
perror(readname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
unlink(readname);
|
||||
unlink(readname.c_str());
|
||||
|
||||
std::atomic<int> is_parsing(0);
|
||||
long long ahead = 0;
|
||||
@@ -1766,25 +1780,25 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
}
|
||||
|
||||
fflush(readfp);
|
||||
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, ¶llel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, ¶llel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized.data(), initial_x.data(), initial_y.data(), maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||
|
||||
initial_offset += ahead;
|
||||
overall_offset += ahead;
|
||||
checkdisk(&readers);
|
||||
ahead = 0;
|
||||
|
||||
snprintf(readname, sizeof(readname), "%s%s", tmpdir, "/read.XXXXXXXX");
|
||||
readfd = mkstemp_cloexec(readname);
|
||||
readname = std::string(tmpdir) + "/read.XXXXXXXX";
|
||||
readfd = mkstemp_cloexec(&readname[0]);
|
||||
if (readfd < 0) {
|
||||
perror(readname);
|
||||
perror(readname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
readfp = fdopen(readfd, "w");
|
||||
if (readfp == NULL) {
|
||||
perror(readname);
|
||||
perror(readname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
unlink(readname);
|
||||
unlink(readname.c_str());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1803,7 +1817,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
fflush(readfp);
|
||||
|
||||
if (ahead > 0) {
|
||||
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, ¶llel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, ¶llel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized.data(), initial_x.data(), initial_y.data(), maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||
|
||||
if (parser_created) {
|
||||
if (pthread_join(parallel_parser, NULL) != 0) {
|
||||
@@ -1912,27 +1926,26 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
// segment+offset to find the data.
|
||||
|
||||
// 2 * CPUS: One per input thread, one per tiling thread
|
||||
long long pool_off[2 * CPUS];
|
||||
std::vector<long long> pool_off(2 * CPUS);
|
||||
for (size_t i = 0; i < 2 * CPUS; i++) {
|
||||
pool_off[i] = 0;
|
||||
}
|
||||
|
||||
char poolname[strlen(tmpdir) + strlen("/pool.XXXXXXXX") + 1];
|
||||
snprintf(poolname, sizeof(poolname), "%s%s", tmpdir, "/pool.XXXXXXXX");
|
||||
std::string poolname = std::string(tmpdir) + "/pool.XXXXXXXX";
|
||||
|
||||
int poolfd = mkstemp_cloexec(poolname);
|
||||
int poolfd = mkstemp_cloexec(&poolname[0]);
|
||||
if (poolfd < 0) {
|
||||
perror(poolname);
|
||||
perror(poolname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
FILE *poolfile = fopen_oflag(poolname, "wb", O_WRONLY | O_CLOEXEC);
|
||||
FILE *poolfile = fopen_oflag(poolname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||
if (poolfile == NULL) {
|
||||
perror(poolname);
|
||||
perror(poolname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
unlink(poolname);
|
||||
unlink(poolname.c_str());
|
||||
std::atomic<long long> poolpos(0);
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
@@ -2160,36 +2173,34 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
fprintf(stderr, "Merging index \r");
|
||||
}
|
||||
|
||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
||||
std::string indexname = std::string(tmpdir) + "/index.XXXXXXXX";
|
||||
|
||||
int indexfd = mkstemp_cloexec(indexname);
|
||||
int indexfd = mkstemp_cloexec(&indexname[0]);
|
||||
if (indexfd < 0) {
|
||||
perror(indexname);
|
||||
perror(indexname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
FILE *indexfile = fopen_oflag(indexname, "wb", O_WRONLY | O_CLOEXEC);
|
||||
FILE *indexfile = fopen_oflag(indexname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||
if (indexfile == NULL) {
|
||||
perror(indexname);
|
||||
perror(indexname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
|
||||
unlink(indexname);
|
||||
unlink(indexname.c_str());
|
||||
|
||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
||||
std::string geomname = std::string(tmpdir) + "/geom.XXXXXXXX";
|
||||
|
||||
int geomfd = mkstemp_cloexec(geomname);
|
||||
int geomfd = mkstemp_cloexec(&geomname[0]);
|
||||
if (geomfd < 0) {
|
||||
perror(geomname);
|
||||
perror(geomname.c_str());
|
||||
exit(EXIT_CLOSE);
|
||||
}
|
||||
FILE *geomfile = fopen_oflag(geomname, "wb", O_WRONLY | O_CLOEXEC);
|
||||
FILE *geomfile = fopen_oflag(geomname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||
if (geomfile == NULL) {
|
||||
perror(geomname);
|
||||
perror(geomname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
unlink(geomname);
|
||||
unlink(geomname.c_str());
|
||||
|
||||
unsigned iz = 0, ix = 0, iy = 0;
|
||||
choose_first_zoom(file_bbox, file_bbox1, file_bbox2, readers, &iz, &ix, &iy, minzoom, buffer);
|
||||
@@ -2676,8 +2687,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
madvise(geom, indexpos, MADV_SEQUENTIAL);
|
||||
madvise(geom, indexpos, MADV_WILLNEED);
|
||||
|
||||
struct drop_state ds[maxzoom + 1];
|
||||
prep_drop_states(ds, maxzoom, basezoom, droprate);
|
||||
std::vector<struct drop_state> ds(maxzoom + 1);
|
||||
prep_drop_states(ds.data(), maxzoom, basezoom, droprate);
|
||||
|
||||
if (drop_denser > 0) {
|
||||
std::vector<drop_densest> ddv;
|
||||
@@ -2695,7 +2706,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
|
||||
previndex = map[ip].ix;
|
||||
} else {
|
||||
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom, gamma);
|
||||
int feature_minzoom = calc_feature_minzoom(&map[ip], ds.data(), maxzoom, gamma);
|
||||
geom[map[ip].end - 1] = feature_minzoom;
|
||||
}
|
||||
}
|
||||
@@ -2720,7 +2731,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
if (ip > 0 && map[ip].start != map[ip - 1].end) {
|
||||
fprintf(stderr, "Mismatched index at %lld: %lld vs %lld\n", ip, map[ip].start, map[ip].end);
|
||||
}
|
||||
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom, gamma);
|
||||
int feature_minzoom = calc_feature_minzoom(&map[ip], ds.data(), maxzoom, gamma);
|
||||
geom[map[ip].end - 1] = feature_minzoom;
|
||||
}
|
||||
}
|
||||
@@ -2743,8 +2754,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
exit(EXIT_STAT);
|
||||
}
|
||||
|
||||
int fd[TEMP_FILES];
|
||||
off_t size[TEMP_FILES];
|
||||
std::vector<int> fd(TEMP_FILES);
|
||||
std::vector<off_t> size(TEMP_FILES);
|
||||
|
||||
fd[0] = geomfd;
|
||||
size[0] = geomst.st_size;
|
||||
@@ -2757,7 +2768,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
||||
std::atomic<unsigned> midx(0);
|
||||
std::atomic<unsigned> midy(0);
|
||||
std::vector<strategy> strategies;
|
||||
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos, shared_nodes_bloom, basezoom, droprate, unidecode_data, &drop_by_attribute_as_needed_attribute, drop_by_attribute_descending);
|
||||
int written = traverse_zooms(fd.data(), size.data(), stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off.data(), initial_x.data(), initial_y.data(), simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos, shared_nodes_bloom, basezoom, droprate, unidecode_data, &drop_by_attribute_as_needed_attribute, drop_by_attribute_descending);
|
||||
|
||||
if (maxzoom != written) {
|
||||
if (written > minzoom) {
|
||||
@@ -2967,59 +2978,7 @@ void parse_json_source(const char *arg, struct source &src) {
|
||||
json_end(jp);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
#ifdef MTRACE
|
||||
mtrace();
|
||||
#endif
|
||||
|
||||
av = argv;
|
||||
init_cpus();
|
||||
|
||||
extern int optind;
|
||||
extern char *optarg;
|
||||
int i;
|
||||
|
||||
char *name = NULL;
|
||||
char *description = NULL;
|
||||
char *layername = NULL;
|
||||
char *out_mbtiles = NULL;
|
||||
char *out_dir = NULL;
|
||||
sqlite3 *outdb = NULL;
|
||||
int maxzoom = 14;
|
||||
int minzoom = 0;
|
||||
int basezoom = -1;
|
||||
double basezoom_marker_width = 1;
|
||||
int force = 0;
|
||||
int forcetable = 0;
|
||||
double droprate = 2.5;
|
||||
double gamma = 0;
|
||||
int buffer = 5;
|
||||
const char *tmpdir = "/tmp";
|
||||
const char *attribution = NULL;
|
||||
std::vector<source> sources;
|
||||
const char *prefilter = NULL;
|
||||
const char *postfilter = NULL;
|
||||
bool guess_maxzoom = false;
|
||||
int minimum_maxzoom = 0;
|
||||
bool guess_cluster_maxzoom = false;
|
||||
|
||||
std::set<std::string> exclude, include;
|
||||
std::unordered_map<std::string, int> attribute_types;
|
||||
std::unordered_map<std::string, attribute_op> attribute_accum;
|
||||
std::map<std::string, std::string> attribute_descriptions;
|
||||
int exclude_all = 0;
|
||||
int read_parallel = 0;
|
||||
int files_open_at_start;
|
||||
json_object *filter = NULL;
|
||||
|
||||
memsize = calc_memsize();
|
||||
|
||||
for (i = 0; i < 256; i++) {
|
||||
prevent[i] = 0;
|
||||
additional[i] = 0;
|
||||
}
|
||||
|
||||
static struct option long_options_orig[] = {
|
||||
static const struct option long_options_orig[] = {
|
||||
{"Output tileset", 0, 0, 0},
|
||||
{"output", required_argument, 0, 'o'},
|
||||
{"output-to-directory", required_argument, 0, 'e'},
|
||||
@@ -3075,7 +3034,6 @@ int main(int argc, char **argv) {
|
||||
{"Filtering features by attributes", 0, 0, 0},
|
||||
{"feature-filter-file", required_argument, 0, 'J'},
|
||||
{"feature-filter", required_argument, 0, 'j'},
|
||||
{"unidecode-data", required_argument, 0, '~'},
|
||||
|
||||
{"Dropping a fixed fraction of features by zoom level", 0, 0, 0},
|
||||
{"drop-rate", required_argument, 0, 'r'},
|
||||
@@ -3178,6 +3136,8 @@ int main(int argc, char **argv) {
|
||||
{"no-progress-indicator", no_argument, 0, 'Q'},
|
||||
{"progress-interval", required_argument, 0, 'U'},
|
||||
{"json-progress", no_argument, 0, 'u'},
|
||||
|
||||
{"Version", 0, 0, 0},
|
||||
{"version", no_argument, 0, 'v'},
|
||||
|
||||
{"", 0, 0, 0},
|
||||
@@ -3186,33 +3146,91 @@ int main(int argc, char **argv) {
|
||||
{"check-polygons", no_argument, &additional[A_DEBUG_POLYGON], 1},
|
||||
{"no-polygon-splitting", no_argument, &prevent[P_POLYGON_SPLIT], 1},
|
||||
{"prefer-radix-sort", no_argument, &additional[A_PREFER_RADIX_SORT], 1},
|
||||
{"unidecode-data", required_argument, 0, '~'},
|
||||
{"help", no_argument, 0, 'H'},
|
||||
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
// the options above, with the usage message headings removed
|
||||
static struct option long_options[sizeof(long_options_orig) / sizeof(long_options_orig[0])];
|
||||
static char getopt_str[sizeof(long_options_orig) / sizeof(long_options_orig[0]) * 2 + 1];
|
||||
|
||||
void usage(char **argv, int status) {
|
||||
static const char *const forms[] = {
|
||||
"[options] [file.json ...]",
|
||||
NULL,
|
||||
};
|
||||
static const struct usage_required_option required[] = {
|
||||
{"output", "output.mbtiles", 1},
|
||||
{"output-to-directory", "directory", 1},
|
||||
{NULL, NULL, 0},
|
||||
};
|
||||
|
||||
print_usage(stderr, argv[0], forms, long_options_orig, required);
|
||||
exit(status);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
#ifdef MTRACE
|
||||
mtrace();
|
||||
#endif
|
||||
|
||||
if (argc == 1) {
|
||||
// with no arguments at all, there is nothing to complain about
|
||||
// specifically, so say in general what the arguments should be
|
||||
usage(argv, EXIT_ARGS);
|
||||
}
|
||||
|
||||
av = argv;
|
||||
init_cpus();
|
||||
|
||||
extern int optind;
|
||||
extern char *optarg;
|
||||
int i;
|
||||
|
||||
char *name = NULL;
|
||||
char *description = NULL;
|
||||
char *layername = NULL;
|
||||
char *out_mbtiles = NULL;
|
||||
char *out_dir = NULL;
|
||||
sqlite3 *outdb = NULL;
|
||||
int maxzoom = 14;
|
||||
int minzoom = 0;
|
||||
int basezoom = -1;
|
||||
double basezoom_marker_width = 1;
|
||||
int force = 0;
|
||||
int forcetable = 0;
|
||||
double droprate = 2.5;
|
||||
double gamma = 0;
|
||||
int buffer = 5;
|
||||
const char *tmpdir = "/tmp";
|
||||
const char *attribution = NULL;
|
||||
std::vector<source> sources;
|
||||
const char *prefilter = NULL;
|
||||
const char *postfilter = NULL;
|
||||
bool guess_maxzoom = false;
|
||||
int minimum_maxzoom = 0;
|
||||
bool guess_cluster_maxzoom = false;
|
||||
|
||||
std::set<std::string> exclude, include;
|
||||
std::unordered_map<std::string, int> attribute_types;
|
||||
std::unordered_map<std::string, attribute_op> attribute_accum;
|
||||
std::map<std::string, std::string> attribute_descriptions;
|
||||
int exclude_all = 0;
|
||||
int read_parallel = 0;
|
||||
int files_open_at_start;
|
||||
json_object *filter = NULL;
|
||||
|
||||
memsize = calc_memsize();
|
||||
|
||||
for (i = 0; i < 256; i++) {
|
||||
prevent[i] = 0;
|
||||
additional[i] = 0;
|
||||
}
|
||||
|
||||
strip_usage_headings(long_options_orig, long_options);
|
||||
|
||||
{
|
||||
size_t out = 0;
|
||||
size_t cout = 0;
|
||||
for (size_t lo = 0; long_options_orig[lo].name != NULL; lo++) {
|
||||
if (long_options_orig[lo].val != 0) {
|
||||
long_options[out++] = long_options_orig[lo];
|
||||
|
||||
if (long_options_orig[lo].val > ' ') {
|
||||
getopt_str[cout++] = long_options_orig[lo].val;
|
||||
|
||||
if (long_options_orig[lo].has_arg == required_argument) {
|
||||
getopt_str[cout++] = ':';
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
long_options[out] = {0, 0, 0, 0};
|
||||
getopt_str[cout] = '\0';
|
||||
|
||||
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
|
||||
if (long_options[lo].flag != NULL) {
|
||||
if (*long_options[lo].flag != 0) {
|
||||
@@ -3231,9 +3249,10 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
|
||||
std::string commandline = format_commandline(argc, argv);
|
||||
std::string getopt_str = getopt_string(long_options);
|
||||
|
||||
int option_index = 0;
|
||||
while ((i = getopt_long(argc, argv, getopt_str, long_options, &option_index)) != -1) {
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), long_options, &option_index)) != -1) {
|
||||
switch (i) {
|
||||
case 0:
|
||||
break;
|
||||
@@ -3663,43 +3682,11 @@ int main(int argc, char **argv) {
|
||||
set_attribute_accum(attribute_accum, optarg, argv);
|
||||
break;
|
||||
|
||||
default: {
|
||||
default:
|
||||
if (i != 'H' && i != '?') {
|
||||
fprintf(stderr, "Unknown option -%c\n", i);
|
||||
}
|
||||
int width = 7 + strlen(argv[0]);
|
||||
fprintf(stderr, "Usage: %s [options] [file.json ...]", argv[0]);
|
||||
for (size_t lo = 0; long_options_orig[lo].name != NULL && strlen(long_options_orig[lo].name) > 0; lo++) {
|
||||
if (long_options_orig[lo].val == 0) {
|
||||
fprintf(stderr, "\n %s\n ", long_options_orig[lo].name);
|
||||
width = 8;
|
||||
continue;
|
||||
}
|
||||
if (width + strlen(long_options_orig[lo].name) + 9 >= 80) {
|
||||
fprintf(stderr, "\n ");
|
||||
width = 8;
|
||||
}
|
||||
width += strlen(long_options_orig[lo].name) + 9;
|
||||
if (strcmp(long_options_orig[lo].name, "output") == 0) {
|
||||
fprintf(stderr, " --%s=output.mbtiles", long_options_orig[lo].name);
|
||||
width += 9;
|
||||
} else if (long_options_orig[lo].has_arg) {
|
||||
fprintf(stderr, " [--%s=...]", long_options_orig[lo].name);
|
||||
} else {
|
||||
fprintf(stderr, " [--%s]", long_options_orig[lo].name);
|
||||
}
|
||||
}
|
||||
if (width + 16 >= 80) {
|
||||
fprintf(stderr, "\n ");
|
||||
width = 8;
|
||||
}
|
||||
fprintf(stderr, "\n");
|
||||
if (i == 'H') {
|
||||
exit(EXIT_SUCCESS);
|
||||
} else {
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
}
|
||||
usage(argv, i == 'H' ? EXIT_SUCCESS : EXIT_ARGS);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+623
-766
File diff suppressed because it is too large
Load Diff
+4
-1
@@ -701,7 +701,10 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
|
||||
m.strategies_json = stringify_strategies(strategies);
|
||||
|
||||
if (std::isinf(droprate)) {
|
||||
droprate = LLONG_MAX;
|
||||
// JSON has no representation for infinity, so substitute a huge
|
||||
// finite value. The cast is explicit because LLONG_MAX itself is
|
||||
// not representable as a double and rounds up to 2^63.
|
||||
droprate = (double) LLONG_MAX;
|
||||
}
|
||||
if (basezoom != maxzoom || droprate != 2.5 || retain_points_multiplier != 1) {
|
||||
m.decisions_json = std::string("{") +
|
||||
|
||||
@@ -97,12 +97,21 @@ struct mvt_value {
|
||||
long long sint_value;
|
||||
bool bool_value;
|
||||
int null_value;
|
||||
// Initializing string_value initializes the union's full width, which
|
||||
// the static_assert below checks. Setting only a narrower member (a
|
||||
// double, say) would leave the remaining bytes indeterminate, and the
|
||||
// implicit copy constructor copies the union as a whole, so those
|
||||
// bytes get read even when they aren't the member in use.
|
||||
struct {
|
||||
size_t off;
|
||||
size_t len;
|
||||
} string_value;
|
||||
} string_value = {0, 0};
|
||||
} numeric_value;
|
||||
|
||||
static_assert(sizeof(numeric_value) == sizeof(numeric_value.string_value),
|
||||
"string_value must span the whole union, since its default member "
|
||||
"initializer is what initializes the union");
|
||||
|
||||
std::string get_string_value() const {
|
||||
if (type == mvt_string) {
|
||||
return std::string(*s, numeric_value.string_value.off, numeric_value.string_value.len);
|
||||
|
||||
+65
-40
@@ -12,6 +12,7 @@
|
||||
#include "text.hpp"
|
||||
#include "read_json.hpp"
|
||||
#include "projection.hpp"
|
||||
#include "usage.hpp"
|
||||
|
||||
extern char *optarg;
|
||||
extern int optind;
|
||||
@@ -31,11 +32,63 @@ std::set<std::string> keep;
|
||||
std::set<std::string> exclude;
|
||||
std::vector<std::string> exclude_prefix;
|
||||
|
||||
static const struct option long_options[] = {
|
||||
{"Output tile", 0, 0, 0},
|
||||
{"output", required_argument, 0, 'o'},
|
||||
{"source-tile", required_argument, 0, 't'},
|
||||
{"no-tile-compression", no_argument, 0, 'd' & 0x1F},
|
||||
{"output-format", required_argument, 0, 'f' & 0x1F},
|
||||
{"pretessellate", no_argument, 0, 'p' & 0x1F},
|
||||
{"no-mlt-feature-sort", no_argument, 0, 'r' & 0x1F},
|
||||
|
||||
{"Tile resolution", 0, 0, 0},
|
||||
{"full-detail", required_argument, 0, 'd'},
|
||||
{"buffer", required_argument, 0, 'b'},
|
||||
|
||||
{"Filtering feature attributes", 0, 0, 0},
|
||||
{"include", required_argument, 0, 'y'},
|
||||
{"exclude", required_argument, 0, 'x'},
|
||||
{"exclude-prefix", required_argument, 0, 'x' & 0x1F},
|
||||
|
||||
{"Modifying feature attributes", 0, 0, 0},
|
||||
{"accumulate-attribute", required_argument, 0, 'E'},
|
||||
|
||||
{"Filtering features", 0, 0, 0},
|
||||
{"feature-filter", required_argument, 0, 'j'},
|
||||
{"feature-filter-file", required_argument, 0, 'J'},
|
||||
{"filter-points-multiplier", no_argument, 0, 'm'},
|
||||
{"deduplicate-by-id", no_argument, 0, 'i' & 0x1F},
|
||||
|
||||
{"Line and polygon simplification", 0, 0, 0},
|
||||
{"line-simplification", required_argument, 0, 'S'},
|
||||
{"tiny-polygon-size", required_argument, 0, 's' & 0x1F},
|
||||
|
||||
{"Reordering features within the tile", 0, 0, 0},
|
||||
{"preserve-input-order", no_argument, 0, 'o' & 0x1F},
|
||||
|
||||
{"", 0, 0, 0},
|
||||
{"unidecode-data", required_argument, 0, 'u' & 0x1F},
|
||||
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
// the options above, with the usage message headings removed
|
||||
static struct option real_long_options[sizeof(long_options) / sizeof(long_options[0])];
|
||||
|
||||
void usage(char **argv) {
|
||||
fprintf(stderr, "Usage: %s -o newtile.pbf.gz tile.pbf.gz oz/ox/oy nz/nx/ny\n", argv[0]);
|
||||
fprintf(stderr, "to create tile nz/nx/ny from tile oz/ox/oy\n");
|
||||
fprintf(stderr, "Usage: %s -o newtile.pbf.gz -t nz/nx/ny tile.pbf.gz oz/ox/oy tile2.pbf.gz oz2/ox2/oy2\n", argv[0]);
|
||||
fprintf(stderr, "to create tile nz/nx/ny from tiles oz/ox/oy and oz2/ox2/oy2\n");
|
||||
static const char *const forms[] = {
|
||||
"[options] tile.pbf.gz oz/ox/oy nz/nx/ny",
|
||||
"[options] --source-tile=nz/nx/ny tile.pbf.gz oz/ox/oy ...",
|
||||
NULL,
|
||||
};
|
||||
static const struct usage_required_option required[] = {
|
||||
{"output", "newtile.pbf.gz", 0},
|
||||
{NULL, NULL, 0},
|
||||
};
|
||||
|
||||
print_usage(stderr, argv[0], forms, long_options, required);
|
||||
fprintf(stderr, "\nThe tile nz/nx/ny is created from the tile or tiles oz/ox/oy that contain it.\n");
|
||||
fprintf(stderr, "In the second form, each source tile is named by a file name and a z/x/y pair.\n");
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
@@ -68,44 +121,11 @@ int main(int argc, char **argv) {
|
||||
|
||||
std::vector<input_tile> sources;
|
||||
|
||||
struct option long_options[] = {
|
||||
{"include", required_argument, 0, 'y'},
|
||||
{"exclude", required_argument, 0, 'x'},
|
||||
{"exclude-prefix", required_argument, 0, 'x' & 0x1F},
|
||||
{"full-detail", required_argument, 0, 'd'},
|
||||
{"buffer", required_argument, 0, 'b'},
|
||||
{"output", required_argument, 0, 'o'},
|
||||
{"filter-points-multiplier", no_argument, 0, 'm'},
|
||||
{"feature-filter", required_argument, 0, 'j'},
|
||||
{"feature-filter-file", required_argument, 0, 'J'},
|
||||
{"preserve-input-order", no_argument, 0, 'o' & 0x1F},
|
||||
{"accumulate-attribute", required_argument, 0, 'E'},
|
||||
{"unidecode-data", required_argument, 0, 'u' & 0x1F},
|
||||
{"line-simplification", required_argument, 0, 'S'},
|
||||
{"tiny-polygon-size", required_argument, 0, 's' & 0x1F},
|
||||
{"source-tile", required_argument, 0, 't'},
|
||||
{"no-tile-compression", no_argument, 0, 'd' & 0x1F},
|
||||
{"deduplicate-by-id", no_argument, 0, 'i' & 0x1F},
|
||||
{"output-format", required_argument, 0, 'f' & 0x1F},
|
||||
{"pretessellate", no_argument, 0, 'p' & 0x1F},
|
||||
{"no-mlt-feature-sort", no_argument, 0, 'r' & 0x1F},
|
||||
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
std::string getopt_str;
|
||||
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
|
||||
if (long_options[lo].val > ' ') {
|
||||
getopt_str.push_back(long_options[lo].val);
|
||||
|
||||
if (long_options[lo].has_arg == required_argument) {
|
||||
getopt_str.push_back(':');
|
||||
}
|
||||
}
|
||||
}
|
||||
strip_usage_headings(long_options, real_long_options);
|
||||
std::string getopt_str = getopt_string(real_long_options);
|
||||
|
||||
int option_index = 0;
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), long_options, &option_index)) != -1) {
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), real_long_options, &option_index)) != -1) {
|
||||
switch (i) {
|
||||
case 'y':
|
||||
keep.insert(optarg);
|
||||
@@ -197,6 +217,11 @@ int main(int argc, char **argv) {
|
||||
std::vector<input_tile> its;
|
||||
int nz, nx, ny;
|
||||
|
||||
if (outfile == NULL) {
|
||||
fprintf(stderr, "%s: must specify -o newtile.pbf.gz\n", argv[0]);
|
||||
usage(argv);
|
||||
}
|
||||
|
||||
if (outtile == NULL) { // single input
|
||||
if (argc - optind != 3) {
|
||||
fprintf(stderr, "Wrong number of arguments\n");
|
||||
|
||||
+10
-2
@@ -665,10 +665,18 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf, std::
|
||||
// VT_POINT extent will be calculated in write_tile from the distance between adjacent features.
|
||||
}
|
||||
|
||||
if (extent <= LLONG_MAX) {
|
||||
// Clamp before converting, since converting a double that is out of range
|
||||
// for a long long is undefined. The bounds are asymmetric: LLONG_MAX is not
|
||||
// representable as a double and rounds up to 2^63, so the upper bound has to
|
||||
// be exclusive, while LLONG_MIN is exactly -2^63 and so can be included.
|
||||
// Areas are signed, so holes that outweigh their rings can make this
|
||||
// negative.
|
||||
if (extent >= (double) LLONG_MIN && extent < (double) LLONG_MAX) {
|
||||
sf.extent = (long long) extent;
|
||||
} else if (extent < 0) {
|
||||
sf.extent = LLONG_MIN;
|
||||
} else {
|
||||
sf.extent = LLONG_MAX;
|
||||
sf.extent = LLONG_MAX; // also the NaN case
|
||||
}
|
||||
|
||||
if (sst->want_dist && sf.t == VT_POLYGON) {
|
||||
|
||||
+83
-58
@@ -45,6 +45,7 @@
|
||||
#include "geometry.hpp"
|
||||
#include "thread.hpp"
|
||||
#include "platform.hpp"
|
||||
#include "usage.hpp"
|
||||
|
||||
int pk = false;
|
||||
int pC = false;
|
||||
@@ -893,7 +894,7 @@ void *join_worker(void *v) {
|
||||
}
|
||||
|
||||
void dispatch_tasks(std::map<zxy, std::vector<std::string>> &tasks, std::vector<std::map<std::string, layermap_entry>> &layermaps, sqlite3 *outdb, const char *outdir, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, sqlite3 *db, std::set<std::string> &exclude, std::set<std::string> &include, int ifmatched, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, json_object *filter, struct tileset_reader *readers, double *minlat, double *minlon, double *maxlat, double *maxlon, double *minlon2, double *maxlon2) {
|
||||
pthread_t pthreads[CPUS];
|
||||
std::vector<pthread_t> pthreads(CPUS);
|
||||
std::vector<arg> args;
|
||||
|
||||
for (size_t i = 0; i < CPUS; i++) {
|
||||
@@ -1254,8 +1255,84 @@ void decode(struct tileset_reader *readers, std::map<std::string, layermap_entry
|
||||
}
|
||||
}
|
||||
|
||||
static const struct option long_options[] = {
|
||||
{"Output tileset", 0, 0, 0},
|
||||
{"output", required_argument, 0, 'o'},
|
||||
{"output-to-directory", required_argument, 0, 'e'},
|
||||
{"force", no_argument, 0, 'f'},
|
||||
|
||||
{"Tileset description and attribution", 0, 0, 0},
|
||||
{"name", required_argument, 0, 'n'},
|
||||
{"attribution", required_argument, 0, 'A'},
|
||||
{"description", required_argument, 0, 'N'},
|
||||
|
||||
{"Input tilesets", 0, 0, 0},
|
||||
{"read-from", required_argument, 0, 'r'},
|
||||
|
||||
{"Zoom levels", 0, 0, 0},
|
||||
{"maximum-zoom", required_argument, 0, 'z'},
|
||||
{"minimum-zoom", required_argument, 0, 'Z'},
|
||||
{"overzoom", no_argument, 0, 'O'},
|
||||
{"buffer", required_argument, 0, 'b'},
|
||||
|
||||
{"Layer names", 0, 0, 0},
|
||||
{"layer", required_argument, 0, 'l'},
|
||||
{"exclude-layer", required_argument, 0, 'L'},
|
||||
{"rename-layer", required_argument, 0, 'R'},
|
||||
|
||||
{"Joining with a CSV file", 0, 0, 0},
|
||||
{"csv", required_argument, 0, 'c'},
|
||||
{"if-matched", no_argument, 0, 'i'},
|
||||
{"empty-csv-columns-are-null", no_argument, &pe, 1},
|
||||
|
||||
{"Filtering feature attributes", 0, 0, 0},
|
||||
{"exclude", required_argument, 0, 'x'},
|
||||
{"include", required_argument, 0, 'y'},
|
||||
{"exclude-all", no_argument, 0, 'X'},
|
||||
{"exclude-all-tile-attributes", no_argument, 0, '~'},
|
||||
{"exclude-all-tile-geometries", no_argument, 0, '~'},
|
||||
|
||||
{"Filtering features by attributes", 0, 0, 0},
|
||||
{"feature-filter-file", required_argument, 0, 'J'},
|
||||
{"feature-filter", required_argument, 0, 'j'},
|
||||
|
||||
{"Setting or disabling tile size limits", 0, 0, 0},
|
||||
{"no-tile-size-limit", no_argument, &pk, 1},
|
||||
{"no-tile-compression", no_argument, &pC, 1},
|
||||
{"output-format", required_argument, 0, '~'},
|
||||
{"pretessellate", no_argument, 0, '~'},
|
||||
{"no-mlt-feature-sort", no_argument, 0, '~'},
|
||||
{"no-tile-stats", no_argument, &pg, 1},
|
||||
{"tile-stats-attributes-limit", required_argument, 0, '~'},
|
||||
{"tile-stats-sample-values-limit", required_argument, 0, '~'},
|
||||
{"tile-stats-values-limit", required_argument, 0, '~'},
|
||||
|
||||
{"Progress indicator", 0, 0, 0},
|
||||
{"quiet", no_argument, 0, 'q'},
|
||||
|
||||
{"", 0, 0, 0},
|
||||
{"prevent", required_argument, 0, 'p'},
|
||||
{"unidecode-data", required_argument, 0, '~'},
|
||||
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
// the options above, with the usage message headings removed
|
||||
static struct option real_long_options[sizeof(long_options) / sizeof(long_options[0])];
|
||||
|
||||
void usage(char **argv) {
|
||||
fprintf(stderr, "Usage: %s [-f] [-i] [-pk] [-pC] [-c joins.csv] [-X] [-x exclude ...] [-y include ...] [-r inputfile.txt ] -o new.mbtiles source.mbtiles ...\n", argv[0]);
|
||||
static const char *const forms[] = {
|
||||
"[options] source.mbtiles ...",
|
||||
"[options] --read-from=inputfile.txt",
|
||||
NULL,
|
||||
};
|
||||
static const struct usage_required_option required[] = {
|
||||
{"output", "new.mbtiles", 1},
|
||||
{"output-to-directory", "directory", 1},
|
||||
{NULL, NULL, 0},
|
||||
};
|
||||
|
||||
print_usage(stderr, argv[0], forms, long_options, required);
|
||||
exit(EXIT_ARGS);
|
||||
}
|
||||
|
||||
@@ -1299,60 +1376,8 @@ int main(int argc, char **argv) {
|
||||
|
||||
std::string set_name, set_description, set_attribution;
|
||||
|
||||
struct option long_options[] = {
|
||||
{"output", required_argument, 0, 'o'},
|
||||
{"output-to-directory", required_argument, 0, 'e'},
|
||||
{"force", no_argument, 0, 'f'},
|
||||
{"overzoom", no_argument, 0, 'O'},
|
||||
{"buffer", required_argument, 0, 'b'},
|
||||
{"if-matched", no_argument, 0, 'i'},
|
||||
{"attribution", required_argument, 0, 'A'},
|
||||
{"name", required_argument, 0, 'n'},
|
||||
{"description", required_argument, 0, 'N'},
|
||||
{"prevent", required_argument, 0, 'p'},
|
||||
{"csv", required_argument, 0, 'c'},
|
||||
{"exclude", required_argument, 0, 'x'},
|
||||
{"exclude-all", no_argument, 0, 'X'},
|
||||
{"include", required_argument, 0, 'y'},
|
||||
{"exclude-all-tile-attributes", no_argument, 0, '~'},
|
||||
{"exclude-all-tile-geometries", no_argument, 0, '~'},
|
||||
{"layer", required_argument, 0, 'l'},
|
||||
{"exclude-layer", required_argument, 0, 'L'},
|
||||
{"quiet", no_argument, 0, 'q'},
|
||||
{"maximum-zoom", required_argument, 0, 'z'},
|
||||
{"minimum-zoom", required_argument, 0, 'Z'},
|
||||
{"feature-filter-file", required_argument, 0, 'J'},
|
||||
{"feature-filter", required_argument, 0, 'j'},
|
||||
{"rename-layer", required_argument, 0, 'R'},
|
||||
{"read-from", required_argument, 0, 'r'},
|
||||
|
||||
{"use-attribute-for-id", required_argument, 0, '~'},
|
||||
|
||||
{"no-tile-size-limit", no_argument, &pk, 1},
|
||||
{"no-tile-compression", no_argument, &pC, 1},
|
||||
{"empty-csv-columns-are-null", no_argument, &pe, 1},
|
||||
{"no-tile-stats", no_argument, &pg, 1},
|
||||
{"tile-stats-attributes-limit", required_argument, 0, '~'},
|
||||
{"tile-stats-sample-values-limit", required_argument, 0, '~'},
|
||||
{"tile-stats-values-limit", required_argument, 0, '~'},
|
||||
{"unidecode-data", required_argument, 0, '~'},
|
||||
{"output-format", required_argument, 0, '~'},
|
||||
{"pretessellate", no_argument, 0, '~'},
|
||||
{"no-mlt-feature-sort", no_argument, 0, '~'},
|
||||
|
||||
{0, 0, 0, 0},
|
||||
};
|
||||
|
||||
std::string getopt_str;
|
||||
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
|
||||
if (long_options[lo].val > ' ') {
|
||||
getopt_str.push_back(long_options[lo].val);
|
||||
|
||||
if (long_options[lo].has_arg == required_argument) {
|
||||
getopt_str.push_back(':');
|
||||
}
|
||||
}
|
||||
}
|
||||
strip_usage_headings(long_options, real_long_options);
|
||||
std::string getopt_str = getopt_string(real_long_options);
|
||||
|
||||
extern int optind;
|
||||
extern char *optarg;
|
||||
@@ -1361,7 +1386,7 @@ int main(int argc, char **argv) {
|
||||
std::string commandline = format_commandline(argc, argv);
|
||||
|
||||
int option_index = 0;
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), long_options, &option_index)) != -1) {
|
||||
while ((i = getopt_long(argc, argv, getopt_str.c_str(), real_long_options, &option_index)) != -1) {
|
||||
switch (i) {
|
||||
case 0:
|
||||
break;
|
||||
@@ -1506,7 +1531,7 @@ int main(int argc, char **argv) {
|
||||
break;
|
||||
|
||||
case '~': {
|
||||
const char *opt = long_options[option_index].name;
|
||||
const char *opt = real_long_options[option_index].name;
|
||||
if (strcmp(opt, "tile-stats-attributes-limit") == 0) {
|
||||
max_tilestats_attributes = atoi(optarg);
|
||||
} else if (strcmp(opt, "tile-stats-sample-values-limit") == 0) {
|
||||
|
||||
@@ -62,9 +62,6 @@ extern "C" {
|
||||
#define COORD_OFFSET (4LL << 32)
|
||||
#define SHIFT_RIGHT(a) ((long long) std::round((double) (a) / (1LL << geometry_scale)))
|
||||
|
||||
#define XSTRINGIFY(s) STRINGIFY(s)
|
||||
#define STRINGIFY(s) #s
|
||||
|
||||
pthread_mutex_t db_lock = PTHREAD_MUTEX_INITIALIZER;
|
||||
pthread_mutex_t var_lock = PTHREAD_MUTEX_INITIALIZER;
|
||||
pthread_mutex_t task_lock = PTHREAD_MUTEX_INITIALIZER;
|
||||
@@ -1748,8 +1745,8 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
||||
|
||||
key_pool key_pool;
|
||||
|
||||
std::atomic<bool> within[child_shards];
|
||||
long long start_geompos[child_shards];
|
||||
std::vector<std::atomic<bool> > within(child_shards);
|
||||
std::vector<long long> start_geompos(child_shards);
|
||||
for (size_t i = 0; i < (size_t) child_shards; i++) {
|
||||
within[i] = false;
|
||||
start_geompos[i] = -1;
|
||||
@@ -1814,10 +1811,10 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
||||
rpa.along = along;
|
||||
rpa.alongminus = alongminus;
|
||||
rpa.buffer = buffer;
|
||||
rpa.within = within;
|
||||
rpa.within = within.data();
|
||||
rpa.geomfile = geomfile;
|
||||
rpa.geompos = geompos;
|
||||
rpa.start_geompos = start_geompos;
|
||||
rpa.start_geompos = start_geompos.data();
|
||||
rpa.oprogress = &oprogress;
|
||||
rpa.todo = todo;
|
||||
rpa.fname = fname;
|
||||
@@ -1862,7 +1859,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
||||
ssize_t which_serial_feature = -1;
|
||||
|
||||
if (prefilter == NULL) {
|
||||
sf = next_feature(geoms, geompos_in, z, tx, ty, initial_x, initial_y, &original_features, &unclipped_features, nextzoom, maxzoom, minzoom, max_zoom_increment, pass, along, alongminus, buffer, within, geomfile, geompos, start_geompos, &oprogress, todo, fname, child_shards, filter, global_stringpool, pool_off, layer_unmaps, first_time, compressed_input, &multiplier_state, tile_stringpool, unidecode_data, next_feature_state, arg->droprate);
|
||||
sf = next_feature(geoms, geompos_in, z, tx, ty, initial_x, initial_y, &original_features, &unclipped_features, nextzoom, maxzoom, minzoom, max_zoom_increment, pass, along, alongminus, buffer, within.data(), geomfile, geompos, start_geompos.data(), &oprogress, todo, fname, child_shards, filter, global_stringpool, pool_off, layer_unmaps, first_time, compressed_input, &multiplier_state, tile_stringpool, unidecode_data, next_feature_state, arg->droprate);
|
||||
} else {
|
||||
sf = parse_feature(prefilter_jp, z, tx, ty, layermaps, tiling_seg, layer_unmaps, postfilter != NULL, key_pool);
|
||||
}
|
||||
@@ -2399,7 +2396,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
||||
if (p.clustered > 0) {
|
||||
serial_val sv, sv2, sv3, sv4;
|
||||
long long point_count = p.clustered + 1;
|
||||
char abbrev[20]; // to_string(LLONG_MAX).length() / 1000 + 1;
|
||||
char abbrev[24]; // fits "%lld" of any long long, including the sign and the NUL
|
||||
|
||||
p.full_keys.push_back(key_pool.pool("clustered"));
|
||||
sv.type = mvt_bool;
|
||||
@@ -2449,7 +2446,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
||||
}
|
||||
|
||||
{
|
||||
pthread_t pthreads[tasks];
|
||||
std::vector<pthread_t> pthreads(tasks);
|
||||
std::vector<simplification_worker_arg> args;
|
||||
args.resize(tasks);
|
||||
for (int i = 0; i < tasks; i++) {
|
||||
@@ -3253,28 +3250,27 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *global_stringpool, std::
|
||||
for (z = iz; z <= maxzoom; z++) {
|
||||
std::atomic<long long> most(0);
|
||||
|
||||
compressor compressors[TEMP_FILES];
|
||||
compressor *sub[TEMP_FILES];
|
||||
std::atomic<long long> subpos[TEMP_FILES];
|
||||
int subfd[TEMP_FILES];
|
||||
std::vector<compressor> compressors(TEMP_FILES);
|
||||
std::vector<compressor *> sub(TEMP_FILES);
|
||||
std::vector<std::atomic<long long> > subpos(TEMP_FILES);
|
||||
std::vector<int> subfd(TEMP_FILES);
|
||||
for (size_t j = 0; j < TEMP_FILES; j++) {
|
||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX" XSTRINGIFY(INT_MAX)) + 1];
|
||||
snprintf(geomname, sizeof(geomname), "%s/geom%zu.XXXXXXXX", tmpdir, j);
|
||||
subfd[j] = mkstemp_cloexec(geomname);
|
||||
// printf("%s\n", geomname);
|
||||
std::string geomname = std::string(tmpdir) + "/geom" + std::to_string(j) + ".XXXXXXXX";
|
||||
subfd[j] = mkstemp_cloexec(&geomname[0]);
|
||||
// printf("%s\n", geomname.c_str());
|
||||
if (subfd[j] < 0) {
|
||||
perror(geomname);
|
||||
perror(geomname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
FILE *fp = fopen_oflag(geomname, "wb", O_WRONLY | O_CLOEXEC);
|
||||
FILE *fp = fopen_oflag(geomname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||
if (fp == NULL) {
|
||||
perror(geomname);
|
||||
perror(geomname.c_str());
|
||||
exit(EXIT_OPEN);
|
||||
}
|
||||
compressors[j] = compressor(fp);
|
||||
sub[j] = &compressors[j];
|
||||
subpos[j] = 0;
|
||||
unlink(geomname);
|
||||
unlink(geomname.c_str());
|
||||
}
|
||||
|
||||
size_t useful_threads = 0;
|
||||
@@ -3342,7 +3338,7 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *global_stringpool, std::
|
||||
std::set<zxy> skip_children_out;
|
||||
|
||||
for (size_t pass = 0;; pass++) {
|
||||
pthread_t pthreads[threads];
|
||||
std::vector<pthread_t> pthreads(threads);
|
||||
std::vector<write_tile_args> args;
|
||||
args.resize(threads);
|
||||
std::atomic<int> running(threads);
|
||||
@@ -3361,8 +3357,8 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *global_stringpool, std::
|
||||
args[thread].outdir = outdir;
|
||||
args[thread].buffer = buffer;
|
||||
args[thread].fname = fname;
|
||||
args[thread].geomfile = sub + thread * (TEMP_FILES / threads);
|
||||
args[thread].geompos = subpos + thread * (TEMP_FILES / threads);
|
||||
args[thread].geomfile = sub.data() + thread * (TEMP_FILES / threads);
|
||||
args[thread].geompos = subpos.data() + thread * (TEMP_FILES / threads);
|
||||
args[thread].todo = todo;
|
||||
args[thread].along = &along; // locked with var_lock
|
||||
args[thread].gamma = zoom_gamma;
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
#include <string.h>
|
||||
#include <set>
|
||||
#include <string>
|
||||
#include "usage.hpp"
|
||||
|
||||
// Options are wrapped to fit within this many columns
|
||||
#define USAGE_WIDTH 80
|
||||
|
||||
// The indentation of the continuation lines of the option list
|
||||
#define USAGE_INDENT 8
|
||||
|
||||
std::string getopt_string(const struct option *long_options) {
|
||||
std::string getopt_str;
|
||||
|
||||
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
|
||||
if (long_options[lo].val > ' ') {
|
||||
getopt_str.push_back(long_options[lo].val);
|
||||
|
||||
if (long_options[lo].has_arg == required_argument) {
|
||||
getopt_str.push_back(':');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return getopt_str;
|
||||
}
|
||||
|
||||
void strip_usage_headings(const struct option *long_options, struct option *real_long_options) {
|
||||
size_t out = 0;
|
||||
|
||||
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
|
||||
if (long_options[lo].val != 0) {
|
||||
real_long_options[out++] = long_options[lo];
|
||||
}
|
||||
}
|
||||
|
||||
real_long_options[out] = {0, 0, 0, 0};
|
||||
}
|
||||
|
||||
// The entry for `name` in the list of options that must be specified,
|
||||
// or NULL if it is an optional option
|
||||
static const struct usage_required_option *required_for(const char *name, const struct usage_required_option *required) {
|
||||
for (size_t i = 0; required != NULL && required[i].name != NULL; i++) {
|
||||
if (strcmp(required[i].name, name) == 0) {
|
||||
return &required[i];
|
||||
}
|
||||
}
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
// "--option", or "--option=placeholder" if the option takes an argument
|
||||
static std::string option_text(const struct option *opt, const struct usage_required_option *req) {
|
||||
std::string text = std::string("--") + opt->name;
|
||||
|
||||
if (opt->has_arg != no_argument) {
|
||||
text += "=";
|
||||
text += (req != NULL && req->placeholder != NULL) ? req->placeholder : "...";
|
||||
}
|
||||
|
||||
return text;
|
||||
}
|
||||
|
||||
// The alternatives that `req` belongs to, as "(--this=... | --that=...)"
|
||||
static std::string alternation_text(const struct option *long_options, const struct usage_required_option *required, int alternation) {
|
||||
std::string text;
|
||||
size_t found = 0;
|
||||
|
||||
for (size_t lo = 0; long_options[lo].name != NULL && long_options[lo].name[0] != '\0'; lo++) {
|
||||
const struct usage_required_option *req = required_for(long_options[lo].name, required);
|
||||
|
||||
if (req != NULL && req->alternation == alternation) {
|
||||
if (found++ > 0) {
|
||||
text += " | ";
|
||||
}
|
||||
|
||||
text += option_text(&long_options[lo], req);
|
||||
}
|
||||
}
|
||||
|
||||
if (found > 1) {
|
||||
text = "(" + text + ")";
|
||||
}
|
||||
|
||||
return text;
|
||||
}
|
||||
|
||||
void print_usage(FILE *out, const char *program, const char *const *forms,
|
||||
const struct option *long_options,
|
||||
const struct usage_required_option *required) {
|
||||
for (size_t f = 0; forms[f] != NULL; f++) {
|
||||
const char *lead = (f == 0) ? "Usage: " : "\n or: ";
|
||||
fprintf(out, "%s%s %s", lead, program, forms[f]);
|
||||
}
|
||||
|
||||
// whatever the forms took up, the option list starts on a line of its own
|
||||
size_t width = USAGE_WIDTH;
|
||||
std::set<int> alternations_listed;
|
||||
|
||||
for (size_t lo = 0; long_options[lo].name != NULL && long_options[lo].name[0] != '\0'; lo++) {
|
||||
if (long_options[lo].val == 0) {
|
||||
fprintf(out, "\n %s\n%*s", long_options[lo].name, USAGE_INDENT, "");
|
||||
width = USAGE_INDENT;
|
||||
continue;
|
||||
}
|
||||
|
||||
const struct usage_required_option *req = required_for(long_options[lo].name, required);
|
||||
std::string text;
|
||||
|
||||
if (req == NULL) {
|
||||
text = "[" + option_text(&long_options[lo], NULL) + "]";
|
||||
} else if (req->alternation == 0) {
|
||||
text = option_text(&long_options[lo], req);
|
||||
} else {
|
||||
if (alternations_listed.count(req->alternation) != 0) {
|
||||
continue; // already listed with the first of its alternatives
|
||||
}
|
||||
alternations_listed.insert(req->alternation);
|
||||
|
||||
text = alternation_text(long_options, required, req->alternation);
|
||||
}
|
||||
|
||||
if (width + 1 + text.size() >= USAGE_WIDTH) {
|
||||
fprintf(out, "\n%*s", USAGE_INDENT, "");
|
||||
width = USAGE_INDENT;
|
||||
}
|
||||
|
||||
fprintf(out, " %s", text.c_str());
|
||||
width += 1 + text.size();
|
||||
}
|
||||
|
||||
fprintf(out, "\n");
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
#ifndef USAGE_HPP
|
||||
#define USAGE_HPP
|
||||
|
||||
#include <stdio.h>
|
||||
#include <getopt.h>
|
||||
#include <string>
|
||||
|
||||
// An option that must be specified rather than being optional, and the
|
||||
// placeholder to show for its argument in the usage message.
|
||||
//
|
||||
// Options that share the same non-zero `alternation` are alternatives to
|
||||
// each other: one of them must be specified, but not more than one, and
|
||||
// they are listed together as `(--this=... | --that=...)`.
|
||||
struct usage_required_option {
|
||||
const char *name;
|
||||
const char *placeholder;
|
||||
int alternation;
|
||||
};
|
||||
|
||||
// Returns the short option string to pass to getopt_long() for the
|
||||
// options in `long_options`, so that the two can't disagree about
|
||||
// which short options exist or take arguments.
|
||||
std::string getopt_string(const struct option *long_options);
|
||||
|
||||
// Copies `long_options` to `real_long_options`, leaving out the headings
|
||||
// of the usage message, which are not real options and so must not be
|
||||
// passed on to getopt_long(). The destination must be at least as large
|
||||
// as the source.
|
||||
void strip_usage_headings(const struct option *long_options, struct option *real_long_options);
|
||||
|
||||
// Prints a usage message for `program` to `out`:
|
||||
//
|
||||
// Usage: program forms[0]
|
||||
// or: program forms[1]
|
||||
// [--some-option] [--another-option=...] ...
|
||||
//
|
||||
// where `forms` is a NULL-terminated list of the ways the non-option
|
||||
// arguments can be given, and the list of options is derived from
|
||||
// `long_options`, the same table that is passed to getopt_long(), so that
|
||||
// the message stays in sync with the options that are really accepted.
|
||||
//
|
||||
// Options named in `required` (a list terminated by a NULL name, or NULL
|
||||
// if there are none) are shown without brackets, using the placeholder
|
||||
// given there for their argument, and grouped with any alternatives to
|
||||
// them. An entry in `long_options` with no `val` is printed as a heading
|
||||
// for the options that follow it, and an entry with an empty name ends
|
||||
// the listing, hiding any options after it.
|
||||
void print_usage(FILE *out, const char *program, const char *const *forms,
|
||||
const struct option *long_options,
|
||||
const struct usage_required_option *required);
|
||||
|
||||
#endif
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
#ifndef VERSION_HPP
|
||||
#define VERSION_HPP
|
||||
|
||||
#define VERSION "v2.80.0"
|
||||
#define VERSION "v2.81.0"
|
||||
|
||||
#endif
|
||||
|
||||
Reference in New Issue
Block a user