Compare commits

...
Author SHA1 Message Date
Erica Fischer e60ca59502 Fix neglected --accumulate-attribute when binning 2024-09-18 14:35:39 -07:00
Erica Fischer d3dfe7f45d More testing to make sure cluster sizes make it all the way through 2024-09-18 12:36:38 -07:00
Erica Fischer 1f9be23aba Forgot to update this test 2024-09-18 10:49:20 -07:00
Erica Fischer 4ff265912d Make sure the accumulate test still works with a different prefix 2024-09-18 10:40:51 -07:00
Erica Fischer 59026112f6 Make the numeric accumulation prefix configurable 2024-09-18 10:36:09 -07:00
Erica Fischer 21bf44720d Fix count/mean accumulation 2024-09-17 22:15:17 -07:00
Erica Fischer 172f7382b5 Merge remote-tracking branch 'origin/main' into more-attribute-accumulation 2024-09-17 17:26:39 -07:00
Erica Fischer 19c144761b Make sure IDs make it through on the bins 2024-09-17 17:23:54 -07:00
Erica Fischer aa3ef1fbbf Add binning tests 2024-09-17 17:21:59 -07:00
Erica Fischer 060e2ee1c1 Fix a typo that broke accumulation 2024-09-17 15:16:31 -07:00
Erica Fischer 64b4a820c2 Found a flaw, now to debug 2024-09-17 14:16:58 -07:00
Erica Fischer 9a8c8df007 Checkpoint before I break everything 2024-09-17 14:04:02 -07:00
Erica Fischer 534242887d Pass bin IDs through to the output (#266)
* Pass bin IDs through to the output

* Update version and changelog
2024-09-16 17:08:56 -07:00
Erica Fischer 27a4f4ecef Handle aggregated attributes with no base attribute in the feature 2024-09-16 17:03:05 -07:00
Erica Fischer a2adb60638 Add means, but somehow I have some counts of 0 2024-09-13 12:39:53 -07:00
Erica Fischer 7a0878da9b Fix backwards conditional 2024-09-13 11:32:51 -07:00
Erica Fischer 56655e6ebf Use the same attribute accumulation logic in binning as in megatiles 2024-09-13 11:11:56 -07:00
Erica Fischer e2cc71b42f Put all that behind an --accumulate-numeric flag 2024-09-12 16:10:24 -07:00
Erica Fischer 9739f342e6 Actually do some accumulation 2024-09-12 15:33:14 -07:00
Erica Fischer 50bf0dc3be More sketching 2024-09-12 14:42:21 -07:00
Erica Fischer 88f8b95f27 More sketching 2024-09-12 14:02:39 -07:00
Erica Fischer b234cb9dac Some sort of test for how all these things interact with each other.
Automatic numeric attribute accumulation does *not* apply to attributes
that have an explicit attribute accumulator set, because the order of
operations is too messy and weird
2024-09-12 11:36:16 -07:00
Erica Fischer becfac9da9 Inching toward attribute accumulation in megatile handling 2024-09-12 11:08:21 -07:00
Erica Fischer 78b4fc19ec Quick bbox reject for bins that can't possibly intersect 2024-09-12 10:08:28 -07:00
Erica Fischer 3e3e7a5e4d Plumb bounding boxes through potential intersections 2024-09-12 09:57:39 -07:00
Erica Fischer 84f6e887a7 Another try at fixing longitude wraparound for bins (#261)
* Another try at fixing longitude wraparound for bins

* Gonna get it right this time

* Forgot to update the comment

* The filter case was not supposed to reinterpret geometry

* Copy antimeridian-crossing geometries to the other side too

* Getting closer to getting antimeridian-crossing polygons right

* Update changelog and version
2024-09-10 14:25:57 -07:00
Erica Fischer 51fcf142df Fix bad interaction between dynamic dropping and limiting by truncation (#260)
* Fix bad interaction between dynamic dropping and limiting by truncation

* Don't enforce the tile size limit here if they said not to
2024-09-05 13:39:53 -07:00
Erica Fischer 5b18eea673 Work in progress on binning features in overzoom (#258)
* Factoring out tilestats management from GeoJSON file reading

* Move code around so overzoom can link against parse_layers

* Read the file of bins

* Plumb the bins through to overzoom()

* Some zip code bins to test with

* (Currently non-functional) test of binning

* Starting to spell out the bin matching loop

* Can't flatten points, so don't flatten bins either

* More fleshing out bin traversal

* Bounding box of tile-relative mvt geometry

* Smallest enclosing tile from bbox

* Most of the bin scan

* Add point in polygon check. It crashes.

* Find the matching bins

* GDAL-style bounding boxes have eaten my brain

* Make some features to bin into

* Increment a count as features are found to be within the bins

* Fix longitude wraparound in overzoom bins

* Fix the tests

* Push off attribute copying until after bin assignment

* Carry sum of numeric attributes into the bins

* Also add mean, min, and max

* Add --calculate-feature-index since I keep needing it for testing

* Add an option to accumulate sum/mean/max/min/count of all numeric attrs

* Don't bake in tippecanoe:mean, since we redo it from sum and count

* Forgot to update this test fixture after removing tiled mean

* Update version and changelog
2024-09-05 12:06:51 -07:00
Monty Taylor f3b575ead4 Minor dockerfile optimizations (#240)
* Replace build-essential in Docker builder image

build-essential is overkill, it pulls in all of the tools one needs to
make to build debian packages, which we are not doing. Even though this
is a builder image, it still takes time to download all the extra crud.

Replace build-essential with make gcc and g++, which are all that are
needed.

* Replace dev packages in final Docker image

The dev packages are only needed in the builder image. They add header
files and docs and things that are not needed in the final image.

Removing them and replacing them with just the runtime libraries reduces
the final image size by about 25% (160M->124M) in a local build.

Additionally, just for simplicity, zlib1g is already part of base
ubuntu minimal, so we don't even have to list it.

* Remove build-essential mention from the README

Just like the in Dockerfile, people don't need all of those packages
to build directly on their system.
2024-08-28 10:35:21 -07:00
Rizki Alhamid f44230f5f5 Fix a bug when reading MultiLineString from .fgb file (#241)
fix logic
2024-08-28 10:34:33 -07:00
Erica Fischer 40bb4ff732 Be more careful to retry when the feature count is exceeded (#257)
* Clip before dealing with multiplier or filters in overzoom

* Be more careful to retry when the feature count is exceeded

* Adjust the estimated total feature count for the multiplier too

* Fix the feature count estimates, I think

* Pass build info into the version string

* Report the actual max zoom of any tiles as the metadata maxzoom

* Revert unneeded renaming to make the diff more readable

* Clean up the adjustments to tile sizes and feature counts

* Update version and changelog

* Dropping a feature into a multiplier cluster still effectively drops it

* Update changelog

* Rethink the changelog description

* Don't try to truncate zooms if we are still tiling at z18
2024-08-20 10:54:16 -07:00
Erica Fischer d891ca7289 Fix latitude bboxes for features that extend beyond the mercator plane (#254)
* Fix latitude bboxes for features that extend beyond the mercator plane

* Update some more test expectations

* Update version and changelog
2024-08-08 15:30:52 -07:00
Erica Fischer bc3ef87c3f Add --generate-variable-depth-tile-pyramid option (#251)
* Track output position at the file level instead of within each tile

* Track file position where the child tile data begins

* Add option and document its intended behavior

* Changing the detail loop to account for stopping early

* I forgot I already added an option for this

* Stop early if we can make a complete tile

* Add a test of zoom truncation with limited feature count

* Forgot to commit the actual code change

* Make room for a vertex count in the header of each serialized tile

* Estimate tile complexity; don't try truncating when unlikely to work

* Be more conservative, because ever retrying a tile is a big speed hit

* If stopping early, don't simplify or clean; leave that to overzoom

* Add tiny polygon reduction / dust to overzoom

* Don't try to stop early in the children if we dropped anything by rate

* Fflush here too before pwriting

* Don't stop early if we ended up dropping any features.

Rework the can-the-next-zoom-stop-early logic to avoid going
one zoom further than needed.

* Fix warning

* Fix warnings

* Oops, checking for the wrong expected return value

* Cleanup from adding line simplification in overzoom

* Current (wrong) behavior when combining coalescing and truncating

* Keep a list of parent tiles to skip rather than truncating

* Now the coalesced tiles in z12 get children in z13

* Don't double-count feature dropping when the zoom level is retried

* Correct README description

* Remove todo about special case below basezoom, which is accounted for

* Be a little more aggressive in drop-densest determination

* Scale tile feature limit for megatiles in the same way as byte limit

* Fully deprecate -detect-shared-borders into an alias

* Track the distances found in the douglas-peucker recursion

* Serialize and deserialize the distance with the vertices

* Revert "Serialize and deserialize the distance with the vertices"

This reverts commit 753f1b7909.

* Revert "Track the distances found in the douglas-peucker recursion"

This reverts commit e5361f8c22.

* Revert "Fully deprecate -detect-shared-borders into an alias"

This reverts commit 0698aeb766.

* Better tracking of whether we failed to make a full-detail tile

* Put a bloom filter in front of the binary search for shared nodes

* Forgot to take out this printf

* Improve dispatch of tiling tasks

* Still dispatch the biggest tasks first

* Track zoom truncation in the strategies list in the tileset metadata

* Prescan for small deltas before doing proper simplification

* Revert "Prescan for small deltas before doing proper simplification"

This reverts commit d1d8238b83.

* Update version and changelog

* Rename to --generate-variable-depth-tile-pyramid
2024-08-06 16:05:52 -07:00
Erica Fischer 50deb9ce63 Add multi-tile input to tippecanoe-overzoom (#249)
* Reviving multi-source-tile overzoom: the clip.cpp side

* Reviving multi-source-tile overzoom: the overzoom.cpp side

* Update readme

* Update version and changelog
2024-07-23 13:21:10 -07:00
Erica Fischer 47e774adc4 Improve the appearance of coalesce-densest-as-needed tiles (#247)
* Start to distinguish fixed cluster density setting from as-needed density

* Make consistent {drop,coalesce}-densest decisions between zooms

* Actually track the previous index instead of just intending to

* Clean up collinearities in coalesced features

* To determine densest, look at actual physical distance, not just index

* Don't actually need the previous index in serial_feature now

* Center of mass of one feature to most distant point of the next

* Add apologetic comment

* Wait, how did the tests pass before?

* Revert "Wait, how did the tests pass before?"

This reverts commit f73c8ee543.

* Add --maximum-string-attribute-length option

* Update version and changelog

* A little more testing to make sure
2024-07-16 13:23:23 -07:00
Drew Good 939579e5a2 Update README (#244) 2024-07-03 11:10:35 -07:00
Erica Fischer e11583df1c Fix hash collision in the string pool (#239)
* Current broken behavior

* More blatant test

* Fix hash collision in string pool

* Update version and changelog
2024-06-07 15:01:14 -07:00
Erica Fischer bb4f220678 Reduce tiling memory (#227)
* Trying to reduce memory in tiling

* Let the simplification workers go out of scope earlier

* Bail out quickly once the maximum feature count is reached

* Update changelog and version
2024-04-03 07:12:26 -07:00
Dane Springmeyer 266ef3c286 Add tippecanoe-overzoom to .gitignore (#224)
add tippecanoe-overzoom to .gitignore
2024-03-25 11:00:18 -07:00
Dane Springmeyer 1519b50791 Upgrade catch from v1.5.7 to v2.13.10 (#223)
upgrade catch from v1.5.7 to v2.13.10
2024-03-25 10:39:11 -07:00
Erica Fischer bd48ba8ea1 Fix accidental loss (at all zooms) of features with an explicit minzoom (#221)
* Fix accidental loss (at all zooms) of features with an explicit minzoom

* Try to stabilize centers and bounding boxes between architectures
2024-03-22 10:20:10 -07:00
Erica Fischer e796400377 Fix null behavior in in/ni expressions (#214)
* Fix null behavior in in/ni expressions

* Add test for null attributes in "ni" filters

* Remove unidecode from "in" expression evaluation again
2024-03-08 13:01:12 -08:00
Erica Fischer 655eccfcc3 Restore the use of unidecode for in/ni operators (#212)
* Restore the use of unidecode for in/ni operators

* Update changelog and version
2024-03-08 09:22:55 -08:00
Erica Fischer 021bb96aeb Allow non-string types to be used in "in" expressions (#211)
* Allow non-string types to be used in "in" expressions

* No unidecode smashing in "in" expressions, though!

* Update version and changelog
2024-03-04 10:16:48 -08:00
Erica Fischer 6a8f1b83d8 Fix some undefined behavior (#209)
* Fix some undefined behavior

* Avoid overflow in line simplification calculations

* Oops, missed a test

* Didn't mean to add that to the Makefile

* Revert "Revert "[ci] test in debug mode (#202)""

This reverts commit c95c328e47.

* Fix reference to out-of-scope pointer

* Fix invalid shift and out-of-bounds vector element reference

* Update changelog and version
2024-03-01 10:11:41 -08:00
Erica Fischer 312e1560a3 Stabilize feature order in overzoom (#210)
* Stabilize feature order in overzoom

* I want my sorts to be stable, please

* Revert "[ci] test in debug mode (#202)"

This reverts commit 853ada87b5.

* No need to reinitialize here
2024-02-29 10:39:10 -08:00
Pal Szabo d9a5c2eed7 highz precision values (#208) 2024-02-28 14:08:34 -08:00
Dane Springmeyer 853ada87b5 [ci] test in debug mode (#202) 2024-02-28 12:54:22 -08:00
Erica Fischer a987197eed Don't swap attributes when reducing tiny polygon dust (#207)
* Don't swap attributes when reducing tiny polygon dust

Because the dust placeholder may be misleadingly far from the feature
that contributed the most area to it

* Remove unused arguments; update changelog
2024-02-26 15:41:26 -08:00
Erica Fischer 2b6630c42c Allow features that would be dropped dynamically to become multiplier features (#199)
* Postpone tagging features as being the first of a multiplier cluster

* Upgrade some dynamically dropped features to multiplier features

* Still don't let it put more features in a cluster than is allowed

* Update tests

* Make the current multiplier cluster size per-layer

* Make sure the first non-empty-geometry in the layer is marked as primary

* Improve comments

* Update changelog and version

* Remove commented out debugging printf

* Factor out duplicated code
2024-02-15 11:34:09 -08:00
Erica Fischer 96f126dd59 FSL-style expressions can use unidecode data to smash case and diacritics (#197)
* Read unidecode data, do some plumbing of it

* More unidecode plumbing

* Do the unidecode smashing, but it doesn't seem to be working

* Ah, that's better!

* Add missing header

* And reorder the includes too

* Shortcut when there is no unidecode data to work with

* Update version and changelog

* Avoid repeated unidecode smashing of the same constant string
2024-02-13 14:18:30 -08:00
Erica Fischer 4e52cbd957 Drop or retain whole multiplier clusters when dropping as needed (#198)
* Prep to track conditions other than just "dropped" or "kept"

* Count up instead of down

* Drop or retain whole multiplier clusters based on their first feature

* Calculate a global feature dropping sequence

* Switch over to using the drop sequence for drop-fraction

* Remove unused arguments for the old drop-fraction implementation

* Fix copy-and-paste bugs, update tests

* Properly incorporate feature_minzoom into the drop sequence, I hope

* Rename drop_by to drop_sequence

* See if sorting within clusters fixes filter stability between zooms

* Remove very chatty debug print

* Update changelog and version

* Use named constants instead of numbers for feature dropping/keeping

* Add comment to explain purpose and method of bit reversal
2024-02-12 10:58:49 -08:00
Erica Fischer e2a7a409c7 Improve tiling speed (#195)
* Add a way to run tippecanoe single-threaded for profiling

* Do less work when the tilestats sample values list is already full

* Save a copy when retrieving the attribute key

* Fewer atomic operations

* Move string hashing from mbtiles to text

* Only do approximate attribute deduplication when writing tiles

* Feature dropping tests are sensitive to exact tile size

* All tile creators now create a string pool for the tile

* Features clipped away to nothing should not participate in that tile

* Revert "Only do approximate attribute deduplication when writing tiles"

This reverts commit c42b34b498.

* Also revert the related test changes

* Revert "Revert "Only do approximate attribute deduplication when writing tiles""

This reverts commit 18509876c3.

* Be more specific about the string hash function

* Use fnv1a instead of std::hash for everything

* Reduce the chance of hash collisions

* Stick a hash search on the front of the tree search in addpool

* Eliminate repeated hashing of the same string

* Switch instead of ifs in json parsing

* A few more cases to populate the hash in addpool

* Store the hash in the tree instead of recalculating

* Add explanatory comment for mysterious argument

* Fewer copies in attribute stringification

* Clean up ancient weirdness in JSON attribute stringification

* More serial_val cleanup

* Pass a serial_feature to rewrite instead of many broken-down arguments

* Get rid of the multiple geometries within `partial`

* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"

This reverts commit 6f4ab9b725.

* Goodbye, struct coalesce

* Revert "Features clipped away to nothing should not participate in that tile"

This reverts commit 124462fbdc.

* Migrating fields from partial to serial_feature

* Name reconciliation between serial_feature and partial

* Replace struct partial with an augmented serial_feature

* Fix some overzealous search-and-replace renaming

* Don't say struct so often

* Remove more of the former partial construction

* Commenting and cleaning up

* Trying again to avoid all these arguments to rewrite

* I swear I did this same thing before and it didn't work.

* More rewrite cleanup

* Exile --detect-shared-borders to its own file

* Add missing headers

* More commenting and cleanup

* More comments

* Sprinkle consts around

* Emplacing and std::moving

* More cleanup

* That shouldn't have worked after a std::move

* Don't need to allocate memory to compare keys

* Reduce use of the global string pool in tiling

* Another avoidable mvt_value construction

* Further reduction to explicit string pool passing

* These reverses are no longer optimizations

* These layernames can all be references

* Don't drag an unused layername string around with every feature

* Heed a compiler warning about potential buffer overflow

* Fix my confusion about which feature's string pool is relevant

* Avoid some unnecessary allocations in attribute accumulation

* Maybe faster serialization?

* Eliminate a comparison

* Do the same here

* Save a couple of allocations when parsing numbers in JSON

* Immediately assign features to layers instead of subdividing later

* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence

* Crunch out more duplicate attribute values when writing out the tile

* Do tilestats for tippecanoe:retain_points_multiplier_first too

* Shell filters need to be real threads, even if nothing else does

* Simplify tippecanoe_minzoom/maxzoom representation

* Update version and changelog
2024-02-07 17:50:35 -08:00
Can Durukandsnyk-bot c5c4698161 [Snyk] Security upgrade centos from 7 to centos8 (#196)
fix: Dockerfile.centos7 to reduce vulnerabilities

The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-CENTOS7-EXPAT-2408807
- https://snyk.io/vuln/SNYK-CENTOS7-EXPAT-3035862
- https://snyk.io/vuln/SNYK-CENTOS7-NSS-2308137
- https://snyk.io/vuln/SNYK-CENTOS7-NSSSYSINIT-2308141
- https://snyk.io/vuln/SNYK-CENTOS7-NSSTOOLS-2308139

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-02-01 14:38:43 -08:00
Erica Fischer 6a2bce8164 Scale the tile size limit up with the multiplier at low zooms (#192)
* Scale the tile size limit up with the multiplier at low zooms

* Add a test to demonstrate that high zoom tiles can't be extra large

* Update changelog and version

* Fail more cleanly when a tile can't be made small enough

* Guard against a cluster where the start marker has been dropped

* Look harder for a working feature interval instead of giving up

* That change to the drop-smallest logic changed a test output

* Update changelog

* Add explanatory comment
2024-01-31 10:23:05 -08:00
David Haberthür ca1aa58a11 Prettify one code line (#187) 2024-01-29 11:34:47 -08:00
Erica Fischer 7d4d264d50 Speeding up tippecanoe-overzoom (#191)
* Speed up mvt_value comparison

* Converting repetitive ifs to cases

* More conversions from ifs to cases

* Optimize the always-true filter case

* Don't convert types of attributes without accumulators

* Unordered map seems to be faster than map

* Add missing header

* Fix some warnings

* Fix the warnings better

* Avoid an int->string->int conversion

* Lazily initialize layer key and values maps when actually needed

* More switches from maps to unordered_maps

* Sure, I'll take the microoptimization

* More emplacement

* Save some copies

* Emplaces and moves

* Lazy linear scan of attributes instead of building a map

* Extra printfs, missing header

* Avoid clipping if the input and output tiles are the same

* But do clip if the tile extent is being reduced

* Make sure I'm not constructing std::strings here at runtime

* More worrying about runtime string construction

* A couple more std::moves

* Const references!

* More const references

* Another std::move

* Make the string_value of mvt_value std::optional

* Reserve storage when decoding

* Provision for different mvt_values to share a string pool

* Use the string pool when decoding

* Avoid another string construction

* Try limiting the depth of the search for duplicate attributes

* Revert "Try limiting the depth of the search for duplicate attributes"

This reverts commit 9ec94a15ff.

* Update changelog

* Fix typo noticed during code review
2024-01-29 11:33:56 -08:00
Erica Fischer cbf222754b Add --accumulate-attribute to tippecanoe-overzoom (#189)
* Starting to factor out attribute accumulation into its own file

* Continuing to factor out attribute accumulation

* Reduce duplicate code

* Plumbing the accumulate-attribute option around

* Call the attribute accumulator

* Test that accumulation works

* Add missing #includes

* Don't sort within individual multiplier clusters

Doing so throws off the spatial distribution of the low zooms

* Docs and changelog

* Add comments
2024-01-23 15:41:36 -08:00
Erica Fischer f957f30f90 Clean up internal naming related to tilestats (#190)
* Get rid of the type_and_string near-synonym for serial_val

* Rename file_keys to the more familiar tilestats

* "tas" (type_and_string) => "sv" (serial_val)

* "fk" (tile_keys) => "ts" (tilestats)

* Revert ""tas" (type_and_string) => "sv" (serial_val)"

This reverts commit 4854c57e22.

* More carefully this time: "tas" (type_and_string) => "sv" (serial_val)
2024-01-23 14:17:19 -08:00
Erica Fischer 679a0d62f2 Make feature ordering cooperate with --retain-points-multiplier (#188)
* Make feature ordering cooperate with --retain-points-multiplier

* Forgot to check in the actual code changes???

* Sort within each multiplier cluster as well as between clusters

* Correct description of behavior in changelog

* Drag original feature sequence along in megatiles for post-filter sort

* Plumb the preserve-input-order flag through overzoom

* Sort in overzoom if requested

* Use within-tile input sequence numbers, not global sequence numbers

* Documentation

* Reverse direction of search to prevent accidental skipping

* Add some comments about converting between attribute representations
2024-01-21 12:09:05 -08:00
Erica Fischer 5d92a17193 Add point retention multiplier (#179)
* Add an option to retain N times as many points as usual at each zoom

* Tests for point multipler with specified and guessed maxzooms

* Work in progress on inverse spatial ordering

* Fix inverse spatial feature order

* --reorder was depending on a feature index that wasn't being preserved

* Separate ordering by feature_minzoom from ordering inverse-spatially

* Add a test for the inverse spatial ordering

* Store the basezoom/droprate/multiplier decisions in tileset metadata

* Progress on adding filters to tippecanoe-overzoom

* Type promotion for comparison

* Look up the attribute value for ordering

* Add test of thinning and ordering features

* Plumb tippecanoe_decisions metadata through pmtiles

* Be careful not to put infinities in JSON

* Fix accidental dropping in what is meant to preserve sparse points

* Start distinguishing true, false, and null in expressions

* Most of the type conversions

* Add boolean conversions

* Literals and conjunctions

* Add filtering to tippecanoe-overzoom

* Add a test of filtering in overzoom

* Fix boolean conjunctions

* Handle the combination of cluster size and filtering

* Rework dot dropping to reconcile density threshold and multiplier

* Revert "Rework dot dropping to reconcile density threshold and multiplier"

This reverts commit f253a66382.

* Retain points by multiplier within each tile, not in global probability

* Test that intends to verify that the multiplier is reversible

* Get the test to detect the discrepancy

* Mark the start of multiplier clusters with a magic attribute

* Add string-contains

* Add in and ni operators

* Revert "Look up the attribute value for ordering"

This reverts commit 56bc73e49a.

* Revert "Type promotion for comparison"

This reverts commit 6f3256f5af.

* Make number formatting in tippecanoe_decisions consistent

* Revert "Add a test for the inverse spatial ordering"

This reverts commit c8047de9ab.

* Revert "Separate ordering by feature_minzoom from ordering inverse-spatially"

This reverts commit 35b19a223c.

* Revert "Fix inverse spatial feature order"

This reverts commit 5978ecdb44.

* Revert "Work in progress on inverse spatial ordering"

This reverts commit fdf230f632.

* Somehow missed the tests associated with that last revert

* Round-robin assign attributes to partials from across the multiplier

* Count the multiplier separately in each layer

* Fix distribution of accumulated attribute across multiplier features

* Update changelog, version, and docs

* Add "is null" and "isnt null" expressions

* Update interpretation of FSL expressions to pass the tests

* Test to assert that polygons are unaffected by the multiplier

* Clean up and comment

* Remove accidental unused case
2024-01-18 15:56:09 -08:00
Erica Fischer e8ca6c6de3 Slightly less compression makes as-needed dropping twice as fast (#182)
* Slightly less compression makes as-needed dropping twice as fast

* Update changelog
2024-01-05 11:07:14 -08:00
Erica Fischer 02e3bac2c0 Reduce memory consumption during tiling (#177)
* Drop duplicate geometries sooner in coalescing-as-needed

* Concatenate geometries before partial cleaning

* Spend less time with two feature representations in memory

* Further reduce in-memory duplication

* Remove another copy

* Don't keep duplicates in memory while coalescing

* Also clear out the mvt layer once it is no longer needed

* Update changelog
2023-12-22 14:20:21 -08:00
180 changed files with 152550 additions and 20092 deletions
+5 -2
View File
@@ -5,7 +5,10 @@ on: [push]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
version: ['Release', 'Debug']
steps:
- uses: actions/checkout@v3
- run: uname -a; make
- run: make test
- run: uname -a; BUILDTYPE=${{ matrix.version }} make
- run: make test
+1
View File
@@ -35,6 +35,7 @@ tile-join
tippecanoe-decode
tippecanoe-enumerate
tippecanoe-json-tool
tippecanoe-overzoom
unit
# Tests
+134
View File
@@ -1,3 +1,137 @@
# 2.62.2
* Pass feature ID through with bins
# 2.62.1
* More work in progress on binning point features in overzoom
# 2.62.0
* Fix another bad interaction, this time between dropping-as-needed and --limit-tile-feature-count
# 2.61.0
* Added --calculate-feature-index option
* Added "count" accumulation type to --accumulate-attribute
* Work in progress on binning of point features in overzoom. Not ready for use yet.
# 2.60.0
* Fix bad interaction between --retain-points-multiplier and stopping early when the tile feature limit is reached
* Fix another bad interaction between --retain-points-multiplier, dropping-as-needed, and variable depth tile pyramids
* Add optional BUILD_INFO string to version
* Reorder overzoom logic to clip before dealing with multiplier and filters
* When --generate-variable-depth-tile-pyramid is in use, report the actual highest zoom generated as tileset maxzoom
# 2.59.0
* Correct `antimeridian_adjusted_bounds` latitude calculation when vertices extend beyond the edge of the Mercator plane
# 2.58.0
* Add --generate-variable-depth-tile-pyramid option
* Add --line-simplification and --tiny-polygon-size options to tippecanoe-overzoom
* Adjust tile feature limit for --retain-points-multiplier
* Tune convergence rate for --coalesce-densest and --drop-densest
* Fix overreported drop and coalesce counts in strategies
# 2.57.0
* Add multi-tile input to tippecanoe-overzoom
# 2.56.0
* Rework --coalesce-densest-as-needed and --drop-densest-as-needed to look better
* Add --maximum-string-attribute-length option
# 2.55.0
* Fix hash collisions in the string pool
# 2.53.0
* Stop trying to add features to the tile after the feature limit is reached
# 2.52.0
* Fix accidental loss (at all zooms) of features that specify an explicit minzoom
# 2.51.0
* Fix null behavior in "in" expressions
* But they are back to not using unidecode
# 2.50.0
* FSL-style "in" expressions use unidecode again
# 2.49.0
* FSL-style "in" expressions now allow numeric comparisons, but they no longer use unidecode to remove diacritics.
# 2.48.0
* Fix some undefined behavior bugs, one of which results in slight changes to line simplification choices
# 2.47.0
* Stabilize feature order in tippecanoe-overzoom when --preserve-feature-order is specified but the sequence attribute is not present
# 2.46.0
* Polygon dust returns to having the attributes of the contributing feature nearest the placeholder instead of the contributing feature with the largest area.
# 2.45.0
* Adjust tile size limit with --retain-points-multiplier dynamically within each tile, to allow multiplier features at high zooms if other features are being dropped as-needed
# 2.44.0
* Add --unidecode-data option to allow case-insensitive filter comparisons of transliterated strings
# 2.43.0
* Change -fraction-as-needed feature dropping to be consistent across tiles and zoom levels, and to follow the same pattern as point dropping by zoom level
* With -as-needed feature dropping, drop or retain entire multiplier clusters instead of individual features
* Sort the features within each multiplier cluster by its retention priority, for more consistency between zoom levels in filtered feature choice
# 2.42.0
* Improve tiling speed
* Generate tilestats for the --retain-points-multiplier attributes
# 2.41.3
* Performance optimizations to tile reading, writing, and overzooming
* Automatically vary the tile size limit by zoom level to match the intended retain-points-multiplier multiplication
* Fix decompression error when bailing out because a tile can't be made small enough
* Search harder for a feature size threshold to make the tile small enough before giving up
# 2.41.2
* Add --accumulate-attribute to tippecanoe-overzoom
* Go back to ordering features within each multiplier cluster spatially, not in the order specified for tile feature order
# 2.41.1
* Make --preserve-input-order, --order-by, --order-descending-by, --order-smallest-first, and --order-largest-first cooperate with --retain-points-multiplier. The clusters will be ordered by their lead feature in the specified sequence. The other features in each cluster will continue to be physically near the lead feature, but ordered as specified within the cluster.
# 2.41.0
* Add Felt-style expression support for -j feature filters
* Add --retain-points-multiplier option
* Add tippecanoe_decisions metadata field to record basezoom, drop rate, and multiplier
* Add multiplier thinning (-m) and feature filters (-j) to tippecanoe-overzoom
# 2.40.0
* Slightly reduce compression aggressiveness to improve as-needed dropping speed
# 2.39.0
* Reduce memory usage during tiling
# 2.38.0
* Tolerate polygon rings with insuffiently many points in input
+2 -2
View File
@@ -1,7 +1,7 @@
FROM ubuntu:22.04 AS tippecanoe-builder
RUN apt-get update \
&& apt-get -y install build-essential libsqlite3-dev zlib1g-dev
&& apt-get -y install make gcc g++ libsqlite3-dev zlib1g-dev
COPY . /tmp/tippecanoe-src
WORKDIR /tmp/tippecanoe-src
@@ -13,7 +13,7 @@ CMD make test
# Using multistage build reduces the docker image size by alot by only copying the needed binaries
FROM ubuntu:22.04
RUN apt-get update \
&& apt-get -y install libsqlite3-dev zlib1g-dev \
&& apt-get -y install libsqlite3-0 \
&& rm -rf /var/lib/apt/lists/*
COPY --from=tippecanoe-builder /tmp/tippecanoe-src/tippecanoe* /usr/local/bin/
COPY --from=tippecanoe-builder /tmp/tippecanoe-src/tile-join /usr/local/bin/
+1 -1
View File
@@ -1,4 +1,4 @@
FROM centos:7
FROM centos:centos8
RUN yum install -y make sqlite-devel zlib-devel bash git gcc-c++
+169 -7
View File
@@ -1,14 +1,15 @@
PREFIX ?= /usr/local
MANDIR ?= $(PREFIX)/share/man/man1/
BUILDTYPE ?= Release
BUILD_INFO ?=
SHELL = /bin/sh
# inherit from env if set
CC := $(CC)
CXX := $(CXX)
CFLAGS := $(CFLAGS) -fPIE
CXXFLAGS := $(CXXFLAGS) -std=c++17 -fPIE
CFLAGS := $(CFLAGS) -fPIE -DBUILD_INFO=$(BUILD_INFO)
CXXFLAGS := $(CXXFLAGS) -std=c++17 -fPIE -DBUILD_INFO=$(BUILD_INFO)
LDFLAGS := $(LDFLAGS)
WARNING_FLAGS := -Wall -Wshadow -Wsign-compare -Wextra -Wunreachable-code -Wuninitialized -Wshadow
RELEASE_FLAGS := -O3 -DNDEBUG
@@ -58,7 +59,7 @@ C = $(wildcard *.c) $(wildcard *.cpp)
INCLUDES = -I/usr/local/include -I.
LIBS = -L/usr/local/lib
tippecanoe: geojson.o jsonpull/jsonpull.o tile.o pool.o mbtiles.o geometry.o projection.o memfile.o mvt.o serial.o main.o text.o dirtiles.o pmtiles_file.o plugin.o read_json.o write_json.o geobuf.o flatgeobuf.o evaluator.o geocsv.o csv.o geojson-loop.o json_logger.o visvalingam.o compression.o clip.o sort.o
tippecanoe: geojson.o jsonpull/jsonpull.o tile.o pool.o mbtiles.o geometry.o projection.o memfile.o mvt.o serial.o main.o text.o dirtiles.o pmtiles_file.o plugin.o read_json.o write_json.o geobuf.o flatgeobuf.o evaluator.o geocsv.o csv.o geojson-loop.o json_logger.o visvalingam.o compression.o clip.o sort.o attribute.o thread.o shared_borders.o
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
tippecanoe-enumerate: enumerate.o
@@ -67,16 +68,16 @@ tippecanoe-enumerate: enumerate.o
tippecanoe-decode: decode.o projection.o mvt.o write_json.o text.o jsonpull/jsonpull.o dirtiles.o pmtiles_file.o
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3
tile-join: tile-join.o projection.o mbtiles.o mvt.o memfile.o dirtiles.o jsonpull/jsonpull.o text.o evaluator.o csv.o write_json.o pmtiles_file.o clip.o
tile-join: tile-join.o projection.o mbtiles.o mvt.o memfile.o dirtiles.o jsonpull/jsonpull.o text.o evaluator.o csv.o write_json.o pmtiles_file.o clip.o attribute.o thread.o
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
tippecanoe-json-tool: jsontool.o jsonpull/jsonpull.o csv.o text.o geojson-loop.o
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
unit: unit.o text.o sort.o mvt.o
unit: unit.o text.o sort.o mvt.o projection.o clip.o attribute.o jsonpull/jsonpull.o evaluator.o
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
tippecanoe-overzoom: overzoom.o mvt.o clip.o
tippecanoe-overzoom: overzoom.o mvt.o clip.o evaluator.o jsonpull/jsonpull.o text.o attribute.o read_json.o projection.o
$(CXX) $(PG) $(LIBS) $(FINAL_FLAGS) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -lm -lz -lsqlite3 -lpthread
-include $(wildcard *.d)
@@ -91,7 +92,7 @@ clean:
rm -f ./tippecanoe ./tippecanoe-* ./tile-join ./unit *.o *.d */*.o */*.d tests/**/*.mbtiles tests/**/*.check
indent:
clang-format -i -style="{BasedOnStyle: Google, IndentWidth: 8, UseTab: Always, AllowShortIfStatementsOnASingleLine: false, ColumnLimit: 0, ContinuationIndentWidth: 8, SpaceAfterCStyleCast: true, IndentCaseLabels: false, AllowShortBlocksOnASingleLine: false, AllowShortFunctionsOnASingleLine: false, SortIncludes: false}" $(filter-out flatgeobuf.cpp,$(C)) $(H)
clang-format -i -style="{BasedOnStyle: Google, IndentWidth: 8, UseTab: Always, AllowShortIfStatementsOnASingleLine: false, ColumnLimit: 0, ContinuationIndentWidth: 8, SpaceAfterCStyleCast: true, IndentCaseLabels: false, AllowShortBlocksOnASingleLine: false, AllowShortFunctionsOnASingleLine: false, SortIncludes: false}" $(filter-out flatgeobuf.cpp,$(C)) $(H) jsonpull/*.[ch]
TESTS = $(wildcard tests/*/out/*.json)
SPACE = $(NULL) $(NULL)
@@ -278,6 +279,22 @@ overzoom-test: tippecanoe-overzoom
./tippecanoe-decode tests/pbf/13-1310-3166.pbf 13 1310 3166 > tests/pbf/13-1310-3166.pbf.json.check
cmp tests/pbf/13-1310-3166.pbf.json.check tests/pbf/13-1310-3166.pbf.json
rm tests/pbf/13-1310-3166.pbf tests/pbf/13-1310-3166.pbf.json.check
# Make sure feature order is stable
./tippecanoe-overzoom --preserve-input-order -o tests/pbf/11-327-791-out.pbf tests/pbf/11-327-791.pbf 11/327/791 11/327/791
./tippecanoe-decode tests/pbf/11-327-791.pbf 11 327 791 > tests/pbf/11-327-791.json
./tippecanoe-decode tests/pbf/11-327-791-out.pbf 11 327 791 > tests/pbf/11-327-791-out.json
cmp tests/pbf/11-327-791.json tests/pbf/11-327-791-out.json
rm tests/pbf/11-327-791.json tests/pbf/11-327-791-out.json tests/pbf/11-327-791-out.pbf
# Basic operation, multiple input form
./tippecanoe-overzoom -o tests/pbf/13-1310-3166.pbf -t 13/1310/3166 tests/pbf/11-327-791.pbf 11/327/791
./tippecanoe-decode tests/pbf/13-1310-3166.pbf 13 1310 3166 > tests/pbf/13-1310-3166.pbf.json.check
cmp tests/pbf/13-1310-3166.pbf.json.check tests/pbf/13-1310-3166.pbf.json
rm tests/pbf/13-1310-3166.pbf tests/pbf/13-1310-3166.pbf.json.check
# Multiple inputs
./tippecanoe-overzoom -o tests/pbf/13-1310-3166-ne.pbf -t 13/1310/3166 tests/pbf/11-327-791.pbf 11/327/791 tests/pbf/0-0-0.pbf 0/0/0
./tippecanoe-decode tests/pbf/13-1310-3166-ne.pbf 13 1310 3166 > tests/pbf/13-1310-3166-ne.pbf.json.check
cmp tests/pbf/13-1310-3166-ne.pbf.json.check tests/pbf/13-1310-3166-ne.pbf.json
rm tests/pbf/13-1310-3166-ne.pbf tests/pbf/13-1310-3166-ne.pbf.json.check
# Different detail and buffer, and attribute stripping
./tippecanoe-overzoom -d8 -b30 -y NAME -y name -y scalerank -o tests/pbf/13-1310-3166-8-30.pbf tests/pbf/11-327-791.pbf 11/327/791 13/1310/3166
./tippecanoe-decode tests/pbf/13-1310-3166-8-30.pbf 13 1310 3166 > tests/pbf/13-1310-3166-8-30.pbf.json.check
@@ -287,6 +304,81 @@ overzoom-test: tippecanoe-overzoom
./tippecanoe-overzoom -o tests/pbf/14-2616-6331.pbf tests/pbf/11-327-791.pbf 11/327/791 14/2616/6331
cmp tests/pbf/14-2616-6331.pbf /dev/null
rm tests/pbf/14-2616-6331.pbf
# Thinning
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
# 9 of them survive as the best of each cluster of 30
# ./tippecanoe -z1 -r30 --retain-points-multiplier 30 -f -e out.dir tests/ne_110m_populated_places/in.json
# cp out.dir/0/0/0.pbf tests/pbf/0-0-0-pop.pbf
./tippecanoe-overzoom -y NAME -m -o tests/pbf/0-0-0-pop-filtered.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
./tippecanoe-decode tests/pbf/0-0-0-pop-filtered.pbf 0 0 0 > tests/pbf/0-0-0-pop-filtered.pbf.json.check
cmp tests/pbf/0-0-0-pop-filtered.pbf.json.check tests/pbf/0-0-0-pop-filtered.pbf.json
rm tests/pbf/0-0-0-pop-filtered.pbf tests/pbf/0-0-0-pop-filtered.pbf.json.check
# Thinning with accumulation
./tippecanoe-overzoom -y NAME -m --accumulate-attribute NAME:comma -o tests/pbf/0-0-0-pop-accum.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
./tippecanoe-decode tests/pbf/0-0-0-pop-accum.pbf 0 0 0 > tests/pbf/0-0-0-pop-accum.pbf.json.check
cmp tests/pbf/0-0-0-pop-accum.pbf.json.check tests/pbf/0-0-0-pop-accum.pbf.json
rm tests/pbf/0-0-0-pop-accum.pbf tests/pbf/0-0-0-pop-accum.pbf.json.check
# Filtering
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
# 27 of them match the filter and are retained
./tippecanoe-overzoom -y NAME -j'{"*":["SCALERANK","eq",0]}' -o tests/pbf/0-0-0-pop-expr.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
./tippecanoe-decode tests/pbf/0-0-0-pop-expr.pbf 0 0 0 > tests/pbf/0-0-0-pop-expr.pbf.json.check
cmp tests/pbf/0-0-0-pop-expr.pbf.json.check tests/pbf/0-0-0-pop-expr.pbf.json
rm tests/pbf/0-0-0-pop-expr.pbf tests/pbf/0-0-0-pop-expr.pbf.json.check
# Filtering with multiplier
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
# 8 features survive into the output, from 9 clusters of 30
./tippecanoe-overzoom -y NAME -y SCALERANK -j'{"*":["SCALERANK","eq",0]}' -m -o tests/pbf/0-0-0-filter-mult.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
./tippecanoe-decode tests/pbf/0-0-0-filter-mult.pbf 0 0 0 > tests/pbf/0-0-0-filter-mult.pbf.json.check
cmp tests/pbf/0-0-0-filter-mult.pbf.json.check tests/pbf/0-0-0-filter-mult.pbf.json
rm tests/pbf/0-0-0-filter-mult.pbf tests/pbf/0-0-0-filter-mult.pbf.json.check
# Filtering with multiplier and preserve-input-order
# 243 features in the source tile tests/pbf/0-0-0-pop.pbf
./tippecanoe-overzoom -y NAME -y SCALERANK -j'{"*":["NAME","cn","e"]}' -m --preserve-input-order -o tests/pbf/0-0-0-filter-mult-order.pbf tests/pbf/0-0-0-pop.pbf 0/0/0 0/0/0
./tippecanoe-decode tests/pbf/0-0-0-filter-mult-order.pbf 0 0 0 > tests/pbf/0-0-0-filter-mult-order.pbf.json.check
cmp tests/pbf/0-0-0-filter-mult-order.pbf.json.check tests/pbf/0-0-0-filter-mult-order.pbf.json
rm tests/pbf/0-0-0-filter-mult-order.pbf tests/pbf/0-0-0-filter-mult-order.pbf.json.check
# Test that overzooming with a multiplier exactly reverses the effect of tiling with a multiplier
./tippecanoe -q -z5 --preserve-point-density-threshold 8 --retain-points-multiplier 3 -f -e tests/muni/out/out.dir tests/muni/muni.json
./tippecanoe -q -z5 --preserve-point-density-threshold 8 -f -o tests/muni/out/out.mbtiles tests/muni/muni.json
./tippecanoe-overzoom -m -o tests/muni/out/out.dir/000.pbf tests/muni/out/out.dir/0/0/0.pbf 0/0/0 0/0/0
./tippecanoe-decode tests/muni/out/out.mbtiles 0 0 0 > tests/muni/out/out.dir/direct.json
./tippecanoe-decode tests/muni/out/out.dir/000.pbf 0 0 0 > tests/muni/out/out.dir/overzoomed.json
cmp tests/muni/out/out.dir/overzoomed.json tests/muni/out/out.dir/direct.json
rm -rf tests/muni/out/out.dir tests/muni/out/out.mbtiles tests/muni/out/out.dir/overzoomed.json tests/muni/out/out.dir/direct.json
# Test filter with null attribute
./tippecanoe-overzoom -j '{"*":["name","ni",[1,5,6,9]]}' -o tests/pbf/12-2145-1391-filter1.pbf tests/pbf/12-2145-1391.pbf 12/2145/1391 12/2145/1391
./tippecanoe-decode tests/pbf/12-2145-1391-filter1.pbf 12 2145 1391 > tests/pbf/12-2145-1391-filter1.pbf.json.check
cmp tests/pbf/12-2145-1391-filter1.pbf.json.check tests/pbf/12-2145-1391-filter1.pbf.json
rm tests/pbf/12-2145-1391-filter1.pbf.json.check tests/pbf/12-2145-1391-filter1.pbf
# Test filter with null attribute in "ni" list
./tippecanoe-overzoom -j '{"*":["name","ni",[1,5,6,9,null]]}' -o tests/pbf/12-2145-1391-filter2.pbf tests/pbf/12-2145-1391.pbf 12/2145/1391 12/2145/1391
./tippecanoe-decode tests/pbf/12-2145-1391-filter2.pbf 12 2145 1391 > tests/pbf/12-2145-1391-filter2.pbf.json.check
cmp tests/pbf/12-2145-1391-filter2.pbf.json.check tests/pbf/12-2145-1391-filter2.pbf.json
rm tests/pbf/12-2145-1391-filter2.pbf.json.check tests/pbf/12-2145-1391-filter2.pbf
# Tiny polygon reduction
./tippecanoe-overzoom --line-simplification=5 --tiny-polygon-size=50 -o tests/pbf/countries-0-0-0.pbf.out tests/pbf/countries-0-0-0.pbf 0/0/0 0/0/0
./tippecanoe-decode tests/pbf/countries-0-0-0.pbf.out 0 0 0 > tests/pbf/countries-0-0-0.pbf.out.json.check
cmp tests/pbf/countries-0-0-0.pbf.out.json.check tests/pbf/countries-0-0-0.pbf.out.json
rm tests/pbf/countries-0-0-0.pbf.out tests/pbf/countries-0-0-0.pbf.out.json.check
# Binning
./tippecanoe-overzoom -o tests/pbf/bin-11-327-791.pbf.out --assign-to-bins tests/pbf/sf-zips.json tests/pbf/muni-11-327-791.pbf 11/327/791 11/327/791
./tippecanoe-decode tests/pbf/bin-11-327-791.pbf.out 11 327 791 > tests/pbf/bin-11-327-791.pbf.out.json.check
cmp tests/pbf/bin-11-327-791.pbf.out.json.check tests/pbf/bin-11-327-791.pbf.out.json
rm tests/pbf/bin-11-327-791.pbf.out.json.check tests/pbf/bin-11-327-791.pbf.out
# Binning with longitude wraparound problems
./tippecanoe-overzoom -o tests/pbf/0-0-0-pop-2-0-1.pbf.out --accumulate-numeric-attributes=tippecanoe --assign-to-bins tests/pbf/h3-2-0-1.geojson tests/pbf/0-0-0.pbf 2/0/1 2/0/1
./tippecanoe-decode tests/pbf/0-0-0-pop-2-0-1.pbf.out 2 0 1 > tests/pbf/0-0-0-pop-2-0-1.pbf.out.json.check
cmp tests/pbf/0-0-0-pop-2-0-1.pbf.out.json.check tests/pbf/0-0-0-pop-2-0-1.pbf.out.json
rm tests/pbf/0-0-0-pop-2-0-1.pbf.out tests/pbf/0-0-0-pop-2-0-1.pbf.out.json.check
./tippecanoe-overzoom -o tests/pbf/0-0-0-pop-1-1-0.pbf.out --accumulate-numeric-attributes=tippecanoe --assign-to-bins tests/pbf/h3-1-1-0.geojson tests/pbf/0-0-0.pbf 1/1/0 1/1/0
./tippecanoe-decode tests/pbf/0-0-0-pop-1-1-0.pbf.out 1 1 0 > tests/pbf/0-0-0-pop-1-1-0.pbf.out.json.check
cmp tests/pbf/0-0-0-pop-1-1-0.pbf.out.json.check tests/pbf/0-0-0-pop-1-1-0.pbf.out.json
rm tests/pbf/0-0-0-pop-1-1-0.pbf.out tests/pbf/0-0-0-pop-1-1-0.pbf.out.json.check
./tippecanoe-overzoom -o tests/pbf/0-0-0-pop-0-0-0.pbf.out --accumulate-numeric-attributes=tippecanoe --assign-to-bins tests/pbf/h3-0-0-0.geojson tests/pbf/0-0-0.pbf 0/0/0 0/0/0
./tippecanoe-decode tests/pbf/0-0-0-pop-0-0-0.pbf.out 0 0 0 > tests/pbf/0-0-0-pop-0-0-0.pbf.out.json.check
cmp tests/pbf/0-0-0-pop-0-0-0.pbf.out.json.check tests/pbf/0-0-0-pop-0-0-0.pbf.out.json
rm tests/pbf/0-0-0-pop-0-0-0.pbf.out tests/pbf/0-0-0-pop-0-0-0.pbf.out.json.check
join-test: tippecanoe tippecanoe-decode tile-join
./tippecanoe -q -f -z12 -o tests/join-population/tabblock_06001420.mbtiles -YALAND10:'Land area' -L'{"file": "tests/join-population/tabblock_06001420.json", "description": "population"}'
@@ -423,6 +515,76 @@ join-test: tippecanoe tippecanoe-decode tile-join
cmp tests/ne_110m_ocean/join/joined.mbtiles.json.check tests/ne_110m_ocean/join/joined.mbtiles.json
rm -f tests/ne_110m_ocean/join/ocean.mbtiles tests/ne_110m_ocean/join/countries.mbtiles tests/ne_110m_ocean/join/joined.mbtiles tests/ne_110m_ocean/join/joined.mbtiles.json.check
accumulate-test:
# there are 144 features with POP1950 in the original dataset
test `grep '"POP1950": [0-9]' tests/ne_110m_populated_places_nulls/in.json | wc -l` == 144
# and 99 without it
test `grep '"POP1950": null' tests/ne_110m_populated_places_nulls/in.json | wc -l` == 99
./tippecanoe -yNAME -yPOP1950 -yclustered:cluster_size -q -z3 -r1.75 -b0 -f -e tests/pbf/accum.dir --accumulate-numeric-attributes=clustered --set-attribute '{"clustered:cluster_size":1}' --accumulate-attribute '{"clustered:cluster_size":"sum"}' --retain-points-multiplier 3 tests/ne_110m_populated_places_nulls/in.json
# at this drop rate, there are 6 points at z0 that have no POP1950s clustered onto them....
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | wc -l` == 78
# 35 of which have no POP1950 at all
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'POP1950' | wc -l` == 35
# 43 of which do have POP1950
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | grep 'POP1950' | wc -l` == 43
# plus 59 that are clustered
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep 'clustered:count:POP1950' | wc -l` == 59
# the 59 clustered POP1950s have a total count of 101
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep 'clustered:count:POP1950' | sed 's/.*"clustered:count:POP1950": //' | awk '{sum += $$1} END {print sum}'` == 101
# we have already established that there are 43 bare POP1950s
# which makes a total of 144, which is the total count expected
#
# meanwhile, regular attribute accumulation.
# there are 137 features in the z0 tile, and they all have clustered:cluster_size
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep 'clustered:cluster_size' | wc -l` == 137
# there are no features that lack it.
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'clustered:cluster_size' | wc -l` == 0
# they add up to the 243 original features
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | sed 's/.*clustered:cluster_size": //' | awk '{sum += $$1} END {print sum}'` == 243
#
# on to the sums:
# in the original data set, the POP1950s that are present add up to 161590
test `grep '"POP1950": [0-9]' tests/ne_110m_populated_places_nulls/in.json | sed 's/.*"POP1950": //' | awk '{sum += $$1} END {print sum}' ` == 161590
# in the z0 tile, the clustered POP1950s add up to 113357
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep 'clustered:sum:POP1950' | sed 's/.*"clustered:sum:POP1950": //' | awk '{sum += $$1} END {print sum}'` == 113357
# and the non-clustered ones add up to 48233
test `./tippecanoe-decode -c tests/pbf/accum.dir/0/0/0.pbf 0 0 0 | grep -v 'clustered:sum:POP1950' | grep POP1950 | sed 's/.*"POP1950": //' | awk '{sum += $$1} END {print sum}'` == 48233
# which is the correct 161590
#
# OK, so do these still hold after megatile filtering?
./tippecanoe-overzoom --accumulate-numeric-attributes=clustered --accumulate-attribute '{"clustered:cluster_size":"sum"}' -m -o tests/pbf/accum-0-0-0.pbf tests/pbf/accum.dir/0/0/0.pbf 0/0/0 0/0/0
# Now there are 40 features with POP1950 clusters
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep 'clustered:count:POP1950' | wc -l` == 40
# There are 4 with bare POP1950
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | grep 'POP1950' | wc -l` == 4
# And 2 with no POP1950 at all
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep -v 'POP1950' | wc -l` == 2
# (which is the same as you get if you don't use -retain-points-multiplier when creating the tileset)
#
# the clustered and megatile-filtered POP1950s add up to 146370
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep 'clustered:sum:POP1950' | sed 's/.*"clustered:sum:POP1950": //' | awk '{sum += $$1} END {print sum}'` == 146370
# the non-clustered but megatile-filtered POP1950s add up to 15220
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | grep -v 'clustered:sum:POP1950' | grep POP1950 | sed 's/.*"POP1950": //' | awk '{sum += $$1} END {print sum}'` == 15220
# which add up to 161590 so we have the right global total
#
# the cluster sizes still add up to the 243 original features
test `./tippecanoe-decode -c tests/pbf/accum-0-0-0.pbf 0 0 0 | sed 's/.*clustered:cluster_size": //' | awk '{sum += $$1} END {print sum}'` == 243
#
# Now on to binning!
./tippecanoe-overzoom --assign-to-bins tests/pbf/h3-0-0-0.geojson --accumulate-numeric-attributes=clustered --accumulate-attribute '{"clustered:cluster_size":"sum"}' -o tests/pbf/bins-0-0-0.pbf tests/pbf/accum.dir/0/0/0.pbf 0/0/0 0/0/0
# Now there are 30 bins with POP1950 clusters
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep 'clustered:count:POP1950' | wc -l` == 44
# There are none with bare POP1950 (which is expected; we should only have summary statistics)
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep -v 'clustered:count:POP1950' | grep 'POP1950' | wc -l` == 0
# And 4 with no POP1950 at all
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep -v 'POP1950' | wc -l` == 4
#
# the clustered and megatile-filtered and binned POP1950s add up to 161590
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | grep 'clustered:sum:POP1950' | sed 's/.*"clustered:sum:POP1950": //' | awk '{sum += $$1} END {print sum}'` == 161590
# which is the right global total
# the cluster sizes still add up to the 243 original features
test `./tippecanoe-decode -c tests/pbf/bins-0-0-0.pbf 0 0 0 | sed 's/.*clustered:cluster_size": //' | awk '{sum += $$1} END {print sum}'` == 243
join-filter-test: tippecanoe tippecanoe-decode tile-join
# Comes out different from the direct tippecanoe run because null attributes are lost
./tippecanoe -q -z0 -f -o tests/feature-filter/out/all.mbtiles tests/feature-filter/in.json
+23 -6
View File
@@ -6,8 +6,11 @@ Builds [vector tilesets](https://github.com/mapbox/vector-tile-spec/) from large
This is the official home of Tippecanoe, developed and actively maintained by [Erica Fischer](https://github.com/e-n-f) at [Felt](https://felt.com).
For a self-hosted, API driven version of Tippecanoe, contact a technical sales engineer at sales@felt.com. Felt produces highly performant, automatically projected versions of your data, and utilizes a rendering engine, built on top of [MapLibre GL JS](https://github.com/maplibre/maplibre-gl-js), to style vector and raster data.
Version 2.0.0 is equivalent to [1.36.0](https://github.com/mapbox/tippecanoe/tree/1.36.0) in the original repository. Thank you Mapbox for the many years of early support.
Intent
------
@@ -34,7 +37,9 @@ Installation
The easiest way to install tippecanoe on OSX is with [Homebrew](http://brew.sh/):
```sh
$ brew install tippecanoe
```
On Ubuntu it will usually be easiest to build from the source repository:
@@ -347,6 +352,7 @@ Parallel processing will also be automatic if the input file is in FlatGeobuf fo
specified maximum zoom and to any levels added beyond that.
* `--extend-zooms-if-still-dropping-maximum=`_count_: Increase the maxzoom if features are still being dropped at that zoom level
by up to _count_ zoom levels.
* `-at` or `--generate-variable-depth-tile-pyramid`: Don't produce child tiles for any tile that should be sufficient to be overzoomed to any higher zoom level. Such tiles will be produced with maximum detail and no simplification or polygon cleaning. Tiles with point features below the basezoom or where any features have to be dropped dynamically, or which contain too many features or bytes with full detail, will be written out with normal detail and split into child tiles. Tilesets generated with this option are suitable for use only with tile servers that will find the appropriate tile to overzoom from and will simplify and clean the geometries appropriately before serving the tile.
* `-R` _zoom_`/`_x_`/`_y_ or `--one-tile=`_zoom_`/`_x_`/`_y_: Set the minzoom and maxzoom to _zoom_ and produce only
the single specified tile at that zoom level.
@@ -378,8 +384,8 @@ zoom level | precision (ft) | precision (m) | map scale
`-z18` | 1.5 in | 4 cm | 1:1250
`-z19` | 0.8 in | 2 cm | 1:600
`-z20` | 0.4 in | 1 cm | 1:300
`-z21` | 0.2 in | 0.5 cm | 1:150
`-z22` | 0.1 in | 0.25 cm | 1:75
`-z21` | 0.4 in | 1 cm | 1:300
`-z22` | 0.4 in | 1 cm | 1:300
### Tile resolution
@@ -418,10 +424,11 @@ be reduced to the maximum that can be used with the specified _maxzoom_.
* `-aI` or `--convert-stringified-ids-to-numbers`: If a feature ID is the string representation of a number, convert it to a plain number to use as the feature ID.
* `--use-attribute-for-id=`*name*: Use the attribute with the specified *name* as if it were specified as the feature ID. (If this attribute is a stringified number, you must also use `-aI` to convert it to a number.)
* `-pN` or `--single-precision`: Write double-precision numeric attribute values to tiles as single-precision to reduce tile size.
* `--maximum-string-attribute-length`=_length_: Truncate string attributes that exceed the specified length in bytes.
### Filtering features by attributes
* `-j` *filter* or `--feature-filter`=*filter*: Check features against a per-layer filter (as defined in the [Mapbox GL Style Specification](https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter)) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer `"*"` apply to all layers. The special variable `$zoom` refers to the current zoom level.
* `-j` *filter* or `--feature-filter`=*filter*: Check features against a per-layer filter (as defined in the [Mapbox GL Style Specification](https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter) or in a Felt filter specification still to be finalized) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer `"*"` apply to all layers. The special variable `$zoom` refers to the current zoom level.
* `-J` *filter-file* or `--feature-filter-file`=*filter-file*: Like `-j`, but read the filter from a file.
Example: to find the Natural Earth countries with low `scalerank` but high `LABELRANK`:
@@ -459,6 +466,7 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
If you use `-Bg`, it will guess a zoom level that will keep at most 50,000 features in the densest tile.
You can also specify a marker-width with `-Bg`*width* to allow fewer features in the densest tile to
compensate for the larger marker, or `-Bf`*number* to allow at most *number* features in the densest tile.
* `--retain-points-multiplier=`_multiple_: Retain the specified multiple of points instead of just the number of points that would ordinarily be retained by the drop rate. These can be thinned out later with the `-m` option to `tippecanoe-overzoom`. The start of each cluster is marked in the feature sequence by the `tippecanoe:retain_points_multiplier_first` attribute. The `--tile-size-limit` will also be extended at low zoom levels to allow for the multiplied features.
* `--drop-denser=`_percentage_: When dropping dots at zoom levels below the base zoom, give the specified _percentage_
preference to retaining points in sparse areas and dropping points in dense areas.
* `--limit-base-zoom-to-maximum-zoom` or `-Pb`: Limit the guessed base zoom not to exceed the maxzoom, even if this would put more than the requested number of features in a base zoom tile.
@@ -525,6 +533,7 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
* `-ag` or `--calculate-feature-density`: Add a new attribute, `tippecanoe_feature_density`, to each feature, to record how densely features are spaced in that area of the tile. You can use this attribute in the style to produce a glowing effect where points are densely packed. It can range from 0 in the sparsest areas to 255 in the densest.
* `-ai` or `--generate-ids`: Add an `id` (a feature ID, not an attribute named `id`) to each feature that does not already have one. There is currently no guarantee that the `id` added will be stable between runs or that it will not conflict with manually-assigned feature IDs. Future versions of Tippecanoe may change the mechanism for allocating IDs.
* `-aX` or `--calculate-feature-index`: Add a `tippecanoe:index` field to each feature, giving its index in the quadkey or hilbert sequence.
### Trying to correct bad source geometry
@@ -699,7 +708,7 @@ uses md2man (`gem install md2man`).
Linux:
sudo apt-get install build-essential libsqlite3-dev zlib1g-dev
sudo apt-get install gcc g++ make libsqlite3-dev zlib1g-dev
Then build:
@@ -984,13 +993,21 @@ to turn those into moderate detail tiles at high zoom levels, for the benefit of
renderers that cannot internally overzoom high-resolution tiles without losing
some of the precision. Running:
tippecanoe-overzoom -o out.mvt.gz inz/inx/iny outz/outx/outy in.mvt.gz
tippecanoe-overzoom -o out.mvt.gz in.mvt.gz inz/inx/iny outz/outx/outy
reads tile `inz/inx/iny` of `in.mvt.gz` and produces tile `outz/outx/outy` of `out.mvt.gz`.
tippecanoe-overzoom -o out.mvt.gz -t outz/outx/outy in.mvt.gz inz/inx/iny in2.mvt.gz in2z/in2x/in2y in3.mvt.gz in3z/in3x/in3y
reads tile `inz/inx/iny` of `in.mvt.gz`, tile `in2z/in2x/in2y` of `in2.mvt.gz`, and tile `in3z/in3x/in3y` of `in3.mvt.gz`,
and produces tile `outz/outx/outy` of `out.mvt.gz` from them.
### Options
* `-b` *buffer*: Set the tile buffer in the output tile (default 5)
* `-d` *detail*: Set the detail of the output tile (default 12)
* `-y` *attribute*: Retain the specified *attribute* in the output features. All attributes that are not named in a `-y` option will be removed.
* `-j` *filter*: Filter features using the same expression syntax as in tippecanoe.
* `-m`: If a tile was created with the `--retain-points-multiplier` option, thin the tile back down to its normal feature count during overzooming. The first feature from each cluster will be retained, unless `-j` is used to specify a filter, in which case the first matching filter from each cluster will be retained instead.
* `--preserve-input-order`: Restore a set of filtered features to its original input order
* `--accumulate-attribute`: Behaves as in `tippecanoe` to sum attributes from the features of a multiplier cluster that are not included in the final output. The attributes from features that are filtered away with `-j` are *not* accumulated onto the output feature.
+232
View File
@@ -0,0 +1,232 @@
#include <string>
#include <unordered_map>
#include "attribute.hpp"
#include "errors.hpp"
#include "serial.hpp"
#include "jsonpull/jsonpull.h"
#include "milo/dtoa_milo.h"
std::map<std::string, attribute_op> numeric_operations = {
{"sum", op_sum},
{"min", op_min},
{"max", op_max},
{"count", op_count},
};
void set_attribute_accum(std::unordered_map<std::string, attribute_op> &attribute_accum, std::string name, std::string type) {
attribute_op t;
if (type == "sum") {
t = op_sum;
} else if (type == "product") {
t = op_product;
} else if (type == "mean") {
t = op_mean;
} else if (type == "max") {
t = op_max;
} else if (type == "min") {
t = op_min;
} else if (type == "concat") {
t = op_concat;
} else if (type == "comma") {
t = op_comma;
} else if (type == "count") {
t = op_count;
} else {
fprintf(stderr, "Attribute method (%s) must be sum, product, mean, max, min, concat, comma, or count\n", type.c_str());
exit(EXIT_ARGS);
}
attribute_accum.insert(std::pair<std::string, attribute_op>(name, t));
}
void set_attribute_accum(std::unordered_map<std::string, attribute_op> &attribute_accum, const char *arg, char **argv) {
if (*arg == '{') {
json_pull *jp = json_begin_string(arg);
json_object *o = json_read_tree(jp);
if (o == NULL) {
fprintf(stderr, "%s: -E%s: %s\n", *argv, arg, jp->error);
exit(EXIT_JSON);
}
if (o->type != JSON_HASH) {
fprintf(stderr, "%s: -E%s: not a JSON object\n", *argv, arg);
exit(EXIT_JSON);
}
for (size_t i = 0; i < o->value.object.length; i++) {
json_object *k = o->value.object.keys[i];
json_object *v = o->value.object.values[i];
if (k->type != JSON_STRING) {
fprintf(stderr, "%s: -E%s: key %zu not a string\n", *argv, arg, i);
exit(EXIT_JSON);
}
if (v->type != JSON_STRING) {
fprintf(stderr, "%s: -E%s: value %zu not a string\n", *argv, arg, i);
exit(EXIT_JSON);
}
set_attribute_accum(attribute_accum, k->value.string.string, v->value.string.string);
}
json_free(o);
json_end(jp);
return;
}
const char *s = strchr(arg, ':');
if (s == NULL) {
fprintf(stderr, "-E%s option must be in the form -Ename:method\n", arg);
exit(EXIT_ARGS);
}
std::string name = std::string(arg, s - arg);
std::string type = std::string(s + 1);
set_attribute_accum(attribute_accum, name, type);
}
void preserve_attribute(attribute_op const &op, std::string const &key, serial_val const &val, std::vector<std::string> &full_keys, std::vector<serial_val> &full_values, std::unordered_map<std::string, accum_state> &attribute_accum_state) {
for (size_t i = 0; i < full_keys.size(); i++) {
if (key == full_keys[i]) {
switch (op) {
case op_sum:
full_values[i].s = milo::dtoa_milo(atof(full_values[i].s.c_str()) + atof(val.s.c_str()));
full_values[i].type = mvt_double;
return;
case op_product:
full_values[i].s = milo::dtoa_milo(atof(full_values[i].s.c_str()) * atof(val.s.c_str()));
full_values[i].type = mvt_double;
return;
case op_max: {
double existing = atof(full_values[i].s.c_str());
double maybe = atof(val.s.c_str());
if (maybe > existing) {
full_values[i].s = val.s.c_str();
full_values[i].type = mvt_double;
}
return;
}
case op_min: {
double existing = atof(full_values[i].s.c_str());
double maybe = atof(val.s.c_str());
if (maybe < existing) {
full_values[i].s = val.s.c_str();
full_values[i].type = mvt_double;
}
return;
}
case op_mean: {
auto state = attribute_accum_state.find(key);
if (state == attribute_accum_state.end()) {
accum_state s;
s.sum = atof(full_values[i].s.c_str()) + atof(val.s.c_str());
s.count = 2;
attribute_accum_state.insert(std::pair<std::string, accum_state>(key, s));
full_values[i].s = milo::dtoa_milo(s.sum / s.count);
full_values[i].type = mvt_double;
} else {
state->second.sum += atof(val.s.c_str());
state->second.count += 1;
full_values[i].s = milo::dtoa_milo(state->second.sum / state->second.count);
full_values[i].type = mvt_double;
}
return;
}
case op_concat:
full_values[i].s += val.s;
full_values[i].type = mvt_string;
return;
case op_comma:
full_values[i].s += std::string(",") + val.s;
full_values[i].type = mvt_string;
return;
case op_count: {
auto state = attribute_accum_state.find(key);
if (state == attribute_accum_state.end()) { // not already present
accum_state s;
s.count = 2;
attribute_accum_state.insert(std::pair<std::string, accum_state>(key, s));
full_values[i].s = std::to_string(s.count);
} else { // already present, incrementing
state->second.count += 1;
full_values[i].s = std::to_string(state->second.count);
}
return;
}
}
}
}
// not found, so we are making a new value
serial_val sv;
switch (op) {
case op_sum:
case op_product:
case op_max:
case op_min:
sv.s = val.s;
sv.type = mvt_double;
break;
case op_mean: {
auto state = attribute_accum_state.find(key);
if (state == attribute_accum_state.end()) {
accum_state s;
s.sum = atof(val.s.c_str());
s.count = 1;
attribute_accum_state.insert(std::pair<std::string, accum_state>(key, s));
sv.s = milo::dtoa_milo(s.sum / s.count);
} else {
state->second.sum += atof(val.s.c_str());
state->second.count += 1;
sv.s = milo::dtoa_milo(state->second.sum / state->second.count);
}
break;
}
case op_concat:
sv.s = val.s;
sv.type = mvt_string;
break;
case op_comma:
sv.s = val.s;
sv.type = mvt_string;
break;
case op_count: {
auto state = attribute_accum_state.find(key);
if (state == attribute_accum_state.end()) { // not already present
accum_state s;
s.count = 1;
attribute_accum_state.insert(std::pair<std::string, accum_state>(key, s));
sv.s = std::to_string(s.count);
} else { // already present, incrementing
state->second.count += 1;
sv.s = std::to_string(state->second.count);
}
sv.type = mvt_double;
break;
}
}
full_keys.push_back(key);
full_values.push_back(sv);
}
+32
View File
@@ -0,0 +1,32 @@
#ifndef ATTRIBUTE_HPP
#define ATTRIBUTE_HPP
#include <vector>
#include <unordered_map>
#include <map>
enum attribute_op {
op_sum,
op_product,
op_mean,
op_concat,
op_comma,
op_max,
op_min,
op_count,
};
struct accum_state {
double sum = 0;
double count = 0;
};
struct serial_val;
void set_attribute_accum(std::unordered_map<std::string, attribute_op> &attribute_accum, std::string name, std::string type);
void set_attribute_accum(std::unordered_map<std::string, attribute_op> &attribute_accum, const char *arg, char **argv);
void preserve_attribute(attribute_op const &op, const std::string &key, serial_val const &val, std::vector<std::string> &full_keys, std::vector<serial_val> &full_values, std::unordered_map<std::string, accum_state> &attribute_accum_state);
extern std::map<std::string, attribute_op> numeric_operations;
#endif
+16245 -8793
View File
File diff suppressed because it is too large Load Diff
+1150 -111
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -277,7 +277,7 @@ void decode(char *fname, int z, unsigned x, unsigned y, std::set<std::string> co
struct stat st;
std::vector<zxy> tiles;
char *pmtiles_map;
char *pmtiles_map = NULL;
std::vector<pmtiles::entry_zxy> entries;
bool is_pmtiles = false;
+4 -1
View File
@@ -176,7 +176,7 @@ std::vector<zxy> enumerate_dirtiles(const char *fname, int minzoom, int maxzoom)
closedir(d1);
}
std::sort(tiles.begin(), tiles.end());
std::stable_sort(tiles.begin(), tiles.end());
return tiles;
}
@@ -318,6 +318,9 @@ void dir_write_metadata(const char *outdir, const metadata &m) {
if (m.strategies_json.size() > 0) {
out(state, "strategies", m.strategies_json);
}
if (m.decisions_json.size() > 0) {
out(state, "tippecanoe_decisions", m.decisions_json);
}
out(state, "format", m.format);
out(state, "generator", m.generator);
out(state, "generator_options", m.generator_options);
+478 -41
View File
@@ -1,39 +1,217 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <map>
#include <unordered_map>
#include <functional>
#include "mvt.hpp"
#include "evaluator.hpp"
#include "errors.hpp"
#include "milo/dtoa_milo.h"
#include "text.hpp"
int compare(mvt_value one, json_object *two, bool &fail) {
if (one.type == mvt_string) {
static std::string mvt_value_to_string(mvt_value const &one, bool &fail, std::vector<std::string> const &unidecode_data) {
switch (one.type) {
case mvt_string:
if (unidecode_data.size() > 0) {
return unidecode_smash(unidecode_data, one.c_str());
} else {
return one.get_string_value();
}
case mvt_float:
return milo::dtoa_milo(one.numeric_value.float_value);
case mvt_double:
return milo::dtoa_milo(one.numeric_value.double_value);
case mvt_int:
return std::to_string(one.numeric_value.int_value);
case mvt_uint:
return std::to_string(one.numeric_value.uint_value);
case mvt_sint:
return std::to_string(one.numeric_value.sint_value);
case mvt_bool:
return one.numeric_value.bool_value ? "true" : "false";
case mvt_null:
fail = true; // null op string => null
return "";
case mvt_no_such_key:
default:
fprintf(stderr, "unhandled mvt_type %d\n", one.type);
exit(EXIT_IMPOSSIBLE);
}
}
// Alter the JSON parse tree in place to replace the original match string
// with its unidecode-smashed version.
//
// To avoid repeated re-smashings of the same JSON object, objects that have
// already been smashed are marked by setting their refcon to the unidecode data.
static void smash(std::vector<std::string> const &unidecode_data, json_object *j) {
if (j->value.string.refcon == (void *) &unidecode_data) {
return;
}
std::string s = unidecode_smash(unidecode_data, j->value.string.string);
j->value.string.string = (char *) realloc(j->value.string.string, s.size() + 1);
if (j->value.string.string == NULL) {
perror("realloc for unidecode_smash");
exit(EXIT_MEMORY);
}
strcpy(j->value.string.string, s.c_str());
j->value.string.refcon = (void *) &unidecode_data;
}
int compare_fsl(mvt_value const &one, json_object *two, bool &fail, std::vector<std::string> const &unidecode_data) {
// In FSL expressions, the attribute value is coerced to the type
// of the JSON literal value it is being compared to.
//
// If it cannot be converted, the comparison returns null
// (which is distinct from true and false but is falsy if
// it is the final output value).
if (two->type == JSON_NULL) {
fail = true; // anything op null => null
return 0;
}
if (two->type == JSON_NUMBER) {
double lhs;
switch (one.type) {
case mvt_string: {
char *endptr = NULL;
const char *s = one.c_str();
lhs = strtod(s, &endptr);
if (endptr == s) {
fail = true; // non-numeric-string op number => null
return 0;
}
} break;
case mvt_float:
lhs = one.numeric_value.float_value;
break;
case mvt_double:
lhs = one.numeric_value.double_value;
break;
case mvt_int:
lhs = one.numeric_value.int_value;
break;
case mvt_uint:
lhs = one.numeric_value.uint_value;
break;
case mvt_sint:
lhs = one.numeric_value.sint_value;
break;
case mvt_bool:
lhs = one.numeric_value.bool_value;
break;
case mvt_null:
fail = true; // null op number => null
return 0;
default:
fprintf(stderr, "unhandled mvt_type %d\n", one.type);
exit(EXIT_IMPOSSIBLE);
}
if (lhs < two->value.number.number) {
return -1;
} else if (lhs > two->value.number.number) {
return 1;
} else {
return 0;
}
}
if (two->type == JSON_STRING) {
std::string lhs = mvt_value_to_string(one, fail, unidecode_data);
if (unidecode_data.size() > 0) {
smash(unidecode_data, two);
}
return strcmp(lhs.c_str(), two->value.string.string);
}
if (two->type == JSON_TRUE || two->type == JSON_FALSE) {
bool lhs;
switch (one.type) {
case mvt_string:
lhs = one.get_string_view().size() > 0;
break;
case mvt_float:
lhs = one.numeric_value.float_value != 0;
break;
case mvt_double:
lhs = one.numeric_value.double_value != 0;
break;
case mvt_int:
lhs = one.numeric_value.int_value != 0;
break;
case mvt_uint:
lhs = one.numeric_value.uint_value != 0;
break;
case mvt_sint:
lhs = one.numeric_value.sint_value != 0;
break;
case mvt_bool:
lhs = one.numeric_value.bool_value;
break;
case mvt_null:
fail = true; // null op bool => null
return 0;
case mvt_no_such_key:
default:
fprintf(stderr, "unhandled mvt_type %d\n", one.type);
exit(EXIT_IMPOSSIBLE);
}
bool rhs = two->type == JSON_TRUE;
return lhs - rhs;
}
fprintf(stderr, "unhandled JSON type %d\n", two->type);
exit(EXIT_IMPOSSIBLE);
}
int compare(mvt_value const &one, json_object *two, bool &fail) {
switch (one.type) {
case mvt_string:
if (two->type != JSON_STRING) {
fail = true;
return false; // string vs non-string
}
return strcmp(one.string_value.c_str(), two->value.string.string);
}
return strcmp(one.c_str(), two->value.string.string);
if (one.type == mvt_double || one.type == mvt_float || one.type == mvt_int || one.type == mvt_uint || one.type == mvt_sint) {
case mvt_double:
case mvt_float:
case mvt_int:
case mvt_uint:
case mvt_sint:
if (two->type != JSON_NUMBER) {
fail = true;
return false; // number vs non-number
}
double v;
if (one.type == mvt_double) {
switch (one.type) {
case mvt_double:
v = one.numeric_value.double_value;
} else if (one.type == mvt_float) {
break;
case mvt_float:
v = one.numeric_value.float_value;
} else if (one.type == mvt_int) {
break;
case mvt_int:
v = one.numeric_value.int_value;
} else if (one.type == mvt_uint) {
break;
case mvt_uint:
v = one.numeric_value.uint_value;
} else if (one.type == mvt_sint) {
break;
case mvt_sint:
v = one.numeric_value.sint_value;
} else {
break;
case mvt_no_such_key:
default:
fprintf(stderr, "Internal error: bad mvt type %d\n", one.type);
exit(EXIT_IMPOSSIBLE);
}
@@ -45,32 +223,65 @@ int compare(mvt_value one, json_object *two, bool &fail) {
} else {
return 0;
}
}
if (one.type == mvt_bool) {
case mvt_bool:
if (two->type != JSON_TRUE && two->type != JSON_FALSE) {
fail = true;
return false; // bool vs non-bool
}
bool b = two->type != JSON_FALSE;
return one.numeric_value.bool_value > b;
}
{
bool b = two->type != JSON_FALSE;
return one.numeric_value.bool_value > b;
}
if (one.type == mvt_null) {
case mvt_null:
if (two->type != JSON_NULL) {
fail = true;
return false; // null vs non-null
}
return 0; // null equals null
case mvt_no_such_key:
default:
break;
}
fprintf(stderr, "Internal error: bad mvt type %d\n", one.type);
exit(EXIT_IMPOSSIBLE);
}
bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::set<std::string> &exclude_attributes) {
// 0: false
// 1: true
// -1: incomparable (sql null), treated as false in final output
static int eval(std::function<mvt_value(std::string const &)> feature, json_object *f, std::set<std::string> &exclude_attributes, std::vector<std::string> const &unidecode_data) {
if (f != NULL) {
if (f->type == JSON_TRUE) {
return 1;
} else if (f->type == JSON_FALSE) {
return 0;
} else if (f->type == JSON_NULL) {
return -1;
}
if (f->type == JSON_NUMBER) {
if (f->value.number.number == 0) {
return 0;
} else {
return 1;
}
}
if (f->type == JSON_STRING) {
if (f->value.string.string[0] == '\0') {
return 0;
} else {
return 1;
}
}
}
if (f == NULL || f->type != JSON_ARRAY) {
fprintf(stderr, "Filter is not an array: %s\n", json_stringify(f));
exit(EXIT_FILTER);
@@ -81,6 +292,163 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
exit(EXIT_FILTER);
}
// FSL conjunctions
if (f->value.array.length == 3 &&
f->value.array.array[1]->type == JSON_STRING &&
(strcmp(f->value.array.array[1]->value.string.string, "or") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "and") == 0)) {
int lhs;
if (f->value.array.array[0]->type == JSON_STRING) {
// if LHS of a boolean conjunction is a string, it is an attribute reference
mvt_value ff = feature(std::string(f->value.array.array[0]->value.string.string));
if (ff.type != mvt_no_such_key) {
if (ff.type == mvt_bool) {
lhs = ff.numeric_value.bool_value;
} else {
lhs = -1; // not boolean: null
}
} else {
lhs = -1; // not found: null
}
} else {
lhs = eval(feature, f->value.array.array[0], exclude_attributes, unidecode_data);
}
int rhs = eval(feature, f->value.array.array[2], exclude_attributes, unidecode_data);
if (lhs < 0 && rhs < 0) {
return -1; // null op null => null
}
if (strcmp(f->value.array.array[1]->value.string.string, "or") == 0) {
return (lhs > 0) || (rhs > 0);
} else {
return (lhs > 0) && (rhs > 0);
}
}
// FSL comparators
if (f->value.array.length == 3 &&
f->value.array.array[0]->type == JSON_STRING &&
f->value.array.array[1]->type == JSON_STRING &&
(strcmp(f->value.array.array[1]->value.string.string, "lt") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "gt") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "le") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "ge") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "eq") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "ne") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "cn") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "nc") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "in") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "ni") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "is") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "isnt") == 0 ||
false)) {
mvt_value lhs = feature(std::string(f->value.array.array[0]->value.string.string));
if (lhs.type == mvt_no_such_key) {
lhs.type = mvt_null;
}
if (f->value.array.array[2]->type == JSON_NULL && strcmp(f->value.array.array[1]->value.string.string, "is") == 0) {
return lhs.type == mvt_null; // null is null => true, anything is null => false
}
if (f->value.array.array[2]->type == JSON_NULL && strcmp(f->value.array.array[1]->value.string.string, "isnt") == 0) {
return lhs.type != mvt_null; // null isnt null => false, anything isnt null => true
}
bool fail = false;
if (f->value.array.array[2]->type == JSON_STRING &&
(strcmp(f->value.array.array[1]->value.string.string, "cn") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "nc") == 0)) {
if (lhs.type == mvt_null) {
return -1; // null cn/nc anything => null
}
std::string s = mvt_value_to_string(lhs, fail, unidecode_data);
if (fail) {
return -1; // null cn anything => false
}
bool contains;
if (unidecode_data.size() > 0) {
smash(unidecode_data, f->value.array.array[2]);
}
contains = strstr(s.c_str(), f->value.array.array[2]->value.string.string);
if (strcmp(f->value.array.array[1]->value.string.string, "cn") == 0) {
return contains;
} else {
return !contains;
}
}
if (f->value.array.array[2]->type == JSON_ARRAY &&
(strcmp(f->value.array.array[1]->value.string.string, "in") == 0 ||
strcmp(f->value.array.array[1]->value.string.string, "ni") == 0)) {
// no null check here, since null can be in or not in
bool contains = false;
for (size_t i = 0; i < f->value.array.array[2]->value.array.length; i++) {
if (lhs.type == mvt_null) {
if (f->value.array.array[2]->value.array.array[i]->type == JSON_NULL) {
contains = true; // null is null
break;
} else {
contains = false; // null isnt non-null
}
} else {
fail = false;
static std::vector<std::string> no_unidecode_data;
int cmp = compare_fsl(lhs, f->value.array.array[2]->value.array.array[i], fail, no_unidecode_data);
if (fail) {
contains = false; // non-null isnt null
} else if (cmp == 0) {
contains = true;
break;
}
}
}
if (strcmp(f->value.array.array[1]->value.string.string, "in") == 0) {
return contains;
} else {
return !contains;
}
}
if (lhs.type == mvt_null) {
return -1; // null compared to anything => null
}
int cmp = compare_fsl(lhs, f->value.array.array[2], fail, unidecode_data);
if (fail) {
return -1; // null
}
if (strcmp(f->value.array.array[1]->value.string.string, "eq") == 0) {
return cmp == 0;
}
if (strcmp(f->value.array.array[1]->value.string.string, "ne") == 0) {
return cmp != 0;
}
if (strcmp(f->value.array.array[1]->value.string.string, "gt") == 0) {
return cmp > 0;
}
if (strcmp(f->value.array.array[1]->value.string.string, "ge") == 0) {
return cmp >= 0;
}
if (strcmp(f->value.array.array[1]->value.string.string, "lt") == 0) {
return cmp < 0;
}
if (strcmp(f->value.array.array[1]->value.string.string, "le") == 0) {
return cmp <= 0;
}
fprintf(stderr, "expression fsl comparison: can't happen %s\n", f->value.array.array[1]->value.string.string);
exit(EXIT_IMPOSSIBLE);
}
if (f->value.array.array[0]->type != JSON_STRING) {
fprintf(stderr, "Filter operation is not a string: %s\n", json_stringify(f));
exit(EXIT_FILTER);
@@ -98,7 +466,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
fprintf(stderr, "\"has\" key is not a string: %s\n", json_stringify(f));
exit(EXIT_FILTER);
}
return feature.count(std::string(f->value.array.array[1]->value.string.string)) != 0;
return feature(std::string(f->value.array.array[1]->value.string.string)).type != mvt_no_such_key;
}
if (strcmp(f->value.array.array[0]->value.string.string, "!has") == 0) {
@@ -106,7 +474,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
fprintf(stderr, "\"!has\" key is not a string: %s\n", json_stringify(f));
exit(EXIT_FILTER);
}
return feature.count(std::string(f->value.array.array[1]->value.string.string)) == 0;
return feature(std::string(f->value.array.array[1]->value.string.string)).type == mvt_no_such_key;
}
}
@@ -125,8 +493,8 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
exit(EXIT_FILTER);
}
auto ff = feature.find(std::string(f->value.array.array[1]->value.string.string));
if (ff == feature.end()) {
mvt_value ff = feature(std::string(f->value.array.array[1]->value.string.string));
if (ff.type == mvt_no_such_key) {
static bool warned = false;
if (!warned) {
const char *s = json_stringify(f);
@@ -141,7 +509,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
}
bool fail = false;
int cmp = compare(ff->second, f->value.array.array[2], fail);
int cmp = compare(ff, f->value.array.array[2], fail);
if (fail) {
static bool warned = false;
@@ -192,17 +560,19 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
}
for (size_t i = 1; i < f->value.array.length; i++) {
bool out = eval(feature, f->value.array.array[i], exclude_attributes);
int out = eval(feature, f->value.array.array[i], exclude_attributes, unidecode_data);
if (strcmp(f->value.array.array[0]->value.string.string, "all") == 0) {
v = v && out;
if (!v) {
break;
}
} else {
v = v || out;
if (v) {
break;
if (out >= 0) { // nulls are ignored in boolean and/or expressions
if (strcmp(f->value.array.array[0]->value.string.string, "all") == 0) {
v = v && out;
if (!v) {
break;
}
} else {
v = v || out;
if (v) {
break;
}
}
}
}
@@ -226,8 +596,8 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
exit(EXIT_FILTER);
}
auto ff = feature.find(std::string(f->value.array.array[1]->value.string.string));
if (ff == feature.end()) {
mvt_value ff = feature(std::string(f->value.array.array[1]->value.string.string));
if (ff.type == mvt_no_such_key) {
static bool warned = false;
if (!warned) {
const char *s = json_stringify(f);
@@ -244,7 +614,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
bool found = false;
for (size_t i = 2; i < f->value.array.length; i++) {
bool fail = false;
int cmp = compare(ff->second, f->value.array.array[i], fail);
int cmp = compare(ff, f->value.array.array[i], fail);
if (fail) {
static bool warned = false;
@@ -281,7 +651,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
exit(EXIT_FILTER);
}
bool ok = eval(feature, f->value.array.array[2], exclude_attributes);
bool ok = eval(feature, f->value.array.array[2], exclude_attributes, unidecode_data) > 0;
if (!ok) {
exclude_attributes.insert(f->value.array.array[1]->value.string.string);
}
@@ -293,7 +663,7 @@ bool eval(std::map<std::string, mvt_value> const &feature, json_object *f, std::
exit(EXIT_FILTER);
}
bool evaluate(std::map<std::string, mvt_value> const &feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes) {
bool evaluate(std::function<mvt_value(std::string const &)> feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes, std::vector<std::string> const &unidecode_data) {
if (filter == NULL || filter->type != JSON_HASH) {
fprintf(stderr, "Error: filter is not a hash: %s\n", json_stringify(filter));
exit(EXIT_JSON);
@@ -304,12 +674,12 @@ bool evaluate(std::map<std::string, mvt_value> const &feature, std::string const
f = json_hash_get(filter, layer.c_str());
if (ok && f != NULL) {
ok = eval(feature, f, exclude_attributes);
ok = eval(feature, f, exclude_attributes, unidecode_data) > 0;
}
f = json_hash_get(filter, "*");
if (ok && f != NULL) {
ok = eval(feature, f, exclude_attributes);
ok = eval(feature, f, exclude_attributes, unidecode_data) > 0;
}
return ok;
@@ -346,3 +716,70 @@ json_object *parse_filter(const char *s) {
json_end(jp);
return filter;
}
bool evaluate(std::unordered_map<std::string, mvt_value> const &feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes, std::vector<std::string> const &unidecode_data) {
std::function<mvt_value(std::string const &)> getter = [&](std::string const &key) {
auto f = feature.find(key);
if (f != feature.end()) {
return f->second;
} else {
mvt_value v;
v.type = mvt_no_such_key;
v.numeric_value.null_value = 0;
return v;
}
};
return evaluate(getter, layer, filter, exclude_attributes, unidecode_data);
}
bool evaluate(mvt_feature const &feat, mvt_layer const &layer, json_object *filter, std::set<std::string> &exclude_attributes, int z, std::vector<std::string> const &unidecode_data) {
std::function<mvt_value(std::string const &)> getter = [&](std::string const &key) {
const static std::string dollar_id = "$id";
if (key == dollar_id && feat.has_id) {
mvt_value v;
v.type = mvt_uint;
v.numeric_value.uint_value = feat.id;
return v;
}
const static std::string dollar_type = "$type";
if (key == dollar_type) {
mvt_value v;
v.type = mvt_string;
if (feat.type == mvt_point) {
const static std::string point = "Point";
v.set_string_value(point);
} else if (feat.type == mvt_linestring) {
const static std::string linestring = "LineString";
v.set_string_value(linestring);
} else if (feat.type == mvt_polygon) {
const static std::string polygon = "Polygon";
v.set_string_value(polygon);
}
return v;
}
const static std::string dollar_zoom = "$zoom";
if (key == dollar_zoom) {
mvt_value v2;
v2.type = mvt_uint;
v2.numeric_value.uint_value = z;
return v2;
}
for (size_t i = 0; i + 1 < feat.tags.size(); i += 2) {
if (layer.keys[feat.tags[i]] == key) {
return layer.values[feat.tags[i + 1]];
}
}
mvt_value v;
v.type = mvt_no_such_key;
v.numeric_value.null_value = 0;
return v;
};
return evaluate(getter, layer.name, filter, exclude_attributes, unidecode_data);
}
+4 -2
View File
@@ -1,14 +1,16 @@
#ifndef EVALUATOR_HPP
#define EVALUATOR HPP
#include <map>
#include <unordered_map>
#include <string>
#include <set>
#include "jsonpull/jsonpull.h"
#include "mvt.hpp"
bool evaluate(std::map<std::string, mvt_value> const &feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes);
bool evaluate(std::unordered_map<std::string, mvt_value> const &feature, std::string const &layer, json_object *filter, std::set<std::string> &exclude_attributes, std::vector<std::string> const &unidecode_data);
json_object *parse_filter(const char *s);
json_object *read_filter(const char *fname);
bool evaluate(mvt_feature const &feat, mvt_layer const &layer, json_object *filter, std::set<std::string> &exclude_attributes, int z, std::vector<std::string> const &unidecode_data);
#endif
+9 -9
View File
@@ -7,6 +7,7 @@
#include "milo/dtoa_milo.h"
#include "main.hpp"
#include "errors.hpp"
#include "thread.hpp"
static constexpr uint8_t magicbytes[8] = { 0x66, 0x67, 0x62, 0x03, 0x66, 0x67, 0x62, 0x01 };
@@ -75,13 +76,13 @@ drawvec readGeometry(const FlatGeobuf::Geometry *geometry, FlatGeobuf::GeometryT
if (geometry_type == FlatGeobuf::GeometryType_Point) {
return readPoints(geometry);
} if (geometry_type == FlatGeobuf::GeometryType_MultiPoint) {
} else if (geometry_type == FlatGeobuf::GeometryType_MultiPoint) {
return readPoints(geometry);
} if (geometry_type == FlatGeobuf::GeometryType_LineString) {
} else if (geometry_type == FlatGeobuf::GeometryType_LineString) {
return readLinePart(geometry);
} else if (h_geometry_type == FlatGeobuf::GeometryType_MultiLineString) {
} else if (geometry_type == FlatGeobuf::GeometryType_MultiLineString) {
return readLinePart(geometry);
} if (geometry_type == FlatGeobuf::GeometryType_Polygon) {
} else if (geometry_type == FlatGeobuf::GeometryType_Polygon) {
return readLinePart(geometry);
} else if (geometry_type == FlatGeobuf::GeometryType_MultiPolygon) {
// if it is a GeometryCollection, parse Parts, ignore XY
@@ -131,7 +132,6 @@ void readFeature(const FlatGeobuf::Feature *feature, long long feature_sequence_
serial_feature sf;
sf.layer = layer;
sf.layername = layername;
sf.segment = sst->segment;
if (feature_sequence_id >= 0) {
sf.has_id = true;
@@ -139,8 +139,8 @@ void readFeature(const FlatGeobuf::Feature *feature, long long feature_sequence_
sf.has_id = false;
}
sf.id = feature_sequence_id;
sf.has_tippecanoe_minzoom = false;
sf.has_tippecanoe_maxzoom = false;
sf.tippecanoe_minzoom = -1;
sf.tippecanoe_maxzoom = -1;
sf.feature_minzoom = false;
sf.seq = (*sst->layer_seq);
sf.geometry = dv;
@@ -246,7 +246,7 @@ void readFeature(const FlatGeobuf::Feature *feature, long long feature_sequence_
sf.full_keys = full_keys;
sf.full_values = full_values;
serialize_feature(sst, sf);
serialize_feature(sst, sf, layername);
}
struct fgb_queued_feature {
@@ -303,7 +303,7 @@ void fgbRunQueue() {
}
for (size_t i = 0; i < CPUS; i++) {
if (pthread_create(&pthreads[i], NULL, fgb_run_parse_feature, &qra[i]) != 0) {
if (thread_create(&pthreads[i], NULL, fgb_run_parse_feature, &qra[i]) != 0) {
perror("pthread_create");
exit(EXIT_PTHREAD);
}
+9 -12
View File
@@ -15,6 +15,7 @@
#include "jsonpull/jsonpull.h"
#include "text.hpp"
#include "errors.hpp"
#include "thread.hpp"
#define POINT 0
#define MULTIPOINT 1
@@ -379,12 +380,11 @@ void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<s
serial_feature sf;
sf.layer = layer;
sf.layername = layername;
sf.segment = sst->segment;
sf.has_id = has_id;
sf.id = id;
sf.has_tippecanoe_minzoom = false;
sf.has_tippecanoe_maxzoom = false;
sf.tippecanoe_minzoom = -1;
sf.tippecanoe_maxzoom = -1;
sf.feature_minzoom = false;
sf.seq = *(sst->layer_seq);
sf.geometry = dv[i].dv;
@@ -400,19 +400,17 @@ void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<s
if (o != NULL) {
json_object *min = json_hash_get(o, "minzoom");
if (min != NULL && (min->type == JSON_NUMBER)) {
sf.has_tippecanoe_minzoom = true;
sf.tippecanoe_minzoom = integer_zoom(sst->fname, milo::dtoa_milo(min->value.number.number));
}
json_object *max = json_hash_get(o, "maxzoom");
if (max != NULL && (max->type == JSON_NUMBER)) {
sf.has_tippecanoe_maxzoom = true;
sf.tippecanoe_maxzoom = integer_zoom(sst->fname, milo::dtoa_milo(max->value.number.number));
}
json_object *tlayer = json_hash_get(o, "layer");
if (tlayer != NULL && (tlayer->type == JSON_STRING)) {
sf.layername = tlayer->value.string.string;
layername = tlayer->value.string.string;
}
}
@@ -420,7 +418,7 @@ void readFeature(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<s
json_end(jp);
}
serialize_feature(sst, sf);
serialize_feature(sst, sf, layername);
}
}
@@ -465,7 +463,7 @@ void runQueue() {
}
for (size_t i = 0; i < CPUS; i++) {
if (pthread_create(&pthreads[i], NULL, run_parse_feature, &qra[i]) != 0) {
if (thread_create(&pthreads[i], NULL, run_parse_feature, &qra[i]) != 0) {
perror("pthread_create");
exit(EXIT_PTHREAD);
}
@@ -506,17 +504,16 @@ void outBareGeometry(drawvec const &dv, int type, struct serialization_state *ss
serial_feature sf;
sf.layer = layer;
sf.layername = layername;
sf.segment = sst->segment;
sf.has_id = false;
sf.has_tippecanoe_minzoom = false;
sf.has_tippecanoe_maxzoom = false;
sf.tippecanoe_minzoom = -1;
sf.tippecanoe_maxzoom = -1;
sf.feature_minzoom = false;
sf.seq = (*sst->layer_seq);
sf.geometry = dv;
sf.t = type;
serialize_feature(sst, sf);
serialize_feature(sst, sf, layername);
}
void readFeatureCollection(protozero::pbf_reader &pbf, size_t dim, double e, std::vector<std::string> &keys, std::vector<struct serialization_state> *sst, int layer, std::string layername) {
+3 -4
View File
@@ -115,12 +115,11 @@ void parse_geocsv(std::vector<struct serialization_state> &sst, std::string fnam
serial_feature sf;
sf.layer = layer;
sf.layername = layername;
sf.segment = sst[0].segment;
sf.has_id = false;
sf.id = 0;
sf.has_tippecanoe_minzoom = false;
sf.has_tippecanoe_maxzoom = false;
sf.tippecanoe_minzoom = -1;
sf.tippecanoe_maxzoom = -1;
sf.feature_minzoom = false;
sf.seq = *(sst[0].layer_seq);
sf.geometry = dv;
@@ -128,7 +127,7 @@ void parse_geocsv(std::vector<struct serialization_state> &sst, std::string fnam
sf.full_keys = full_keys;
sf.full_values = full_values;
serialize_feature(&sst[0], sf);
serialize_feature(&sst[0], sf, layername);
}
if (fname.size() != 0) {
+14 -48
View File
@@ -10,7 +10,6 @@
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <string.h>
#include <fcntl.h>
#include <ctype.h>
#include <errno.h>
@@ -41,7 +40,7 @@
#include "milo/dtoa_milo.h"
#include "errors.hpp"
int serialize_geojson_feature(struct serialization_state *sst, json_object *geometry, json_object *properties, json_object *id, int layer, json_object *tippecanoe, json_object *feature, std::string layername) {
int serialize_geojson_feature(struct serialization_state *sst, json_object *geometry, json_object *properties, json_object *id, int layer, json_object *tippecanoe, json_object *feature, std::string const &layername) {
json_object *geometry_type = json_hash_get(geometry, "type");
if (geometry_type == NULL) {
static int warned = 0;
@@ -81,7 +80,7 @@ int serialize_geojson_feature(struct serialization_state *sst, json_object *geom
int tippecanoe_minzoom = -1;
int tippecanoe_maxzoom = -1;
std::string tippecanoe_layername;
std::string tippecanoe_layername = layername;
if (tippecanoe != NULL) {
json_object *min = json_hash_get(tippecanoe, "minzoom");
@@ -106,7 +105,8 @@ int serialize_geojson_feature(struct serialization_state *sst, json_object *geom
if (id->type == JSON_NUMBER) {
if (id->value.number.number >= 0) {
char *err = NULL;
id_value = strtoull(milo::dtoa_milo(id->value.number.number).c_str(), &err, 10);
std::string id_number = milo::dtoa_milo(id->value.number.number);
id_value = strtoull(id_number.c_str(), &err, 10);
if (id->value.number.large_unsigned != 0) {
id_value = id->value.number.large_unsigned;
@@ -182,36 +182,18 @@ int serialize_geojson_feature(struct serialization_state *sst, json_object *geom
nprop = properties->value.object.length;
}
std::vector<char *> metakey;
metakey.resize(nprop);
std::vector<std::string> keys;
std::vector<serial_val> values;
std::vector<std::string> metaval;
metaval.resize(nprop);
std::vector<int> metatype;
metatype.resize(nprop);
size_t m = 0;
keys.reserve(nprop);
values.reserve(nprop);
for (size_t i = 0; i < nprop; i++) {
if (properties->value.object.keys[i]->type == JSON_STRING) {
std::string s(properties->value.object.keys[i]->value.string.string);
serial_val sv = stringify_value(properties->value.object.values[i], sst->fname, sst->line, feature);
int type = -1;
std::string val;
stringify_value(properties->value.object.values[i], type, val, sst->fname, sst->line, feature);
if (type >= 0) {
metakey[m] = properties->value.object.keys[i]->value.string.string;
metatype[m] = type;
metaval[m] = val;
m++;
} else {
metakey[m] = properties->value.object.keys[i]->value.string.string;
metatype[m] = mvt_null;
metaval[m] = "null";
m++;
}
keys.emplace_back(properties->value.object.keys[i]->value.string.string);
values.push_back(std::move(sv));
}
}
@@ -224,31 +206,15 @@ int serialize_geojson_feature(struct serialization_state *sst, json_object *geom
sf.t = mb_geometry[t];
sf.has_id = has_id;
sf.id = id_value;
sf.has_tippecanoe_minzoom = (tippecanoe_minzoom != -1);
sf.tippecanoe_minzoom = tippecanoe_minzoom;
sf.has_tippecanoe_maxzoom = (tippecanoe_maxzoom != -1);
sf.tippecanoe_maxzoom = tippecanoe_maxzoom;
sf.geometry = dv;
sf.feature_minzoom = 0; // Will be filled in during index merging
sf.seq = *(sst->layer_seq);
sf.full_keys = std::move(keys);
sf.full_values = std::move(values);
if (tippecanoe_layername.size() != 0) {
sf.layername = tippecanoe_layername;
} else {
sf.layername = layername;
}
for (size_t i = 0; i < m; i++) {
sf.full_keys.push_back(metakey[i]);
serial_val sv;
sv.type = metatype[i];
sv.s = metaval[i];
sf.full_values.push_back(sv);
}
return serialize_feature(sst, sf);
return serialize_feature(sst, sf, tippecanoe_layername);
}
void check_crs(json_object *j, const char *reading) {
+65 -443
View File
@@ -1,7 +1,6 @@
#include <iostream>
#include <fstream>
#include <string>
#include <stack>
#include <vector>
#include <map>
#include <algorithm>
@@ -19,9 +18,8 @@
#include "main.hpp"
#include "options.hpp"
#include "errors.hpp"
#include "projection.hpp"
drawvec decode_geometry(char **meta, int z, unsigned tx, unsigned ty, long long *bbox, unsigned initial_x, unsigned initial_y) {
drawvec decode_geometry(const char **meta, int z, unsigned tx, unsigned ty, long long *bbox, unsigned initial_x, unsigned initial_y) {
drawvec out;
bbox[0] = LLONG_MAX;
@@ -56,18 +54,10 @@ drawvec decode_geometry(char **meta, int z, unsigned tx, unsigned ty, long long
wwy -= ty << (32 - z);
}
if (wwx < bbox[0]) {
bbox[0] = wwx;
}
if (wwy < bbox[1]) {
bbox[1] = wwy;
}
if (wwx > bbox[2]) {
bbox[2] = wwx;
}
if (wwy > bbox[3]) {
bbox[3] = wwy;
}
bbox[0] = std::min(wwx, bbox[0]);
bbox[1] = std::min(wwy, bbox[1]);
bbox[2] = std::max(wwx, bbox[2]);
bbox[3] = std::max(wwy, bbox[3]);
d.x = wwx;
d.y = wwy;
@@ -79,28 +69,6 @@ drawvec decode_geometry(char **meta, int z, unsigned tx, unsigned ty, long long
return out;
}
/* pnpoly:
Copyright (c) 1970-2003, Wm. Randolph Franklin
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimers.
Redistributions in binary form must reproduce the above copyright notice in the documentation and/or other materials provided with the distribution.
The name of W. Randolph Franklin may not be used to endorse or promote products derived from this Software without specific prior written permission.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
int pnpoly(const drawvec &vert, size_t start, size_t nvert, long long testx, long long testy) {
size_t i, j;
bool c = false;
for (i = 0, j = nvert - 1; i < nvert; j = i++) {
if (((vert[i + start].y > testy) != (vert[j + start].y > testy)) &&
(testx < (vert[j + start].x - vert[i + start].x) * (testy - vert[i + start].y) / (double) (vert[j + start].y - vert[i + start].y) + vert[i + start].x))
c = !c;
}
return c;
}
void check_polygon(drawvec &geom) {
geom = remove_noop(geom, VT_POLYGON, 0);
@@ -118,13 +86,13 @@ void check_polygon(drawvec &geom) {
mapbox::geometry::linear_ring<long long> lr;
for (size_t k = i; k < j; k++) {
lr.push_back(mapbox::geometry::point<long long>(geom[k].x, geom[k].y));
lr.push_back({geom[k].x, geom[k].y});
}
if (lr.size() >= 3) {
mapbox::geometry::polygon<long long> p;
p.push_back(lr);
mp.push_back(p);
p.push_back(std::move(lr));
mp.push_back(std::move(p));
}
}
@@ -167,7 +135,7 @@ void check_polygon(drawvec &geom) {
}
if (!on_edge) {
fprintf(stderr, "%lld,%lld at %lld not in outer ring (%lld to %lld)\n", geom[k].x, geom[k].y, (long long) k, (long long) outer_start, (long long) (outer_start + outer_len));
fprintf(stderr, "%lld,%lld at %lld not in outer ring (%lld to %lld)\n", (long long) geom[k].x, (long long) geom[k].y, (long long) k, (long long) outer_start, (long long) (outer_start + outer_len));
}
}
}
@@ -176,127 +144,7 @@ void check_polygon(drawvec &geom) {
}
}
drawvec reduce_tiny_poly(drawvec &geom, int z, int detail, bool *still_needs_simplification, bool *reduced_away, double *accum_area, serial_feature *this_feature, serial_feature *tiny_feature) {
drawvec out;
const double pixel = (1LL << (32 - detail - z)) * (double) tiny_polygon_size;
bool includes_real = false;
bool includes_dust = false;
bool included_last_outer = false;
*still_needs_simplification = false;
*reduced_away = false;
for (size_t i = 0; i < geom.size(); i++) {
if (geom[i].op == VT_MOVETO) {
size_t j;
for (j = i + 1; j < geom.size(); j++) {
if (geom[j].op != VT_LINETO) {
break;
}
}
double area = get_area(geom, i, j);
// XXX There is an ambiguity here: If the area of a ring is 0 and it is followed by holes,
// we don't know whether the area-0 ring was a hole too or whether it was the outer ring
// that these subsequent holes are somehow being subtracted from. I hope that if a polygon
// was simplified down to nothing, its holes also became nothing.
if (area != 0) {
// These are pixel coordinates, so area > 0 for the outer ring.
// If the outer ring of a polygon was reduced to a pixel, its
// inner rings must just have their area de-accumulated rather
// than being drawn since we don't really know where they are.
// i.e., this outer ring is small enough that we are including it
// in a tiny polygon rather than letting it represent itself,
// OR it is an inner ring and we haven't output an outer ring for it to be
// cut out of, so we are just subtracting its area from the tiny polygon
// rather than trying to deal with it geometrically
if ((area > 0 && area <= pixel * pixel) || (area < 0 && !included_last_outer)) {
*accum_area += area;
*reduced_away = true;
if (area > 0 && *accum_area > pixel * pixel) {
// XXX use centroid;
out.push_back(draw(VT_MOVETO, geom[i].x - pixel / 2, geom[i].y - pixel / 2));
out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2 + pixel, geom[i].y - pixel / 2));
out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2 + pixel, geom[i].y - pixel / 2 + pixel));
out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2, geom[i].y - pixel / 2 + pixel));
out.push_back(draw(VT_LINETO, geom[i].x - pixel / 2, geom[i].y - pixel / 2));
includes_dust = true;
*accum_area -= pixel * pixel;
}
if (area > 0) {
included_last_outer = false;
}
}
// i.e., this ring is large enough that it gets to represent itself
// or it is a tiny hole out of a real polygon, which we are still treating
// as a real geometry because otherwise we can accumulate enough tiny holes
// that we will drop the next several outer rings getting back up to 0.
else {
for (size_t k = i; k < j && k < geom.size(); k++) {
out.push_back(geom[k]);
}
// which means that the overall polygon has a real geometry,
// which means that it gets to be simplified.
*still_needs_simplification = true;
includes_real = true;
if (area > 0) {
included_last_outer = true;
}
}
} else {
// area is 0: doesn't count as either having been reduced away,
// since it was probably just degenerate from having been clipped,
// or as needing simplification, since it produces no output.
}
i = j - 1;
} else {
fprintf(stderr, "how did we get here with %d in %d?\n", geom[i].op, (int) geom.size());
for (size_t n = 0; n < geom.size(); n++) {
fprintf(stderr, "%d/%lld/%lld ", geom[n].op, geom[n].x, geom[n].y);
}
fprintf(stderr, "\n");
out.push_back(geom[i]);
includes_real = true;
}
}
if (!includes_real) {
if (includes_dust) {
// this geometry is just dust, so if there is another feature that
// contributed to the dust that is larger than this feature,
// keep its attributes instead of this one that just happened to be
// the one that hit the threshold of survival.
if (tiny_feature->extent > this_feature->extent) {
*this_feature = *tiny_feature;
tiny_feature->extent = 0;
}
} else {
// this is a feature that we are throwing away, so hang on to it
// attributes if it is bigger than the biggest one we threw away so far
if (this_feature->extent > tiny_feature->extent) {
*tiny_feature = *this_feature;
}
}
}
return out;
}
int quick_check(long long *bbox, int z, long long buffer) {
int quick_check(const long long *bbox, int z, long long buffer) {
long long min = 0;
long long area = 1LL << (32 - z);
@@ -334,127 +182,10 @@ bool point_within_tile(long long x, long long y, int z) {
return x >= 0 && y >= 0 && x < area && y < area;
}
double distance_from_line(long long point_x, long long point_y, long long segA_x, long long segA_y, long long segB_x, long long segB_y) {
long long p2x = segB_x - segA_x;
long long p2y = segB_y - segA_y;
double something = p2x * p2x + p2y * p2y;
double u = (0 == something) ? 0 : ((point_x - segA_x) * p2x + (point_y - segA_y) * p2y) / (something);
if (u >= 1) {
u = 1;
} else if (u <= 0) {
u = 0;
}
double x = segA_x + u * p2x;
double y = segA_y + u * p2y;
double dx = x - point_x;
double dy = y - point_y;
double out = std::round(sqrt(dx * dx + dy * dy) * 16.0) / 16.0;
return out;
}
// https://github.com/Project-OSRM/osrm-backend/blob/733d1384a40f/Algorithms/DouglasePeucker.cpp
static void douglas_peucker(drawvec &geom, int start, int n, double e, size_t kept, size_t retain) {
std::stack<int> recursion_stack;
if (!geom[start + 0].necessary || !geom[start + n - 1].necessary) {
fprintf(stderr, "endpoints not marked necessary\n");
exit(EXIT_IMPOSSIBLE);
}
int prev = 0;
for (int here = 1; here < n; here++) {
if (geom[start + here].necessary) {
recursion_stack.push(prev);
recursion_stack.push(here);
prev = here;
if (prevent[P_SIMPLIFY_SHARED_NODES]) {
if (retain > 0) {
retain--;
}
}
}
}
// These segments are put on the stack from start to end,
// independent of winding, so note that anything that uses
// "retain" to force it to keep at least N points will
// keep a different set of points when wound one way than
// when wound the other way.
while (!recursion_stack.empty()) {
// pop next element
int second = recursion_stack.top();
recursion_stack.pop();
int first = recursion_stack.top();
recursion_stack.pop();
double max_distance = -1;
int farthest_element_index;
// find index idx of element with max_distance
int i;
if (geom[start + first] < geom[start + second]) {
farthest_element_index = first;
for (i = first + 1; i < second; i++) {
double temp_dist = distance_from_line(geom[start + i].x, geom[start + i].y, geom[start + first].x, geom[start + first].y, geom[start + second].x, geom[start + second].y);
double distance = std::fabs(temp_dist);
if ((distance > e || kept < retain) && (distance > max_distance || (distance == max_distance && geom[start + i] < geom[start + farthest_element_index]))) {
farthest_element_index = i;
max_distance = distance;
}
}
} else {
farthest_element_index = second;
for (i = second - 1; i > first; i--) {
double temp_dist = distance_from_line(geom[start + i].x, geom[start + i].y, geom[start + second].x, geom[start + second].y, geom[start + first].x, geom[start + first].y);
double distance = std::fabs(temp_dist);
if ((distance > e || kept < retain) && (distance > max_distance || (distance == max_distance && geom[start + i] < geom[start + farthest_element_index]))) {
farthest_element_index = i;
max_distance = distance;
}
}
}
if (max_distance >= 0) {
// mark idx as necessary
geom[start + farthest_element_index].necessary = 1;
kept++;
if (geom[start + first] < geom[start + second]) {
if (1 < farthest_element_index - first) {
recursion_stack.push(first);
recursion_stack.push(farthest_element_index);
}
if (1 < second - farthest_element_index) {
recursion_stack.push(farthest_element_index);
recursion_stack.push(second);
}
} else {
if (1 < second - farthest_element_index) {
recursion_stack.push(farthest_element_index);
recursion_stack.push(second);
}
if (1 < farthest_element_index - first) {
recursion_stack.push(first);
recursion_stack.push(farthest_element_index);
}
}
}
}
}
// If any line segment crosses a tile boundary, add a node there
// that cannot be simplified away, to prevent the edge of any
// feature from jumping abruptly at the tile boundary.
drawvec impose_tile_boundaries(drawvec &geom, long long extent) {
drawvec impose_tile_boundaries(const drawvec &geom, long long extent) {
drawvec out;
for (size_t i = 0; i < geom.size(); i++) {
@@ -469,11 +200,11 @@ drawvec impose_tile_boundaries(drawvec &geom, long long extent) {
if (c > 1) { // clipped
if (x1 != geom[i - 1].x || y1 != geom[i - 1].y) {
out.push_back(draw(VT_LINETO, x1, y1));
out.emplace_back(VT_LINETO, x1, y1);
out[out.size() - 1].necessary = 1;
}
if (x2 != geom[i - 0].x || y2 != geom[i - 0].y) {
out.push_back(draw(VT_LINETO, x2, y2));
out.emplace_back(VT_LINETO, x2, y2);
out[out.size() - 1].necessary = 1;
}
}
@@ -485,7 +216,7 @@ drawvec impose_tile_boundaries(drawvec &geom, long long extent) {
return out;
}
drawvec simplify_lines(drawvec &geom, int z, int tx, int ty, int detail, bool mark_tile_bounds, double simplification, size_t retain, drawvec const &shared_nodes, struct node *shared_nodes_map, size_t nodepos) {
drawvec simplify_lines(drawvec &geom, int z, int tx, int ty, int detail, bool mark_tile_bounds, double simplification, size_t retain, drawvec const &shared_nodes, struct node *shared_nodes_map, size_t nodepos, std::string const &shared_nodes_bloom) {
int res = 1 << (32 - detail - z);
long long area = 1LL << (32 - z);
@@ -522,12 +253,16 @@ drawvec simplify_lines(drawvec &geom, int z, int tx, int ty, int detail, bool ma
d.y += ty * (1LL << (32 - z));
}
// to quadkey
struct node n;
n.index = encode_quadkey((unsigned) d.x, (unsigned) d.y);
n.index = encode_vertex((unsigned) d.x, (unsigned) d.y);
size_t bloom_ix = n.index % (shared_nodes_bloom.size() * 8);
unsigned char bloom_mask = 1 << (bloom_ix & 7);
bloom_ix >>= 3;
if (bsearch(&n, shared_nodes_map, nodepos / sizeof(node), sizeof(node), nodecmp) != NULL) {
geom[i].necessary = true;
if (shared_nodes_bloom[bloom_ix] & bloom_mask) {
if (bsearch(&n, shared_nodes_map, nodepos / sizeof(node), sizeof(node), nodecmp) != NULL) {
geom[i].necessary = true;
}
}
}
}
@@ -560,24 +295,24 @@ drawvec simplify_lines(drawvec &geom, int z, int tx, int ty, int detail, bool ma
if (additional[A_VISVALINGAM]) {
visvalingam(geom, i, j, scale, retain);
} else {
douglas_peucker(geom, i, j - i, res * simplification, 2, retain);
douglas_peucker(geom, i, j - i, res * simplification, 2, retain, prevent[P_SIMPLIFY_SHARED_NODES]);
}
}
i = j - 1;
}
}
drawvec out;
size_t out = 0;
for (size_t i = 0; i < geom.size(); i++) {
if (geom[i].necessary) {
out.push_back(geom[i]);
geom[out++] = geom[i];
}
}
return out;
geom.resize(out);
return geom;
}
drawvec reorder_lines(drawvec &geom) {
drawvec reorder_lines(const drawvec &geom) {
// Only reorder simple linestrings with a single moveto
if (geom.size() == 0) {
@@ -623,154 +358,6 @@ drawvec reorder_lines(drawvec &geom) {
return geom;
}
drawvec fix_polygon(drawvec &geom) {
int outer = 1;
drawvec out;
for (size_t i = 0; i < geom.size(); i++) {
if (geom[i].op == VT_CLOSEPATH) {
outer = 1;
} else if (geom[i].op == VT_MOVETO) {
// Find the end of the ring
size_t j;
for (j = i + 1; j < geom.size(); j++) {
if (geom[j].op != VT_LINETO) {
break;
}
}
// A polygon ring must contain at least three points
// (and really should contain four). If this one does
// not have any, avoid a division by zero trying to
// calculate the centroid below.
if (j - i < 1) {
i = j - 1;
outer = 0;
continue;
}
// Make a temporary copy of the ring.
// Close it if it isn't closed.
drawvec ring;
for (size_t a = i; a < j; a++) {
ring.push_back(geom[a]);
}
if (j - i != 0 && (ring[0].x != ring[j - i - 1].x || ring[0].y != ring[j - i - 1].y)) {
ring.push_back(ring[0]);
}
// A polygon ring at this point should contain at least four points.
// Flesh it out with some vertex copies if it doesn't.
while (ring.size() < 4) {
ring.push_back(ring[0]);
}
// Reverse ring if winding order doesn't match
// inner/outer expectation
bool reverse_ring = false;
if (prevent[P_USE_SOURCE_POLYGON_WINDING]) {
// GeoJSON winding is reversed from vector winding
reverse_ring = true;
} else if (prevent[P_REVERSE_SOURCE_POLYGON_WINDING]) {
// GeoJSON winding is reversed from vector winding
reverse_ring = false;
} else {
double area = get_area(ring, 0, ring.size());
if ((area > 0) != outer) {
reverse_ring = true;
}
}
if (reverse_ring) {
drawvec tmp;
for (int a = ring.size() - 1; a >= 0; a--) {
tmp.push_back(ring[a]);
}
ring = tmp;
}
// Now we are rotating the ring to make the first/last point
// one that would be unlikely to be simplified away.
// calculate centroid
// a + 1 < size() because point 0 is duplicated at the end
long long xtotal = 0;
long long ytotal = 0;
long long count = 0;
for (size_t a = 0; a + 1 < ring.size(); a++) {
xtotal += ring[a].x;
ytotal += ring[a].y;
count++;
}
xtotal /= count;
ytotal /= count;
// figure out which point is furthest from the centroid
long long dist2 = 0;
long long furthest = 0;
for (size_t a = 0; a + 1 < ring.size(); a++) {
// division by 16 because these are z0 coordinates and we need to avoid overflow
long long xd = (ring[a].x - xtotal) / 16;
long long yd = (ring[a].y - ytotal) / 16;
long long d2 = xd * xd + yd * yd;
if (d2 > dist2 || (d2 == dist2 && ring[a] < ring[furthest])) {
dist2 = d2;
furthest = a;
}
}
// then figure out which point is furthest from *that*,
// which will hopefully be a good origin point since it should be
// at a far edge of the shape.
long long dist2b = 0;
long long furthestb = 0;
for (size_t a = 0; a + 1 < ring.size(); a++) {
// division by 16 because these are z0 coordinates and we need to avoid overflow
long long xd = (ring[a].x - ring[furthest].x) / 16;
long long yd = (ring[a].y - ring[furthest].y) / 16;
long long d2 = xd * xd + yd * yd;
if (d2 > dist2b || (d2 == dist2b && ring[a] < ring[furthestb])) {
dist2b = d2;
furthestb = a;
}
}
// rotate ring so the furthest point is the duplicated one.
// the idea is that simplification will then be more efficient,
// never wasting the start and end points, which are always retained,
// on a point that has little impact on the shape.
// Copy ring into output, fixing the moveto/lineto ops if necessary because of
// reversal or closing
for (size_t a = 0; a < ring.size(); a++) {
size_t a2 = (a + furthestb) % (ring.size() - 1);
if (a == 0) {
out.push_back(draw(VT_MOVETO, ring[a2].x, ring[a2].y));
} else {
out.push_back(draw(VT_LINETO, ring[a2].x, ring[a2].y));
}
}
// Next ring or polygon begins on the non-lineto that ended this one
// and is not an outer ring unless there is a terminator first
i = j - 1;
outer = 0;
} else {
fprintf(stderr, "Internal error: polygon ring begins with %d, not moveto\n", geom[i].op);
exit(EXIT_IMPOSSIBLE);
}
}
return out;
}
#if 0
std::vector<drawvec> chop_polygon(std::vector<drawvec> &geoms) {
while (1) {
@@ -1011,6 +598,41 @@ draw centerOfMass(const drawvec &dv, size_t start, size_t end, draw centre) {
}
}
draw center_of_mass_mp(const drawvec &dv) {
double ringx = 0, ringy = 0;
size_t ringcount = 0;
for (size_t i = 0; i < dv.size(); i++) {
if (dv[i].op == VT_MOVETO) {
double xsum = dv[i].x, ysum = dv[i].y;
ssize_t count = 1;
size_t j;
for (j = i + 1; j < dv.size(); j++) {
if (dv[j].op != VT_LINETO) {
break;
} else {
xsum += dv[j].x;
ysum += dv[j].y;
count++;
}
}
double area = get_area(dv, i, j);
draw centroid(VT_MOVETO, std::llround(xsum / count), std::llround(ysum / count));
draw center = centerOfMass(dv, i, j, centroid);
ringx += center.x * area;
ringy += center.y * area;
ringcount += area;
i = j - 1;
}
}
draw center(VT_MOVETO, ringx / ringcount, ringy / ringcount);
return center;
}
double label_goodness(const drawvec &dv, long long x, long long y) {
int nesting = 0;
@@ -1223,7 +845,7 @@ drawvec polygon_to_anchor(const drawvec &geom) {
std::vector<candidate> candidates;
for (size_t pass = 0; pass < 4; pass++) {
std::sort(points.begin(), points.end(), sorty_sorter(pass));
std::stable_sort(points.begin(), points.end(), sorty_sorter(pass));
for (size_t i = 1; i < points.size(); i++) {
double dx = points[i].x - points[i - 1].x;
@@ -1247,7 +869,7 @@ drawvec polygon_to_anchor(const drawvec &geom) {
// segment, if we find one whose midpoint is inside the polygon and
// far enough from any edge to be good enough, stop looking.
std::sort(candidates.begin(), candidates.end());
std::stable_sort(candidates.begin(), candidates.end());
// only check the top 50 stride midpoints, since this list can be quite large
for (size_t i = 0; i < candidates.size() && i < 50; i++) {
double maybe_goodness = label_goodness(geom, candidates[i].x, candidates[i].y);
+43 -11
View File
@@ -8,6 +8,8 @@
#include <sqlite3.h>
#include <stdio.h>
#include <mvt.hpp>
#include "jsonpull/jsonpull.h"
#include "attribute.hpp"
#define VT_POINT 1
#define VT_LINE 2
@@ -65,22 +67,23 @@ struct draw {
typedef std::vector<draw> drawvec;
struct serial_feature;
drawvec decode_geometry(char **meta, int z, unsigned tx, unsigned ty, long long *bbox, unsigned initial_x, unsigned initial_y);
drawvec decode_geometry(const char **meta, int z, unsigned tx, unsigned ty, long long *bbox, unsigned initial_x, unsigned initial_y);
void to_tile_scale(drawvec &geom, int z, int detail);
drawvec from_tile_scale(drawvec const &geom, int z, int detail);
drawvec remove_noop(drawvec geom, int type, int shift);
drawvec clip_point(drawvec &geom, int z, long long buffer);
drawvec clean_or_clip_poly(drawvec &geom, int z, int buffer, bool clip, bool try_scaling);
drawvec close_poly(drawvec &geom);
drawvec reduce_tiny_poly(drawvec &geom, int z, int detail, bool *still_needs_simplification, bool *reduced_away, double *accum_area, serial_feature *this_feature, serial_feature *tiny_feature);
drawvec reduce_tiny_poly(const drawvec &geom, int z, int detail, bool *still_needs_simplification, bool *reduced_away, double *accum_area, double tiny_polygon_size);
int clip(long long *x0, long long *y0, long long *x1, long long *y1, long long xmin, long long ymin, long long xmax, long long ymax);
drawvec clip_lines(drawvec &geom, int z, long long buffer);
drawvec stairstep(drawvec &geom, int z, int detail);
bool point_within_tile(long long x, long long y, int z);
int quick_check(long long *bbox, int z, long long buffer);
drawvec simplify_lines(drawvec &geom, int z, int tx, int ty, int detail, bool mark_tile_bounds, double simplification, size_t retain, drawvec const &shared_nodes, struct node *shared_nodes_map, size_t nodepos);
drawvec reorder_lines(drawvec &geom);
drawvec fix_polygon(drawvec &geom);
int quick_check(const long long *bbox, int z, long long buffer);
void douglas_peucker(drawvec &geom, int start, int n, double e, size_t kept, size_t retain, bool prevent_simplify_shared_nodes);
drawvec simplify_lines(drawvec &geom, int z, int tx, int ty, int detail, bool mark_tile_bounds, double simplification, size_t retain, drawvec const &shared_nodes, struct node *shared_nodes_map, size_t nodepos, std::string const &shared_nodes_bloom);
drawvec reorder_lines(const drawvec &geom);
drawvec fix_polygon(const drawvec &geom, bool use_winding, bool reverse_winding);
std::vector<drawvec> chop_polygon(std::vector<drawvec> &geoms);
void check_polygon(drawvec &geom);
double get_area(const drawvec &geom, size_t i, size_t j);
@@ -98,12 +101,41 @@ void visvalingam(drawvec &ls, size_t start, size_t end, double threshold, size_t
int pnpoly(const drawvec &vert, size_t start, size_t nvert, long long testx, long long testy);
double distance_from_line(long long point_x, long long point_y, long long segA_x, long long segA_y, long long segB_x, long long segB_y);
std::string overzoom(mvt_tile tile, int oz, int ox, int oy, int nz, int nx, int ny,
int detail, int buffer, std::set<std::string> const &keep, bool do_compress,
std::vector<std::pair<unsigned, unsigned>> *next_overzoomed_tiles);
struct input_tile {
std::string tile;
int z;
int x;
int y;
};
std::string overzoom(std::string s, int oz, int ox, int oy, int nz, int nx, int ny,
struct source_tile {
mvt_tile tile;
int z;
int x;
int y;
};
std::string overzoom(std::vector<source_tile> const &tiles, int nz, int nx, int ny,
int detail, int buffer, std::set<std::string> const &keep, bool do_compress,
std::vector<std::pair<unsigned, unsigned>> *next_overzoomed_tiles);
std::vector<std::pair<unsigned, unsigned>> *next_overzoomed_tiles,
bool demultiply, json_object *filter, bool preserve_input_order,
std::unordered_map<std::string, attribute_op> const &attribute_accum,
std::vector<std::string> const &unidecode_data, double simplification,
double tiny_polygon_size, std::vector<mvt_layer> const &bins,
std::string const &accumulate_numeric);
std::string overzoom(std::vector<input_tile> const &tiles, int nz, int nx, int ny,
int detail, int buffer, std::set<std::string> const &keep, bool do_compress,
std::vector<std::pair<unsigned, unsigned>> *next_overzoomed_tiles,
bool demultiply, json_object *filter, bool preserve_input_order,
std::unordered_map<std::string, attribute_op> const &attribute_accum,
std::vector<std::string> const &unidecode_data, double simplification,
double tiny_polygon_size, std::vector<mvt_layer> const &bins,
std::string const &accumulate_numeric);
draw center_of_mass_mp(const drawvec &dv);
void get_quadkey_bounds(long long xmin, long long ymin, long long xmax, long long ymax,
unsigned long long *start, unsigned long long *end);
#endif
+105 -74
View File
@@ -10,6 +10,15 @@
#define BUFFER 10000
struct string {
char *buf;
size_t n;
size_t nalloc;
};
static void string_init(struct string *s);
static void string_free(struct string *s);
json_pull *json_begin(ssize_t (*read)(struct json_pull *, char *buffer, size_t n), void *source) {
json_pull *j = malloc(sizeof(json_pull));
if (j == NULL) {
@@ -33,6 +42,13 @@ json_pull *json_begin(ssize_t (*read)(struct json_pull *, char *buffer, size_t n
exit(EXIT_FAILURE);
}
j->number_buffer = malloc(sizeof(struct string));
if (j->number_buffer == NULL) {
perror("Out of memory");
exit(EXIT_FAILURE);
}
string_init(j->number_buffer);
return j;
}
@@ -88,6 +104,9 @@ json_pull *json_begin_string(const char *s) {
}
void json_end(json_pull *p) {
string_free(p->number_buffer);
free(p->number_buffer);
json_free(p->root);
free(p->buffer);
free(p);
@@ -103,7 +122,7 @@ static inline int read_wrap(json_pull *j) {
return c;
}
#define SIZE_FOR(i, size) ((size_t)((((i) + 7) & ~7) * size))
#define SIZE_FOR(i, size) ((size_t) ((((i) + 7) & ~7) * size))
static json_object *fabricate_object(json_pull *jp, json_object *parent, json_type type) {
json_object *o = malloc(sizeof(struct json_object));
@@ -215,12 +234,6 @@ json_object *json_hash_get(json_object *o, const char *s) {
return NULL;
}
struct string {
char *buf;
size_t n;
size_t nalloc;
};
static void string_init(struct string *s) {
s->nalloc = 500;
s->buf = malloc(s->nalloc);
@@ -292,38 +305,45 @@ json_object *json_read_separators(json_pull *j, json_separator_callback cb, void
}
again:
/////////////////////////// Whitespace
do {
c = read_wrap(j);
if (c == EOF) {
if (j->container != NULL) {
j->error = "Reached EOF without all containers being closed";
}
return NULL;
c = read_wrap(j);
if (c == EOF) {
if (j->container != NULL) {
j->error = "Reached EOF without all containers being closed";
}
// Byte-order mark
if (c == 0xEF) {
int c2 = peek(j);
if (c2 == 0xBB) {
return NULL;
}
switch (c) {
/////////////////////////// Byte order mark
case 0xEF: {
int c2 = peek(j);
if (c2 == 0xBB) {
c2 = read_wrap(j);
c2 = peek(j);
if (c2 == 0xBF) {
c2 = read_wrap(j);
c2 = peek(j);
if (c2 == 0xBF) {
c2 = read_wrap(j);
c = ' ';
continue;
}
c = ' ';
goto again;
}
j->error = "Corrupt byte-order mark found";
return NULL;
}
} while (c == ' ' || c == '\t' || c == '\r' || c == '\n' || c == 0x1E);
j->error = "Corrupt byte-order mark found";
return NULL;
}
/////////////////////////// Arrays
/////////////////////////// Whitespace
if (c == '[') {
case ' ':
case '\t':
case '\r':
case '\n':
case 0x1E:
goto again;
/////////////////////////// Arrays
case '[': {
json_object *o = add_object(j, JSON_ARRAY);
if (o == NULL) {
return NULL;
@@ -336,7 +356,9 @@ again:
}
goto again;
} else if (c == ']') {
}
case ']': {
if (j->container == NULL) {
j->error = "Found ] at top level";
return NULL;
@@ -359,9 +381,9 @@ again:
return ret;
}
/////////////////////////// Hashes
/////////////////////////// Hashes
if (c == '{') {
case '{': {
json_object *o = add_object(j, JSON_HASH);
if (o == NULL) {
return NULL;
@@ -374,7 +396,9 @@ again:
}
goto again;
} else if (c == '}') {
}
case '}': {
if (j->container == NULL) {
j->error = "Found } at top level";
return NULL;
@@ -397,9 +421,9 @@ again:
return ret;
}
/////////////////////////// Null
/////////////////////////// Null
if (c == 'n') {
case 'n': {
if (read_wrap(j) != 'u' || read_wrap(j) != 'l' || read_wrap(j) != 'l') {
j->error = "Found misspelling of null";
return NULL;
@@ -408,9 +432,9 @@ again:
return add_object(j, JSON_NULL);
}
/////////////////////////// NaN
/////////////////////////// NaN
if (c == 'N') {
case 'N': {
if (read_wrap(j) != 'a' || read_wrap(j) != 'N') {
j->error = "Found misspelling of NaN";
return NULL;
@@ -420,9 +444,9 @@ again:
return NULL;
}
/////////////////////////// Infinity
/////////////////////////// Infinity
if (c == 'I') {
case 'I': {
if (read_wrap(j) != 'n' || read_wrap(j) != 'f' || read_wrap(j) != 'i' ||
read_wrap(j) != 'n' || read_wrap(j) != 'i' || read_wrap(j) != 't' ||
read_wrap(j) != 'y') {
@@ -434,9 +458,9 @@ again:
return NULL;
}
/////////////////////////// True
/////////////////////////// True
if (c == 't') {
case 't': {
if (read_wrap(j) != 'r' || read_wrap(j) != 'u' || read_wrap(j) != 'e') {
j->error = "Found misspelling of true";
return NULL;
@@ -445,9 +469,9 @@ again:
return add_object(j, JSON_TRUE);
}
/////////////////////////// False
/////////////////////////// False
if (c == 'f') {
case 'f': {
if (read_wrap(j) != 'a' || read_wrap(j) != 'l' || read_wrap(j) != 's' || read_wrap(j) != 'e') {
j->error = "Found misspelling of false";
return NULL;
@@ -456,9 +480,9 @@ again:
return add_object(j, JSON_FALSE);
}
/////////////////////////// Comma
/////////////////////////// Comma
if (c == ',') {
case ',': {
if (j->container != NULL) {
if (j->container->expect != JSON_COMMA) {
j->error = "Found unexpected comma";
@@ -479,9 +503,9 @@ again:
goto again;
}
/////////////////////////// Colon
/////////////////////////// Colon
if (c == ':') {
case ':': {
if (j->container == NULL) {
j->error = "Found colon at top level";
return NULL;
@@ -501,71 +525,80 @@ again:
goto again;
}
/////////////////////////// Numbers
/////////////////////////// Numbers
if (c == '-' || (c >= '0' && c <= '9')) {
struct string val;
string_init(&val);
case '-':
case '0':
case '1':
case '2':
case '3':
case '4':
case '5':
case '6':
case '7':
case '8':
case '9': {
j->number_buffer->n = 0;
int decimal = 0;
if (c == '-') {
string_append(&val, c);
string_append(j->number_buffer, c);
c = read_wrap(j);
}
if (c == '0') {
string_append(&val, c);
string_append(j->number_buffer, c);
} else if (c >= '1' && c <= '9') {
string_append(&val, c);
string_append(j->number_buffer, c);
c = peek(j);
while (c >= '0' && c <= '9') {
string_append(&val, read_wrap(j));
string_append(j->number_buffer, read_wrap(j));
c = peek(j);
}
}
if (peek(j) == '.') {
string_append(&val, read_wrap(j));
string_append(j->number_buffer, read_wrap(j));
decimal = 1;
c = peek(j);
if (c < '0' || c > '9') {
j->error = "Decimal point without digits";
string_free(&val);
string_free(j->number_buffer);
return NULL;
}
while (c >= '0' && c <= '9') {
string_append(&val, read_wrap(j));
string_append(j->number_buffer, read_wrap(j));
c = peek(j);
}
}
c = peek(j);
if (c == 'e' || c == 'E') {
string_append(&val, read_wrap(j));
string_append(j->number_buffer, read_wrap(j));
decimal = 1;
c = peek(j);
if (c == '+' || c == '-') {
string_append(&val, read_wrap(j));
string_append(j->number_buffer, read_wrap(j));
}
c = peek(j);
if (c < '0' || c > '9') {
j->error = "Exponent without digits";
string_free(&val);
string_free(j->number_buffer);
return NULL;
}
while (c >= '0' && c <= '9') {
string_append(&val, read_wrap(j));
string_append(j->number_buffer, read_wrap(j));
c = peek(j);
}
}
json_object *n = add_object(j, JSON_NUMBER);
if (n != NULL) {
n->value.number.number = atof(val.buf);
n->value.number.number = atof(j->number_buffer->buf);
n->value.number.large_signed = 0;
n->value.number.large_unsigned = 0;
@@ -575,7 +608,7 @@ again:
if (!decimal && n->value.number.number > MAX_SAFE_INTEGER) {
errno = 0;
char *err = NULL;
unsigned long long ull = strtoull(val.buf, &err, 10);
unsigned long long ull = strtoull(j->number_buffer->buf, &err, 10);
if (errno == 0 && (err == NULL || *err == '\0')) {
n->value.number.large_unsigned = ull;
}
@@ -583,22 +616,18 @@ again:
if (!decimal && n->value.number.number < MIN_SAFE_INTEGER) {
errno = 0;
char *err = NULL;
long long ll = strtoll(val.buf, &err, 10);
long long ll = strtoll(j->number_buffer->buf, &err, 10);
if (errno == 0 && (err == NULL || *err == '\0')) {
n->value.number.large_signed = ll;
}
}
string_free(&val);
} else {
string_free(&val);
}
return n;
}
/////////////////////////// Strings
/////////////////////////// Strings
if (c == '"') {
case '"': {
struct string val;
string_init(&val);
@@ -726,11 +755,13 @@ again:
json_object *s = add_object(j, JSON_STRING);
if (s != NULL) {
s->value.string.string = val.buf;
s->value.string.refcon = NULL;
} else {
string_free(&val);
}
return s;
}
}
j->error = "Found unexpected character";
return NULL;
+3
View File
@@ -38,6 +38,7 @@ typedef struct json_object {
struct {
char *string;
void *refcon; // reference constant for caller's use
} string;
struct {
@@ -68,6 +69,8 @@ typedef struct json_pull {
json_object *container;
json_object *root;
struct string *number_buffer;
} json_pull;
json_pull *json_begin_file(FILE *f);
+60 -100
View File
@@ -49,7 +49,6 @@
#include "tile.hpp"
#include "pool.hpp"
#include "projection.hpp"
#include "version.hpp"
#include "memfile.hpp"
#include "main.hpp"
#include "geojson.hpp"
@@ -66,6 +65,8 @@
#include "errors.hpp"
#include "read_json.hpp"
#include "sort.hpp"
#include "attribute.hpp"
#include "thread.hpp"
static int low_detail = 12;
static int full_detail = -1;
@@ -93,6 +94,10 @@ unsigned int drop_denser = 0;
std::map<std::string, serial_val> set_attributes;
unsigned long long preserve_point_density_threshold = 0;
long long extend_zooms_max = 0;
int retain_points_multiplier = 1;
std::vector<std::string> unidecode_data;
size_t maximum_string_attribute_length = 0;
std::string accumulate_numeric;
std::vector<order_field> order_by;
bool order_reverse;
@@ -303,7 +308,6 @@ int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, d
for (ssize_t i = maxzoom; i >= 0; i--) {
ds[i].seq++;
}
ssize_t chosen = maxzoom + 1;
for (ssize_t i = maxzoom; i >= 0; i--) {
if (ds[i].seq < 0) {
feature_minzoom = i + 1;
@@ -317,7 +321,6 @@ int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, d
ds[j].previndex = ix->ix;
}
chosen = i + 1;
break;
} else {
ds[i].seq -= ds[i].interval;
@@ -330,7 +333,7 @@ int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, d
// we will go ahead and push it out.
if (preserve_point_density_threshold > 0) {
for (ssize_t i = 0; i < chosen && i < maxzoom; i++) {
for (ssize_t i = 0; i < feature_minzoom && i < maxzoom; i++) {
if (ix->ix - ds[i].previndex > ((1LL << (32 - i)) / preserve_point_density_threshold) * ((1LL << (32 - i)) / preserve_point_density_threshold)) {
feature_minzoom = i;
@@ -448,7 +451,7 @@ void *run_sort(void *v) {
return NULL;
}
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::unordered_map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
long long segs[CPUS + 1];
segs[0] = 0;
segs[CPUS] = len;
@@ -480,10 +483,10 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
sst.resize(CPUS);
pthread_t pthreads[CPUS];
std::vector<std::set<type_and_string> > file_subkeys;
std::vector<std::set<serial_val> > file_subkeys;
for (size_t i = 0; i < CPUS; i++) {
file_subkeys.push_back(std::set<type_and_string>());
file_subkeys.push_back(std::set<serial_val>());
}
for (size_t i = 0; i < CPUS; i++) {
@@ -518,7 +521,7 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
}
for (size_t i = 0; i < CPUS; i++) {
if (pthread_create(&pthreads[i], NULL, run_parse_json, &pja[i]) != 0) {
if (thread_create(&pthreads[i], NULL, run_parse_json, &pja[i]) != 0) {
perror("pthread_create");
exit(EXIT_PTHREAD);
}
@@ -649,7 +652,7 @@ struct read_parallel_arg {
unsigned *initial_y = NULL;
std::string layername = "";
bool uses_gamma = false;
std::map<std::string, int> const *attribute_types = NULL;
std::unordered_map<std::string, int> const *attribute_types = NULL;
double *dist_sum = NULL;
size_t *dist_count = NULL;
double *area_sum = NULL;
@@ -693,7 +696,7 @@ void *run_read_parallel(void *v) {
return NULL;
}
void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::atomic<int> *is_parsing, pthread_t *parallel_parser, bool &parser_created, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > &layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::atomic<int> *is_parsing, pthread_t *parallel_parser, bool &parser_created, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > &layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::unordered_map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
// This has to kick off an intermediate thread to start the parser threads,
// so the main thread can get back to reading the next input stage while
// the intermediate thread waits for the completion of the parser threads.
@@ -735,7 +738,7 @@ void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::ato
rpa->want_dist = want_dist;
rpa->filters = filters;
if (pthread_create(parallel_parser, NULL, run_read_parallel, rpa) != 0) {
if (thread_create(parallel_parser, NULL, run_read_parallel, rpa) != 0) {
perror("pthread_create");
exit(EXIT_PTHREAD);
}
@@ -931,7 +934,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
}
for (size_t a = 0; a < CPUS; a++) {
if (pthread_create(&pthreads[a], NULL, run_sort, &args[a]) != 0) {
if (thread_create(&pthreads[a], NULL, run_sort, &args[a]) != 0) {
perror("pthread_create");
exit(EXIT_PTHREAD);
}
@@ -1237,7 +1240,7 @@ int vertexcmp(const void *void1, const void *void2) {
return 0;
}
std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzoom, int basezoom, double basezoom_marker_width, sqlite3 *outdb, const char *outdir, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, json_object *filter, double droprate, int buffer, const char *tmpdir, double gamma, int read_parallel, int forcetable, const char *attribution, bool uses_gamma, long long *file_bbox, long long *file_bbox1, long long *file_bbox2, const char *prefilter, const char *postfilter, const char *description, bool guess_maxzoom, bool guess_cluster_maxzoom, std::map<std::string, int> const *attribute_types, const char *pgm, std::map<std::string, attribute_op> const *attribute_accum, std::map<std::string, std::string> const &attribute_descriptions, std::string const &commandline, int minimum_maxzoom) {
std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzoom, int basezoom, double basezoom_marker_width, sqlite3 *outdb, const char *outdir, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, json_object *filter, double droprate, int buffer, const char *tmpdir, double gamma, int read_parallel, int forcetable, const char *attribution, bool uses_gamma, long long *file_bbox, long long *file_bbox1, long long *file_bbox2, const char *prefilter, const char *postfilter, const char *description, bool guess_maxzoom, bool guess_cluster_maxzoom, std::unordered_map<std::string, int> const *attribute_types, const char *pgm, std::unordered_map<std::string, attribute_op> const *attribute_accum, std::map<std::string, std::string> const &attribute_descriptions, std::string const &commandline, int minimum_maxzoom) {
int ret = EXIT_SUCCESS;
std::vector<struct reader> readers;
@@ -1335,7 +1338,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
// To distinguish a null value
{
struct stringpool p;
memfile_write(r->treefile, &p, sizeof(struct stringpool));
bool in_memory;
memfile_write(r->treefile, &p, sizeof(struct stringpool), in_memory);
}
r->file_bbox[0] = r->file_bbox[1] = UINT_MAX;
@@ -2066,7 +2070,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
#endif
struct node n;
n.index = encode_quadkey((unsigned) x, (unsigned) y);
n.index = encode_vertex((unsigned) x, (unsigned) y);
fwrite_check((char *) &n, sizeof(struct node), 1, readers[0].nodefile, &readers[0].nodepos, "vertices");
}
@@ -2080,6 +2084,9 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
fprintf(stderr, "Merging nodes \r");
}
std::string shared_nodes_bloom;
shared_nodes_bloom.resize(34567891); // circa 34MB, size nowhere near a power of 2
// Sort nodes that can't be simplified away; scan the list to remove duplicates
FILE *shared_nodes;
@@ -2145,6 +2152,11 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
fwrite_check((void *) &here, sizeof(here), 1, shared_nodes, &nodepos, "shared nodes");
written = here;
size_t bloom_ix = here.index % (shared_nodes_bloom.size() * 8);
unsigned char bloom_mask = 1 << (bloom_ix & 7);
bloom_ix >>= 3;
shared_nodes_bloom[bloom_ix] |= bloom_mask;
#if 0
unsigned wx, wy;
decode_quadkey(here.index, &wx, &wy);
@@ -2215,6 +2227,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
std::atomic<long long> geompos(0);
/* initial tile is normally 0/0/0 but can be iz/ix/iy if limited to one tile */
long long estimated_complexity = 0; // to be replaced after writing the data
fwrite_check(&estimated_complexity, sizeof(estimated_complexity), 1, geomfile, &geompos, fname);
serialize_int(geomfile, iz, &geompos, fname);
serialize_uint(geomfile, ix, &geompos, fname);
serialize_uint(geomfile, iy, &geompos, fname);
@@ -2224,6 +2238,13 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
/* end of tile */
serialize_ulong_long(geomfile, 0, &geompos, fname); // EOF
estimated_complexity = geompos;
fflush(geomfile);
if (pwrite(fileno(geomfile), &estimated_complexity, sizeof(estimated_complexity), 0) != sizeof(estimated_complexity)) {
perror("pwrite estimated complexity");
exit(EXIT_WRITE);
}
if (fclose(geomfile) != 0) {
perror("fclose geom");
exit(EXIT_CLOSE);
@@ -2701,7 +2722,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
}
}
std::sort(ddv.begin(), ddv.end());
std::stable_sort(ddv.begin(), ddv.end());
size_t i = 0;
for (int z = 0; z <= basezoom; z++) {
@@ -2757,7 +2778,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
std::atomic<unsigned> midx(0);
std::atomic<unsigned> midy(0);
std::vector<strategy> strategies;
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos);
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos, shared_nodes_bloom, basezoom, droprate, unidecode_data);
if (maxzoom != written) {
if (written > minzoom) {
@@ -2825,7 +2846,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
ai->second.maxzoom = maxzoom;
}
metadata m = make_metadata(fname, minzoom, maxzoom, minlat, minlon, maxlat, maxlon, minlat2, minlon2, maxlat2, maxlon2, midlat, midlon, attribution, merged_lm, true, description, !prevent[P_TILE_STATS], attribute_descriptions, "tippecanoe", commandline, strategies);
metadata m = make_metadata(fname, minzoom, maxzoom, minlat, minlon, maxlat, maxlon, minlat2, minlon2, maxlat2, maxlon2, midlat, midlon, attribution, merged_lm, true, description, !prevent[P_TILE_STATS], attribute_descriptions, "tippecanoe", commandline, strategies, basezoom, droprate, retain_points_multiplier);
if (outdb != NULL) {
mbtiles_write_metadata(outdb, m, forcetable);
} else {
@@ -2845,7 +2866,7 @@ static bool has_name(struct option *long_options, int *pl) {
return false;
}
void set_attribute_type(std::map<std::string, int> &attribute_types, const char *arg) {
void set_attribute_type(std::unordered_map<std::string, int> &attribute_types, const char *arg) {
const char *s = strchr(arg, ':');
if (s == NULL) {
fprintf(stderr, "-T%s option must be in the form -Tname:type\n", arg);
@@ -2872,79 +2893,6 @@ void set_attribute_type(std::map<std::string, int> &attribute_types, const char
attribute_types.insert(std::pair<std::string, int>(name, t));
}
void set_attribute_accum(std::map<std::string, attribute_op> &attribute_accum, std::string name, std::string type) {
attribute_op t;
if (type == "sum") {
t = op_sum;
} else if (type == "product") {
t = op_product;
} else if (type == "mean") {
t = op_mean;
} else if (type == "max") {
t = op_max;
} else if (type == "min") {
t = op_min;
} else if (type == "concat") {
t = op_concat;
} else if (type == "comma") {
t = op_comma;
} else {
fprintf(stderr, "Attribute method (%s) must be sum, product, mean, max, min, concat, or comma\n", type.c_str());
exit(EXIT_ARGS);
}
attribute_accum.insert(std::pair<std::string, attribute_op>(name, t));
}
void set_attribute_accum(std::map<std::string, attribute_op> &attribute_accum, const char *arg) {
if (*arg == '{') {
json_pull *jp = json_begin_string(arg);
json_object *o = json_read_tree(jp);
if (o == NULL) {
fprintf(stderr, "%s: -E%s: %s\n", *av, arg, jp->error);
exit(EXIT_JSON);
}
if (o->type != JSON_HASH) {
fprintf(stderr, "%s: -E%s: not a JSON object\n", *av, arg);
exit(EXIT_JSON);
}
for (size_t i = 0; i < o->value.object.length; i++) {
json_object *k = o->value.object.keys[i];
json_object *v = o->value.object.values[i];
if (k->type != JSON_STRING) {
fprintf(stderr, "%s: -E%s: key %zu not a string\n", *av, arg, i);
exit(EXIT_JSON);
}
if (v->type != JSON_STRING) {
fprintf(stderr, "%s: -E%s: value %zu not a string\n", *av, arg, i);
exit(EXIT_JSON);
}
set_attribute_accum(attribute_accum, k->value.string.string, v->value.string.string);
}
json_free(o);
json_end(jp);
return;
}
const char *s = strchr(arg, ':');
if (s == NULL) {
fprintf(stderr, "-E%s option must be in the form -Ename:method\n", arg);
exit(EXIT_ARGS);
}
std::string name = std::string(arg, s - arg);
std::string type = std::string(s + 1);
set_attribute_accum(attribute_accum, name, type);
}
void set_attribute_value(const char *arg) {
if (*arg == '{') {
json_pull *jp = json_begin_string(arg);
@@ -2969,10 +2917,8 @@ void set_attribute_value(const char *arg) {
exit(EXIT_JSON);
}
serial_val val;
stringify_value(v, val.type, val.s, "json", 1, o);
set_attributes.insert(std::pair<std::string, serial_val>(k->value.string.string, val));
serial_val val = stringify_value(v, "json", 1, o);
set_attributes.emplace(k->value.string.string, val);
}
json_free(o);
@@ -3078,8 +3024,8 @@ int main(int argc, char **argv) {
bool guess_cluster_maxzoom = false;
std::set<std::string> exclude, include;
std::map<std::string, int> attribute_types;
std::map<std::string, attribute_op> attribute_accum;
std::unordered_map<std::string, int> attribute_types;
std::unordered_map<std::string, attribute_op> attribute_accum;
std::map<std::string, std::string> attribute_descriptions;
int exclude_all = 0;
int read_parallel = 0;
@@ -3121,6 +3067,7 @@ int main(int argc, char **argv) {
{"smallest-maximum-zoom-guess", required_argument, 0, '~'},
{"extend-zooms-if-still-dropping", no_argument, &additional[A_EXTEND_ZOOMS], 1},
{"extend-zooms-if-still-dropping-maximum", required_argument, 0, '~'},
{"generate-variable-depth-tile-pyramid", no_argument, &additional[A_VARIABLE_DEPTH_PYRAMID], 1},
{"one-tile", required_argument, 0, 'R'},
{"Tile resolution", 0, 0, 0},
@@ -3138,18 +3085,22 @@ int main(int argc, char **argv) {
{"attribute-type", required_argument, 0, 'T'},
{"attribute-description", required_argument, 0, 'Y'},
{"accumulate-attribute", required_argument, 0, 'E'},
{"accumulate-numeric-attributes", required_argument, 0, '~'},
{"empty-csv-columns-are-null", no_argument, &prevent[P_EMPTY_CSV_COLUMNS], 1},
{"convert-stringified-ids-to-numbers", no_argument, &additional[A_CONVERT_NUMERIC_IDS], 1},
{"use-attribute-for-id", required_argument, 0, '~'},
{"single-precision", no_argument, &prevent[P_SINGLE_PRECISION], 1},
{"set-attribute", required_argument, 0, '~'},
{"maximum-string-attribute-length", required_argument, 0, '~'},
{"Filtering features by attributes", 0, 0, 0},
{"feature-filter-file", required_argument, 0, 'J'},
{"feature-filter", required_argument, 0, 'j'},
{"unidecode-data", required_argument, 0, '~'},
{"Dropping a fixed fraction of features by zoom level", 0, 0, 0},
{"drop-rate", required_argument, 0, 'r'},
{"retain-points-multiplier", required_argument, 0, '~'},
{"base-zoom", required_argument, 0, 'B'},
{"drop-denser", required_argument, 0, '~'},
{"limit-base-zoom-to-maximum-zoom", no_argument, &prevent[P_BASEZOOM_ABOVE_MAXZOOM], 1},
@@ -3207,6 +3158,7 @@ int main(int argc, char **argv) {
{"Adding calculated attributes", 0, 0, 0},
{"calculate-feature-density", no_argument, &additional[A_CALCULATE_FEATURE_DENSITY], 1},
{"generate-ids", no_argument, &additional[A_GENERATE_IDS], 1},
{"calculate-feature-index", no_argument, &additional[A_CALCULATE_INDEX], 1},
{"Trying to correct bad source geometry", 0, 0, 0},
{"detect-longitude-wraparound", no_argument, &additional[A_DETECT_WRAPAROUND], 1},
@@ -3370,6 +3322,14 @@ int main(int argc, char **argv) {
preserve_point_density_threshold = atoll_require(optarg, "Preserve point density threshold");
} else if (strcmp(opt, "extend-zooms-if-still-dropping-maximum") == 0) {
extend_zooms_max = atoll_require(optarg, "Maximum number by which to extend zooms");
} else if (strcmp(opt, "retain-points-multiplier") == 0) {
retain_points_multiplier = atoll_require(optarg, "Multiply the fraction of points retained by zoom level");
} else if (strcmp(opt, "unidecode-data") == 0) {
unidecode_data = read_unidecode(optarg);
} else if (strcmp(opt, "maximum-string-attribute-length") == 0) {
maximum_string_attribute_length = atoll_require(optarg, "Maximum string attribute length");
} else if (strcmp(opt, "accumulate-numeric-attributes") == 0) {
accumulate_numeric = optarg;
} else {
fprintf(stderr, "%s: Unrecognized option --%s\n", argv[0], opt);
exit(EXIT_ARGS);
@@ -3652,7 +3612,7 @@ int main(int argc, char **argv) {
}
case 'v':
fprintf(stderr, "tippecanoe %s\n", VERSION);
fprintf(stderr, "tippecanoe %s\n", version_str().c_str());
exit(EXIT_SUCCESS);
case 'P':
@@ -3692,7 +3652,7 @@ int main(int argc, char **argv) {
break;
case 'E':
set_attribute_accum(attribute_accum, optarg);
set_attribute_accum(attribute_accum, optarg, argv);
break;
default: {
+3
View File
@@ -62,6 +62,9 @@ extern size_t limit_tile_feature_count;
extern size_t limit_tile_feature_count_at_maxzoom;
extern std::map<std::string, serial_val> set_attributes;
extern long long extend_zooms_max;
extern int retain_points_multiplier;
extern size_t maximum_string_attribute_length;
extern std::string accumulate_numeric;
struct order_field {
std::string name;
+36 -5
View File
@@ -5,6 +5,8 @@ like these \[la]MADE_WITH.md\[ra]\&.
.PP
This is the official home of Tippecanoe, developed and actively maintained by Erica Fischer \[la]https://github.com/e-n-f\[ra] at Felt \[la]https://felt.com\[ra]\&.
.PP
For a self\-hosted, API driven version of Tippecanoe, contact a technical sales engineer at \[la]sales@felt.com\[ra]\&. Felt produces highly performant, automatically projected versions of your data, and utilizes a rendering engine, built on top of MapLibre GL JS \[la]https://github.com/maplibre/maplibre-gl-js\[ra], to style vector and raster data.
.PP
Version 2.0.0 is equivalent to 1.36.0 \[la]https://github.com/mapbox/tippecanoe/tree/1.36.0\[ra] in the original repository. Thank you Mapbox for the many years of early support.
.SH Intent
.PP
@@ -29,7 +31,11 @@ significant travel corridor.
.PP
The easiest way to install tippecanoe on OSX is with Homebrew \[la]http://brew.sh/\[ra]:
.PP
.RS
.nf
$ brew install tippecanoe
.fi
.RE
.PP
On Ubuntu it will usually be easiest to build from the source repository:
.PP
@@ -410,6 +416,8 @@ specified maximum zoom and to any levels added beyond that.
\fB\fC\-\-extend\-zooms\-if\-still\-dropping\-maximum=\fR\fIcount\fP: Increase the maxzoom if features are still being dropped at that zoom level
by up to \fIcount\fP zoom levels.
.IP \(bu 2
\fB\fC\-at\fR or \fB\fC\-\-generate\-variable\-depth\-tile\-pyramid\fR: Don't produce child tiles for any tile that should be sufficient to be overzoomed to any higher zoom level. Such tiles will be produced with maximum detail and no simplification or polygon cleaning. Tiles with point features below the basezoom or where any features have to be dropped dynamically, or which contain too many features or bytes with full detail, will be written out with normal detail and split into child tiles. Tilesets generated with this option are suitable for use only with tile servers that will find the appropriate tile to overzoom from and will simplify and clean the geometries appropriately before serving the tile.
.IP \(bu 2
\fB\fC\-R\fR \fIzoom\fP\fB\fC/\fR\fIx\fP\fB\fC/\fR\fIy\fP or \fB\fC\-\-one\-tile=\fR\fIzoom\fP\fB\fC/\fR\fIx\fP\fB\fC/\fR\fIy\fP: Set the minzoom and maxzoom to \fIzoom\fP and produce only
the single specified tile at that zoom level.
.RE
@@ -467,8 +475,8 @@ zoom level precision (ft) precision (m) map scale
\fB\fC\-z18\fR 1.5 in 4 cm 1:1250
\fB\fC\-z19\fR 0.8 in 2 cm 1:600
\fB\fC\-z20\fR 0.4 in 1 cm 1:300
\fB\fC\-z21\fR 0.2 in 0.5 cm 1:150
\fB\fC\-z22\fR 0.1 in 0.25 cm 1:75
\fB\fC\-z21\fR 0.4 in 1 cm 1:300
\fB\fC\-z22\fR 0.4 in 1 cm 1:300
.TE
.SS Tile resolution
.RS
@@ -522,11 +530,13 @@ The attributes and values may also be specified as JSON keys and values: \fB\fC\
\fB\fC\-\-use\-attribute\-for\-id=\fR\fIname\fP: Use the attribute with the specified \fIname\fP as if it were specified as the feature ID. (If this attribute is a stringified number, you must also use \fB\fC\-aI\fR to convert it to a number.)
.IP \(bu 2
\fB\fC\-pN\fR or \fB\fC\-\-single\-precision\fR: Write double\-precision numeric attribute values to tiles as single\-precision to reduce tile size.
.IP \(bu 2
\fB\fC\-\-maximum\-string\-attribute\-length\fR=\fIlength\fP: Truncate string attributes that exceed the specified length in bytes.
.RE
.SS Filtering features by attributes
.RS
.IP \(bu 2
\fB\fC\-j\fR \fIfilter\fP or \fB\fC\-\-feature\-filter\fR=\fIfilter\fP: Check features against a per\-layer filter (as defined in the Mapbox GL Style Specification \[la]https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter\[ra]) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer \fB\fC"*"\fR apply to all layers. The special variable \fB\fC$zoom\fR refers to the current zoom level.
\fB\fC\-j\fR \fIfilter\fP or \fB\fC\-\-feature\-filter\fR=\fIfilter\fP: Check features against a per\-layer filter (as defined in the Mapbox GL Style Specification \[la]https://docs.mapbox.com/mapbox-gl-js/style-spec/#other-filter\[ra] or in a Felt filter specification still to be finalized) and only include those that match. Any features in layers that have no filter specified will be passed through. Filters for the layer \fB\fC"*"\fR apply to all layers. The special variable \fB\fC$zoom\fR refers to the current zoom level.
.IP \(bu 2
\fB\fC\-J\fR \fIfilter\-file\fP or \fB\fC\-\-feature\-filter\-file\fR=\fIfilter\-file\fP: Like \fB\fC\-j\fR, but read the filter from a file.
.RE
@@ -574,6 +584,8 @@ If you use \fB\fC\-Bg\fR, it will guess a zoom level that will keep at most 50,0
You can also specify a marker\-width with \fB\fC\-Bg\fR\fIwidth\fP to allow fewer features in the densest tile to
compensate for the larger marker, or \fB\fC\-Bf\fR\fInumber\fP to allow at most \fInumber\fP features in the densest tile.
.IP \(bu 2
\fB\fC\-\-retain\-points\-multiplier=\fR\fImultiple\fP: Retain the specified multiple of points instead of just the number of points that would ordinarily be retained by the drop rate. These can be thinned out later with the \fB\fC\-m\fR option to \fB\fCtippecanoe\-overzoom\fR\&. The start of each cluster is marked in the feature sequence by the \fB\fCtippecanoe:retain_points_multiplier_first\fR attribute. The \fB\fC\-\-tile\-size\-limit\fR will also be extended at low zoom levels to allow for the multiplied features.
.IP \(bu 2
\fB\fC\-\-drop\-denser=\fR\fIpercentage\fP: When dropping dots at zoom levels below the base zoom, give the specified \fIpercentage\fP
preference to retaining points in sparse areas and dropping points in dense areas.
.IP \(bu 2
@@ -682,6 +694,8 @@ the line or polygon within one tile unit of its proper location. You can probabl
\fB\fC\-ag\fR or \fB\fC\-\-calculate\-feature\-density\fR: Add a new attribute, \fB\fCtippecanoe_feature_density\fR, to each feature, to record how densely features are spaced in that area of the tile. You can use this attribute in the style to produce a glowing effect where points are densely packed. It can range from 0 in the sparsest areas to 255 in the densest.
.IP \(bu 2
\fB\fC\-ai\fR or \fB\fC\-\-generate\-ids\fR: Add an \fB\fCid\fR (a feature ID, not an attribute named \fB\fCid\fR) to each feature that does not already have one. There is currently no guarantee that the \fB\fCid\fR added will be stable between runs or that it will not conflict with manually\-assigned feature IDs. Future versions of Tippecanoe may change the mechanism for allocating IDs.
.IP \(bu 2
\fB\fC\-aX\fR or \fB\fC\-\-calculate\-feature\-index\fR: Add a \fB\fCtippecanoe:index\fR field to each feature, giving its index in the quadkey or hilbert sequence.
.RE
.SS Trying to correct bad source geometry
.RS
@@ -892,7 +906,7 @@ Linux:
.PP
.RS
.nf
sudo apt\-get install build\-essential libsqlite3\-dev zlib1g\-dev
sudo apt\-get install gcc g++ make libsqlite3\-dev zlib1g\-dev
.fi
.RE
.PP
@@ -1247,11 +1261,20 @@ some of the precision. Running:
.PP
.RS
.nf
tippecanoe\-overzoom \-o out.mvt.gz inz/inx/iny outz/outx/outy in.mvt.gz
tippecanoe\-overzoom \-o out.mvt.gz in.mvt.gz inz/inx/iny outz/outx/outy
.fi
.RE
.PP
reads tile \fB\fCinz/inx/iny\fR of \fB\fCin.mvt.gz\fR and produces tile \fB\fCoutz/outx/outy\fR of \fB\fCout.mvt.gz\fR\&.
.PP
.RS
.nf
tippecanoe\-overzoom \-o out.mvt.gz \-t outz/outx/outy in.mvt.gz inz/inx/iny in2.mvt.gz in2z/in2x/in2y in3.mvt.gz in3z/in3x/in3y
.fi
.RE
.PP
reads tile \fB\fCinz/inx/iny\fR of \fB\fCin.mvt.gz\fR, tile \fB\fCin2z/in2x/in2y\fR of \fB\fCin2.mvt.gz\fR, and tile \fB\fCin3z/in3x/in3y\fR of \fB\fCin3.mvt.gz\fR,
and produces tile \fB\fCoutz/outx/outy\fR of \fB\fCout.mvt.gz\fR from them.
.SS Options
.RS
.IP \(bu 2
@@ -1260,4 +1283,12 @@ reads tile \fB\fCinz/inx/iny\fR of \fB\fCin.mvt.gz\fR and produces tile \fB\fCou
\fB\fC\-d\fR \fIdetail\fP: Set the detail of the output tile (default 12)
.IP \(bu 2
\fB\fC\-y\fR \fIattribute\fP: Retain the specified \fIattribute\fP in the output features. All attributes that are not named in a \fB\fC\-y\fR option will be removed.
.IP \(bu 2
\fB\fC\-j\fR \fIfilter\fP: Filter features using the same expression syntax as in tippecanoe.
.IP \(bu 2
\fB\fC\-m\fR: If a tile was created with the \fB\fC\-\-retain\-points\-multiplier\fR option, thin the tile back down to its normal feature count during overzooming. The first feature from each cluster will be retained, unless \fB\fC\-j\fR is used to specify a filter, in which case the first matching filter from each cluster will be retained instead.
.IP \(bu 2
\fB\fC\-\-preserve\-input\-order\fR: Restore a set of filtered features to its original input order
.IP \(bu 2
\fB\fC\-\-accumulate\-attribute\fR: Behaves as in \fB\fCtippecanoe\fR to sum attributes from the features of a multiplier cluster that are not included in the final output. The attributes from features that are filtered away with \fB\fC\-j\fR are \fInot\fP accumulated onto the output feature.
.RE
+116 -75
View File
@@ -7,6 +7,8 @@
#include <stdlib.h>
#include <string.h>
#include <sqlite3.h>
#include <cmath>
#include <climits>
#include <vector>
#include <string>
#include <set>
@@ -99,19 +101,6 @@ sqlite3 *mbtiles_open(char *dbname, char **argv, int forcetable) {
return outdb;
}
unsigned long long fnv1a(std::string const &s) {
// Store tiles by a hash of their contents (fnv1a 64-bit)
// http://www.isthe.com/chongo/tech/comp/fnv/
const unsigned long long fnv_offset_basis = 14695981039346656037u;
const unsigned long long fnv_prime = 1099511628211u;
unsigned long long h = fnv_offset_basis;
for (size_t i = 0; i < s.size(); i++) {
h ^= (unsigned char) s[i];
h *= fnv_prime;
}
return h;
}
void mbtiles_write_tile(sqlite3 *outdb, int z, int tx, int ty, const char *data, int size) {
std::string hash = std::to_string(fnv1a(std::string(data, size)));
@@ -194,21 +183,21 @@ void mbtiles_erase_zoom(sqlite3 *outdb, int z) {
}
}
bool type_and_string::operator<(const type_and_string &o) const {
if (string < o.string) {
bool serial_val::operator<(const serial_val &o) const {
if (s < o.s) {
return true;
}
if (string == o.string && type < o.type) {
if (s == o.s && type < o.type) {
return true;
}
return false;
}
bool type_and_string::operator!=(const type_and_string &o) const {
bool serial_val::operator!=(const serial_val &o) const {
if (type != o.type) {
return true;
}
if (string != o.string) {
if (s != o.s) {
return true;
}
return false;
@@ -259,7 +248,7 @@ void tilestats(std::map<std::string, layermap_entry> const &layermap1, size_t el
state.nospace = true;
state.json_write_string(geomtype);
size_t attrib_count = layer.second.file_keys.size();
size_t attrib_count = layer.second.tilestats.size();
if (attrib_count > max_tilestats_attributes) {
attrib_count = max_tilestats_attributes;
}
@@ -275,7 +264,7 @@ void tilestats(std::map<std::string, layermap_entry> const &layermap1, size_t el
state.json_write_array();
size_t attrs = 0;
for (auto attribute : layer.second.file_keys) {
for (auto attribute : layer.second.tilestats) {
if (attrs == elements) {
break;
}
@@ -336,15 +325,15 @@ void tilestats(std::map<std::string, layermap_entry> const &layermap1, size_t el
vals++;
state.nospace = true;
state.json_write_stringified(value.string);
state.json_write_stringified(value.s);
} else {
std::string trunc = truncate16(value.string, 256);
std::string trunc = truncate16(value.s, 256);
if (trunc.size() == value.string.size()) {
if (trunc.size() == value.s.size()) {
vals++;
state.nospace = true;
state.json_write_string(value.string);
state.json_write_string(value.s);
}
}
}
@@ -455,6 +444,14 @@ std::string stringify_strategies(std::vector<strategy> const &strategies) {
any = true;
}
if (strategies[i].truncated_zooms > 0) {
state.nospace = true;
state.json_write_string("truncated_zooms");
state.nospace = true;
state.json_write_number(strategies[i].truncated_zooms);
any = true;
}
state.nospace = true;
state.json_end_hash();
}
@@ -601,6 +598,17 @@ void mbtiles_write_metadata(sqlite3 *db, const metadata &m, bool forcetable) {
sqlite3_free(sql);
}
if (m.decisions_json.size() > 0) {
sql = sqlite3_mprintf("INSERT INTO metadata (name, value) VALUES ('tippecanoe_decisions', %Q);", m.decisions_json.c_str());
if (sqlite3_exec(db, sql, NULL, NULL, &err) != SQLITE_OK) {
fprintf(stderr, "set decisions: %s\n", err);
if (!forcetable) {
exit(EXIT_SQLITE);
}
}
sqlite3_free(sql);
}
if (m.vector_layers_json.size() > 0 || m.tilestats_json.size() > 0) {
std::string json;
json_writer state(&json);
@@ -642,7 +650,22 @@ void mbtiles_write_metadata(sqlite3 *db, const metadata &m, bool forcetable) {
}
}
metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minlat, double minlon, double maxlat, double maxlon, double minlat2, double minlon2, double maxlat2, double maxlon2, double midlat, double midlon, const char *attribution, std::map<std::string, layermap_entry> const &layermap, bool vector, const char *description, bool do_tilestats, std::map<std::string, std::string> const &attribute_descriptions, std::string const &program, std::string const &commandline, std::vector<strategy> const &strategies) {
static double sixdig(double val) {
return std::round(val * 1e6) / 1e6;
}
#define str(x) #x
#define xstr(x) str(x)
std::string version_str() {
std::string s = VERSION;
std::string build_info = xstr(BUILD_INFO);
if (build_info.size() > 0) {
s += " " + build_info;
}
return s;
}
metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minlat, double minlon, double maxlat, double maxlon, double minlat2, double minlon2, double maxlat2, double maxlon2, double midlat, double midlon, const char *attribution, std::map<std::string, layermap_entry> const &layermap, bool vector, const char *description, bool do_tilestats, std::map<std::string, std::string> const &attribute_descriptions, std::string const &program, std::string const &commandline, std::vector<strategy> const &strategies, int basezoom, double droprate, int retain_points_multiplier) {
metadata m;
m.name = fname;
@@ -654,29 +677,40 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
m.minzoom = minzoom;
m.maxzoom = maxzoom;
m.minlat = minlat;
m.minlon = minlon;
m.maxlat = maxlat;
m.maxlon = maxlon;
m.minlat = sixdig(minlat);
m.minlon = sixdig(minlon);
m.maxlat = sixdig(maxlat);
m.maxlon = sixdig(maxlon);
m.minlat2 = minlat2;
m.minlon2 = minlon2;
m.maxlat2 = maxlat2;
m.maxlon2 = maxlon2;
m.minlat2 = sixdig(minlat2);
m.minlon2 = sixdig(minlon2);
m.maxlat2 = sixdig(maxlat2);
m.maxlon2 = sixdig(maxlon2);
m.center_lat = midlat;
m.center_lon = midlon;
m.center_lat = sixdig(midlat);
m.center_lon = sixdig(midlon);
m.center_z = maxzoom;
if (attribution != NULL) {
m.attribution = attribution;
}
m.generator = program + " " + VERSION;
m.generator = program + " " + version_str();
m.generator_options = commandline;
m.strategies_json = stringify_strategies(strategies);
if (isinf(droprate)) {
droprate = LLONG_MAX;
}
if (basezoom != maxzoom || droprate != 2.5 || retain_points_multiplier != 1) {
m.decisions_json = std::string("{") +
"\"basezoom\":" + milo::dtoa_milo(basezoom) + "," +
"\"droprate\":" + milo::dtoa_milo(droprate) + "," +
"\"retain_points_multiplier\":" + std::to_string(retain_points_multiplier) +
std::string("}");
}
if (vector) {
{
json_writer state(&m.vector_layers_json);
@@ -690,7 +724,7 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
}
for (size_t i = 0; i < lnames.size(); i++) {
auto fk = layermap.find(lnames[i]);
auto ts = layermap.find(lnames[i]);
state.nospace = true;
state.json_write_hash();
@@ -702,17 +736,17 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
state.nospace = true;
state.json_write_string("description");
state.nospace = true;
state.json_write_string(fk->second.description);
state.json_write_string(ts->second.description);
state.nospace = true;
state.json_write_string("minzoom");
state.nospace = true;
state.json_write_signed(fk->second.minzoom);
state.json_write_signed(ts->second.minzoom);
state.nospace = true;
state.json_write_string("maxzoom");
state.nospace = true;
state.json_write_signed(fk->second.maxzoom);
state.json_write_signed(ts->second.maxzoom);
state.nospace = true;
state.json_write_string("fields");
@@ -721,7 +755,7 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
bool first = true;
size_t attribute_count = 0;
for (auto j = fk->second.file_keys.begin(); j != fk->second.file_keys.end(); ++j) {
for (auto j = ts->second.tilestats.begin(); j != ts->second.tilestats.end(); ++j) {
if (first) {
first = false;
}
@@ -828,35 +862,35 @@ std::map<std::string, layermap_entry> merge_layermaps(std::vector<std::map<std::
exit(EXIT_IMPOSSIBLE);
}
for (auto fk = map->second.file_keys.begin(); fk != map->second.file_keys.end(); ++fk) {
std::string attribname = fk->first;
for (auto ts = map->second.tilestats.begin(); ts != map->second.tilestats.end(); ++ts) {
std::string attribname = ts->first;
if (trunc) {
attribname = truncate16(attribname, 256);
}
auto fk2 = out_entry->second.file_keys.find(attribname);
auto ts2 = out_entry->second.tilestats.find(attribname);
if (fk2 == out_entry->second.file_keys.end()) {
out_entry->second.file_keys.insert(std::pair<std::string, type_and_string_stats>(attribname, fk->second));
if (ts2 == out_entry->second.tilestats.end()) {
out_entry->second.tilestats.insert(std::pair<std::string, tilestat>(attribname, ts->second));
} else {
for (auto val : fk->second.sample_values) {
auto pt = std::lower_bound(fk2->second.sample_values.begin(), fk2->second.sample_values.end(), val);
if (pt == fk2->second.sample_values.end() || *pt != val) { // not found
fk2->second.sample_values.insert(pt, val);
for (auto val : ts->second.sample_values) {
auto pt = std::lower_bound(ts2->second.sample_values.begin(), ts2->second.sample_values.end(), val);
if (pt == ts2->second.sample_values.end() || *pt != val) { // not found
ts2->second.sample_values.insert(pt, val);
if (fk2->second.sample_values.size() > max_tilestats_sample_values) {
fk2->second.sample_values.pop_back();
if (ts2->second.sample_values.size() > max_tilestats_sample_values) {
ts2->second.sample_values.pop_back();
}
}
}
fk2->second.type |= fk->second.type;
ts2->second.type |= ts->second.type;
if (fk->second.min < fk2->second.min) {
fk2->second.min = fk->second.min;
if (ts->second.min < ts2->second.min) {
ts2->second.min = ts->second.min;
}
if (fk->second.max > fk2->second.max) {
fk2->second.max = fk->second.max;
if (ts->second.max > ts2->second.max) {
ts2->second.max = ts->second.max;
}
}
}
@@ -877,41 +911,48 @@ std::map<std::string, layermap_entry> merge_layermaps(std::vector<std::map<std::
return out;
}
void add_to_file_keys(std::map<std::string, type_and_string_stats> &file_keys, std::string const &attrib, type_and_string const &val) {
void add_to_tilestats(std::map<std::string, tilestat> &tilestats, std::string const &attrib, serial_val const &val) {
if (val.type == mvt_null) {
return;
}
auto fka = file_keys.find(attrib);
if (fka == file_keys.end()) {
file_keys.insert(std::pair<std::string, type_and_string_stats>(attrib, type_and_string_stats()));
fka = file_keys.find(attrib);
auto tsa = tilestats.find(attrib);
if (tsa == tilestats.end()) {
tilestats.insert(std::pair<std::string, tilestat>(attrib, tilestat()));
tsa = tilestats.find(attrib);
}
if (fka == file_keys.end()) {
if (tsa == tilestats.end()) {
fprintf(stderr, "Can't happen (tilestats)\n");
exit(EXIT_IMPOSSIBLE);
}
if (val.type == mvt_double) {
double d = atof(val.string.c_str());
double d = atof(val.s.c_str());
if (d < fka->second.min) {
fka->second.min = d;
if (d < tsa->second.min) {
tsa->second.min = d;
}
if (d > fka->second.max) {
fka->second.max = d;
if (d > tsa->second.max) {
tsa->second.max = d;
}
}
auto pt = std::lower_bound(fka->second.sample_values.begin(), fka->second.sample_values.end(), val);
if (pt == fka->second.sample_values.end() || *pt != val) { // not found
fka->second.sample_values.insert(pt, val);
if (fka->second.sample_values.size() > max_tilestats_sample_values) {
fka->second.sample_values.pop_back();
auto pt = std::lower_bound(tsa->second.sample_values.begin(), tsa->second.sample_values.end(), val);
if (pt == tsa->second.sample_values.end() || *pt != val) { // not found
if (tsa->second.sample_values.size() >= max_tilestats_sample_values) {
if (pt == tsa->second.sample_values.end()) {
// insertion point would be at the end,
// and the list is already full, so do nothing
} else {
// bump the former last value, insert this one
tsa->second.sample_values.insert(pt, val);
tsa->second.sample_values.pop_back();
}
} else {
tsa->second.sample_values.insert(pt, val);
}
}
fka->second.type |= (1 << val.type);
tsa->second.type |= (1 << val.type);
}
+7 -15
View File
@@ -9,16 +9,8 @@ extern size_t max_tilestats_attributes;
extern size_t max_tilestats_sample_values;
extern size_t max_tilestats_values;
struct type_and_string {
int type = 0;
std::string string = "";
bool operator<(const type_and_string &o) const;
bool operator!=(const type_and_string &o) const;
};
struct type_and_string_stats {
std::vector<type_and_string> sample_values = std::vector<type_and_string>(); // sorted
struct tilestat {
std::vector<serial_val> sample_values = std::vector<serial_val>(); // sorted
double min = INFINITY;
double max = -INFINITY;
int type = 0;
@@ -26,7 +18,7 @@ struct type_and_string_stats {
struct layermap_entry {
size_t id = 0;
std::map<std::string, type_and_string_stats> file_keys{};
std::map<std::string, tilestat> tilestats{};
int minzoom = 0;
int maxzoom = 0;
std::string description = "";
@@ -66,6 +58,7 @@ struct metadata {
std::string vector_layers_json;
std::string tilestats_json;
std::string decisions_json;
};
#include "tile.hpp"
@@ -75,7 +68,7 @@ sqlite3 *mbtiles_open(char *dbname, char **argv, int forcetable);
void mbtiles_write_tile(sqlite3 *outdb, int z, int tx, int ty, const char *data, int size);
void mbtiles_erase_zoom(sqlite3 *outdb, int z);
metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minlat, double minlon, double maxlat, double maxlon, double minlat2, double minlon2, double maxlat2, double maxlon2, double midlat, double midlon, const char *attribution, std::map<std::string, layermap_entry> const &layermap, bool vector, const char *description, bool do_tilestats, std::map<std::string, std::string> const &attribute_descriptions, std::string const &program, std::string const &commandline, std::vector<strategy> const &strategies);
metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minlat, double minlon, double maxlat, double maxlon, double minlat2, double minlon2, double maxlat2, double maxlon2, double midlat, double midlon, const char *attribution, std::map<std::string, layermap_entry> const &layermap, bool vector, const char *description, bool do_tilestats, std::map<std::string, std::string> const &attribute_descriptions, std::string const &program, std::string const &commandline, std::vector<strategy> const &strategies, int basezoom, double droprate, int retain_points_multiplier);
void mbtiles_write_metadata(sqlite3 *db, const metadata &m, bool forcetable);
void mbtiles_close(sqlite3 *outdb, const char *pgm);
@@ -83,8 +76,7 @@ void mbtiles_close(sqlite3 *outdb, const char *pgm);
std::map<std::string, layermap_entry> merge_layermaps(std::vector<std::map<std::string, layermap_entry> > const &maps);
std::map<std::string, layermap_entry> merge_layermaps(std::vector<std::map<std::string, layermap_entry> > const &maps, bool trunc);
void add_to_file_keys(std::map<std::string, type_and_string_stats> &file_keys, std::string const &layername, type_and_string const &val);
unsigned long long fnv1a(std::string const &s);
void add_to_tilestats(std::map<std::string, tilestat> &tilestats, std::string const &layername, serial_val const &val);
std::string version_str();
#endif
+3 -1
View File
@@ -49,10 +49,11 @@ int memfile_close(struct memfile *file) {
return 0;
}
int memfile_write(struct memfile *file, void *s, long long len) {
int memfile_write(struct memfile *file, void *s, long long len, bool &in_memory) {
// If it is full, append to the file.
// If it is not full yet, append to the string in memory.
in_memory = false;
if (file->fp != NULL) {
if (fwrite(s, sizeof(char), len, file->fp) != (size_t) len) {
return 0;
@@ -61,6 +62,7 @@ int memfile_write(struct memfile *file, void *s, long long len) {
} else {
file->map.append(std::string((char *) s, len));
file->off += len;
in_memory = true;
}
return len;
+1 -1
View File
@@ -15,7 +15,7 @@ struct memfile {
struct memfile *memfile_open(int fd);
int memfile_close(struct memfile *file);
int memfile_write(struct memfile *file, void *s, long long len);
int memfile_write(struct memfile *file, void *s, long long len, bool &in_memory);
void memfile_full(struct memfile *file);
#endif
+293 -77
View File
@@ -14,6 +14,8 @@
#include "protozero/pbf_writer.hpp"
#include "milo/dtoa_milo.h"
#include "errors.hpp"
#include "serial.hpp"
#include "text.hpp"
mvt_geometry::mvt_geometry(int nop, long long nx, long long ny) {
this->op = nop;
@@ -89,7 +91,7 @@ int compress(std::string const &input, std::string &output, bool gz) {
deflate_s.opaque = Z_NULL;
deflate_s.avail_in = 0;
deflate_s.next_in = Z_NULL;
deflateInit2(&deflate_s, Z_BEST_COMPRESSION, Z_DEFLATED, gz ? 31 : 15, 8, Z_DEFAULT_STRATEGY);
deflateInit2(&deflate_s, Z_DEFAULT_COMPRESSION, Z_DEFLATED, gz ? 31 : 15, 8, Z_DEFAULT_STRATEGY);
deflate_s.next_in = (Bytef *) input.data();
deflate_s.avail_in = input.size();
size_t length = 0;
@@ -109,7 +111,7 @@ int compress(std::string const &input, std::string &output, bool gz) {
return 0;
}
bool mvt_tile::decode(std::string &message, bool &was_compressed) {
bool mvt_tile::decode(const std::string &message, bool &was_compressed) {
layers.clear();
std::string src;
@@ -126,6 +128,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
}
protozero::pbf_reader reader(src);
std::shared_ptr<std::string> string_pool = std::make_shared<std::string>();
while (reader.next()) {
switch (reader.tag()) {
@@ -156,7 +159,12 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
switch (value_reader.tag()) {
case 1: /* string */
value.type = mvt_string;
value.string_value = value_reader.get_string();
value.s = string_pool;
{
auto v = value_reader.get_view();
std::string_view sv(v.data(), v.size());
value.set_string_value(sv);
}
break;
case 2: /* float */
@@ -195,7 +203,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
}
}
layer.values.push_back(value);
layer.values.push_back(std::move(value));
break;
}
@@ -223,6 +231,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
case 2: /* tag */
{
auto pi = feature_reader.get_packed_uint32();
feature.tags.reserve(std::distance(pi.first, pi.second));
for (auto it = pi.first; it != pi.second; ++it) {
feature.tags.push_back(*it);
}
@@ -236,6 +245,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
case 4: /* geometry */
{
auto pi = feature_reader.get_packed_uint32();
geoms.reserve(std::distance(pi.first, pi.second));
for (auto it = pi.first; it != pi.second; ++it) {
geoms.push_back(*it);
}
@@ -249,6 +259,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
}
long long px = 0, py = 0;
feature.geometry.reserve(geoms.size()); // probably not quite right, but still plausible
for (size_t g = 0; g < geoms.size(); g++) {
uint32_t geom = geoms[g];
uint32_t op = geom & 7;
@@ -260,14 +271,14 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
py += protozero::decode_zigzag32(geoms[g + 2]);
g += 2;
feature.geometry.push_back(mvt_geometry(op, px, py));
feature.geometry.emplace_back(op, px, py);
}
} else {
feature.geometry.push_back(mvt_geometry(op, 0, 0));
feature.geometry.emplace_back(op, 0, 0);
}
}
layer.features.push_back(feature);
layer.features.push_back(std::move(feature));
break;
}
@@ -277,14 +288,7 @@ bool mvt_tile::decode(std::string &message, bool &was_compressed) {
}
}
for (size_t i = 0; i < layer.keys.size(); i++) {
layer.key_map.insert(std::pair<std::string, size_t>(layer.keys[i], i));
}
for (size_t i = 0; i < layer.values.size(); i++) {
layer.value_map.insert(std::pair<mvt_value, size_t>(layer.values[i], i));
}
layers.push_back(layer);
layers.push_back(std::move(layer));
break;
}
@@ -304,15 +308,18 @@ struct sorted_value {
bool operator<(const sorted_value &sv) const {
if (val < sv.val) {
return true;
} else {
return false;
} else if (val == sv.val) {
if (orig < sv.orig) {
return true;
}
}
return false;
}
};
std::string mvt_tile::encode() {
std::string data;
protozero::pbf_writer writer(data);
for (size_t i = 0; i < layers.size(); i++) {
@@ -334,41 +341,58 @@ std::string mvt_tile::encode() {
protozero::pbf_writer value_writer(value_string);
mvt_value &pbv = layers[i].values[v];
if (pbv.type == mvt_string) {
value_writer.add_string(1, pbv.string_value);
} else if (pbv.type == mvt_float) {
switch (pbv.type) {
case mvt_string:
value_writer.add_string(1, pbv.get_string_value());
break;
case mvt_float:
value_writer.add_float(2, pbv.numeric_value.float_value);
} else if (pbv.type == mvt_double) {
break;
case mvt_double:
value_writer.add_double(3, pbv.numeric_value.double_value);
} else if (pbv.type == mvt_int) {
break;
case mvt_int:
value_writer.add_int64(4, pbv.numeric_value.int_value);
} else if (pbv.type == mvt_uint) {
break;
case mvt_uint:
value_writer.add_uint64(5, pbv.numeric_value.uint_value);
} else if (pbv.type == mvt_sint) {
break;
case mvt_sint:
value_writer.add_sint64(6, pbv.numeric_value.sint_value);
} else if (pbv.type == mvt_bool) {
break;
case mvt_bool:
value_writer.add_bool(7, pbv.numeric_value.bool_value);
} else if (pbv.type == mvt_null) {
break;
case mvt_null:
fprintf(stderr, "Internal error: trying to write null attribute to tile\n");
exit(EXIT_IMPOSSIBLE);
} else {
default:
fprintf(stderr, "Internal error: trying to write undefined attribute type to tile\n");
exit(EXIT_IMPOSSIBLE);
}
sorted_value sv;
sv.val = value_string;
sv.val = std::move(value_string);
sv.orig = v;
sorted_values.push_back(sv);
sorted_values.push_back(std::move(sv));
}
std::sort(sorted_values.begin(), sorted_values.end());
std::stable_sort(sorted_values.begin(), sorted_values.end());
std::vector<size_t> mapping;
mapping.resize(sorted_values.size());
size_t value_index = 0;
for (size_t v = 0; v < sorted_values.size(); v++) {
mapping[sorted_values[v].orig] = v;
mapping[sorted_values[v].orig] = value_index;
layer_writer.add_message(4, sorted_values[v].val);
// crunch out duplicates that were missed by the hashing
while (v + 1 < sorted_values.size() && sorted_values[v].val == sorted_values[v + 1].val) {
mapping[sorted_values[v + 1].orig] = value_index;
v++;
}
value_index++;
}
for (size_t f = 0; f < layers[i].features.size(); f++) {
@@ -455,15 +479,70 @@ bool mvt_value::operator<(const mvt_value &o) const {
return true;
}
if (type == o.type) {
if ((type == mvt_string && string_value < o.string_value) ||
(type == mvt_float && numeric_value.float_value < o.numeric_value.float_value) ||
(type == mvt_double && numeric_value.double_value < o.numeric_value.double_value) ||
(type == mvt_int && numeric_value.int_value < o.numeric_value.int_value) ||
(type == mvt_uint && numeric_value.uint_value < o.numeric_value.uint_value) ||
(type == mvt_sint && numeric_value.sint_value < o.numeric_value.sint_value) ||
(type == mvt_bool && numeric_value.bool_value < o.numeric_value.bool_value) ||
(type == mvt_null && numeric_value.null_value < o.numeric_value.null_value)) {
return true;
switch (type) {
case mvt_string:
return get_string_view() < o.get_string_view();
case mvt_float:
return numeric_value.float_value < o.numeric_value.float_value;
case mvt_double:
return numeric_value.double_value < o.numeric_value.double_value;
case mvt_int:
return numeric_value.int_value < o.numeric_value.int_value;
case mvt_uint:
return numeric_value.uint_value < o.numeric_value.uint_value;
case mvt_sint:
return numeric_value.sint_value < o.numeric_value.sint_value;
case mvt_bool:
return numeric_value.bool_value < o.numeric_value.bool_value;
case mvt_null:
return numeric_value.null_value < o.numeric_value.null_value;
default:
fprintf(stderr, "mvt_value::operator<<: can't happen\n");
exit(EXIT_IMPOSSIBLE);
}
}
return false;
}
bool mvt_value::operator==(const mvt_value &o) const {
if (type == o.type) {
switch (type) {
case mvt_string:
return get_string_view() == o.get_string_view();
case mvt_float:
return numeric_value.float_value == o.numeric_value.float_value;
case mvt_double:
return numeric_value.double_value == o.numeric_value.double_value;
case mvt_int:
return numeric_value.int_value == o.numeric_value.int_value;
case mvt_uint:
return numeric_value.uint_value == o.numeric_value.uint_value;
case mvt_sint:
return numeric_value.sint_value == o.numeric_value.sint_value;
case mvt_bool:
return numeric_value.bool_value == o.numeric_value.bool_value;
case mvt_null:
return numeric_value.null_value == o.numeric_value.null_value;
default:
fprintf(stderr, "mvt_value::operator==: can't happen\n");
exit(EXIT_IMPOSSIBLE);
}
}
@@ -492,61 +571,58 @@ static std::string quote(std::string const &s) {
}
std::string mvt_value::toString() const {
if (type == mvt_string) {
return quote(string_value);
} else if (type == mvt_int) {
switch (type) {
case mvt_string:
return quote(get_string_value());
case mvt_int:
return std::to_string(numeric_value.int_value);
} else if (type == mvt_double) {
case mvt_double: {
double v = numeric_value.double_value;
if (v == (long long) v) {
return std::to_string((long long) v);
} else {
return milo::dtoa_milo(v);
}
} else if (type == mvt_float) {
}
case mvt_float: {
double v = numeric_value.float_value;
if (v == (long long) v) {
return std::to_string((long long) v);
} else {
return milo::dtoa_milo(v);
}
} else if (type == mvt_sint) {
}
case mvt_sint:
return std::to_string(numeric_value.sint_value);
} else if (type == mvt_uint) {
case mvt_uint:
return std::to_string(numeric_value.uint_value);
} else if (type == mvt_bool) {
case mvt_bool:
return numeric_value.bool_value ? "true" : "false";
} else if (type == mvt_null) {
case mvt_null:
return "null";
} else {
return "unknown";
default:
return "unknown " + std::to_string(type);
}
}
void mvt_layer::tag(mvt_feature &feature, std::string key, mvt_value value) {
size_t ko, vo;
std::map<std::string, size_t>::iterator ki = key_map.find(key);
std::map<mvt_value, size_t>::iterator vi = value_map.find(value);
if (ki == key_map.end()) {
ko = keys.size();
void mvt_layer::tag(mvt_feature &feature, std::string const &key, mvt_value const &value) {
size_t key_hash = fnv1a(key) % key_dedup.size();
if (key_dedup[key_hash] >= 0 &&
keys[key_dedup[key_hash]] == key) {
} else {
key_dedup[key_hash] = keys.size();
keys.push_back(key);
key_map.insert(std::pair<std::string, size_t>(key, ko));
} else {
ko = ki->second;
}
feature.tags.push_back(key_dedup[key_hash]);
if (vi == value_map.end()) {
vo = values.size();
size_t value_hash = std::hash<mvt_value>()(value) % value_dedup.size();
if (value_dedup[value_hash] >= 0 &&
values[value_dedup[value_hash]] == value) {
} else {
value_dedup[value_hash] = values.size();
values.push_back(value);
value_map.insert(std::pair<mvt_value, size_t>(value, vo));
} else {
vo = vi->second;
}
feature.tags.push_back(ko);
feature.tags.push_back(vo);
feature.tags.push_back(value_dedup[value_hash]);
}
bool is_integer(const char *s, long long *v) {
@@ -620,10 +696,17 @@ bool is_unsigned_integer(const char *s, unsigned long long *v) {
return 1;
}
mvt_value stringified_to_mvt_value(int type, const char *s) {
// This converts a serial_val-style attribute value to an mvt_value
// to store in a tile. If the value is numeric, it tries to choose
// the type (int, uint, sint, float, or double) that will give the
// smallest representation in the tile without losing precision,
// regardless of how the value was represented in the original source.
mvt_value stringified_to_mvt_value(int type, const char *s, std::shared_ptr<std::string> const &tile_stringpool) {
mvt_value tv;
tv.s = tile_stringpool;
if (type == mvt_double) {
switch (type) {
case mvt_double: {
long long v;
unsigned long long uv;
if (is_unsigned_integer(s, &uv)) {
@@ -662,16 +745,149 @@ mvt_value stringified_to_mvt_value(int type, const char *s) {
}
}
}
} else if (type == mvt_bool) {
} break;
case mvt_bool:
tv.type = mvt_bool;
tv.numeric_value.bool_value = (s[0] == 't');
} else if (type == mvt_null) {
break;
case mvt_null:
tv.type = mvt_null;
tv.numeric_value.null_value = 0;
} else {
break;
default:
tv.type = mvt_string;
tv.string_value = s;
tv.set_string_value(s);
}
return tv;
}
// This converts a mvt_value attribute value from a tile back into
// a serial_val for more convenient parsing and comparison without
// having to handle all of the vector tile numeric types separately.
// All numeric types are given the type mvt_double in the serial_val
// whether the actual value is integer or floating point.
serial_val mvt_value_to_serial_val(mvt_value const &v) {
serial_val sv;
switch (v.type) {
case mvt_string:
sv.type = mvt_string;
sv.s = v.get_string_value();
break;
case mvt_float:
sv.type = mvt_double;
sv.s = milo::dtoa_milo(v.numeric_value.float_value);
break;
case mvt_double:
sv.type = mvt_double;
sv.s = milo::dtoa_milo(v.numeric_value.double_value);
break;
case mvt_int:
sv.type = mvt_double;
sv.s = std::to_string(v.numeric_value.int_value);
break;
case mvt_uint:
sv.type = mvt_double;
sv.s = std::to_string(v.numeric_value.uint_value);
break;
case mvt_sint:
sv.type = mvt_double;
sv.s = std::to_string(v.numeric_value.sint_value);
break;
case mvt_bool:
sv.type = mvt_bool;
sv.s = v.numeric_value.bool_value ? "true" : "false";
break;
case mvt_null:
sv.type = mvt_null;
sv.s = "null";
break;
default:
fprintf(stderr, "unhandled mvt_type %d\n", v.type);
exit(EXIT_IMPOSSIBLE);
}
return sv;
}
// This extracts an integer value from an mvt_value
long long mvt_value_to_long_long(mvt_value const &v) {
switch (v.type) {
case mvt_string:
return atoll(v.c_str());
case mvt_float:
return v.numeric_value.float_value;
case mvt_double:
return v.numeric_value.double_value;
case mvt_int:
return v.numeric_value.int_value;
case mvt_uint:
return v.numeric_value.uint_value;
case mvt_sint:
return v.numeric_value.sint_value;
case mvt_bool:
return v.numeric_value.bool_value;
case mvt_null:
return 0;
default:
fprintf(stderr, "unhandled mvt_type %d\n", v.type);
exit(EXIT_IMPOSSIBLE);
}
}
// This extracts a double value from an mvt_value
double mvt_value_to_double(mvt_value const &v) {
switch (v.type) {
case mvt_string:
return atof(v.c_str());
case mvt_float:
return v.numeric_value.float_value;
case mvt_double:
return v.numeric_value.double_value;
case mvt_int:
return v.numeric_value.int_value;
case mvt_uint:
return v.numeric_value.uint_value;
case mvt_sint:
return v.numeric_value.sint_value;
case mvt_bool:
return v.numeric_value.bool_value;
case mvt_null:
return 0;
default:
fprintf(stderr, "unhandled mvt_type %d\n", v.type);
exit(EXIT_IMPOSSIBLE);
}
}
void get_bbox(std::vector<mvt_geometry> const &geom,
long long *xmin, long long *ymin, long long *xmax, long long *ymax,
int z, int tx, int ty, int detail) {
*xmin = LLONG_MAX;
*ymin = LLONG_MAX;
*xmax = 0;
*ymax = 0;
for (auto const &g : geom) {
if (g.op == mvt_moveto || g.op == mvt_lineto) {
long long x = g.x;
long long y = g.y;
// to world scale
x = x * (1LL << (32 - z - detail));
y = y * (1LL << (32 - z - detail));
// to world origin
if (z > 0) {
x += (1LL << (32 - z)) * tx;
y += (1LL << (32 - z)) * ty;
}
*xmin = std::min(*xmin, x);
*ymin = std::min(*ymin, y);
*xmax = std::max(*xmax, x);
*ymax = std::max(*ymax, y);
}
}
}
+99 -9
View File
@@ -3,9 +3,15 @@
#include <sqlite3.h>
#include <string>
#include <map>
#include <string_view>
#include <unordered_map>
#include <set>
#include <vector>
#include <optional>
#include <memory>
#include "errors.hpp"
#include "text.hpp"
struct mvt_value;
struct mvt_layer;
@@ -48,7 +54,8 @@ struct mvt_feature {
int /* mvt_geometry_type */ type = 0;
unsigned long long id = 0;
bool has_id = false;
bool dropped = false;
int dropped = 0;
size_t seq = 0; // used for ordering in overzoom
mvt_feature() {
has_id = false;
@@ -65,11 +72,14 @@ enum mvt_value_type {
mvt_sint,
mvt_bool,
mvt_null,
mvt_no_such_key,
};
struct mvt_value {
mvt_value_type type;
std::string string_value;
std::shared_ptr<std::string> s;
union {
float float_value;
double double_value;
@@ -78,18 +88,89 @@ struct mvt_value {
long long sint_value;
bool bool_value;
int null_value;
struct {
size_t off;
size_t len;
} string_value;
} numeric_value;
std::string get_string_value() const {
return std::string(*s, numeric_value.string_value.off, numeric_value.string_value.len);
}
std::string_view get_string_view() const {
return std::string_view(s->c_str() + numeric_value.string_value.off, numeric_value.string_value.len);
}
const char *c_str() const {
return s->c_str() + numeric_value.string_value.off;
}
void set_string_value(const std::string_view &val) {
if (s == nullptr) {
s = std::make_shared<std::string>();
}
type = mvt_string;
numeric_value.string_value.off = s->size();
numeric_value.string_value.len = val.size();
s->append(val);
s->push_back('\0');
}
bool is_numeric() const {
return type == mvt_float ||
type == mvt_double ||
type == mvt_int ||
type == mvt_uint ||
type == mvt_sint;
}
bool operator<(const mvt_value &o) const;
bool operator==(const mvt_value &o) const;
std::string toString() const;
mvt_value() {
this->type = mvt_double;
this->string_value = "";
this->numeric_value.double_value = 0;
}
};
template <>
struct std::hash<mvt_value> {
std::size_t operator()(const mvt_value &k) const {
switch (k.type) {
case mvt_string:
return fnv1a(k.c_str(), 0);
case mvt_float:
return fnv1a(sizeof(float), (void *) &k.numeric_value.float_value);
case mvt_double:
return fnv1a(sizeof(double), (void *) &k.numeric_value.double_value);
case mvt_int:
return fnv1a(sizeof(long long), (void *) &k.numeric_value.int_value);
case mvt_uint:
return fnv1a(sizeof(unsigned long long), (void *) &k.numeric_value.uint_value);
case mvt_sint:
return fnv1a(sizeof(long long), (void *) &k.numeric_value.sint_value);
case mvt_bool:
return fnv1a(sizeof(bool), (void *) &k.numeric_value.bool_value);
case mvt_null:
return fnv1a(sizeof(int), (void *) &k.numeric_value.null_value);
default:
fprintf(stderr, "mvt_value hash can't happen\n");
exit(EXIT_IMPOSSIBLE);
}
}
};
struct mvt_layer {
int version = 0;
std::string name = "";
@@ -99,18 +180,18 @@ struct mvt_layer {
long long extent = 0;
// Add a key-value pair to a feature, using this layer's constant pool
void tag(mvt_feature &feature, std::string key, mvt_value value);
void tag(mvt_feature &feature, std::string const &key, mvt_value const &value);
// For tracking the key-value constants already used in this layer
std::map<std::string, size_t> key_map{};
std::map<mvt_value, size_t> value_map{};
std::vector<ssize_t> key_dedup = std::vector<ssize_t>(65536, -1);
std::vector<ssize_t> value_dedup = std::vector<ssize_t>(65536, -1);
};
struct mvt_tile {
std::vector<mvt_layer> layers{};
std::string encode();
bool decode(std::string &message, bool &was_compressed);
bool decode(const std::string &message, bool &was_compressed);
};
bool is_compressed(std::string const &data);
@@ -118,8 +199,17 @@ int decompress(std::string const &input, std::string &output);
int compress(std::string const &input, std::string &output, bool gz);
int dezig(unsigned n);
mvt_value stringified_to_mvt_value(int type, const char *s);
mvt_value stringified_to_mvt_value(int type, const char *s, std::shared_ptr<std::string> const &tile_stringpool);
long long mvt_value_to_long_long(mvt_value const &v);
double mvt_value_to_double(mvt_value const &v);
bool is_integer(const char *s, long long *v);
bool is_unsigned_integer(const char *s, unsigned long long *v);
struct serial_val;
serial_val mvt_value_to_serial_val(mvt_value const &v);
void get_bbox(std::vector<mvt_geometry> const &geom,
long long *xmin, long long *ymin, long long *xmax, long long *ymax,
int z, int tx, int ty, int detail);
#endif
+39 -37
View File
@@ -1,51 +1,53 @@
#ifndef OPTIONS_HPP
#define OPTIONS_HPP
#define A_COALESCE ((int) 'c')
#define A_REVERSE ((int) 'r')
#define A_REORDER ((int) 'o')
#define A_LINE_DROP ((int) 'l')
#define A_DEBUG_POLYGON ((int) '@')
#define A_POLYGON_DROP ((int) 'p')
#define A_DETECT_SHARED_BORDERS ((int) 'b')
#define A_PREFER_RADIX_SORT ((int) 'R')
#define A_CALCULATE_FEATURE_DENSITY ((int) 'g')
#define A_INCREASE_GAMMA_AS_NEEDED ((int) 'G')
#define A_DROP_DENSEST_AS_NEEDED ((int) 's')
#define A_DROP_FRACTION_AS_NEEDED ((int) 'd')
#define A_DROP_SMALLEST_AS_NEEDED ((int) 'n')
#define A_COALESCE_DENSEST_AS_NEEDED ((int) 'S')
#define A_COALESCE_SMALLEST_AS_NEEDED ((int) 'N')
#define A_COALESCE_FRACTION_AS_NEEDED ((int) 'D')
#define A_GRID_LOW_ZOOMS ((int) 'L')
#define A_DETECT_WRAPAROUND ((int) 'w')
#define A_EXTEND_ZOOMS ((int) 'e')
#define A_CLUSTER_DENSEST_AS_NEEDED ((int) 'C')
#define A_GENERATE_IDS ((int) 'i')
#define A_COALESCE_FRACTION_AS_NEEDED ((int) 'D')
#define A_INCREASE_GAMMA_AS_NEEDED ((int) 'G')
#define A_CONVERT_NUMERIC_IDS ((int) 'I')
#define A_HILBERT ((int) 'h')
#define A_VISVALINGAM ((int) 'v')
#define A_GRID_LOW_ZOOMS ((int) 'L')
#define A_COALESCE_SMALLEST_AS_NEEDED ((int) 'N')
#define A_GENERATE_POLYGON_LABEL_POINTS ((int) 'P')
#define A_PREFER_RADIX_SORT ((int) 'R')
#define A_COALESCE_DENSEST_AS_NEEDED ((int) 'S')
#define A_CALCULATE_INDEX ((int) 'X')
#define A_DETECT_SHARED_BORDERS ((int) 'b')
#define A_COALESCE ((int) 'c')
#define A_DROP_FRACTION_AS_NEEDED ((int) 'd')
#define A_EXTEND_ZOOMS ((int) 'e')
#define A_CALCULATE_FEATURE_DENSITY ((int) 'g')
#define A_HILBERT ((int) 'h')
#define A_GENERATE_IDS ((int) 'i')
#define A_LINE_DROP ((int) 'l')
#define A_DROP_SMALLEST_AS_NEEDED ((int) 'n')
#define A_REORDER ((int) 'o')
#define A_POLYGON_DROP ((int) 'p')
#define A_REVERSE ((int) 'r')
#define A_DROP_DENSEST_AS_NEEDED ((int) 's')
#define A_VARIABLE_DEPTH_PYRAMID ((int) 't')
#define A_VISVALINGAM ((int) 'v')
#define A_DETECT_WRAPAROUND ((int) 'w')
#define P_SIMPLIFY ((int) 's')
#define P_SIMPLIFY_LOW ((int) 'S')
#define P_SIMPLIFY_SHARED_NODES ((int) 'n')
#define P_SINGLE_PRECISION ((int) 'N')
#define P_FEATURE_LIMIT ((int) 'f')
#define P_KILOBYTE_LIMIT ((int) 'k')
#define P_DYNAMIC_DROP ((int) 'd')
#define P_INPUT_ORDER ((int) 'i')
#define P_POLYGON_SPLIT ((int) 'p')
#define P_CLIPPING ((int) 'c')
#define P_DUPLICATION ((int) 'D')
#define P_TINY_POLYGON_REDUCTION ((int) 't')
#define P_TINY_POLYGON_REDUCTION_AT_MAXZOOM ((int) 'T')
#define P_TILE_COMPRESSION ((int) 'C')
#define P_TILE_STATS ((int) 'g')
#define P_USE_SOURCE_POLYGON_WINDING ((int) 'w')
#define P_DUPLICATION ((int) 'D')
#define P_SINGLE_PRECISION ((int) 'N')
#define P_SIMPLIFY_LOW ((int) 'S')
#define P_TINY_POLYGON_REDUCTION_AT_MAXZOOM ((int) 'T')
#define P_REVERSE_SOURCE_POLYGON_WINDING ((int) 'W')
#define P_EMPTY_CSV_COLUMNS ((int) 'e')
#define P_BASEZOOM_ABOVE_MAXZOOM ((int) 'b')
#define P_CLIPPING ((int) 'c')
#define P_DYNAMIC_DROP ((int) 'd')
#define P_EMPTY_CSV_COLUMNS ((int) 'e')
#define P_FEATURE_LIMIT ((int) 'f')
#define P_TILE_STATS ((int) 'g')
#define P_INPUT_ORDER ((int) 'i')
#define P_KILOBYTE_LIMIT ((int) 'k')
#define P_SIMPLIFY_SHARED_NODES ((int) 'n')
#define P_POLYGON_SPLIT ((int) 'p')
#define P_SIMPLIFY ((int) 's')
#define P_TINY_POLYGON_REDUCTION ((int) 't')
#define P_USE_SOURCE_POLYGON_WINDING ((int) 'w')
extern int prevent[256];
extern int additional[256];
+181 -32
View File
@@ -6,26 +6,76 @@
#include "errors.hpp"
#include "mvt.hpp"
#include "geometry.hpp"
#include "evaluator.hpp"
#include "attribute.hpp"
#include "text.hpp"
#include "read_json.hpp"
extern char *optarg;
extern int optind;
int detail = 12; // tippecanoe-style: mvt extent == 1 << detail
int buffer = 5; // tippecanoe-style: mvt buffer == extent * buffer / 256;
bool demultiply = false;
std::string filter;
bool preserve_input_order = false;
std::unordered_map<std::string, attribute_op> attribute_accum;
std::vector<std::string> unidecode_data;
std::vector<mvt_layer> bins;
std::string accumulate_numeric;
std::set<std::string> keep;
void usage(char **argv) {
fprintf(stderr, "Usage: %s -o newtile.pbf.gz tile.pbf.gz oz/ox/oy nz/nx/ny\n", argv[0]);
fprintf(stderr, "to create tile nz/nx/ny from tile oz/ox/oy\n");
fprintf(stderr, "Usage: %s -o newtile.pbf.gz -t nz/nx/ny tile.pbf.gz oz/ox/oy tile2.pbf.gz oz2/ox2/oy2\n", argv[0]);
fprintf(stderr, "to create tile nz/nx/ny from tiles oz/ox/oy and oz2/ox2/oy2\n");
exit(EXIT_FAILURE);
}
int main(int argc, char **argv) {
int i;
const char *outtile = NULL;
const char *outfile = NULL;
double simplification = 0;
double tiny_polygon_size = 0;
std::string assign_to_bins;
while ((i = getopt(argc, argv, "y:o:d:b:")) != -1) {
std::vector<input_tile> sources;
struct option long_options[] = {
{"include", required_argument, 0, 'y'},
{"full-detail", required_argument, 0, 'd'},
{"buffer", required_argument, 0, 'b'},
{"output", required_argument, 0, 'o'},
{"filter-points-multiplier", no_argument, 0, 'm'},
{"feature-filter", required_argument, 0, 'j'},
{"preserve-input-order", no_argument, 0, 'o' & 0x1F},
{"accumulate-attribute", required_argument, 0, 'E'},
{"unidecode-data", required_argument, 0, 'u' & 0x1F},
{"line-simplification", required_argument, 0, 'S'},
{"tiny-polygon-size", required_argument, 0, 's' & 0x1F},
{"source-tile", required_argument, 0, 't'},
{"assign-to-bins", required_argument, 0, 'b' & 0x1F},
{"accumulate-numeric-attributes", required_argument, 0, 'a' & 0x1F},
{0, 0, 0, 0},
};
std::string getopt_str;
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
if (long_options[lo].val > ' ') {
getopt_str.push_back(long_options[lo].val);
if (long_options[lo].has_arg == required_argument) {
getopt_str.push_back(':');
}
}
}
int option_index = 0;
while ((i = getopt_long(argc, argv, getopt_str.c_str(), long_options, &option_index)) != -1) {
switch (i) {
case 'y':
keep.insert(optarg);
@@ -43,55 +93,154 @@ int main(int argc, char **argv) {
buffer = atoi(optarg);
break;
case 'm':
demultiply = true;
break;
case 'j':
filter = optarg;
break;
case 'o' & 0x1F:
preserve_input_order = true;
break;
case 'E':
set_attribute_accum(attribute_accum, optarg, argv);
break;
case 'u' & 0x1F:
unidecode_data = read_unidecode(optarg);
break;
case 't':
outtile = optarg;
break;
case 's' & 0x1F:
tiny_polygon_size = atof(optarg);
break;
case 'S':
simplification = atof(optarg);
break;
case 'b' & 0x1F:
assign_to_bins = optarg;
break;
case 'a' & 0x1F:
accumulate_numeric = optarg;
break;
default:
fprintf(stderr, "Unrecognized flag -%c\n", i);
usage(argv);
}
}
if (argc - optind != 3) {
usage(argv);
}
if (outfile == NULL) {
usage(argv);
}
const char *infile = argv[optind + 0];
int oz, ox, oy;
if (sscanf(argv[optind + 1], "%d/%d/%d", &oz, &ox, &oy) != 3) {
fprintf(stderr, "%s: not in z/x/y form\n", argv[optind + 1]);
usage(argv);
}
std::vector<input_tile> its;
int nz, nx, ny;
if (sscanf(argv[optind + 2], "%d/%d/%d", &nz, &nx, &ny) != 3) {
fprintf(stderr, "%s: not in z/x/y form\n", argv[optind + 2]);
usage(argv);
if (outtile == NULL) { // single input
if (argc - optind != 3) {
fprintf(stderr, "Wrong number of arguments\n");
usage(argv);
}
const char *infile = argv[optind + 0];
int oz, ox, oy;
if (sscanf(argv[optind + 1], "%d/%d/%d", &oz, &ox, &oy) != 3) {
fprintf(stderr, "%s: not in z/x/y form\n", argv[optind + 1]);
usage(argv);
}
if (sscanf(argv[optind + 2], "%d/%d/%d", &nz, &nx, &ny) != 3) {
fprintf(stderr, "%s: not in z/x/y form\n", argv[optind + 2]);
usage(argv);
}
input_tile s;
s.tile = infile;
s.z = oz;
s.x = ox;
s.y = oy;
sources.push_back(s);
} else { // multiple inputs
if ((argc - optind) % 2 != 0) {
fprintf(stderr, "Unpaired arguments\n");
usage(argv);
}
if (sscanf(outtile, "%d/%d/%d", &nz, &nx, &ny) != 3) {
fprintf(stderr, "%s: not in z/x/y form\n", outtile);
usage(argv);
}
for (i = optind; i + 1 < argc; i += 2) {
int oz, ox, oy;
if (sscanf(argv[i + 1], "%d/%d/%d", &oz, &ox, &oy) != 3) {
fprintf(stderr, "%s: not in z/x/y form\n", argv[i + 1]);
usage(argv);
}
input_tile s;
s.tile = argv[i];
s.z = oz;
s.x = ox;
s.y = oy;
sources.push_back(s);
}
}
std::string tile;
char buf[1000];
int len;
if (assign_to_bins.size() != 0) {
FILE *f = fopen(assign_to_bins.c_str(), "r");
if (f == NULL) {
perror(assign_to_bins.c_str());
exit(EXIT_OPEN);
}
FILE *f = fopen(infile, "rb");
if (f == NULL) {
perror(infile);
exit(EXIT_FAILURE);
bins = parse_layers(f, nz, nx, ny, 1LL << detail, true);
fclose(f);
}
while ((len = fread(buf, sizeof(char), 1000, f)) > 0) {
tile.append(std::string(buf, len));
json_object *json_filter = NULL;
if (filter.size() > 0) {
json_filter = parse_filter(filter.c_str());
}
fclose(f);
f = fopen(outfile, "wb");
for (auto const &s : sources) {
std::string tile;
char buf[1000];
int len;
FILE *f = fopen(s.tile.c_str(), "rb");
if (f == NULL) {
perror(s.tile.c_str());
exit(EXIT_FAILURE);
}
while ((len = fread(buf, sizeof(char), 1000, f)) > 0) {
tile.append(std::string(buf, len));
}
fclose(f);
input_tile t = s;
t.tile = std::move(tile);
its.push_back(std::move(t));
}
std::string out = overzoom(its, nz, nx, ny, detail, buffer, keep, true, NULL, demultiply, json_filter, preserve_input_order, attribute_accum, unidecode_data, simplification, tiny_polygon_size, bins, accumulate_numeric);
FILE *f = fopen(outfile, "wb");
if (f == NULL) {
perror(outfile);
exit(EXIT_FAILURE);
}
std::string out = overzoom(tile, oz, ox, oy, nz, nx, ny, detail, buffer, keep, true, NULL);
fwrite(out.c_str(), sizeof(char), out.size(), f);
fclose(f);
+72 -233
View File
@@ -25,6 +25,7 @@
#include "geometry.hpp"
#include "serial.hpp"
#include "errors.hpp"
#include "thread.hpp"
extern "C" {
#include "jsonpull/jsonpull.h"
@@ -73,232 +74,74 @@ void *run_writer(void *a) {
return NULL;
}
// XXX deduplicate
static std::vector<mvt_geometry> to_feature(drawvec &geom) {
std::vector<mvt_geometry> out;
for (size_t i = 0; i < geom.size(); i++) {
out.push_back(mvt_geometry(geom[i].op, geom[i].x, geom[i].y));
}
return out;
}
// Reads from the postfilter
std::vector<mvt_layer> parse_layers(int fd, int z, unsigned x, unsigned y, std::vector<std::map<std::string, layermap_entry>> *layermaps, size_t tiling_seg, std::vector<std::vector<std::string>> *layer_unmaps, int extent) {
std::map<std::string, mvt_layer> ret;
FILE *f = fdopen(fd, "r");
if (f == NULL) {
perror("fdopen filter output");
exit(EXIT_OPEN);
}
json_pull *jp = json_begin_file(f);
while (1) {
json_object *j = json_read(jp);
if (j == NULL) {
if (jp->error != NULL) {
fprintf(stderr, "Filter output:%d: %s: ", jp->line, jp->error);
if (jp->root != NULL) {
json_context(jp->root);
} else {
fprintf(stderr, "\n");
}
exit(EXIT_JSON);
}
std::vector<mvt_layer> out = parse_layers(f, z, x, y, extent, false);
json_free(jp->root);
break;
}
json_object *type = json_hash_get(j, "type");
if (type == NULL || type->type != JSON_STRING) {
continue;
}
if (strcmp(type->value.string.string, "Feature") != 0) {
continue;
}
json_object *geometry = json_hash_get(j, "geometry");
if (geometry == NULL) {
fprintf(stderr, "Filter output:%d: filtered feature with no geometry: ", jp->line);
json_context(j);
json_free(j);
exit(EXIT_JSON);
}
json_object *properties = json_hash_get(j, "properties");
if (properties == NULL || (properties->type != JSON_HASH && properties->type != JSON_NULL)) {
fprintf(stderr, "Filter output:%d: feature without properties hash: ", jp->line);
json_context(j);
json_free(j);
exit(EXIT_JSON);
}
json_object *geometry_type = json_hash_get(geometry, "type");
if (geometry_type == NULL) {
fprintf(stderr, "Filter output:%d: null geometry (additional not reported): ", jp->line);
json_context(j);
exit(EXIT_JSON);
}
if (geometry_type->type != JSON_STRING) {
fprintf(stderr, "Filter output:%d: geometry type is not a string: ", jp->line);
json_context(j);
exit(EXIT_JSON);
}
json_object *coordinates = json_hash_get(geometry, "coordinates");
if (coordinates == NULL || coordinates->type != JSON_ARRAY) {
fprintf(stderr, "Filter output:%d: feature without coordinates array: ", jp->line);
json_context(j);
exit(EXIT_JSON);
}
int t;
for (t = 0; t < GEOM_TYPES; t++) {
if (strcmp(geometry_type->value.string.string, geometry_names[t]) == 0) {
break;
}
}
if (t >= GEOM_TYPES) {
fprintf(stderr, "Filter output:%d: Can't handle geometry type %s: ", jp->line, geometry_type->value.string.string);
json_context(j);
exit(EXIT_JSON);
}
std::string layername = "unknown";
json_object *tippecanoe = json_hash_get(j, "tippecanoe");
json_object *layer = NULL;
if (tippecanoe != NULL) {
layer = json_hash_get(tippecanoe, "layer");
if (layer != NULL && layer->type == JSON_STRING) {
layername = std::string(layer->value.string.string);
}
}
if (ret.count(layername) == 0) {
mvt_layer l;
l.name = layername;
l.version = 2;
l.extent = extent;
ret.insert(std::pair<std::string, mvt_layer>(layername, l));
}
auto l = ret.find(layername);
drawvec dv;
parse_geometry(t, coordinates, dv, VT_MOVETO, "Filter output", jp->line, j);
if (mb_geometry[t] == VT_POLYGON) {
dv = fix_polygon(dv);
}
// Scale and offset geometry from global to tile
for (size_t i = 0; i < dv.size(); i++) {
long long scale = 1LL << (32 - z);
dv[i].x = std::round((dv[i].x - scale * x) * extent / (double) scale);
dv[i].y = std::round((dv[i].y - scale * y) * extent / (double) scale);
}
if (mb_geometry[t] == VT_POLYGON) {
// we can try scaling up because these are tile coordinates
dv = clean_or_clip_poly(dv, 0, 0, false, true);
if (dv.size() < 3) {
dv.clear();
}
}
dv = remove_noop(dv, mb_geometry[t], 0);
if (mb_geometry[t] == VT_POLYGON) {
dv = close_poly(dv);
}
if (dv.size() > 0) {
mvt_feature feature;
feature.type = mb_geometry[t];
feature.geometry = to_feature(dv);
json_object *id = json_hash_get(j, "id");
if (id != NULL && id->type == JSON_NUMBER) {
feature.id = id->value.number.number;
if (id->value.number.large_unsigned > 0) {
feature.id = id->value.number.large_unsigned;
}
feature.has_id = true;
}
std::map<std::string, layermap_entry> &layermap = (*layermaps)[tiling_seg];
if (layermap.count(layername) == 0) {
layermap_entry lme = layermap_entry(layermap.size());
lme.minzoom = z;
lme.maxzoom = z;
layermap.insert(std::pair<std::string, layermap_entry>(layername, lme));
if (lme.id >= (*layer_unmaps)[tiling_seg].size()) {
(*layer_unmaps)[tiling_seg].resize(lme.id + 1);
(*layer_unmaps)[tiling_seg][lme.id] = layername;
}
}
auto fk = layermap.find(layername);
if (fk == layermap.end()) {
fprintf(stderr, "Internal error: layer %s not found\n", layername.c_str());
exit(EXIT_IMPOSSIBLE);
}
if (z < fk->second.minzoom) {
fk->second.minzoom = z;
}
if (z > fk->second.maxzoom) {
fk->second.maxzoom = z;
}
if (feature.type == mvt_point) {
fk->second.points++;
} else if (feature.type == mvt_linestring) {
fk->second.lines++;
} else if (feature.type == mvt_polygon) {
fk->second.polygons++;
}
for (size_t i = 0; i < properties->value.object.length; i++) {
int tp = -1;
std::string s;
stringify_value(properties->value.object.values[i], tp, s, "Filter output", jp->line, j);
// Nulls can be excluded here because this is the postfilter
// and it is nearly time to create the vector representation
if (tp >= 0 && tp != mvt_null) {
mvt_value v = stringified_to_mvt_value(tp, s.c_str());
l->second.tag(feature, std::string(properties->value.object.keys[i]->value.string.string), v);
type_and_string attrib;
attrib.type = tp;
attrib.string = s;
add_to_file_keys(fk->second.file_keys, std::string(properties->value.object.keys[i]->value.string.string), attrib);
}
}
l->second.features.push_back(feature);
}
json_free(j);
}
json_end(jp);
if (fclose(f) != 0) {
perror("fclose postfilter output");
exit(EXIT_CLOSE);
}
std::vector<mvt_layer> final;
for (auto a : ret) {
final.push_back(a.second);
for (auto const &layer : out) {
std::string layername = layer.name;
std::map<std::string, layermap_entry> &layermap = (*layermaps)[tiling_seg];
if (layermap.count(layername) == 0) {
layermap_entry lme = layermap_entry(layermap.size());
lme.minzoom = z;
lme.maxzoom = z;
layermap.insert(std::pair<std::string, layermap_entry>(layername, lme));
if (lme.id >= (*layer_unmaps)[tiling_seg].size()) {
(*layer_unmaps)[tiling_seg].resize(lme.id + 1);
(*layer_unmaps)[tiling_seg][lme.id] = layername;
}
}
auto ts = layermap.find(layername);
if (ts == layermap.end()) {
fprintf(stderr, "Internal error: layer %s not found\n", layername.c_str());
exit(EXIT_IMPOSSIBLE);
}
if (z < ts->second.minzoom) {
ts->second.minzoom = z;
}
if (z > ts->second.maxzoom) {
ts->second.maxzoom = z;
}
for (auto const &feature : layer.features) {
if (feature.type == mvt_point) {
ts->second.points++;
} else if (feature.type == mvt_linestring) {
ts->second.lines++;
} else if (feature.type == mvt_polygon) {
ts->second.polygons++;
}
for (size_t i = 0; i + 1 < feature.tags.size(); i += 2) {
const std::string &key = layer.keys[feature.tags[i]];
const mvt_value &val = layer.values[feature.tags[i + 1]];
// Nulls can be excluded here because this is the postfilter
// and it is nearly time to create the vector representation
if (val.type != mvt_null) {
add_to_tilestats(ts->second.tilestats, key, mvt_value_to_serial_val(val));
}
}
}
}
return final;
return out;
}
// Reads from the prefilter
@@ -382,7 +225,7 @@ serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::
drawvec dv;
parse_geometry(t, coordinates, dv, VT_MOVETO, "Filter output", jp->line, j);
if (mb_geometry[t] == VT_POLYGON) {
dv = fix_polygon(dv);
dv = fix_polygon(dv, false, false);
}
// Scale and offset geometry from global to tile
@@ -433,7 +276,9 @@ serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::
json_object *dropped = json_hash_get(tippecanoe, "dropped");
if (dropped != NULL && dropped->type == JSON_TRUE) {
sf.dropped = true;
sf.dropped = FEATURE_DROPPED; // dropped
} else {
sf.dropped = FEATURE_KEPT; // kept
}
}
@@ -478,49 +323,42 @@ serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::
}
}
auto fk = layermap.find(layername);
if (fk == layermap.end()) {
auto ts = layermap.find(layername);
if (ts == layermap.end()) {
fprintf(stderr, "Internal error: layer %s not found\n", layername.c_str());
exit(EXIT_IMPOSSIBLE);
}
sf.layer = fk->second.id;
sf.layer = ts->second.id;
if (z < fk->second.minzoom) {
fk->second.minzoom = z;
if (z < ts->second.minzoom) {
ts->second.minzoom = z;
}
if (z > fk->second.maxzoom) {
fk->second.maxzoom = z;
if (z > ts->second.maxzoom) {
ts->second.maxzoom = z;
}
if (!postfilter) {
if (sf.t == mvt_point) {
fk->second.points++;
ts->second.points++;
} else if (sf.t == mvt_linestring) {
fk->second.lines++;
ts->second.lines++;
} else if (sf.t == mvt_polygon) {
fk->second.polygons++;
ts->second.polygons++;
}
}
for (size_t i = 0; i < properties->value.object.length; i++) {
serial_val v;
v.type = -1;
stringify_value(properties->value.object.values[i], v.type, v.s, "Filter output", jp->line, j);
serial_val v = stringify_value(properties->value.object.values[i], "Filter output", jp->line, j);
// Nulls can be excluded here because the expression evaluation filter
// would have already run before prefiltering
if (v.type >= 0 && v.type != mvt_null) {
if (v.type != mvt_null) {
sf.full_keys.push_back(std::string(properties->value.object.keys[i]->value.string.string));
sf.full_values.push_back(v);
type_and_string attrib;
attrib.string = v.s;
attrib.type = v.type;
if (!postfilter) {
add_to_file_keys(fk->second.file_keys, std::string(properties->value.object.keys[i]->value.string.string), attrib);
add_to_tilestats(ts->second.tilestats, std::string(properties->value.object.keys[i]->value.string.string), v);
}
}
}
@@ -643,6 +481,7 @@ std::vector<mvt_layer> filter_layers(const char *filter, std::vector<mvt_layer>
wa.extent = extent;
pthread_t writer;
// this does need to be a real thread, so we can pipe both to and from it
if (pthread_create(&writer, NULL, run_writer, &wa) != 0) {
perror("pthread_create (filter writer)");
exit(EXIT_PTHREAD);
+3 -3
View File
@@ -166,11 +166,11 @@ struct entryv3 {
}
};
struct {
struct entryv3_cmp {
bool operator()(entryv3 a, entryv3 b) const {
return a.tile_id < b.tile_id;
}
} entryv3_cmp;
};
struct entry_zxy {
uint8_t z;
@@ -388,7 +388,7 @@ inline uint64_t zxy_to_tileid(uint8_t z, uint32_t x, uint32_t y) {
if (z > 31) {
throw std::overflow_error("tile zoom exceeds 64-bit limit");
}
if (x > (1 << z) - 1 || y > (1 << z) - 1) {
if (x > (1U << z) - 1 || y > (1U << z) - 1) {
throw std::overflow_error("tile x/y outside zoom level bounds");
}
uint64_t acc = 0;
+15 -2
View File
@@ -103,6 +103,13 @@ std::string metadata_to_pmtiles_json(metadata m) {
state.json_write_string("strategies");
state.json_write_json(m.strategies_json);
}
if (m.decisions_json.size() > 0) {
state.json_comma_newline();
state.json_write_string("tippecanoe_decisions");
state.json_write_json(m.decisions_json);
}
out(state, "generator", m.generator);
out(state, "generator_options", m.generator_options);
@@ -167,7 +174,7 @@ void mbtiles_map_image_to_pmtiles(char *fname, metadata m, bool tile_compression
sqlite3_finalize(stmt);
}
std::sort(tile_ids.begin(), tile_ids.end());
std::stable_sort(tile_ids.begin(), tile_ids.end());
std::unordered_map<std::string, std::pair<unsigned long long, unsigned long>> hash_to_offset_len;
std::vector<pmtiles::entryv3> entries;
@@ -256,7 +263,7 @@ void mbtiles_map_image_to_pmtiles(char *fname, metadata m, bool tile_compression
// finalize PMTiles archive.
{
std::sort(entries.begin(), entries.end(), pmtiles::entryv3_cmp);
std::stable_sort(entries.begin(), entries.end(), pmtiles::entryv3_cmp());
std::string root_bytes;
std::string leaves_bytes;
@@ -428,6 +435,12 @@ sqlite3 *pmtilesmeta2tmp(const char *fname, const char *pmtiles_map) {
fprintf(stderr, "set %s in metadata: %s\n", key, err);
}
sqlite3_free(sql);
} else if (strcmp(key, "tippecanoe_decisions") == 0 && o->value.object.values[i]->type == JSON_HASH) {
sql = sqlite3_mprintf("INSERT INTO metadata (name, value) VALUES ('tippecanoe_decisions', %Q);", json_stringify(o->value.object.values[i]));
if (sqlite3_exec(db, sql, NULL, NULL, &err) != SQLITE_OK) {
fprintf(stderr, "set %s in metadata: %s\n", key, err);
}
sqlite3_free(sql);
} else if (o->value.object.keys[i]->type != JSON_STRING || o->value.object.values[i]->type != JSON_STRING) {
fprintf(stderr, "%s\n", key);
fprintf(stderr, "%s: non-string in metadata\n", fname);
+47 -34
View File
@@ -7,32 +7,33 @@
#include "memfile.hpp"
#include "pool.hpp"
#include "errors.hpp"
#include "text.hpp"
int swizzlecmp(const char *a, const char *b) {
ssize_t alen = strlen(a);
ssize_t blen = strlen(b);
if (strcmp(a, b) == 0) {
return 0;
inline long long swizzlecmp(const char *a, int atype, unsigned long long ahash, const char *b, int btype, unsigned long long bhash) {
if ((long long) ahash == (long long) bhash) {
if (atype == btype) {
return strcmp(a, b);
} else {
return atype - btype;
}
} else {
return (long long) (ahash - bhash);
}
long long hash1 = 0, hash2 = 0;
for (ssize_t i = alen - 1; i >= 0; i--) {
hash1 = (hash1 * 37 + a[i]) & INT_MAX;
}
for (ssize_t i = blen - 1; i >= 0; i--) {
hash2 = (hash2 * 37 + b[i]) & INT_MAX;
}
int h1 = hash1, h2 = hash2;
if (h1 == h2) {
return strcmp(a, b);
}
return h1 - h2;
}
long long addpool(struct memfile *poolfile, struct memfile *treefile, const char *s, char type) {
long long addpool(struct memfile *poolfile, struct memfile *treefile, const char *s, char type, std::vector<ssize_t> &dedup) {
unsigned long long hash = fnv1a(s, type);
size_t hash_off = hash % dedup.size();
if (dedup[hash_off] >= 0 &&
poolfile->map[dedup[hash_off]] == type &&
strcmp(poolfile->map.c_str() + dedup[hash_off] + 1, s) == 0) {
// printf("hit for %s\n", s);
return dedup[hash_off];
} else {
// printf("miss for %s\n", s);
}
unsigned long *sp = &treefile->tree;
size_t depth = 0;
@@ -43,17 +44,17 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
}
while (*sp != 0) {
int cmp = swizzlecmp(s, poolfile->map.c_str() + ((struct stringpool *) (treefile->map.c_str() + *sp))->off + 1);
if (cmp == 0) {
cmp = type - (poolfile->map.c_str() + ((struct stringpool *) (treefile->map.c_str() + *sp))->off)[0];
}
long long cmp = swizzlecmp(s, type, hash,
poolfile->map.c_str() + ((struct stringpool *) (treefile->map.c_str() + *sp))->off + 1,
(poolfile->map.c_str() + ((struct stringpool *) (treefile->map.c_str() + *sp))->off)[0],
((struct stringpool *) (treefile->map.c_str() + *sp))->hash);
if (cmp < 0) {
sp = &(((struct stringpool *) (treefile->map.c_str() + *sp))->left);
} else if (cmp > 0) {
sp = &(((struct stringpool *) (treefile->map.c_str() + *sp))->right);
} else {
dedup[hash_off] = ((struct stringpool *) (treefile->map.c_str() + *sp))->off;
return ((struct stringpool *) (treefile->map.c_str() + *sp))->off;
}
@@ -65,14 +66,20 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
// the pool is full yet.
long long off = poolfile->off;
if (memfile_write(poolfile, &type, 1) < 0) {
bool in_memory = false;
if (memfile_write(poolfile, &type, 1, in_memory) < 0) {
perror("memfile write");
exit(EXIT_WRITE);
}
if (memfile_write(poolfile, (void *) s, strlen(s) + 1) < 0) {
if (memfile_write(poolfile, (void *) s, strlen(s) + 1, in_memory) < 0) {
perror("memfile write");
exit(EXIT_WRITE);
}
if (in_memory) {
dedup[hash_off] = off;
}
return off;
}
}
@@ -95,11 +102,12 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
// to the newly-added strings.
long long off = poolfile->off;
if (memfile_write(poolfile, &type, 1) < 0) {
bool in_memory;
if (memfile_write(poolfile, &type, 1, in_memory) < 0) {
perror("memfile write");
exit(EXIT_WRITE);
}
if (memfile_write(poolfile, (void *) s, strlen(s) + 1) < 0) {
if (memfile_write(poolfile, (void *) s, strlen(s) + 1, in_memory) < 0) {
perror("memfile write");
exit(EXIT_WRITE);
}
@@ -115,14 +123,18 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
}
long long off = poolfile->off;
if (memfile_write(poolfile, &type, 1) < 0) {
bool in_memory = false;
if (memfile_write(poolfile, &type, 1, in_memory) < 0) {
perror("memfile write");
exit(EXIT_WRITE);
}
if (memfile_write(poolfile, (void *) s, strlen(s) + 1) < 0) {
if (memfile_write(poolfile, (void *) s, strlen(s) + 1, in_memory) < 0) {
perror("memfile write");
exit(EXIT_WRITE);
}
if (in_memory) {
dedup[hash_off] = off;
}
if (off >= LONG_MAX || treefile->off >= LONG_MAX) {
// Tree or pool is bigger than 2GB
@@ -138,9 +150,10 @@ long long addpool(struct memfile *poolfile, struct memfile *treefile, const char
tsp.left = 0;
tsp.right = 0;
tsp.off = off;
tsp.hash = hash;
long long p = treefile->off;
if (memfile_write(treefile, &tsp, sizeof(struct stringpool)) < 0) {
if (memfile_write(treefile, &tsp, sizeof(struct stringpool), in_memory) < 0) {
perror("memfile write");
exit(EXIT_WRITE);
}
+3 -1
View File
@@ -4,9 +4,11 @@
struct stringpool {
unsigned long left = 0;
unsigned long right = 0;
unsigned long off = 0;
size_t hash; // hash of the string at this node
};
long long addpool(struct memfile *poolfile, struct memfile *treefile, const char *s, char type);
long long addpool(struct memfile *poolfile, struct memfile *treefile, const char *s, char type, std::vector<ssize_t> &dedup);
#endif
+4
View File
@@ -217,3 +217,7 @@ void set_projection_or_exit(const char *optarg) {
exit(EXIT_ARGS);
}
}
unsigned long long encode_vertex(unsigned int wx, unsigned int wy) {
return (((unsigned long long) wx) << 32) | wy;
}
+2
View File
@@ -26,4 +26,6 @@ void decode_quadkey(unsigned long long index, unsigned *wx, unsigned *wy);
unsigned long long encode_hilbert(unsigned int wx, unsigned int wy);
void decode_hilbert(unsigned long long index, unsigned *wx, unsigned *wy);
unsigned long long encode_vertex(unsigned int wx, unsigned int wy);
#endif
+269 -36
View File
@@ -13,6 +13,7 @@
#include "mvt.hpp"
#include "milo/dtoa_milo.h"
#include "errors.hpp"
#include "serial.hpp"
const char *geometry_names[GEOM_TYPES] = {
"Point",
@@ -116,54 +117,286 @@ void parse_geometry(int t, json_object *j, drawvec &out, int op, const char *fna
}
}
void stringify_value(json_object *value, int &type, std::string &stringified, const char *reading, int line, json_object *feature) {
// This is used to convert a JSON attribute value into a serial_val-style
// type and stringified value. All numeric values, even if they are integers,
// even integers that are too large to fit in a double but will still be
// stringified with their original precision, are recorded here as mvt_double.
serial_val stringify_value(json_object *value, const char *reading, int line, json_object *feature) {
serial_val sv;
if (value != NULL) {
int vt = value->type;
std::string val;
if (vt == JSON_STRING) {
val = value->value.string.string;
} else if (vt == JSON_NUMBER) {
if (value->value.number.large_unsigned != 0) {
val = std::to_string(value->value.number.large_unsigned);
} else if (value->value.number.large_signed != 0) {
val = std::to_string(value->value.number.large_signed);
} else {
val = milo::dtoa_milo(value->value.number.number);
}
} else if (vt == JSON_TRUE) {
val = "true";
} else if (vt == JSON_FALSE) {
val = "false";
} else if (vt == JSON_NULL) {
val = "null";
} else {
const char *v = json_stringify(value);
val = std::string(v);
free((void *) v); // stringify
}
sv.type = mvt_string;
sv.s = value->value.string.string;
if (vt == JSON_STRING) {
type = mvt_string;
stringified = val;
std::string err = check_utf8(val);
if (err != "") {
std::string err = check_utf8(sv.s);
if (err.size() > 0) {
fprintf(stderr, "%s:%d: %s: ", reading, line, err.c_str());
json_context(feature);
exit(EXIT_UTF8);
}
} else if (vt == JSON_NUMBER) {
type = mvt_double;
stringified = val;
} else if (vt == JSON_TRUE || vt == JSON_FALSE) {
type = mvt_bool;
stringified = val;
sv.type = mvt_double;
if (value->value.number.large_unsigned != 0) {
sv.s = std::to_string(value->value.number.large_unsigned);
} else if (value->value.number.large_signed != 0) {
sv.s = std::to_string(value->value.number.large_signed);
} else {
sv.s = milo::dtoa_milo(value->value.number.number);
}
} else if (vt == JSON_TRUE) {
sv.type = mvt_bool;
sv.s = "true";
} else if (vt == JSON_FALSE) {
sv.type = mvt_bool;
sv.s = "false";
} else if (vt == JSON_NULL) {
type = mvt_null;
stringified = "null";
sv.type = mvt_null;
sv.s = "null";
} else {
type = mvt_string;
stringified = val;
sv.type = mvt_string;
const char *v = json_stringify(value);
sv.s = std::string(v);
free((void *) v); // stringify
}
}
return sv;
}
// XXX deduplicate
static std::vector<mvt_geometry> to_feature(drawvec &geom) {
std::vector<mvt_geometry> out;
for (size_t i = 0; i < geom.size(); i++) {
out.push_back(mvt_geometry(geom[i].op, geom[i].x, geom[i].y));
}
return out;
}
std::vector<mvt_layer> parse_layers(FILE *fp, int z, unsigned x, unsigned y, int extent, bool fix_longitudes) {
std::map<std::string, mvt_layer> ret;
std::shared_ptr<std::string> tile_stringpool = std::make_shared<std::string>();
json_pull *jp = json_begin_file(fp);
while (1) {
json_object *j = json_read(jp);
if (j == NULL) {
if (jp->error != NULL) {
fprintf(stderr, "Filter output:%d: %s: ", jp->line, jp->error);
if (jp->root != NULL) {
json_context(jp->root);
} else {
fprintf(stderr, "\n");
}
exit(EXIT_JSON);
}
json_free(jp->root);
break;
}
json_object *type = json_hash_get(j, "type");
if (type == NULL || type->type != JSON_STRING) {
continue;
}
if (strcmp(type->value.string.string, "Feature") != 0) {
continue;
}
json_object *geometry = json_hash_get(j, "geometry");
if (geometry == NULL) {
fprintf(stderr, "Filter output:%d: filtered feature with no geometry: ", jp->line);
json_context(j);
json_free(j);
exit(EXIT_JSON);
}
json_object *properties = json_hash_get(j, "properties");
if (properties == NULL || (properties->type != JSON_HASH && properties->type != JSON_NULL)) {
fprintf(stderr, "Filter output:%d: feature without properties hash: ", jp->line);
json_context(j);
json_free(j);
exit(EXIT_JSON);
}
json_object *geometry_type = json_hash_get(geometry, "type");
if (geometry_type == NULL) {
fprintf(stderr, "Filter output:%d: null geometry (additional not reported): ", jp->line);
json_context(j);
exit(EXIT_JSON);
}
if (geometry_type->type != JSON_STRING) {
fprintf(stderr, "Filter output:%d: geometry type is not a string: ", jp->line);
json_context(j);
exit(EXIT_JSON);
}
json_object *coordinates = json_hash_get(geometry, "coordinates");
if (coordinates == NULL || coordinates->type != JSON_ARRAY) {
fprintf(stderr, "Filter output:%d: feature without coordinates array: ", jp->line);
json_context(j);
exit(EXIT_JSON);
}
int t;
for (t = 0; t < GEOM_TYPES; t++) {
if (strcmp(geometry_type->value.string.string, geometry_names[t]) == 0) {
break;
}
}
if (t >= GEOM_TYPES) {
fprintf(stderr, "Filter output:%d: Can't handle geometry type %s: ", jp->line, geometry_type->value.string.string);
json_context(j);
exit(EXIT_JSON);
}
std::string layername = "unknown";
json_object *tippecanoe = json_hash_get(j, "tippecanoe");
json_object *layer = NULL;
if (tippecanoe != NULL) {
layer = json_hash_get(tippecanoe, "layer");
if (layer != NULL && layer->type == JSON_STRING) {
layername = std::string(layer->value.string.string);
}
}
if (ret.count(layername) == 0) {
mvt_layer l;
l.name = layername;
l.version = 2;
l.extent = extent;
ret.insert(std::pair<std::string, mvt_layer>(layername, l));
}
auto l = ret.find(layername);
drawvec dv;
parse_geometry(t, coordinates, dv, VT_MOVETO, "Filter output", jp->line, j);
// handle longitude wraparound
//
// this is supposed to be data for a single tile,
// so any jump from the left hand side edge of the world
// to the right edge, or vice versa, is unexpected,
// so move it to the other side.
if (fix_longitudes && mb_geometry[t] == VT_POLYGON) {
const long long quarter_world = 1LL << 30;
const long long world = 1LL << 32;
bool copy_to_left = false;
bool copy_to_right = false;
for (size_t i = 0; i < dv.size(); i++) {
// is this vertex on a different side of the world
// than the first vertex? then shift this one to match
if (i > 0) {
if ((dv[0].x < quarter_world) && (dv[i].x > 3 * quarter_world)) {
dv[i].x -= world;
}
if ((dv[0].x > 3 * quarter_world) && (dv[i].x < quarter_world)) {
dv[i].x += world;
}
}
// does it stick off the edge of the world?
// then we need another copy on the other side of the world
if (dv[i].x < 0) {
copy_to_right = true;
}
if (dv[i].x > world) {
copy_to_left = true;
}
}
if (copy_to_left) {
size_t n = dv.size();
for (size_t i = 0; i < n; i++) {
dv.emplace_back(dv[i].op, dv[i].x - world, (long long) dv[i].y);
}
}
if (copy_to_right) {
size_t n = dv.size();
for (size_t i = 0; i < n; i++) {
dv.emplace_back(dv[i].op, dv[i].x + world, (long long) dv[i].y);
}
}
}
if (mb_geometry[t] == VT_POLYGON) {
dv = fix_polygon(dv, false, false);
}
// Offset and scale geometry from global to tile
for (size_t i = 0; i < dv.size(); i++) {
long long scale = 1LL << (32 - z);
// offset to tile
dv[i].x -= scale * x;
dv[i].y -= scale * y;
// scale to tile
dv[i].x = std::round(dv[i].x * extent / (double) scale);
dv[i].y = std::round(dv[i].y * extent / (double) scale);
}
if (mb_geometry[t] == VT_POLYGON) {
// don't try scaling up because we may have coordinates
// on the other side of the world
dv = clean_or_clip_poly(dv, z, 256, true, false);
if (dv.size() < 3) {
dv.clear();
}
}
dv = remove_noop(dv, mb_geometry[t], 0);
if (mb_geometry[t] == VT_POLYGON) {
dv = close_poly(dv);
}
if (dv.size() > 0) {
mvt_feature feature;
feature.type = mb_geometry[t];
feature.geometry = to_feature(dv);
json_object *id = json_hash_get(j, "id");
if (id != NULL && id->type == JSON_NUMBER) {
feature.id = id->value.number.number;
if (id->value.number.large_unsigned > 0) {
feature.id = id->value.number.large_unsigned;
}
feature.has_id = true;
}
for (size_t i = 0; i < properties->value.object.length; i++) {
serial_val sv = stringify_value(properties->value.object.values[i], "Filter output", jp->line, j);
// Nulls can be excluded here because this is the postfilter
// and it is nearly time to create the vector representation
if (sv.type != mvt_null) {
mvt_value v = stringified_to_mvt_value(sv.type, sv.s.c_str(), tile_stringpool);
l->second.tag(feature, std::string(properties->value.object.keys[i]->value.string.string), v);
}
}
l->second.features.push_back(feature);
}
json_free(j);
}
json_end(jp);
std::vector<mvt_layer> final;
for (auto a : ret) {
final.push_back(a.second);
}
return final;
}
+2 -1
View File
@@ -12,5 +12,6 @@ extern int mb_geometry[GEOM_TYPES];
void json_context(json_object *j);
void parse_geometry(int t, json_object *j, drawvec &out, int op, const char *fname, int line, json_object *feature);
std::vector<mvt_layer> parse_layers(FILE *fp, int z, unsigned x, unsigned y, int extent, bool fix_longitudes);
void stringify_value(json_object *value, int &type, std::string &stringified, const char *reading, int line, json_object *feature);
serial_val stringify_value(json_object *value, const char *reading, int line, json_object *feature);
+91 -72
View File
@@ -23,6 +23,7 @@
#include "evaluator.hpp"
#include "milo/dtoa_milo.h"
#include "errors.hpp"
#include "text.hpp"
// Offset coordinates to keep them positive
#define COORD_OFFSET (4LL << 32)
@@ -52,25 +53,16 @@ void serialize_long_long(FILE *out, long long n, std::atomic<long long> *fpos, c
}
void serialize_ulong_long(FILE *out, unsigned long long zigzag, std::atomic<long long> *fpos, const char *fname) {
while (1) {
unsigned char b = zigzag & 0x7F;
if ((zigzag >> 7) != 0) {
b |= 0x80;
if (putc(b, out) == EOF) {
fprintf(stderr, "%s: Write to temporary file failed: %s\n", fname, strerror(errno));
exit(EXIT_WRITE);
}
*fpos += 1;
zigzag >>= 7;
} else {
if (putc(b, out) == EOF) {
fprintf(stderr, "%s: Write to temporary file failed: %s\n", fname, strerror(errno));
exit(EXIT_WRITE);
}
*fpos += 1;
break;
}
char buf[10]; // ceil(64 / 7)
char *s = buf;
while (zigzag >= 0x80) {
*s++ = (zigzag & 0x7F) | 0x80;
zigzag >>= 7;
}
*s++ = zigzag;
fwrite_check(buf, sizeof(char), s - buf, out, fpos, fname);
}
void serialize_byte(FILE *out, signed char n, std::atomic<long long> *fpos, const char *fname) {
@@ -89,17 +81,16 @@ size_t fwrite_check(const void *ptr, size_t size, size_t nitems, std::string &st
}
void serialize_ulong_long(std::string &out, unsigned long long zigzag) {
while (1) {
unsigned char b = zigzag & 0x7F;
if ((zigzag >> 7) != 0) {
b |= 0x80;
out += b;
zigzag >>= 7;
} else {
out += b;
break;
}
char buf[10]; // ceil(64 / 7)
char *s = buf;
while (zigzag >= 0x80) {
*s++ = (zigzag & 0x7F) | 0x80;
zigzag >>= 7;
}
*s++ = zigzag;
out.append(buf, s - buf);
}
void serialize_long_long(std::string &out, long long n) {
@@ -122,19 +113,19 @@ void serialize_uint(std::string &out, unsigned n) {
// read from memory
void deserialize_int(char **f, int *n) {
void deserialize_int(const char **f, int *n) {
long long ll;
deserialize_long_long(f, &ll);
*n = ll;
}
void deserialize_long_long(char **f, long long *n) {
void deserialize_long_long(const char **f, long long *n) {
unsigned long long zigzag = 0;
deserialize_ulong_long(f, &zigzag);
*n = protozero::decode_zigzag64(zigzag);
}
void deserialize_ulong_long(char **f, unsigned long long *zigzag) {
void deserialize_ulong_long(const char **f, unsigned long long *zigzag) {
*zigzag = 0;
int shift = 0;
@@ -152,13 +143,13 @@ void deserialize_ulong_long(char **f, unsigned long long *zigzag) {
}
}
void deserialize_uint(char **f, unsigned *n) {
void deserialize_uint(const char **f, unsigned *n) {
unsigned long long v;
deserialize_ulong_long(f, &v);
*n = v;
}
void deserialize_byte(char **f, signed char *n) {
void deserialize_byte(const char **f, signed char *n) {
memcpy(n, *f, sizeof(signed char));
*f += sizeof(signed char);
}
@@ -187,7 +178,6 @@ std::string serialize_feature(serial_feature *sf, long long wx, long long wy) {
#define FLAG_LAYER 7
#define FLAG_LABEL_POINT 6
#define FLAG_SEQ 5
#define FLAG_INDEX 4
#define FLAG_EXTENT 3
#define FLAG_ID 2
@@ -197,21 +187,18 @@ std::string serialize_feature(serial_feature *sf, long long wx, long long wy) {
long long layer = 0;
layer |= sf->layer << FLAG_LAYER;
layer |= (sf->label_point != 0) << FLAG_LABEL_POINT;
layer |= (sf->seq != 0) << FLAG_SEQ;
layer |= (sf->index != 0) << FLAG_INDEX;
layer |= (sf->extent != 0) << FLAG_EXTENT;
layer |= sf->has_id << FLAG_ID;
layer |= sf->has_tippecanoe_minzoom << FLAG_MINZOOM;
layer |= sf->has_tippecanoe_maxzoom << FLAG_MAXZOOM;
layer |= (sf->tippecanoe_minzoom != -1) << FLAG_MINZOOM;
layer |= (sf->tippecanoe_maxzoom != -1) << FLAG_MAXZOOM;
serialize_long_long(s, layer);
if (sf->seq != 0) {
serialize_long_long(s, sf->seq);
}
if (sf->has_tippecanoe_minzoom) {
serialize_long_long(s, sf->seq);
if (sf->tippecanoe_minzoom != -1) {
serialize_int(s, sf->tippecanoe_minzoom);
}
if (sf->has_tippecanoe_maxzoom) {
if (sf->tippecanoe_maxzoom != -1) {
serialize_int(s, sf->tippecanoe_maxzoom);
}
if (sf->has_id) {
@@ -224,6 +211,7 @@ std::string serialize_feature(serial_feature *sf, long long wx, long long wy) {
if (sf->index != 0) {
serialize_ulong_long(s, sf->index);
serialize_ulong_long(s, sf->gap);
}
if (sf->label_point != 0) {
serialize_ulong_long(s, sf->label_point);
@@ -244,17 +232,15 @@ std::string serialize_feature(serial_feature *sf, long long wx, long long wy) {
return s;
}
serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y) {
serial_feature deserialize_feature(std::string const &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y) {
serial_feature sf;
char *cp = (char *) geoms.c_str();
const char *cp = geoms.c_str();
deserialize_byte(&cp, &sf.t);
deserialize_long_long(&cp, &sf.layer);
sf.seq = 0;
if (sf.layer & (1 << FLAG_SEQ)) {
deserialize_long_long(&cp, &sf.seq);
}
deserialize_long_long(&cp, &sf.seq);
sf.tippecanoe_minzoom = -1;
sf.tippecanoe_maxzoom = -1;
@@ -274,6 +260,7 @@ serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx,
deserialize_int(&cp, &sf.segment);
sf.index = 0;
sf.gap = 0;
sf.label_point = 0;
sf.extent = 0;
@@ -281,6 +268,7 @@ serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx,
if (sf.layer & (1 << FLAG_INDEX)) {
deserialize_ulong_long(&cp, &sf.index);
deserialize_ulong_long(&cp, &sf.gap);
}
if (sf.layer & (1 << FLAG_LABEL_POINT)) {
deserialize_ulong_long(&cp, &sf.label_point);
@@ -419,13 +407,13 @@ static void add_scaled_node(struct reader *r, serialization_state *sst, draw g)
long long y = SHIFT_LEFT(g.y);
struct node n;
n.index = encode_quadkey((unsigned) x, (unsigned) y);
n.index = encode_vertex((unsigned) x, (unsigned) y);
fwrite_check((char *) &n, sizeof(struct node), 1, r->nodefile, &r->nodepos, sst->fname);
}
// called from frontends
int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
int serialize_feature(struct serialization_state *sst, serial_feature &sf, std::string const &layername) {
struct reader *r = &(*sst->readers)[sst->segment];
sf.bbox[0] = LLONG_MAX;
@@ -435,12 +423,15 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
for (size_t i = 0; i < sf.geometry.size(); i++) {
if (sf.geometry[i].op == VT_MOVETO || sf.geometry[i].op == VT_LINETO) {
if (sf.geometry[i].y > 0 && sf.geometry[i].y < 0xFFFFFFFF) {
// standard -180 to 180 world plane
// standard -180 to 180 world plane
//
// mask X, since it wraps around
// pin Y, since it doesn't
long long x = sf.geometry[i].x & 0xFFFFFFFF;
long long y = sf.geometry[i].y & 0xFFFFFFFF;
long long x = sf.geometry[i].x & 0xFFFFFFFF;
long long y = std::max(std::min(sf.geometry[i].y, 0xFFFFFFFFLL), 0LL);
{
r->file_bbox1[0] = std::min(r->file_bbox1[0], x);
r->file_bbox1[1] = std::min(r->file_bbox1[1], y);
r->file_bbox1[2] = std::max(r->file_bbox1[2], x);
@@ -469,7 +460,7 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
// This has to happen after scaling so that the wraparound detection has happened first.
// Otherwise the inner/outer calculation will be confused by bad geometries.
if (sf.t == VT_POLYGON) {
scaled_geometry = fix_polygon(scaled_geometry);
scaled_geometry = fix_polygon(scaled_geometry, prevent[P_USE_SOURCE_POLYGON_WINDING], prevent[P_REVERSE_SOURCE_POLYGON_WINDING]);
}
for (auto &c : clipbboxes) {
@@ -620,7 +611,7 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
locs.push_back(encode_index(SHIFT_LEFT(scaled_geometry[i].x), SHIFT_LEFT(scaled_geometry[i].y)));
}
}
std::sort(locs.begin(), locs.end());
std::stable_sort(locs.begin(), locs.end());
size_t n = 0;
double sum = 0;
for (size_t i = 1; i < locs.size(); i++) {
@@ -683,10 +674,6 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
*(sst->area_sum) += extent;
}
if (!prevent[P_INPUT_ORDER]) {
sf.seq = 0;
}
unsigned long long bbox_index;
long long midx, midy;
@@ -694,6 +681,13 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
// keep old behavior, which loses one bit of precision at the bottom
midx = (sf.bbox[0] / 2 + sf.bbox[2] / 2) & ((1LL << 32) - 1);
midy = (sf.bbox[1] / 2 + sf.bbox[3] / 2) & ((1LL << 32) - 1);
} else if ((additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED]) && sf.t == VT_POLYGON) {
// This probably should really apply to all polygons,
// but I hate to change the feature sequence in all the
// test fixtures again
draw scaled_center = center_of_mass_mp(scaled_geometry);
midx = SHIFT_LEFT(scaled_center.x) & ((1LL << 32) - 1);
midy = SHIFT_LEFT(scaled_center.y) & ((1LL << 32) - 1);
} else {
// To reduce the chances of giving multiple polygons or linestrings
// the same index, use an arbitrary but predictable point from the
@@ -719,6 +713,14 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
}
bbox_index = encode_index(midx, midy);
if (additional[A_CALCULATE_INDEX]) {
sf.full_keys.push_back("tippecanoe:index");
serial_val sv;
sv.type = mvt_double;
sv.s = std::to_string(bbox_index);
sf.full_values.push_back(sv);
}
if (sf.t == VT_POLYGON && additional[A_GENERATE_POLYGON_LABEL_POINTS]) {
drawvec dv = polygon_to_anchor(scaled_geometry);
@@ -729,17 +731,30 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
}
}
if (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED] || additional[A_CALCULATE_FEATURE_DENSITY] || additional[A_DROP_SMALLEST_AS_NEEDED] || additional[A_COALESCE_SMALLEST_AS_NEEDED] || additional[A_INCREASE_GAMMA_AS_NEEDED] || additional[A_GENERATE_POLYGON_LABEL_POINTS] || sst->uses_gamma || cluster_distance != 0) {
if (additional[A_DROP_DENSEST_AS_NEEDED] ||
additional[A_COALESCE_DENSEST_AS_NEEDED] ||
additional[A_CLUSTER_DENSEST_AS_NEEDED] ||
additional[A_CALCULATE_FEATURE_DENSITY] ||
additional[A_DROP_SMALLEST_AS_NEEDED] ||
additional[A_COALESCE_SMALLEST_AS_NEEDED] ||
additional[A_DROP_FRACTION_AS_NEEDED] ||
additional[A_COALESCE_FRACTION_AS_NEEDED] ||
prevent[P_DYNAMIC_DROP] ||
additional[A_INCREASE_GAMMA_AS_NEEDED] ||
additional[A_GENERATE_POLYGON_LABEL_POINTS] ||
sst->uses_gamma ||
retain_points_multiplier > 1 ||
cluster_distance != 0) {
sf.index = bbox_index;
} else {
sf.index = 0;
}
if (sst->layermap->count(sf.layername) == 0) {
sst->layermap->insert(std::pair<std::string, layermap_entry>(sf.layername, layermap_entry(sst->layermap->size())));
if (sst->layermap->count(layername) == 0) {
sst->layermap->emplace(layername, layermap_entry(sst->layermap->size()));
}
auto ai = sst->layermap->find(sf.layername);
auto ai = sst->layermap->find(layername);
if (ai != sst->layermap->end()) {
sf.layer = ai->second.id;
@@ -753,7 +768,7 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
}
}
} else {
fprintf(stderr, "Internal error: can't find layer name %s\n", sf.layername.c_str());
fprintf(stderr, "Internal error: can't find layer name %s\n", layername.c_str());
exit(EXIT_IMPOSSIBLE);
}
@@ -837,18 +852,22 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
if (!sst->filters) {
for (size_t i = 0; i < sf.full_keys.size(); i++) {
type_and_string attrib;
attrib.type = sf.full_values[i].type;
attrib.string = sf.full_values[i].s;
auto ts = sst->layermap->find(layername);
add_to_tilestats(ts->second.tilestats, sf.full_keys[i], sf.full_values[i]);
}
}
auto fk = sst->layermap->find(sf.layername);
add_to_file_keys(fk->second.file_keys, sf.full_keys[i], attrib);
if (maximum_string_attribute_length > 0) {
for (size_t i = 0; i < sf.full_keys.size(); i++) {
if (sf.full_values[i].type == mvt_string && sf.full_values[i].s.size() > maximum_string_attribute_length) {
sf.full_values[i].s = truncate_string(sf.full_values[i].s, maximum_string_attribute_length);
}
}
}
for (size_t i = 0; i < sf.full_keys.size(); i++) {
sf.keys.push_back(addpool(r->poolfile, r->treefile, sf.full_keys[i].c_str(), mvt_string));
sf.values.push_back(addpool(r->poolfile, r->treefile, sf.full_values[i].s.c_str(), sf.full_values[i].type));
sf.keys.push_back(addpool(r->poolfile, r->treefile, sf.full_keys[i].c_str(), mvt_string, r->key_dedup));
sf.values.push_back(addpool(r->poolfile, r->treefile, sf.full_values[i].s.c_str(), sf.full_values[i].type, r->value_dedup));
}
long long geomstart = r->geompos;
@@ -892,7 +911,7 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
return 1;
}
void coerce_value(std::string const &key, int &vt, std::string &val, std::map<std::string, int> const *attribute_types) {
void coerce_value(std::string const &key, int &vt, std::string &val, std::unordered_map<std::string, int> const *attribute_types) {
auto a = (*attribute_types).find(key);
if (a != attribute_types->end()) {
if (a->second == mvt_string) {
+75 -24
View File
@@ -25,15 +25,30 @@ void serialize_ulong_long(std::string &out, unsigned long long n);
void serialize_byte(std::string &out, signed char n);
void serialize_uint(std::string &out, unsigned n);
void deserialize_int(char **f, int *n);
void deserialize_long_long(char **f, long long *n);
void deserialize_ulong_long(char **f, unsigned long long *n);
void deserialize_uint(char **f, unsigned *n);
void deserialize_byte(char **f, signed char *n);
void deserialize_int(const char **f, int *n);
void deserialize_long_long(const char **f, long long *n);
void deserialize_ulong_long(const char **f, unsigned long long *n);
void deserialize_uint(const char **f, unsigned *n);
void deserialize_byte(const char **f, signed char *n);
// This is the main representation of attribute values in memory and
// in the string pool. The type is one of the mvt_value type (mvt_string,
// mvt_double, mvt_bool, or mvt_null). Note that all numeric values,
// whether integer or floating point, use mvt_double here.
struct serial_val {
int type = 0;
std::string s = "";
int type;
std::string s;
bool operator<(const serial_val &o) const;
bool operator!=(const serial_val &o) const;
serial_val() {
type = 0;
}
serial_val(int t, const std::string &val)
: type(t), s(val) {
}
};
struct serial_feature {
@@ -47,31 +62,66 @@ struct serial_feature {
bool has_id = false;
unsigned long long id = 0;
bool has_tippecanoe_minzoom = false;
int tippecanoe_minzoom = 0;
bool has_tippecanoe_maxzoom = false;
int tippecanoe_maxzoom = 0;
int tippecanoe_minzoom = -1;
int tippecanoe_maxzoom = -1;
drawvec geometry = drawvec();
unsigned long long index = 0;
unsigned long long gap = 0; // filled in during z0. square of planar distance
unsigned long long label_point = 0;
long long extent = 0;
// These fields are not directly serialized, but are used
// to create the keys and values references into the string pool
// during initial serialization
std::vector<std::string> full_keys{};
std::vector<serial_val> full_values{};
// These fields are generated from full_keys and full_values
// during initial serialization and then replace the string
// representations:
std::vector<long long> keys{};
std::vector<long long> values{};
// XXX This isn't serialized. Should it be here?
// These fields are used during tiling,
// but are not serialized and are not expected
// to be provided by frontends:
long long bbox[4] = {0, 0, 0, 0};
std::vector<std::string> full_keys{};
std::vector<serial_val> full_values{};
std::string layername = "";
bool dropped = false;
drawvec edge_nodes;
drawvec edge_nodes; // what nodes at the tile edge were added during clipping?
#define FEATURE_DROPPED -1
#define FEATURE_KEPT 0
// <0: dropped
// 0: kept
// >0: sequence number of additional feature kept by retain-points-multiplier
int dropped = FEATURE_DROPPED; // was this feature dropped by rate?
// unsigned long long drop_by; // dot-dropping priority
bool reduced; // is polygon dust
bool coalesced; // was coalesced from multiple features
int line_detail; // current tile resolution being used for simplification
int extra_detail; // extra tile resolution to retain in output
int maxzoom;
double spacing; // feature spacing for --calculate-feature-density
double simplification; // simplification level at this zoom level
std::vector<ssize_t> arc_polygon; // used in --detect-shared-borders
ssize_t renamed; // used in --detect-shared-borders logic
long long clustered; // does this feature need the clustered/point_count attributes?
const char *stringpool; // string pool for keys/values lookup
std::shared_ptr<std::string> tile_stringpool; // string pool for mvt_value construction
std::set<std::string> need_tilestats;
std::unordered_map<std::string, accum_state> attribute_accum_state;
int z; // tile being produced
int tx;
int ty;
};
std::string serialize_feature(serial_feature *sf, long long wx, long long wy);
serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y);
serial_feature deserialize_feature(std::string const &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y);
struct reader {
int poolfd = -1;
@@ -94,14 +144,15 @@ struct reader {
std::atomic<long long> nodepos;
long long file_bbox[4] = {0, 0, 0, 0};
long long file_bbox1[4] = {0xFFFFFFFF, 0xFFFFFFFF, 0, 0}; // standard -180 to 180 world plane
long long file_bbox2[4] = {0x1FFFFFFFF, 0xFFFFFFFF, 0x100000000, 0}; // 0 to 360 world plane
struct stat geomst {};
char *geom_map = NULL;
std::vector<ssize_t> key_dedup = std::vector<ssize_t>(655536, -1);
std::vector<ssize_t> value_dedup = std::vector<ssize_t>(655536, -1);
reader()
: geompos(0), indexpos(0), vertexpos(0), nodepos(0) {
}
@@ -168,7 +219,7 @@ struct serialization_state {
std::map<std::string, layermap_entry> *layermap = NULL;
std::map<std::string, int> const *attribute_types = NULL;
std::unordered_map<std::string, int> const *attribute_types = NULL;
std::set<std::string> *exclude = NULL;
std::set<std::string> *include = NULL;
int exclude_all = 0;
@@ -220,7 +271,7 @@ struct node {
int nodecmp(const void *void1, const void *void2);
int serialize_feature(struct serialization_state *sst, serial_feature &sf);
void coerce_value(std::string const &key, int &vt, std::string &val, std::map<std::string, int> const *attribute_types);
int serialize_feature(struct serialization_state *sst, serial_feature &sf, std::string const &layername);
void coerce_value(std::string const &key, int &vt, std::string &val, std::unordered_map<std::string, int> const *attribute_types);
#endif
+575
View File
@@ -0,0 +1,575 @@
#include <stdio.h>
#include <algorithm>
#include "geometry.hpp"
#include "serial.hpp"
#include "options.hpp"
// This should all be removed and replaced with --no-simplification-of-shared-nodes
// Does not fix up moveto/lineto
static drawvec reverse_subring(drawvec const &dv) {
drawvec out;
for (size_t i = dv.size(); i > 0; i--) {
out.push_back(dv[i - 1]);
}
return out;
}
struct edge {
unsigned x1 = 0;
unsigned y1 = 0;
unsigned x2 = 0;
unsigned y2 = 0;
unsigned ring = 0;
edge(unsigned _x1, unsigned _y1, unsigned _x2, unsigned _y2, unsigned _ring) {
x1 = _x1;
y1 = _y1;
x2 = _x2;
y2 = _y2;
ring = _ring;
}
bool operator<(const edge &s) const {
long long cmp = (long long) y1 - s.y1;
if (cmp == 0) {
cmp = (long long) x1 - s.x1;
}
if (cmp == 0) {
cmp = (long long) y2 - s.y2;
}
if (cmp == 0) {
cmp = (long long) x2 - s.x2;
}
return cmp < 0;
}
};
struct edgecmp_ring {
bool operator()(const edge &a, const edge &b) {
long long cmp = (long long) a.y1 - b.y1;
if (cmp == 0) {
cmp = (long long) a.x1 - b.x1;
}
if (cmp == 0) {
cmp = (long long) a.y2 - b.y2;
}
if (cmp == 0) {
cmp = (long long) a.x2 - b.x2;
}
if (cmp == 0) {
cmp = (long long) a.ring - b.ring;
}
return cmp < 0;
}
} edgecmp_ring;
bool edges_same(std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e1, std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e2) {
if ((e2.second - e2.first) != (e1.second - e1.first)) {
return false;
}
while (e1.first != e1.second) {
if (e1.first->ring != e2.first->ring) {
return false;
}
++e1.first;
++e2.first;
}
return true;
}
bool find_common_edges(std::vector<serial_feature> &features, int z, int line_detail, double simplification, int maxzoom, double merge_fraction) {
size_t merge_count = ceil((1 - merge_fraction) * features.size());
for (size_t i = 0; i < features.size(); i++) {
if (features[i].t == VT_POLYGON) {
{
drawvec &g = features[i].geometry;
drawvec out;
for (size_t k = 0; k < g.size(); k++) {
if (g[k].op == VT_LINETO && k > 0 && g[k - 1] == g[k]) {
;
} else {
out.push_back(g[k]);
}
}
features[i].geometry = out;
}
}
}
// Construct a mapping from all polygon edges to the set of rings
// that each edge appears in. (The ring number is across all polygons;
// we don't need to look it back up, just to tell where it changes.)
std::vector<edge> edges;
size_t ring = 0;
for (size_t i = 0; i < features.size(); i++) {
if (features[i].t == VT_POLYGON) {
{
for (size_t k = 0; k + 1 < features[i].geometry.size(); k++) {
if (features[i].geometry[k].op == VT_MOVETO) {
ring++;
}
if (features[i].geometry[k + 1].op == VT_LINETO) {
drawvec dv;
if (features[i].geometry[k] < features[i].geometry[k + 1]) {
dv.push_back(features[i].geometry[k]);
dv.push_back(features[i].geometry[k + 1]);
} else {
dv.push_back(features[i].geometry[k + 1]);
dv.push_back(features[i].geometry[k]);
}
edges.push_back(edge(dv[0].x, dv[0].y, dv[1].x, dv[1].y, ring));
}
}
}
}
}
std::stable_sort(edges.begin(), edges.end(), edgecmp_ring);
std::set<draw> necessaries;
// Now mark all the points where the set of rings using the edge on one side
// is not the same as the set of rings using the edge on the other side.
for (size_t i = 0; i < features.size(); i++) {
if (features[i].t == VT_POLYGON) {
{
drawvec &g = features[i].geometry;
for (size_t k = 0; k < g.size(); k++) {
g[k].necessary = 0;
}
for (size_t a = 0; a < g.size(); a++) {
if (g[a].op == VT_MOVETO) {
size_t b;
for (b = a + 1; b < g.size(); b++) {
if (g[b].op != VT_LINETO) {
break;
}
}
// -1 because of duplication at the end
size_t s = b - a - 1;
if (s > 0) {
drawvec left;
if (g[a + (s - 1) % s] < g[a]) {
left.push_back(g[a + (s - 1) % s]);
left.push_back(g[a]);
} else {
left.push_back(g[a]);
left.push_back(g[a + (s - 1) % s]);
}
if (left[1] < left[0]) {
fprintf(stderr, "left misordered\n");
}
std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e1 = std::equal_range(edges.begin(), edges.end(), edge(left[0].x, left[0].y, left[1].x, left[1].y, 0));
for (size_t k = 0; k < s; k++) {
drawvec right;
if (g[a + k] < g[a + k + 1]) {
right.push_back(g[a + k]);
right.push_back(g[a + k + 1]);
} else {
right.push_back(g[a + k + 1]);
right.push_back(g[a + k]);
}
std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e2 = std::equal_range(edges.begin(), edges.end(), edge(right[0].x, right[0].y, right[1].x, right[1].y, 0));
if (right[1] < right[0]) {
fprintf(stderr, "left misordered\n");
}
if (e1.first == e1.second || e2.first == e2.second) {
fprintf(stderr, "Internal error: polygon edge lookup failed for %lld,%lld to %lld,%lld or %lld,%lld to %lld,%lld\n", (long long) left[0].x, (long long) left[0].y, (long long) left[1].x, (long long) left[1].y, (long long) right[0].x, (long long) right[0].y, (long long) right[1].x, (long long) right[1].y);
exit(EXIT_IMPOSSIBLE);
}
if (!edges_same(e1, e2)) {
g[a + k].necessary = 1;
necessaries.insert(g[a + k]);
}
e1 = e2;
}
}
a = b - 1;
}
}
}
}
}
edges.clear();
std::map<drawvec, size_t> arcs;
std::multimap<ssize_t, size_t> merge_candidates; // from arc to serial_feature
// Roll rings that include a necessary point around so they start at one
for (size_t i = 0; i < features.size(); i++) {
if (features[i].t == VT_POLYGON) {
{
drawvec &g = features[i].geometry;
for (size_t k = 0; k < g.size(); k++) {
if (necessaries.count(g[k]) != 0) {
g[k].necessary = 1;
}
}
for (size_t k = 0; k < g.size(); k++) {
if (g[k].op == VT_MOVETO) {
ssize_t necessary = -1;
ssize_t lowest = k;
size_t l;
for (l = k + 1; l < g.size(); l++) {
if (g[l].op != VT_LINETO) {
break;
}
if (g[l].necessary) {
necessary = l;
}
if (g[l] < g[lowest]) {
lowest = l;
}
}
if (necessary < 0) {
necessary = lowest;
// Add a necessary marker if there was none in the ring,
// so the arc code below can find it.
g[lowest].necessary = 1;
}
{
drawvec tmp;
// l - 1 because the endpoint is duplicated
for (size_t m = necessary; m < l - 1; m++) {
tmp.push_back(g[m]);
}
for (ssize_t m = k; m < necessary; m++) {
tmp.push_back(g[m]);
}
// replace the endpoint
tmp.push_back(g[necessary]);
if (tmp.size() != l - k) {
fprintf(stderr, "internal error shifting ring\n");
exit(EXIT_IMPOSSIBLE);
}
for (size_t m = 0; m < tmp.size(); m++) {
if (m == 0) {
tmp[m].op = VT_MOVETO;
} else {
tmp[m].op = VT_LINETO;
}
g[k + m] = tmp[m];
}
}
// Now peel off each set of segments from one necessary point to the next
// into an "arc" as in TopoJSON
for (size_t m = k; m < l; m++) {
if (!g[m].necessary) {
fprintf(stderr, "internal error in arc building\n");
exit(EXIT_IMPOSSIBLE);
}
drawvec arc;
size_t n;
for (n = m; n < l; n++) {
arc.push_back(g[n]);
if (n > m && g[n].necessary) {
break;
}
}
auto f = arcs.find(arc);
if (f == arcs.end()) {
drawvec arc2 = reverse_subring(arc);
auto f2 = arcs.find(arc2);
if (f2 == arcs.end()) {
// Add new arc
size_t added = arcs.size() + 1;
arcs.insert(std::pair<drawvec, size_t>(arc, added));
features[i].arc_polygon.push_back(added);
merge_candidates.insert(std::pair<ssize_t, size_t>(added, i));
} else {
features[i].arc_polygon.push_back(-(ssize_t) f2->second);
merge_candidates.insert(std::pair<ssize_t, size_t>(-(ssize_t) f2->second, i));
}
} else {
features[i].arc_polygon.push_back(f->second);
merge_candidates.insert(std::pair<ssize_t, size_t>(f->second, i));
}
m = n - 1;
}
features[i].arc_polygon.push_back(0);
k = l - 1;
}
}
}
}
}
// Simplify each arc
std::vector<drawvec> simplified_arcs;
for (auto ai = arcs.begin(); ai != arcs.end(); ++ai) {
if (simplified_arcs.size() < ai->second + 1) {
simplified_arcs.resize(ai->second + 1);
}
drawvec dv = ai->first;
for (size_t i = 0; i < dv.size(); i++) {
if (i == 0) {
dv[i].op = VT_MOVETO;
} else {
dv[i].op = VT_LINETO;
}
}
if (!(prevent[P_SIMPLIFY] || (z == maxzoom && prevent[P_SIMPLIFY_LOW]) || (z < maxzoom && additional[A_GRID_LOW_ZOOMS]))) {
// tx and ty are 0 here because we aren't trying to do anything with the shared_nodes_map
simplified_arcs[ai->second] = simplify_lines(dv, z, 0, 0, line_detail, !(prevent[P_CLIPPING] || prevent[P_DUPLICATION]), simplification, 4, drawvec(), NULL, 0, "");
} else {
simplified_arcs[ai->second] = dv;
}
}
// If necessary, merge some adjacent polygons into some other polygons
struct merge_order {
ssize_t edge = 0;
unsigned long long gap = 0;
size_t p1 = 0;
size_t p2 = 0;
bool operator<(const merge_order &m) const {
return gap < m.gap;
}
};
std::vector<merge_order> order;
for (ssize_t i = 0; i < (ssize_t) simplified_arcs.size(); i++) {
auto r1 = merge_candidates.equal_range(i);
for (auto r1i = r1.first; r1i != r1.second; ++r1i) {
auto r2 = merge_candidates.equal_range(-i);
for (auto r2i = r2.first; r2i != r2.second; ++r2i) {
if (r1i->second != r2i->second) {
merge_order mo;
mo.edge = i;
if (features[r1i->second].index > features[r2i->second].index) {
mo.gap = features[r1i->second].index - features[r2i->second].index;
} else {
mo.gap = features[r2i->second].index - features[r1i->second].index;
}
mo.p1 = r1i->second;
mo.p2 = r2i->second;
order.push_back(mo);
}
}
}
}
std::stable_sort(order.begin(), order.end());
size_t merged = 0;
for (size_t o = 0; o < order.size(); o++) {
if (merged >= merge_count) {
break;
}
size_t i = order[o].p1;
while (features[i].renamed >= 0) {
i = features[i].renamed;
}
size_t i2 = order[o].p2;
while (features[i2].renamed >= 0) {
i2 = features[i2].renamed;
}
for (size_t j = 0; j < features[i].arc_polygon.size() && merged < merge_count; j++) {
if (features[i].arc_polygon[j] == order[o].edge) {
{
// XXX snap links
if (features[order[o].p2].arc_polygon.size() > 0) {
// This has to merge the ring that contains the anti-arc to this arc
// into the current ring, and then add whatever other rings were in
// that feature on to the end.
//
// This can't be good for keeping parent-child relationships among
// the rings in order, but Wagyu should sort that out later
std::vector<ssize_t> additions;
std::vector<ssize_t> &here = features[i].arc_polygon;
std::vector<ssize_t> &other = features[i2].arc_polygon;
#if 0
printf("seeking %zd\n", features[i].arc_polygon[j]);
printf("before: ");
for (size_t k = 0; k < here.size(); k++) {
printf("%zd ", here[k]);
}
printf("\n");
printf("other: ");
for (size_t k = 0; k < other.size(); k++) {
printf("%zd ", other[k]);
}
printf("\n");
#endif
for (size_t k = 0; k < other.size(); k++) {
size_t l;
for (l = k; l < other.size(); l++) {
if (other[l] == 0) {
break;
}
}
if (l >= other.size()) {
l--;
}
#if 0
for (size_t m = k; m <= l; m++) {
printf("%zd ", other[m]);
}
printf("\n");
#endif
size_t m;
for (m = k; m <= l; m++) {
if (other[m] == -features[i].arc_polygon[j]) {
break;
}
}
if (m <= l) {
// Found the shared arc
here.erase(here.begin() + j);
size_t off = 0;
for (size_t n = m + 1; n < l; n++) {
here.insert(here.begin() + j + off, other[n]);
off++;
}
for (size_t n = k; n < m; n++) {
here.insert(here.begin() + j + off, other[n]);
off++;
}
} else {
// Looking at some other ring
for (size_t n = k; n <= l; n++) {
additions.push_back(other[n]);
}
}
k = l;
}
features[i2].arc_polygon.clear();
features[i2].renamed = i;
merged++;
for (size_t k = 0; k < additions.size(); k++) {
features[i].arc_polygon.push_back(additions[k]);
}
#if 0
printf("after: ");
for (size_t k = 0; k < here.size(); k++) {
printf("%zd ", here[k]);
}
printf("\n");
#endif
#if 0
for (size_t k = 0; k + 1 < here.size(); k++) {
if (here[k] != 0 && here[k + 1] != 0) {
if (simplified_arcs[here[k + 1]][0] != simplified_arcs[here[k]][simplified_arcs[here[k]].size() - 1]) {
printf("error from %zd to %zd\n", here[k], here[k + 1]);
}
}
}
#endif
}
}
}
}
}
// Turn the arc representations of the polygons back into standard polygon geometries
for (size_t i = 0; i < features.size(); i++) {
if (features[i].t == VT_POLYGON) {
features[i].geometry.clear();
bool at_start = true;
draw first(-1, 0, 0);
for (size_t j = 0; j < features[i].arc_polygon.size(); j++) {
ssize_t p = features[i].arc_polygon[j];
if (p == 0) {
if (first.op >= 0) {
features[i].geometry.push_back(first);
first = draw(-1, 0, 0);
}
at_start = true;
} else if (p > 0) {
for (size_t k = 0; k + 1 < simplified_arcs[p].size(); k++) {
if (at_start) {
features[i].geometry.push_back(draw(VT_MOVETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y));
first = draw(VT_LINETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y);
} else {
features[i].geometry.push_back(draw(VT_LINETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y));
}
at_start = 0;
}
} else { /* p < 0 */
for (ssize_t k = simplified_arcs[-p].size() - 1; k > 0; k--) {
if (at_start) {
features[i].geometry.push_back(draw(VT_MOVETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y));
first = draw(VT_LINETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y);
} else {
features[i].geometry.push_back(draw(VT_LINETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y));
}
at_start = 0;
}
}
}
}
}
if (merged >= merge_count) {
return true;
} else {
return false;
}
}
+1
View File
@@ -0,0 +1 @@
bool find_common_edges(std::vector<serial_feature> &features, int z, int line_detail, double simplification, int maxzoom, double merge_fraction);
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "3.757458,-84.969764,359.744029,85.007238",
"antimeridian_adjusted_bounds": "3.757458,-85.051129,359.744029,85.051129",
"bounds": "-179.320808,-85.051129,177.449262,85.051129",
"center": "-179.320808,-69.033211,14",
"description": "tests/accumulate/out/--set-attribute_thecomma%3aNEWVALUE_--accumulate-attribute_thecomma%3acomma.json.check.mbtiles",
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "3.757458,-84.969764,359.744029,85.007238",
"antimeridian_adjusted_bounds": "3.757458,-85.051129,359.744029,85.051129",
"bounds": "-179.320808,-85.051129,177.449262,85.051129",
"center": "0.000000,0.000000,0",
"description": "tests/accumulate/out/-z0_--set-attribute_%7b%22num%22%3a5,%22str%22%3a%22abc%22}.json.check.mbtiles",
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "3.757458,-84.969764,359.744029,85.007238",
"antimeridian_adjusted_bounds": "3.757458,-85.051129,359.744029,85.051129",
"bounds": "-179.320808,-85.051129,177.449262,85.051129",
"center": "0.000000,0.000000,0",
"description": "tests/accumulate/out/-z0_--set-attribute_num%3a5_--set-attribute_str%3aabc.json.check.mbtiles",
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "3.757458,-84.969764,359.744029,85.007238",
"antimeridian_adjusted_bounds": "3.757458,-85.051129,359.744029,85.051129",
"bounds": "-179.320808,-85.051129,177.449262,85.051129",
"center": "-157.500000,20.489949,3",
"description": "tests/accumulate/out/-z3_--accumulate-attribute_%7b%22thesum%22%3a%22sum%22,%22theproduct%22%3a%22product%22}.json.check.mbtiles",
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "3.757458,-84.969764,359.744029,85.007238",
"antimeridian_adjusted_bounds": "3.757458,-85.051129,359.744029,85.051129",
"bounds": "-179.320808,-85.051129,177.449262,85.051129",
"center": "112.500000,20.489949,3",
"description": "tests/accumulate/out/-z3_-Ethesum%3asum_-Etheproduct%3aproduct_-Ethemax%3amax_-Ethemin%3amin_-Ethemean%3amean_-Etheconcat%3aconcat_-Ethecomma%3acomma_-r1_-K100.json.check.mbtiles",
@@ -10,6 +10,7 @@
"minzoom": "0",
"name": "tests/accumulate/out/-z3_-Ethesum%3asum_-Etheproduct%3aproduct_-Ethemax%3amax_-Ethemin%3amin_-Ethemean%3amean_-Etheconcat%3aconcat_-Ethecomma%3acomma_-r1_-K100.json.check.mbtiles",
"strategies": "[{\"coalesced_as_needed\":78},{\"coalesced_as_needed\":65},{\"coalesced_as_needed\":43},{\"coalesced_as_needed\":17}]",
"tippecanoe_decisions": "{\"basezoom\":3,\"droprate\":1,\"retain_points_multiplier\":1}",
"type": "overlay",
"version": "2"
}, "features": [
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "3.757458,-84.969764,359.744029,85.007238",
"antimeridian_adjusted_bounds": "3.757458,-85.051129,359.744029,85.051129",
"bounds": "-179.320808,-85.051129,177.449262,85.051129",
"center": "-174.375000,-52.349536,5",
"description": "tests/accumulate/out/-z5_-Ethesum%3asum_-Etheproduct%3aproduct_-Ethemax%3amax_-Ethemin%3amin_-Ethemean%3amean_-Etheconcat%3aconcat_-Ethecomma%3acomma.json.check.mbtiles",
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-175.220564,-41.299988,179.216647,64.150024",
"antimeridian_adjusted_bounds": "-175.220564,-41.299988,179.216647,85.051129",
"bounds": "-180.000000,-41.299988,180.000000,85.051129",
"center": "0.000000,0.000000,0",
"description": "tests/csv/out-null.mbtiles",
+1 -1
View File
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-175.220564,-41.299988,179.216647,64.150024",
"antimeridian_adjusted_bounds": "-175.220564,-41.299988,179.216647,85.051129",
"bounds": "-180.000000,-41.299988,180.000000,85.051129",
"center": "0.000000,0.000000,0",
"description": "tests/csv/out.mbtiles",
+1 -1
View File
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "180.000000,85.051129,-180.000000,-85.051129",
"antimeridian_adjusted_bounds": "-135.232962,-85.051129,-135.232962,-85.051129",
"bounds": "-135.232962,-85.051129,-135.232962,-85.051129",
"center": "-135.232962,-85.051129,0",
"description": "tests/invalid-polygon/out/-z0.json.check.mbtiles",
+1 -1
View File
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "180.000000,85.051129,-180.000000,-85.051129",
"antimeridian_adjusted_bounds": "0.000000,-85.051129,0.000000,-85.051129",
"bounds": "-180.000000,-85.051129,180.000000,-85.051129",
"center": "0.000000,-85.051129,0",
"format": "pbf",
+1
View File
@@ -8,6 +8,7 @@
"maxzoom": "0",
"minzoom": "0",
"name": "tests/layer-json/out.mbtiles",
"tippecanoe_decisions": "{\"basezoom\":0,\"droprate\":1,\"retain_points_multiplier\":1}",
"type": "overlay",
"version": "2"
}, "features": [
@@ -0,0 +1,20 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "0.000000,0.000000,0.000000,0.000000",
"bounds": "0.000000,0.000000,0.000000,0.000000",
"center": "0.000000,0.000000,0",
"description": "tests/longattr/out/-z0_--maximum-string-attribute-length_20.json.check.mbtiles",
"format": "pbf",
"generator_options": "./tippecanoe -q -a@ -f -o tests/longattr/out/-z0_--maximum-string-attribute-length_20.json.check.mbtiles -z0 --maximum-string-attribute-length 20 tests/longattr/sherlock.json",
"json": "{\"vector_layers\":[{\"id\":\"sherlock\",\"description\":\"\",\"minzoom\":0,\"maxzoom\":0,\"fields\":{\"sherlock\":\"String\"}}],\"tilestats\":{\"layerCount\":1,\"layers\":[{\"layer\":\"sherlock\",\"count\":1,\"geometry\":\"Point\",\"attributeCount\":1,\"attributes\":[{\"attribute\":\"sherlock\",\"count\":1,\"type\":\"string\",\"values\":[]}]}]}}",
"maxzoom": "0",
"minzoom": "0",
"name": "tests/longattr/out/-z0_--maximum-string-attribute-length_20.json.check.mbtiles",
"type": "overlay",
"version": "2"
}, "features": [
{ "type": "FeatureCollection", "properties": { "zoom": 0, "x": 0, "y": 0 }, "features": [
{ "type": "FeatureCollection", "properties": { "layer": "sherlock", "version": 2, "extent": 4096 }, "features": [
{ "type": "Feature", "properties": { "sherlock": "The Project Gutenber" }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
] }
] }
] }
@@ -9,389 +9,13 @@
"maxzoom": "0",
"minzoom": "0",
"name": "tests/loop/out/-z0_-O200_--drop-fraction-as-needed.json.check.mbtiles",
"strategies": "[{\"dropped_as_needed\":811,\"feature_count_desired\":1000}]",
"strategies": "[{\"dropped_as_needed\":999,\"feature_count_desired\":1000}]",
"type": "overlay",
"version": "2"
}, "features": [
{ "type": "FeatureCollection", "properties": { "zoom": 0, "x": 0, "y": 0 }, "features": [
{ "type": "FeatureCollection", "properties": { "layer": "in", "version": 2, "extent": 4096 }, "features": [
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
,
{ "type": "Feature", "properties": { }, "geometry": { "type": "Point", "coordinates": [ 0.966797, 0.966751 ] } }
] }
] }
] }
+2 -2
View File
@@ -1,9 +1,9 @@
[
{ "zoom": 11, "x": 326, "y": 791, "bytes": 376, "compressed": true, "layers": { "muni": { "points": 14, "lines": 0, "polygons": 0, "extent": 4096 } } }
,
{ "zoom": 11, "x": 327, "y": 792, "bytes": 6533, "compressed": true, "layers": { "muni": { "points": 528, "lines": 0, "polygons": 0, "extent": 4096 } } }
{ "zoom": 11, "x": 327, "y": 792, "bytes": 6543, "compressed": true, "layers": { "muni": { "points": 528, "lines": 0, "polygons": 0, "extent": 4096 } } }
,
{ "zoom": 11, "x": 327, "y": 791, "bytes": 45265, "compressed": true, "layers": { "muni": { "points": 4284, "lines": 0, "polygons": 0, "extent": 4096 }, "subway": { "points": 19, "lines": 0, "polygons": 0, "extent": 4096 } } }
{ "zoom": 11, "x": 327, "y": 791, "bytes": 45475, "compressed": true, "layers": { "muni": { "points": 4284, "lines": 0, "polygons": 0, "extent": 4096 }, "subway": { "points": 19, "lines": 0, "polygons": 0, "extent": 4096 } } }
,
{ "zoom": 11, "x": 954, "y": 791, "bytes": 75, "compressed": true, "layers": { "muni": { "points": 12, "lines": 0, "polygons": 0, "extent": 4096 } } }
]
+1
View File
@@ -10,6 +10,7 @@
"minzoom": "11",
"name": "tests/muni/out/-Z11_-z13_-B15.json.check.mbtiles",
"strategies": "[{},{},{},{},{},{},{},{},{},{},{},{\"dropped_by_rate\":4727},{\"dropped_by_rate\":4643},{\"dropped_by_rate\":4210}]",
"tippecanoe_decisions": "{\"basezoom\":15,\"droprate\":2.5,\"retain_points_multiplier\":1}",
"type": "overlay",
"version": "2"
}, "features": [
+1
View File
@@ -10,6 +10,7 @@
"minzoom": "11",
"name": "tests/muni/out/-Z11_-z13_-Bf2000.json.check.mbtiles",
"strategies": "[{},{},{},{},{},{},{},{},{},{},{},{\"dropped_by_rate\":2910},{},{}]",
"tippecanoe_decisions": "{\"basezoom\":12,\"droprate\":2.5,\"retain_points_multiplier\":1}",
"type": "overlay",
"version": "2"
}, "features": [
+1 -1
View File
@@ -9,7 +9,7 @@
"maxzoom": "13",
"minzoom": "11",
"name": "tests/muni/out/-Z11_-z13_-M10000.json.check.mbtiles",
"strategies": "[{},{},{},{},{},{},{},{},{},{},{},{\"dropped_by_rate\":18540,\"detail_reduced\":4,\"tile_size_desired\":10857},{\"dropped_by_rate\":6391,\"detail_reduced\":3,\"tile_size_desired\":10700},{}]",
"strategies": "[{},{},{},{},{},{},{},{},{},{},{},{\"dropped_by_rate\":4080,\"detail_reduced\":4,\"tile_size_desired\":10887},{\"dropped_by_rate\":2974,\"detail_reduced\":3,\"tile_size_desired\":10721},{}]",
"type": "overlay",
"version": "2"
}, "features": [
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
+1
View File
@@ -10,6 +10,7 @@
"minzoom": "11",
"name": "tests/muni/out/-Z11_-z13_-rf2000.json.check.mbtiles",
"strategies": "[{},{},{},{},{},{},{},{},{},{},{},{\"dropped_by_rate\":2542},{\"dropped_by_rate\":1547},{}]",
"tippecanoe_decisions": "{\"basezoom\":13,\"droprate\":1.4510341140028376,\"retain_points_multiplier\":1}",
"type": "overlay",
"version": "2"
}, "features": [
@@ -10,6 +10,7 @@
"minzoom": "11",
"name": "tests/muni/out/-Z11_-z13_-rf2000_-Bg.json.check.mbtiles",
"strategies": "[{},{},{},{},{},{},{},{},{},{},{},{\"dropped_by_rate\":2546},{},{}]",
"tippecanoe_decisions": "{\"basezoom\":12,\"droprate\":2.1055,\"retain_points_multiplier\":1}",
"type": "overlay",
"version": "2"
}, "features": [
@@ -10,6 +10,7 @@
"minzoom": "11",
"name": "tests/muni/out/-Z11_-z13_-rf2000_-g2.json.check.mbtiles",
"strategies": "[{},{},{},{},{},{},{},{},{},{},{},{\"dropped_by_rate\":1349,\"dropped_by_gamma\":963},{\"dropped_by_rate\":747,\"dropped_by_gamma\":752},{\"dropped_by_gamma\":514}]",
"tippecanoe_decisions": "{\"basezoom\":13,\"droprate\":1.176222768016331,\"retain_points_multiplier\":1}",
"type": "overlay",
"version": "2"
}, "features": [
+1
View File
@@ -10,6 +10,7 @@
"minzoom": "0",
"name": "tests/muni/out/-r1_-K20.json.check.mbtiles",
"strategies": "[{\"coalesced_as_needed\":4608},{\"coalesced_as_needed\":4608},{\"coalesced_as_needed\":4608},{\"coalesced_as_needed\":4608},{\"coalesced_as_needed\":4608},{\"coalesced_as_needed\":4607},{\"coalesced_as_needed\":4607},{\"coalesced_as_needed\":4606},{\"coalesced_as_needed\":6005},{\"coalesced_as_needed\":5393},{\"coalesced_as_needed\":5012},{\"coalesced_as_needed\":4727},{\"coalesced_as_needed\":4577},{\"coalesced_as_needed\":4003},{\"coalesced_as_needed\":3005}]",
"tippecanoe_decisions": "{\"basezoom\":14,\"droprate\":1,\"retain_points_multiplier\":1}",
"type": "overlay",
"version": "2"
}, "features": [
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "0.023802,-85.040752,359.950216,83.645130",
"antimeridian_adjusted_bounds": "0.023802,-85.051129,359.950216,83.645130",
"bounds": "-180.000000,-85.051129,180.000000,83.645130",
"center": "16.875000,44.951199,5",
"description": "tests/ne_110m_admin_0_countries/out/-R5%2f17%2f11.json.check.mbtiles",
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "0.023802,-85.040752,359.950216,83.645130",
"antimeridian_adjusted_bounds": "0.023802,-85.051129,359.950216,83.645130",
"bounds": "-110.039063,26.980829,-92.021484,51.998410",
"center": "-92.021484,26.980829,0",
"description": "tests/ne_110m_admin_0_countries/out/-z0_--clip-bounding-box_-110,27,-92,52.json.check.mbtiles",
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "0.023802,-85.040752,359.950216,83.645130",
"antimeridian_adjusted_bounds": "0.023802,-85.051129,359.950216,83.645130",
"bounds": "-180.000000,-85.051129,180.000000,83.645130",
"center": "0.000000,0.000000,0",
"description": "tests/ne_110m_admin_0_countries/out/-z0_--tiny-polygon-size_50_--simplification_50.json.check.mbtiles",
@@ -61,7 +61,7 @@
,
{ "type": "Feature", "properties": { "scalerank": 1, "featurecla": "Admin-0 country", "labelrank": 3, "sovereignt": "Ivory Coast", "sov_a3": "CIV", "adm0_dif": 0, "level": 2, "type": "Sovereign country", "admin": "Ivory Coast", "adm0_a3": "CIV", "geou_dif": 0, "geounit": "Ivory Coast", "gu_a3": "CIV", "su_dif": 0, "subunit": "Ivory Coast", "su_a3": "CIV", "brk_diff": 0, "name": "Côte d'Ivoire", "name_long": "Côte d'Ivoire", "brk_a3": "CIV", "brk_name": "Côte d'Ivoire", "abbrev": "I.C.", "postal": "CI", "formal_en": "Republic of Ivory Coast", "formal_fr": "Republic of Cote D'Ivoire", "name_sort": "Côte d'Ivoire", "mapcolor7": 4, "mapcolor8": 6, "mapcolor9": 3, "mapcolor13": 3, "pop_est": 20617068, "gdp_md_est": 33850, "pop_year": -99, "lastcensus": 1998, "gdp_year": -99, "economy": "6. Developing region", "income_grp": "4. Lower middle income", "wikipedia": -99, "iso_a2": "CI", "iso_a3": "CIV", "iso_n3": "384", "un_a3": "384", "wb_a2": "CI", "wb_a3": "CIV", "woe_id": -99, "adm0_a3_is": "CIV", "adm0_a3_us": "CIV", "adm0_a3_un": -99, "adm0_a3_wb": -99, "continent": "Africa", "region_un": "Africa", "subregion": "Western Africa", "region_wb": "Sub-Saharan Africa", "name_len": 13, "long_len": 13, "abbrev_len": 4, "tiny": -99, "homepart": 1 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -7.910156, 10.314919 ], [ -2.812500, 9.622414 ], [ -2.812500, 5.003394 ], [ -7.910156, 10.314919 ] ] ] } }
,
{ "type": "Feature", "properties": { "scalerank": 1, "featurecla": "Admin-0 country", "labelrank": 4, "sovereignt": "Antarctica", "sov_a3": "ATA", "adm0_dif": 0, "level": 2, "type": "Indeterminate", "admin": "Antarctica", "adm0_a3": "ATA", "geou_dif": 0, "geounit": "Antarctica", "gu_a3": "ATA", "su_dif": 0, "subunit": "Antarctica", "su_a3": "ATA", "brk_diff": 0, "name": "Antarctica", "name_long": "Antarctica", "brk_a3": "ATA", "brk_name": "Antarctica", "abbrev": "Ant.", "postal": "AQ", "note_brk": "Multiple claims held in abeyance", "name_sort": "Antarctica", "mapcolor7": 4, "mapcolor8": 5, "mapcolor9": 1, "mapcolor13": -99, "pop_est": 3802, "gdp_md_est": 760.4, "pop_year": -99, "lastcensus": -99, "gdp_year": -99, "economy": "6. Developing region", "income_grp": "2. High income: nonOECD", "wikipedia": -99, "iso_a2": "AQ", "iso_a3": "ATA", "iso_n3": "010", "un_a3": "-099", "wb_a2": "-99", "wb_a3": "-99", "woe_id": -99, "adm0_a3_is": "ATA", "adm0_a3_us": "ATA", "adm0_a3_un": -99, "adm0_a3_wb": -99, "continent": "Antarctica", "region_un": "Antarctica", "subregion": "Antarctica", "region_wb": "Antarctica", "name_len": 10, "long_len": 10, "abbrev_len": 4, "tiny": -99, "homepart": 1 }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ -46.669922, -77.823323 ], [ -43.330078, -80.027655 ], [ -52.822266, -80.969904 ], [ -46.669922, -77.823323 ] ] ], [ [ [ -60.644531, -79.624056 ], [ -64.511719, -80.928426 ], [ -66.269531, -80.253391 ], [ -60.644531, -79.624056 ] ] ], [ [ [ 135.087891, -65.293468 ], [ 168.398438, -70.959697 ], [ 166.992188, -78.750659 ], [ 159.785156, -80.942273 ], [ 180.000000, -85.051129 ], [ 180.000000, -84.714152 ], [ 180.966797, -84.142939 ], [ 185.625000, -84.532994 ], [ 187.031250, -84.079819 ], [ 187.031250, -85.622069 ], [ 180.000000, -85.622069 ], [ -180.000000, -85.622069 ], [ -187.031250, -85.622069 ], [ -187.031250, -84.319608 ], [ -180.000000, -84.714152 ], [ -180.000000, -85.051129 ], [ -179.033203, -84.142939 ], [ -174.375000, -84.532994 ], [ -169.980469, -83.886366 ], [ -162.597656, -85.051129 ], [ -148.535156, -85.608630 ], [ -143.173828, -85.051129 ], [ -156.796875, -81.106811 ], [ -150.644531, -81.334844 ], [ -146.777344, -79.920548 ], [ -158.378906, -76.880775 ], [ -151.347656, -77.389504 ], [ -135.175781, -74.307353 ], [ -113.906250, -73.726595 ], [ -100.634766, -75.297735 ], [ -103.710938, -72.607120 ], [ -74.882812, -73.873717 ], [ -65.302734, -68.528235 ], [ -61.347656, -74.116047 ], [ -77.255859, -76.720223 ], [ -78.046875, -79.187834 ], [ -58.183594, -83.215693 ], [ -49.746094, -81.723188 ], [ -42.802734, -82.082145 ], [ -28.564453, -80.342262 ], [ -35.332031, -78.116408 ], [ -10.283203, -71.272595 ], [ 37.177734, -69.162558 ], [ 56.337891, -65.982270 ], [ 68.906250, -67.941650 ], [ 69.873047, -72.262310 ], [ 87.978516, -66.196009 ], [ 135.087891, -65.293468 ] ] ], [ [ [ -163.125000, -78.224513 ], [ -160.224609, -78.699106 ], [ -159.169922, -79.496652 ], [ -163.125000, -78.224513 ] ] ], [ [ [ -70.224609, -68.879358 ], [ -73.037109, -72.235514 ], [ -74.970703, -71.663663 ], [ -70.224609, -68.879358 ] ] ] ] } }
{ "type": "Feature", "properties": { "scalerank": 1, "featurecla": "Admin-0 country", "labelrank": 4, "sovereignt": "Antarctica", "sov_a3": "ATA", "adm0_dif": 0, "level": 2, "type": "Indeterminate", "admin": "Antarctica", "adm0_a3": "ATA", "geou_dif": 0, "geounit": "Antarctica", "gu_a3": "ATA", "su_dif": 0, "subunit": "Antarctica", "su_a3": "ATA", "brk_diff": 0, "name": "Antarctica", "name_long": "Antarctica", "brk_a3": "ATA", "brk_name": "Antarctica", "abbrev": "Ant.", "postal": "AQ", "note_brk": "Multiple claims held in abeyance", "name_sort": "Antarctica", "mapcolor7": 4, "mapcolor8": 5, "mapcolor9": 1, "mapcolor13": -99, "pop_est": 3802, "gdp_md_est": 760.4, "pop_year": -99, "lastcensus": -99, "gdp_year": -99, "economy": "6. Developing region", "income_grp": "2. High income: nonOECD", "wikipedia": -99, "iso_a2": "AQ", "iso_a3": "ATA", "iso_n3": "010", "un_a3": "-099", "wb_a2": "-99", "wb_a3": "-99", "woe_id": -99, "adm0_a3_is": "ATA", "adm0_a3_us": "ATA", "adm0_a3_un": -99, "adm0_a3_wb": -99, "continent": "Antarctica", "region_un": "Antarctica", "subregion": "Antarctica", "region_wb": "Antarctica", "name_len": 10, "long_len": 10, "abbrev_len": 4, "tiny": -99, "homepart": 1 }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ -46.669922, -77.823323 ], [ -43.330078, -80.027655 ], [ -52.822266, -80.969904 ], [ -46.669922, -77.823323 ] ] ], [ [ [ -60.644531, -79.624056 ], [ -64.511719, -80.928426 ], [ -66.269531, -80.253391 ], [ -60.644531, -79.624056 ] ] ], [ [ [ 135.087891, -65.293468 ], [ 171.210938, -71.691293 ], [ 159.785156, -80.942273 ], [ 180.000000, -85.051129 ], [ 180.000000, -84.714152 ], [ 180.966797, -84.142939 ], [ 185.625000, -84.532994 ], [ 187.031250, -84.079819 ], [ 187.031250, -85.622069 ], [ 180.000000, -85.622069 ], [ -180.000000, -85.622069 ], [ -187.031250, -85.622069 ], [ -187.031250, -84.319608 ], [ -180.000000, -84.714152 ], [ -180.000000, -85.051129 ], [ -179.033203, -84.142939 ], [ -174.375000, -84.532994 ], [ -169.980469, -83.886366 ], [ -162.597656, -85.051129 ], [ -148.535156, -85.608630 ], [ -143.173828, -85.051129 ], [ -156.796875, -81.106811 ], [ -150.644531, -81.334844 ], [ -146.777344, -79.920548 ], [ -158.378906, -76.880775 ], [ -151.347656, -77.389504 ], [ -135.175781, -74.307353 ], [ -113.906250, -73.726595 ], [ -100.634766, -75.297735 ], [ -103.710938, -72.607120 ], [ -74.882812, -73.873717 ], [ -65.302734, -68.528235 ], [ -61.347656, -74.116047 ], [ -77.255859, -76.720223 ], [ -78.046875, -79.187834 ], [ -58.183594, -83.215693 ], [ -49.746094, -81.723188 ], [ -42.802734, -82.082145 ], [ -28.564453, -80.342262 ], [ -35.332031, -78.116408 ], [ -10.283203, -71.272595 ], [ 38.671875, -69.778952 ], [ 54.492188, -65.802776 ], [ 68.906250, -67.941650 ], [ 69.873047, -72.262310 ], [ 87.978516, -66.196009 ], [ 135.087891, -65.293468 ] ] ], [ [ [ -163.125000, -78.224513 ], [ -160.224609, -78.699106 ], [ -159.169922, -79.496652 ], [ -163.125000, -78.224513 ] ] ], [ [ [ -70.224609, -68.879358 ], [ -73.037109, -72.235514 ], [ -74.970703, -71.663663 ], [ -70.224609, -68.879358 ] ] ] ] } }
,
{ "type": "Feature", "properties": { "scalerank": 1, "featurecla": "Admin-0 country", "labelrank": 3, "sovereignt": "Ecuador", "sov_a3": "ECU", "adm0_dif": 0, "level": 2, "type": "Sovereign country", "admin": "Ecuador", "adm0_a3": "ECU", "geou_dif": 0, "geounit": "Ecuador", "gu_a3": "ECU", "su_dif": 0, "subunit": "Ecuador", "su_a3": "ECU", "brk_diff": 0, "name": "Ecuador", "name_long": "Ecuador", "brk_a3": "ECU", "brk_name": "Ecuador", "abbrev": "Ecu.", "postal": "EC", "formal_en": "Republic of Ecuador", "name_sort": "Ecuador", "mapcolor7": 1, "mapcolor8": 5, "mapcolor9": 2, "mapcolor13": 12, "pop_est": 14573101, "gdp_md_est": 107700, "pop_year": -99, "lastcensus": 2010, "gdp_year": -99, "economy": "6. Developing region", "income_grp": "3. Upper middle income", "wikipedia": -99, "iso_a2": "EC", "iso_a3": "ECU", "iso_n3": "218", "un_a3": "218", "wb_a2": "EC", "wb_a3": "ECU", "woe_id": -99, "adm0_a3_is": "ECU", "adm0_a3_us": "ECU", "adm0_a3_un": -99, "adm0_a3_wb": -99, "continent": "South America", "region_un": "Americas", "subregion": "South America", "region_wb": "Latin America & Caribbean", "name_len": 7, "long_len": 7, "abbrev_len": 4, "tiny": -99, "homepart": 1 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -80.068359, 0.790990 ], [ -75.410156, -0.175781 ], [ -80.068359, -4.302591 ], [ -80.068359, 0.790990 ] ] ] } }
,
File diff suppressed because one or more lines are too long
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "0.023802,-85.040752,359.950216,83.645130",
"antimeridian_adjusted_bounds": "0.023802,-85.051129,359.950216,83.645130",
"bounds": "-180.000000,-85.051129,180.000000,83.645130",
"center": "45.000000,33.256630,2",
"description": "tests/ne_110m_admin_0_countries/out/-z2_--convert-polygons-to-label-points.json.check.mbtiles",
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "0.023802,-85.040752,359.950216,83.645130",
"antimeridian_adjusted_bounds": "0.023802,-85.051129,359.950216,83.645130",
"bounds": "-180.000000,-85.051129,180.000000,83.645130",
"center": "11.250000,48.378236,4",
"description": "tests/ne_110m_admin_0_countries/out/-z4_-yname_--drop-polygons.json.check.mbtiles",
@@ -1,5 +1,5 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "0.023802,-85.040752,359.950216,83.645130",
"antimeridian_adjusted_bounds": "0.023802,-85.051129,359.950216,83.645130",
"bounds": "-180.000000,-85.051129,180.000000,83.645130",
"center": "11.250000,48.378236,4",
"description": "tests/ne_110m_admin_0_countries/out/-z4_-yname_--grid-low-zooms_-D8.json.check.mbtiles",
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More