* Divide-and-conquer polygon cleaning
* Catch the case where the gap can't be increased further
* Catch the case where we try to keep impossibly many features
* Make label points earlier in the tiling process
* Another case where it could try to drop even after already limiting.
* And do not coalesce on impossibly small geometries
* Add missing return
* Update version and changelog
* Add a flag to use an H3 index for the feature index
* Give mvt_value and serial_val a double-with-count concept
* Switch mean over to internal accumulation state
* Get rid of the attribute accumulation map
* Change vectors of features to vectors of pointers to features
* Fix --coalesce
* Revert "Add a flag to use an H3 index for the feature index"
This reverts commit b9b48f42c9.
* Update version and changelog
* Plumb a clip bounding box around through overzoom
* Actually do some clipping
* Add a test
* Fix post-binning clipping
* Factoring out geometry parsing from feature parsing
* Accept a clip polygon argument to tippecanoe-overzoom
* Progress in the direction of polygon clipping
* Fix the wagyu flags. We need intersection, not union
* Remove debug spew
* Clip points to polygon bounds too
* Copy the geometric binning code to serve as intersection-finding code
* Add clipper2 for linestring clipping
* Compiles, but does not actually seem to clip. Hmm.
* Oh, it helps if I actually call the function
* Add clipping tests
* Add missing fixture, and don't crash if it is missing
* Remember to do polygon clipping after binning too
* Fix scaling before post-binning clipping. Add test.
* Remove unused parts of clipper
* Rename for consistency
* Revert accidentally added line
* Clip the clip regions to the tile bounds to reduce their complexity
* Add a test of clipping the clip region down to the tile boundary
* Update version and changelog
* Strip out unwanted attributes earlier in the process
* Skip aggregations whose attributes have been excluded
* Forgot one
* Add some more tests
* Update version and changelog
* Add a tippecanoe-decode option to restrict which attributes to decode
* Plumb buffer and feature limit around
* Check the feature limit
* Clarifying cases where output detail can be unspecified
* Clip bins to the tile buffer instead of just passing them through
* Add missing include
* Missed some tests
* Add --no-tile-compression option to tippecanoe-overzoom
* Update version and changelog
* Progress on plumbing a string pool for full_keys through
* More plumbing for key_pool
* Don't keep features with identical locations as multiplier features
* Revert "Don't keep features with identical locations as multiplier features"
This reverts commit 413f0c8024.
* Adjust calculated maxzoom to account for duplicate feature locations
* Update changelog and version
* Add a test affected by the maxzoom change with duplicate locations
* Round the drop rate a little for cross-platform test consistency
* Add an all-mvt_value attribute accumulation path
* Only bin by ID, not geometrically
* A little cleanup; changelog and version; test
* Remove accidental double-conversion
* Replace duplicated code with template
* Update changelog
* Bin more aggressively if a point doesn't meet the pnpoly test
* Don't clip the points if we are binning
* Fix output of features added to the bin after its closure
* Update tests
* Update version and changelog
* Plumb bounding boxes through potential intersections
* Quick bbox reject for bins that can't possibly intersect
* Inching toward attribute accumulation in megatile handling
* Some sort of test for how all these things interact with each other.
Automatic numeric attribute accumulation does *not* apply to attributes
that have an explicit attribute accumulator set, because the order of
operations is too messy and weird
* More sketching
* More sketching
* Actually do some accumulation
* Put all that behind an --accumulate-numeric flag
* Use the same attribute accumulation logic in binning as in megatiles
* Fix backwards conditional
* Add means, but somehow I have some counts of 0
* Handle aggregated attributes with no base attribute in the feature
* Checkpoint before I break everything
* Found a flaw, now to debug
* Fix a typo that broke accumulation
* Add binning tests
* Make sure IDs make it through on the bins
* Fix count/mean accumulation
* Make the numeric accumulation prefix configurable
* Make sure the accumulate test still works with a different prefix
* Forgot to update this test
* More testing to make sure cluster sizes make it all the way through
* Fix neglected --accumulate-attribute when binning
* Mark unexercised attribute accumulation cases as "can't happen"
* Factor out numeric preservation
* Attrs with the accumulation prefix are just preserved, not accumulated
* Test behavior of prefixed attributes
* Plumbing for exclude and exclude-prefix
* Implement and test attribute prefix stripping in overzoom
* Update version and changelog
* For debugging, make an attribute list of source feature IDs
* Revert "For debugging, make an attribute list of source feature IDs"
This reverts commit 65fc99c9d1.
* Factoring out tilestats management from GeoJSON file reading
* Move code around so overzoom can link against parse_layers
* Read the file of bins
* Plumb the bins through to overzoom()
* Some zip code bins to test with
* (Currently non-functional) test of binning
* Starting to spell out the bin matching loop
* Can't flatten points, so don't flatten bins either
* More fleshing out bin traversal
* Bounding box of tile-relative mvt geometry
* Smallest enclosing tile from bbox
* Most of the bin scan
* Add point in polygon check. It crashes.
* Find the matching bins
* GDAL-style bounding boxes have eaten my brain
* Make some features to bin into
* Increment a count as features are found to be within the bins
* Fix longitude wraparound in overzoom bins
* Fix the tests
* Push off attribute copying until after bin assignment
* Carry sum of numeric attributes into the bins
* Also add mean, min, and max
* Add --calculate-feature-index since I keep needing it for testing
* Add an option to accumulate sum/mean/max/min/count of all numeric attrs
* Don't bake in tippecanoe:mean, since we redo it from sum and count
* Forgot to update this test fixture after removing tiled mean
* Update version and changelog
* Clip before dealing with multiplier or filters in overzoom
* Be more careful to retry when the feature count is exceeded
* Adjust the estimated total feature count for the multiplier too
* Fix the feature count estimates, I think
* Pass build info into the version string
* Report the actual max zoom of any tiles as the metadata maxzoom
* Revert unneeded renaming to make the diff more readable
* Clean up the adjustments to tile sizes and feature counts
* Update version and changelog
* Dropping a feature into a multiplier cluster still effectively drops it
* Update changelog
* Rethink the changelog description
* Don't try to truncate zooms if we are still tiling at z18
* Track output position at the file level instead of within each tile
* Track file position where the child tile data begins
* Add option and document its intended behavior
* Changing the detail loop to account for stopping early
* I forgot I already added an option for this
* Stop early if we can make a complete tile
* Add a test of zoom truncation with limited feature count
* Forgot to commit the actual code change
* Make room for a vertex count in the header of each serialized tile
* Estimate tile complexity; don't try truncating when unlikely to work
* Be more conservative, because ever retrying a tile is a big speed hit
* If stopping early, don't simplify or clean; leave that to overzoom
* Add tiny polygon reduction / dust to overzoom
* Don't try to stop early in the children if we dropped anything by rate
* Fflush here too before pwriting
* Don't stop early if we ended up dropping any features.
Rework the can-the-next-zoom-stop-early logic to avoid going
one zoom further than needed.
* Fix warning
* Fix warnings
* Oops, checking for the wrong expected return value
* Cleanup from adding line simplification in overzoom
* Current (wrong) behavior when combining coalescing and truncating
* Keep a list of parent tiles to skip rather than truncating
* Now the coalesced tiles in z12 get children in z13
* Don't double-count feature dropping when the zoom level is retried
* Correct README description
* Remove todo about special case below basezoom, which is accounted for
* Be a little more aggressive in drop-densest determination
* Scale tile feature limit for megatiles in the same way as byte limit
* Fully deprecate -detect-shared-borders into an alias
* Track the distances found in the douglas-peucker recursion
* Serialize and deserialize the distance with the vertices
* Revert "Serialize and deserialize the distance with the vertices"
This reverts commit 753f1b7909.
* Revert "Track the distances found in the douglas-peucker recursion"
This reverts commit e5361f8c22.
* Revert "Fully deprecate -detect-shared-borders into an alias"
This reverts commit 0698aeb766.
* Better tracking of whether we failed to make a full-detail tile
* Put a bloom filter in front of the binary search for shared nodes
* Forgot to take out this printf
* Improve dispatch of tiling tasks
* Still dispatch the biggest tasks first
* Track zoom truncation in the strategies list in the tileset metadata
* Prescan for small deltas before doing proper simplification
* Revert "Prescan for small deltas before doing proper simplification"
This reverts commit d1d8238b83.
* Update version and changelog
* Rename to --generate-variable-depth-tile-pyramid
* Reviving multi-source-tile overzoom: the clip.cpp side
* Reviving multi-source-tile overzoom: the overzoom.cpp side
* Update readme
* Update version and changelog
* Fix some undefined behavior
* Avoid overflow in line simplification calculations
* Oops, missed a test
* Didn't mean to add that to the Makefile
* Revert "Revert "[ci] test in debug mode (#202)""
This reverts commit c95c328e47.
* Fix reference to out-of-scope pointer
* Fix invalid shift and out-of-bounds vector element reference
* Update changelog and version
* Stabilize feature order in overzoom
* I want my sorts to be stable, please
* Revert "[ci] test in debug mode (#202)"
This reverts commit 853ada87b5.
* No need to reinitialize here
* Read unidecode data, do some plumbing of it
* More unidecode plumbing
* Do the unidecode smashing, but it doesn't seem to be working
* Ah, that's better!
* Add missing header
* And reorder the includes too
* Shortcut when there is no unidecode data to work with
* Update version and changelog
* Avoid repeated unidecode smashing of the same constant string
* Add a way to run tippecanoe single-threaded for profiling
* Do less work when the tilestats sample values list is already full
* Save a copy when retrieving the attribute key
* Fewer atomic operations
* Move string hashing from mbtiles to text
* Only do approximate attribute deduplication when writing tiles
* Feature dropping tests are sensitive to exact tile size
* All tile creators now create a string pool for the tile
* Features clipped away to nothing should not participate in that tile
* Revert "Only do approximate attribute deduplication when writing tiles"
This reverts commit c42b34b498.
* Also revert the related test changes
* Revert "Revert "Only do approximate attribute deduplication when writing tiles""
This reverts commit 18509876c3.
* Be more specific about the string hash function
* Use fnv1a instead of std::hash for everything
* Reduce the chance of hash collisions
* Stick a hash search on the front of the tree search in addpool
* Eliminate repeated hashing of the same string
* Switch instead of ifs in json parsing
* A few more cases to populate the hash in addpool
* Store the hash in the tree instead of recalculating
* Add explanatory comment for mysterious argument
* Fewer copies in attribute stringification
* Clean up ancient weirdness in JSON attribute stringification
* More serial_val cleanup
* Pass a serial_feature to rewrite instead of many broken-down arguments
* Get rid of the multiple geometries within `partial`
* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"
This reverts commit 6f4ab9b725.
* Goodbye, struct coalesce
* Revert "Features clipped away to nothing should not participate in that tile"
This reverts commit 124462fbdc.
* Migrating fields from partial to serial_feature
* Name reconciliation between serial_feature and partial
* Replace struct partial with an augmented serial_feature
* Fix some overzealous search-and-replace renaming
* Don't say struct so often
* Remove more of the former partial construction
* Commenting and cleaning up
* Trying again to avoid all these arguments to rewrite
* I swear I did this same thing before and it didn't work.
* More rewrite cleanup
* Exile --detect-shared-borders to its own file
* Add missing headers
* More commenting and cleanup
* More comments
* Sprinkle consts around
* Emplacing and std::moving
* More cleanup
* That shouldn't have worked after a std::move
* Don't need to allocate memory to compare keys
* Reduce use of the global string pool in tiling
* Another avoidable mvt_value construction
* Further reduction to explicit string pool passing
* These reverses are no longer optimizations
* These layernames can all be references
* Don't drag an unused layername string around with every feature
* Heed a compiler warning about potential buffer overflow
* Fix my confusion about which feature's string pool is relevant
* Avoid some unnecessary allocations in attribute accumulation
* Maybe faster serialization?
* Eliminate a comparison
* Do the same here
* Save a couple of allocations when parsing numbers in JSON
* Immediately assign features to layers instead of subdividing later
* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence
* Crunch out more duplicate attribute values when writing out the tile
* Do tilestats for tippecanoe:retain_points_multiplier_first too
* Shell filters need to be real threads, even if nothing else does
* Simplify tippecanoe_minzoom/maxzoom representation
* Update version and changelog
* Speed up mvt_value comparison
* Converting repetitive ifs to cases
* More conversions from ifs to cases
* Optimize the always-true filter case
* Don't convert types of attributes without accumulators
* Unordered map seems to be faster than map
* Add missing header
* Fix some warnings
* Fix the warnings better
* Avoid an int->string->int conversion
* Lazily initialize layer key and values maps when actually needed
* More switches from maps to unordered_maps
* Sure, I'll take the microoptimization
* More emplacement
* Save some copies
* Emplaces and moves
* Lazy linear scan of attributes instead of building a map
* Extra printfs, missing header
* Avoid clipping if the input and output tiles are the same
* But do clip if the tile extent is being reduced
* Make sure I'm not constructing std::strings here at runtime
* More worrying about runtime string construction
* A couple more std::moves
* Const references!
* More const references
* Another std::move
* Make the string_value of mvt_value std::optional
* Reserve storage when decoding
* Provision for different mvt_values to share a string pool
* Use the string pool when decoding
* Avoid another string construction
* Try limiting the depth of the search for duplicate attributes
* Revert "Try limiting the depth of the search for duplicate attributes"
This reverts commit 9ec94a15ff.
* Update changelog
* Fix typo noticed during code review
* Starting to factor out attribute accumulation into its own file
* Continuing to factor out attribute accumulation
* Reduce duplicate code
* Plumbing the accumulate-attribute option around
* Call the attribute accumulator
* Test that accumulation works
* Add missing #includes
* Don't sort within individual multiplier clusters
Doing so throws off the spatial distribution of the low zooms
* Docs and changelog
* Add comments
* Make feature ordering cooperate with --retain-points-multiplier
* Forgot to check in the actual code changes???
* Sort within each multiplier cluster as well as between clusters
* Correct description of behavior in changelog
* Drag original feature sequence along in megatiles for post-filter sort
* Plumb the preserve-input-order flag through overzoom
* Sort in overzoom if requested
* Use within-tile input sequence numbers, not global sequence numbers
* Documentation
* Reverse direction of search to prevent accidental skipping
* Add some comments about converting between attribute representations
* Add an option to retain N times as many points as usual at each zoom
* Tests for point multipler with specified and guessed maxzooms
* Work in progress on inverse spatial ordering
* Fix inverse spatial feature order
* --reorder was depending on a feature index that wasn't being preserved
* Separate ordering by feature_minzoom from ordering inverse-spatially
* Add a test for the inverse spatial ordering
* Store the basezoom/droprate/multiplier decisions in tileset metadata
* Progress on adding filters to tippecanoe-overzoom
* Type promotion for comparison
* Look up the attribute value for ordering
* Add test of thinning and ordering features
* Plumb tippecanoe_decisions metadata through pmtiles
* Be careful not to put infinities in JSON
* Fix accidental dropping in what is meant to preserve sparse points
* Start distinguishing true, false, and null in expressions
* Most of the type conversions
* Add boolean conversions
* Literals and conjunctions
* Add filtering to tippecanoe-overzoom
* Add a test of filtering in overzoom
* Fix boolean conjunctions
* Handle the combination of cluster size and filtering
* Rework dot dropping to reconcile density threshold and multiplier
* Revert "Rework dot dropping to reconcile density threshold and multiplier"
This reverts commit f253a66382.
* Retain points by multiplier within each tile, not in global probability
* Test that intends to verify that the multiplier is reversible
* Get the test to detect the discrepancy
* Mark the start of multiplier clusters with a magic attribute
* Add string-contains
* Add in and ni operators
* Revert "Look up the attribute value for ordering"
This reverts commit 56bc73e49a.
* Revert "Type promotion for comparison"
This reverts commit 6f3256f5af.
* Make number formatting in tippecanoe_decisions consistent
* Revert "Add a test for the inverse spatial ordering"
This reverts commit c8047de9ab.
* Revert "Separate ordering by feature_minzoom from ordering inverse-spatially"
This reverts commit 35b19a223c.
* Revert "Fix inverse spatial feature order"
This reverts commit 5978ecdb44.
* Revert "Work in progress on inverse spatial ordering"
This reverts commit fdf230f632.
* Somehow missed the tests associated with that last revert
* Round-robin assign attributes to partials from across the multiplier
* Count the multiplier separately in each layer
* Fix distribution of accumulated attribute across multiplier features
* Update changelog, version, and docs
* Add "is null" and "isnt null" expressions
* Update interpretation of FSL expressions to pass the tests
* Test to assert that polygons are unaffected by the multiplier
* Clean up and comment
* Remove accidental unused case
* 16 bits is enough for tile numbers
* Revert "16 bits is enough for tile numbers"
This reverts commit 71a0c4e1cf.
* Check what child tiles each overzoomed tile will have
and don't queue further overzooming of empty tiles
* Clean up naming; use std::move to avoid copying large arrays
* Clip away entire features by bbox. Avoid unnecessary recompression.
* Move parent tile decoding in tile-join out of overzoom proper
* An ever-growing cache of parent tiles
* Limit the size of the cache
* Remove the current reader *before* checking if we can run the queue
* Clean up
* Add missing #include
* Add comment
* When the tile-join cache fills up, evict the least recently used
* Fix microsecond math
* Factoring out tile-join's cache for testing
* Add unit tests for tile-join cache
* Update changelog and version
* Put all of this back in geometry.cpp for conflict resolution
* Stabilize line simplification to behave the same regardless of winding
* Round instead of truncating when clipping lines
* Restore non-Wagyu polygon clipping from prior to 2fdec7d2
* Make it round, not truncate, which reverts the last commit's test diffs
* Clip in floating point, not integers, which makes no difference
* Track nodes added at tile edges during clipping
* Scale geometry up before wagyu to prevent changes from precision loss
* Actually do the shared edge detection
* Fix cases where nodes were not being added at the tile boundary
* One more place I should have rounded
* Narrow down where the discrepancy comes in
* Revert "Narrow down where the discrepancy comes in"
This reverts commit 221c4c5fc0ac9a6567e091c6a94b3d87dc8ade83.
* Another attempt to narrow it down
* The discrepancy seems to be introduced in reordering. No obvious bug
* Was still truncating instead of rounding in projection
* Also makes no difference...
* Just forget that line reversal exists for a minute
* Just reversal no coalescing
* Try clipping in integers instead of floating point
* Don't simplify after coalescing if they said no simplification
* Check whether behavior is consistent with intentional simplification
* Replace more floating point with integer
* Are these three features enough to demonstrate the problem?
* Add a few more nearby borders
* All the features that touch tile 6/16/23
* Stay in integers in line simplification
* More attempts to solve failures to simplify consistently
* Fix most of the overflow errors
* Fix known cases of integer overflow
* All the tests change again
* Pull clipping and scaling code back out into clip.cpp
* Resolve the test conflicts
* Stabilize choice of which three points to keep with different windings
* Almost right, I think!
* Fix collapse of islands to shards
* Self-intersections in the same feature don't count
* Revert "Self-intersections in the same feature don't count"
This reverts commit e04b19916e.
* Don't scale down geometry if we are going to look for shared nodes
* Fix the missing multiply that was keeping simplification from happening
* Fix one more opportunity for overflow
* Somehow I deleted this test?
* Lost this test too
* Clean up debugging printfs
* Restore code sequence from main to make it reviewable
* Remove unneeded rounding
* Update documentation
* This test is no longer useful
* Back to floating point Douglas-Peucker to fix undersimplification
* Try an older ubuntu
* Revert "Try an older ubuntu"
This reverts commit 13fefacfd7.
* Log OS info
* Remove tests that are no longer needed
* Oops, did need that one after all
* Fix the arm vs x86 discrepancy?
* Try another quantization
* Cleanup from review
* Add a test for the actual purpose of this PR
* Add an option to extend zooms if still dropping, but with a limit
* At least when to overzoom, even if not actually doing it yet
* Refactor to give tile-join access to overzoom()
* Didn't work, but *might* have worked
* OK, it did something now
* Ah, there's the bug!
* Hook up pmtiles and dirtiles as overzooming sources
* Add command line option to enable or disable overzooming
* Add (currently broken) test of overzooming in tile-join
* Slightly more abstraction for the tile-join readers
* Factor out duplicated code
* Move construction into a constructor
* More changing accessors to methods
* Reduce magic
* Start tracking a list of the tiles at maxzoom
* I think it worked?
* Add missing #include
* Fix sequence of overzoomed tiles (Y sorts backwards for TMS)
* Don't spend memory on overzooming when we aren't going to use it
* Diff rather than cmp, in the hope of figuring out this broken test
* Keep full coordinate precision if we might extend zooms
* Try a slightly different byte limit
* Make drop-densest more consistent across tile boundaries
* Also affects this test
* Does it behave any differently if it can extend forever?
* I think the discrepancy is a thread-safety problem here
* Revert "Does it behave any differently if it can extend forever?"
This reverts commit 0dff0a0acc.
* Lost this change to the test
* This time for sure!
* Revert "Also affects this test"
This reverts commit cd1f7c2e78.
* Revert "Make drop-densest more consistent across tile boundaries"
This reverts commit 563f7d2bc2.
* Revert "Try a slightly different byte limit"
This reverts commit 2e271213d6.
* Add some more explanatory comments
* Amend the join-test to detect my current bug
* Allow overzooming to complete the zoom if it ever starts
* Forgot to correct the test
* Update changelog and version
* Cleanups from code review
* Remove version number from fixture to fix test
* Starting work on overzooming
* Factor clipping out of geometry.cpp to simplify linkage
* Pull out more geometry functions into now-badly-named clip.cpp
* Not surprisingly, there is a bug
* Found the bug
* Make indent
* Pass attributes through
* Make formatting more consistent
* Fix typos in comments
* Fix the typos better
* Add tippecanoe-overzoom to the install list
* Give overzoom a predictable exit status
* Forgot to translate to and from polygon ring closepaths
* Fix geometry collapse at z21
* Add attribute stripping; don't generate layers if they have no features
* Add docs for tippecanoe-overzoom (as it will be, not as it is)
* Change overzoom to accept input and output files as arguments
* Working on overzooming tests
* Oops
* Hook up and test the detail and buffer options, and the empty-tile case
* Also test attribute stripping
* Fix error message
* Update changelog and version
Erica Fischer