* Add a tippecanoe-decode option to restrict which attributes to decode
* Plumb buffer and feature limit around
* Check the feature limit
* Clarifying cases where output detail can be unspecified
* Clip bins to the tile buffer instead of just passing them through
* Add missing include
* Missed some tests
* Add --no-tile-compression option to tippecanoe-overzoom
* Update version and changelog
* Progress on plumbing a string pool for full_keys through
* More plumbing for key_pool
* Don't keep features with identical locations as multiplier features
* Revert "Don't keep features with identical locations as multiplier features"
This reverts commit 413f0c8024.
* Adjust calculated maxzoom to account for duplicate feature locations
* Update changelog and version
* Add a test affected by the maxzoom change with duplicate locations
* Round the drop rate a little for cross-platform test consistency
* Add an all-mvt_value attribute accumulation path
* Only bin by ID, not geometrically
* A little cleanup; changelog and version; test
* Remove accidental double-conversion
* Replace duplicated code with template
* Update changelog
* Choose the megatile features from those that will be in the next N zooms
* Take fractional zooms into account in multiplier feature choices
* Fix more tests
* Add a flag to retain multiplier features by minimum distance
* Limit feature expansion from multiplier density to 2x
* The multiplier cap was a bad idea
* Revert "The multiplier cap was a bad idea"
This reverts commit 6f8273a4c8.
* Revert "Limit feature expansion from multiplier density to 2x"
This reverts commit a26e41309d.
* Revert "Add a flag to retain multiplier features by minimum distance"
This reverts commit 01f14a4255.
* Remove the multiplier sequence, which should no longer matter
* Revert "Revert "Add a flag to retain multiplier features by minimum distance""
This reverts commit 776da4a1b8.
* Revert "Revert "Limit feature expansion from multiplier density to 2x""
This reverts commit 44a683d808.
* Revert "Revert "The multiplier cap was a bad idea""
This reverts commit 80f7cb1c0e.
* Track two kinds of previous index for next_feature
* Fix multiplier density threshold, I think
* Oh, I didn't git add the code changes
* Update version and changelog
* Try to install sqlite3 to fix the automated build
* Deleted too much
* Only let --preserve-point-density-threshold shift density around
* Remove the density debt concept, since it doesn't help
* Make the drop states a vector instead of an array
* Revert "Make the drop states a vector instead of an array"
This reverts commit 66c7abb6fa.
* Revert "Remove the density debt concept, since it doesn't help"
This reverts commit 707bb0c562.
* Revert "Only let --preserve-point-density-threshold shift density around"
This reverts commit ecf01f2231.
* Bin more aggressively if a point doesn't meet the pnpoly test
* Don't clip the points if we are binning
* Fix output of features added to the bin after its closure
* Update tests
* Update version and changelog
* Plumb bounding boxes through potential intersections
* Quick bbox reject for bins that can't possibly intersect
* Inching toward attribute accumulation in megatile handling
* Some sort of test for how all these things interact with each other.
Automatic numeric attribute accumulation does *not* apply to attributes
that have an explicit attribute accumulator set, because the order of
operations is too messy and weird
* More sketching
* More sketching
* Actually do some accumulation
* Put all that behind an --accumulate-numeric flag
* Use the same attribute accumulation logic in binning as in megatiles
* Fix backwards conditional
* Add means, but somehow I have some counts of 0
* Handle aggregated attributes with no base attribute in the feature
* Checkpoint before I break everything
* Found a flaw, now to debug
* Fix a typo that broke accumulation
* Add binning tests
* Make sure IDs make it through on the bins
* Fix count/mean accumulation
* Make the numeric accumulation prefix configurable
* Make sure the accumulate test still works with a different prefix
* Forgot to update this test
* More testing to make sure cluster sizes make it all the way through
* Fix neglected --accumulate-attribute when binning
* Mark unexercised attribute accumulation cases as "can't happen"
* Factor out numeric preservation
* Attrs with the accumulation prefix are just preserved, not accumulated
* Test behavior of prefixed attributes
* Plumbing for exclude and exclude-prefix
* Implement and test attribute prefix stripping in overzoom
* Update version and changelog
* For debugging, make an attribute list of source feature IDs
* Revert "For debugging, make an attribute list of source feature IDs"
This reverts commit 65fc99c9d1.
* Another try at fixing longitude wraparound for bins
* Gonna get it right this time
* Forgot to update the comment
* The filter case was not supposed to reinterpret geometry
* Copy antimeridian-crossing geometries to the other side too
* Getting closer to getting antimeridian-crossing polygons right
* Update changelog and version
* Factoring out tilestats management from GeoJSON file reading
* Move code around so overzoom can link against parse_layers
* Read the file of bins
* Plumb the bins through to overzoom()
* Some zip code bins to test with
* (Currently non-functional) test of binning
* Starting to spell out the bin matching loop
* Can't flatten points, so don't flatten bins either
* More fleshing out bin traversal
* Bounding box of tile-relative mvt geometry
* Smallest enclosing tile from bbox
* Most of the bin scan
* Add point in polygon check. It crashes.
* Find the matching bins
* GDAL-style bounding boxes have eaten my brain
* Make some features to bin into
* Increment a count as features are found to be within the bins
* Fix longitude wraparound in overzoom bins
* Fix the tests
* Push off attribute copying until after bin assignment
* Carry sum of numeric attributes into the bins
* Also add mean, min, and max
* Add --calculate-feature-index since I keep needing it for testing
* Add an option to accumulate sum/mean/max/min/count of all numeric attrs
* Don't bake in tippecanoe:mean, since we redo it from sum and count
* Forgot to update this test fixture after removing tiled mean
* Update version and changelog
* Clip before dealing with multiplier or filters in overzoom
* Be more careful to retry when the feature count is exceeded
* Adjust the estimated total feature count for the multiplier too
* Fix the feature count estimates, I think
* Pass build info into the version string
* Report the actual max zoom of any tiles as the metadata maxzoom
* Revert unneeded renaming to make the diff more readable
* Clean up the adjustments to tile sizes and feature counts
* Update version and changelog
* Dropping a feature into a multiplier cluster still effectively drops it
* Update changelog
* Rethink the changelog description
* Don't try to truncate zooms if we are still tiling at z18
* Track output position at the file level instead of within each tile
* Track file position where the child tile data begins
* Add option and document its intended behavior
* Changing the detail loop to account for stopping early
* I forgot I already added an option for this
* Stop early if we can make a complete tile
* Add a test of zoom truncation with limited feature count
* Forgot to commit the actual code change
* Make room for a vertex count in the header of each serialized tile
* Estimate tile complexity; don't try truncating when unlikely to work
* Be more conservative, because ever retrying a tile is a big speed hit
* If stopping early, don't simplify or clean; leave that to overzoom
* Add tiny polygon reduction / dust to overzoom
* Don't try to stop early in the children if we dropped anything by rate
* Fflush here too before pwriting
* Don't stop early if we ended up dropping any features.
Rework the can-the-next-zoom-stop-early logic to avoid going
one zoom further than needed.
* Fix warning
* Fix warnings
* Oops, checking for the wrong expected return value
* Cleanup from adding line simplification in overzoom
* Current (wrong) behavior when combining coalescing and truncating
* Keep a list of parent tiles to skip rather than truncating
* Now the coalesced tiles in z12 get children in z13
* Don't double-count feature dropping when the zoom level is retried
* Correct README description
* Remove todo about special case below basezoom, which is accounted for
* Be a little more aggressive in drop-densest determination
* Scale tile feature limit for megatiles in the same way as byte limit
* Fully deprecate -detect-shared-borders into an alias
* Track the distances found in the douglas-peucker recursion
* Serialize and deserialize the distance with the vertices
* Revert "Serialize and deserialize the distance with the vertices"
This reverts commit 753f1b7909.
* Revert "Track the distances found in the douglas-peucker recursion"
This reverts commit e5361f8c22.
* Revert "Fully deprecate -detect-shared-borders into an alias"
This reverts commit 0698aeb766.
* Better tracking of whether we failed to make a full-detail tile
* Put a bloom filter in front of the binary search for shared nodes
* Forgot to take out this printf
* Improve dispatch of tiling tasks
* Still dispatch the biggest tasks first
* Track zoom truncation in the strategies list in the tileset metadata
* Prescan for small deltas before doing proper simplification
* Revert "Prescan for small deltas before doing proper simplification"
This reverts commit d1d8238b83.
* Update version and changelog
* Rename to --generate-variable-depth-tile-pyramid
* Reviving multi-source-tile overzoom: the clip.cpp side
* Reviving multi-source-tile overzoom: the overzoom.cpp side
* Update readme
* Update version and changelog
* Start to distinguish fixed cluster density setting from as-needed density
* Make consistent {drop,coalesce}-densest decisions between zooms
* Actually track the previous index instead of just intending to
* Clean up collinearities in coalesced features
* To determine densest, look at actual physical distance, not just index
* Don't actually need the previous index in serial_feature now
* Center of mass of one feature to most distant point of the next
* Add apologetic comment
* Wait, how did the tests pass before?
* Revert "Wait, how did the tests pass before?"
This reverts commit f73c8ee543.
* Add --maximum-string-attribute-length option
* Update version and changelog
* A little more testing to make sure
* Trying to reduce memory in tiling
* Let the simplification workers go out of scope earlier
* Bail out quickly once the maximum feature count is reached
* Update changelog and version
* Fix some undefined behavior
* Avoid overflow in line simplification calculations
* Oops, missed a test
* Didn't mean to add that to the Makefile
* Revert "Revert "[ci] test in debug mode (#202)""
This reverts commit c95c328e47.
* Fix reference to out-of-scope pointer
* Fix invalid shift and out-of-bounds vector element reference
* Update changelog and version
* Stabilize feature order in overzoom
* I want my sorts to be stable, please
* Revert "[ci] test in debug mode (#202)"
This reverts commit 853ada87b5.
* No need to reinitialize here
* Don't swap attributes when reducing tiny polygon dust
Because the dust placeholder may be misleadingly far from the feature
that contributed the most area to it
* Remove unused arguments; update changelog
* Postpone tagging features as being the first of a multiplier cluster
* Upgrade some dynamically dropped features to multiplier features
* Still don't let it put more features in a cluster than is allowed
* Update tests
* Make the current multiplier cluster size per-layer
* Make sure the first non-empty-geometry in the layer is marked as primary
* Improve comments
* Update changelog and version
* Remove commented out debugging printf
* Factor out duplicated code
* Read unidecode data, do some plumbing of it
* More unidecode plumbing
* Do the unidecode smashing, but it doesn't seem to be working
* Ah, that's better!
* Add missing header
* And reorder the includes too
* Shortcut when there is no unidecode data to work with
* Update version and changelog
* Avoid repeated unidecode smashing of the same constant string
* Prep to track conditions other than just "dropped" or "kept"
* Count up instead of down
* Drop or retain whole multiplier clusters based on their first feature
* Calculate a global feature dropping sequence
* Switch over to using the drop sequence for drop-fraction
* Remove unused arguments for the old drop-fraction implementation
* Fix copy-and-paste bugs, update tests
* Properly incorporate feature_minzoom into the drop sequence, I hope
* Rename drop_by to drop_sequence
* See if sorting within clusters fixes filter stability between zooms
* Remove very chatty debug print
* Update changelog and version
* Use named constants instead of numbers for feature dropping/keeping
* Add comment to explain purpose and method of bit reversal
* Add a way to run tippecanoe single-threaded for profiling
* Do less work when the tilestats sample values list is already full
* Save a copy when retrieving the attribute key
* Fewer atomic operations
* Move string hashing from mbtiles to text
* Only do approximate attribute deduplication when writing tiles
* Feature dropping tests are sensitive to exact tile size
* All tile creators now create a string pool for the tile
* Features clipped away to nothing should not participate in that tile
* Revert "Only do approximate attribute deduplication when writing tiles"
This reverts commit c42b34b498.
* Also revert the related test changes
* Revert "Revert "Only do approximate attribute deduplication when writing tiles""
This reverts commit 18509876c3.
* Be more specific about the string hash function
* Use fnv1a instead of std::hash for everything
* Reduce the chance of hash collisions
* Stick a hash search on the front of the tree search in addpool
* Eliminate repeated hashing of the same string
* Switch instead of ifs in json parsing
* A few more cases to populate the hash in addpool
* Store the hash in the tree instead of recalculating
* Add explanatory comment for mysterious argument
* Fewer copies in attribute stringification
* Clean up ancient weirdness in JSON attribute stringification
* More serial_val cleanup
* Pass a serial_feature to rewrite instead of many broken-down arguments
* Get rid of the multiple geometries within `partial`
* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"
This reverts commit 6f4ab9b725.
* Goodbye, struct coalesce
* Revert "Features clipped away to nothing should not participate in that tile"
This reverts commit 124462fbdc.
* Migrating fields from partial to serial_feature
* Name reconciliation between serial_feature and partial
* Replace struct partial with an augmented serial_feature
* Fix some overzealous search-and-replace renaming
* Don't say struct so often
* Remove more of the former partial construction
* Commenting and cleaning up
* Trying again to avoid all these arguments to rewrite
* I swear I did this same thing before and it didn't work.
* More rewrite cleanup
* Exile --detect-shared-borders to its own file
* Add missing headers
* More commenting and cleanup
* More comments
* Sprinkle consts around
* Emplacing and std::moving
* More cleanup
* That shouldn't have worked after a std::move
* Don't need to allocate memory to compare keys
* Reduce use of the global string pool in tiling
* Another avoidable mvt_value construction
* Further reduction to explicit string pool passing
* These reverses are no longer optimizations
* These layernames can all be references
* Don't drag an unused layername string around with every feature
* Heed a compiler warning about potential buffer overflow
* Fix my confusion about which feature's string pool is relevant
* Avoid some unnecessary allocations in attribute accumulation
* Maybe faster serialization?
* Eliminate a comparison
* Do the same here
* Save a couple of allocations when parsing numbers in JSON
* Immediately assign features to layers instead of subdividing later
* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence
* Crunch out more duplicate attribute values when writing out the tile
* Do tilestats for tippecanoe:retain_points_multiplier_first too
* Shell filters need to be real threads, even if nothing else does
* Simplify tippecanoe_minzoom/maxzoom representation
* Update version and changelog
* Scale the tile size limit up with the multiplier at low zooms
* Add a test to demonstrate that high zoom tiles can't be extra large
* Update changelog and version
* Fail more cleanly when a tile can't be made small enough
* Guard against a cluster where the start marker has been dropped
* Look harder for a working feature interval instead of giving up
* That change to the drop-smallest logic changed a test output
* Update changelog
* Add explanatory comment
* Starting to factor out attribute accumulation into its own file
* Continuing to factor out attribute accumulation
* Reduce duplicate code
* Plumbing the accumulate-attribute option around
* Call the attribute accumulator
* Test that accumulation works
* Add missing #includes
* Don't sort within individual multiplier clusters
Doing so throws off the spatial distribution of the low zooms
* Docs and changelog
* Add comments
* Make feature ordering cooperate with --retain-points-multiplier
* Forgot to check in the actual code changes???
* Sort within each multiplier cluster as well as between clusters
* Correct description of behavior in changelog
* Drag original feature sequence along in megatiles for post-filter sort
* Plumb the preserve-input-order flag through overzoom
* Sort in overzoom if requested
* Use within-tile input sequence numbers, not global sequence numbers
* Documentation
* Reverse direction of search to prevent accidental skipping
* Add some comments about converting between attribute representations
* Add an option to retain N times as many points as usual at each zoom
* Tests for point multipler with specified and guessed maxzooms
* Work in progress on inverse spatial ordering
* Fix inverse spatial feature order
* --reorder was depending on a feature index that wasn't being preserved
* Separate ordering by feature_minzoom from ordering inverse-spatially
* Add a test for the inverse spatial ordering
* Store the basezoom/droprate/multiplier decisions in tileset metadata
* Progress on adding filters to tippecanoe-overzoom
* Type promotion for comparison
* Look up the attribute value for ordering
* Add test of thinning and ordering features
* Plumb tippecanoe_decisions metadata through pmtiles
* Be careful not to put infinities in JSON
* Fix accidental dropping in what is meant to preserve sparse points
* Start distinguishing true, false, and null in expressions
* Most of the type conversions
* Add boolean conversions
* Literals and conjunctions
* Add filtering to tippecanoe-overzoom
* Add a test of filtering in overzoom
* Fix boolean conjunctions
* Handle the combination of cluster size and filtering
* Rework dot dropping to reconcile density threshold and multiplier
* Revert "Rework dot dropping to reconcile density threshold and multiplier"
This reverts commit f253a66382.
* Retain points by multiplier within each tile, not in global probability
* Test that intends to verify that the multiplier is reversible
* Get the test to detect the discrepancy
* Mark the start of multiplier clusters with a magic attribute
* Add string-contains
* Add in and ni operators
* Revert "Look up the attribute value for ordering"
This reverts commit 56bc73e49a.
* Revert "Type promotion for comparison"
This reverts commit 6f3256f5af.
* Make number formatting in tippecanoe_decisions consistent
* Revert "Add a test for the inverse spatial ordering"
This reverts commit c8047de9ab.
* Revert "Separate ordering by feature_minzoom from ordering inverse-spatially"
This reverts commit 35b19a223c.
* Revert "Fix inverse spatial feature order"
This reverts commit 5978ecdb44.
* Revert "Work in progress on inverse spatial ordering"
This reverts commit fdf230f632.
* Somehow missed the tests associated with that last revert
* Round-robin assign attributes to partials from across the multiplier
* Count the multiplier separately in each layer
* Fix distribution of accumulated attribute across multiplier features
* Update changelog, version, and docs
* Add "is null" and "isnt null" expressions
* Update interpretation of FSL expressions to pass the tests
* Test to assert that polygons are unaffected by the multiplier
* Clean up and comment
* Remove accidental unused case
* Drop duplicate geometries sooner in coalescing-as-needed
* Concatenate geometries before partial cleaning
* Spend less time with two feature representations in memory
* Further reduce in-memory duplication
* Remove another copy
* Don't keep duplicates in memory while coalescing
* Also clear out the mvt layer once it is no longer needed
* Update changelog
* 16 bits is enough for tile numbers
* Revert "16 bits is enough for tile numbers"
This reverts commit 71a0c4e1cf.
* Check what child tiles each overzoomed tile will have
and don't queue further overzooming of empty tiles
* Clean up naming; use std::move to avoid copying large arrays
* Special-case a longitude wraparound of exactly 360°
* Update version and changelog
* Make tile-join distrust source tilesets' metadata maxzoom and minzoom
* Only detect longitude wraparound within each ring.
Between rings it's OK to jump around from one side of the world
to the other.
* Add test
* Update changelog
* Clip away entire features by bbox. Avoid unnecessary recompression.
* Move parent tile decoding in tile-join out of overzoom proper
* An ever-growing cache of parent tiles
* Limit the size of the cache
* Remove the current reader *before* checking if we can run the queue
* Clean up
* Add missing #include
* Add comment
* When the tile-join cache fills up, evict the least recently used
* Fix microsecond math
* Factoring out tile-join's cache for testing
* Add unit tests for tile-join cache
* Update changelog and version
* Start of externalizing polygon shard detection
* Completely untested external quicksort
* Add unit test for external quicksort
* Remember to clean up temporary files
* Sort and scan the vertices
* Bring over more vertex logic
* Make nodes from vertices
* Checkpoint on switching over to global shared nodes
* Do the thing
* Revert unintended change to coalesced linestring behavior
* Take shared nodes into account in early simplification
* Let it do more sorting in memory
* Fix overnoding of collinear linestrings
* Remove duplicate nodes, since only duplicate vertices now matter
* Remember to delete temporary files
* Fix out of bounds memory access below, apparently
* Fix the actual undefined behavior
* Still running out of memory in one case. Find out where.
* Forgot the conditional
* Try again to make it not run out of memory
* Update version and changelog
* Fix tile-join crash when trying to merge empty tilesets with --overzoom
* Add an option not to reduce tiny polygons to dust at maxzoom
* Add test for prevention of tiny polygon reduction at maxzoom
* Change version number
* Dropping a 0-length feature doesn't count as dropping-as-needed
* Add an option not to reduce tiny polygons to dust at maxzoom
* Add test for prevention of tiny polygon reduction at maxzoom
* Fix accounting for tiny polygons not to include degenerate geometries
* Revert "Add test for prevention of tiny polygon reduction at maxzoom"
This reverts commit f931bbd73e.
* Revert "Add an option not to reduce tiny polygons to dust at maxzoom"
This reverts commit 03f0882bb6.
* Fix tests
* Another test that no longer has any really tiny polygons
* Oops, that broke LineString simplification
* This time for sure!
* Update changelog and version
* Add an option to extend zooms if still dropping, but with a limit
* At least when to overzoom, even if not actually doing it yet
* Refactor to give tile-join access to overzoom()
* Didn't work, but *might* have worked
* OK, it did something now
* Ah, there's the bug!
* Hook up pmtiles and dirtiles as overzooming sources
* Add command line option to enable or disable overzooming
* Add (currently broken) test of overzooming in tile-join
* Slightly more abstraction for the tile-join readers
* Factor out duplicated code
* Move construction into a constructor
* More changing accessors to methods
* Reduce magic
* Start tracking a list of the tiles at maxzoom
* I think it worked?
* Add missing #include
* Fix sequence of overzoomed tiles (Y sorts backwards for TMS)
* Don't spend memory on overzooming when we aren't going to use it
* Diff rather than cmp, in the hope of figuring out this broken test
* Keep full coordinate precision if we might extend zooms
* Try a slightly different byte limit
* Make drop-densest more consistent across tile boundaries
* Also affects this test
* Does it behave any differently if it can extend forever?
* I think the discrepancy is a thread-safety problem here
* Revert "Does it behave any differently if it can extend forever?"
This reverts commit 0dff0a0acc.
* Lost this change to the test
* This time for sure!
* Revert "Also affects this test"
This reverts commit cd1f7c2e78.
* Revert "Make drop-densest more consistent across tile boundaries"
This reverts commit 563f7d2bc2.
* Revert "Try a slightly different byte limit"
This reverts commit 2e271213d6.
* Add some more explanatory comments
* Amend the join-test to detect my current bug
* Allow overzooming to complete the zoom if it ever starts
* Forgot to correct the test
* Update changelog and version
* Cleanups from code review
* Remove version number from fixture to fix test
Erica Fischer