* Scale the tile size limit up with the multiplier at low zooms
* Add a test to demonstrate that high zoom tiles can't be extra large
* Update changelog and version
* Fail more cleanly when a tile can't be made small enough
* Guard against a cluster where the start marker has been dropped
* Look harder for a working feature interval instead of giving up
* That change to the drop-smallest logic changed a test output
* Update changelog
* Add explanatory comment
* Starting to factor out attribute accumulation into its own file
* Continuing to factor out attribute accumulation
* Reduce duplicate code
* Plumbing the accumulate-attribute option around
* Call the attribute accumulator
* Test that accumulation works
* Add missing #includes
* Don't sort within individual multiplier clusters
Doing so throws off the spatial distribution of the low zooms
* Docs and changelog
* Add comments
* Make feature ordering cooperate with --retain-points-multiplier
* Forgot to check in the actual code changes???
* Sort within each multiplier cluster as well as between clusters
* Correct description of behavior in changelog
* Drag original feature sequence along in megatiles for post-filter sort
* Plumb the preserve-input-order flag through overzoom
* Sort in overzoom if requested
* Use within-tile input sequence numbers, not global sequence numbers
* Documentation
* Reverse direction of search to prevent accidental skipping
* Add some comments about converting between attribute representations
* Add an option to retain N times as many points as usual at each zoom
* Tests for point multipler with specified and guessed maxzooms
* Work in progress on inverse spatial ordering
* Fix inverse spatial feature order
* --reorder was depending on a feature index that wasn't being preserved
* Separate ordering by feature_minzoom from ordering inverse-spatially
* Add a test for the inverse spatial ordering
* Store the basezoom/droprate/multiplier decisions in tileset metadata
* Progress on adding filters to tippecanoe-overzoom
* Type promotion for comparison
* Look up the attribute value for ordering
* Add test of thinning and ordering features
* Plumb tippecanoe_decisions metadata through pmtiles
* Be careful not to put infinities in JSON
* Fix accidental dropping in what is meant to preserve sparse points
* Start distinguishing true, false, and null in expressions
* Most of the type conversions
* Add boolean conversions
* Literals and conjunctions
* Add filtering to tippecanoe-overzoom
* Add a test of filtering in overzoom
* Fix boolean conjunctions
* Handle the combination of cluster size and filtering
* Rework dot dropping to reconcile density threshold and multiplier
* Revert "Rework dot dropping to reconcile density threshold and multiplier"
This reverts commit f253a66382.
* Retain points by multiplier within each tile, not in global probability
* Test that intends to verify that the multiplier is reversible
* Get the test to detect the discrepancy
* Mark the start of multiplier clusters with a magic attribute
* Add string-contains
* Add in and ni operators
* Revert "Look up the attribute value for ordering"
This reverts commit 56bc73e49a.
* Revert "Type promotion for comparison"
This reverts commit 6f3256f5af.
* Make number formatting in tippecanoe_decisions consistent
* Revert "Add a test for the inverse spatial ordering"
This reverts commit c8047de9ab.
* Revert "Separate ordering by feature_minzoom from ordering inverse-spatially"
This reverts commit 35b19a223c.
* Revert "Fix inverse spatial feature order"
This reverts commit 5978ecdb44.
* Revert "Work in progress on inverse spatial ordering"
This reverts commit fdf230f632.
* Somehow missed the tests associated with that last revert
* Round-robin assign attributes to partials from across the multiplier
* Count the multiplier separately in each layer
* Fix distribution of accumulated attribute across multiplier features
* Update changelog, version, and docs
* Add "is null" and "isnt null" expressions
* Update interpretation of FSL expressions to pass the tests
* Test to assert that polygons are unaffected by the multiplier
* Clean up and comment
* Remove accidental unused case
* Special-case a longitude wraparound of exactly 360°
* Update version and changelog
* Make tile-join distrust source tilesets' metadata maxzoom and minzoom
* Only detect longitude wraparound within each ring.
Between rings it's OK to jump around from one side of the world
to the other.
* Add test
* Update changelog
* Clip away entire features by bbox. Avoid unnecessary recompression.
* Move parent tile decoding in tile-join out of overzoom proper
* An ever-growing cache of parent tiles
* Limit the size of the cache
* Remove the current reader *before* checking if we can run the queue
* Clean up
* Add missing #include
* Add comment
* When the tile-join cache fills up, evict the least recently used
* Fix microsecond math
* Factoring out tile-join's cache for testing
* Add unit tests for tile-join cache
* Update changelog and version
* Start of externalizing polygon shard detection
* Completely untested external quicksort
* Add unit test for external quicksort
* Remember to clean up temporary files
* Sort and scan the vertices
* Bring over more vertex logic
* Make nodes from vertices
* Checkpoint on switching over to global shared nodes
* Do the thing
* Revert unintended change to coalesced linestring behavior
* Take shared nodes into account in early simplification
* Let it do more sorting in memory
* Fix overnoding of collinear linestrings
* Remove duplicate nodes, since only duplicate vertices now matter
* Remember to delete temporary files
* Fix out of bounds memory access below, apparently
* Fix the actual undefined behavior
* Still running out of memory in one case. Find out where.
* Forgot the conditional
* Try again to make it not run out of memory
* Update version and changelog
* Put all of this back in geometry.cpp for conflict resolution
* Stabilize line simplification to behave the same regardless of winding
* Round instead of truncating when clipping lines
* Restore non-Wagyu polygon clipping from prior to 2fdec7d2
* Make it round, not truncate, which reverts the last commit's test diffs
* Clip in floating point, not integers, which makes no difference
* Track nodes added at tile edges during clipping
* Scale geometry up before wagyu to prevent changes from precision loss
* Actually do the shared edge detection
* Fix cases where nodes were not being added at the tile boundary
* One more place I should have rounded
* Narrow down where the discrepancy comes in
* Revert "Narrow down where the discrepancy comes in"
This reverts commit 221c4c5fc0ac9a6567e091c6a94b3d87dc8ade83.
* Another attempt to narrow it down
* The discrepancy seems to be introduced in reordering. No obvious bug
* Was still truncating instead of rounding in projection
* Also makes no difference...
* Just forget that line reversal exists for a minute
* Just reversal no coalescing
* Try clipping in integers instead of floating point
* Don't simplify after coalescing if they said no simplification
* Check whether behavior is consistent with intentional simplification
* Replace more floating point with integer
* Are these three features enough to demonstrate the problem?
* Add a few more nearby borders
* All the features that touch tile 6/16/23
* Stay in integers in line simplification
* More attempts to solve failures to simplify consistently
* Fix most of the overflow errors
* Fix known cases of integer overflow
* All the tests change again
* Pull clipping and scaling code back out into clip.cpp
* Resolve the test conflicts
* Stabilize choice of which three points to keep with different windings
* Almost right, I think!
* Fix collapse of islands to shards
* Self-intersections in the same feature don't count
* Revert "Self-intersections in the same feature don't count"
This reverts commit e04b19916e.
* Don't scale down geometry if we are going to look for shared nodes
* Fix the missing multiply that was keeping simplification from happening
* Fix one more opportunity for overflow
* Somehow I deleted this test?
* Lost this test too
* Clean up debugging printfs
* Restore code sequence from main to make it reviewable
* Remove unneeded rounding
* Update documentation
* This test is no longer useful
* Back to floating point Douglas-Peucker to fix undersimplification
* Try an older ubuntu
* Revert "Try an older ubuntu"
This reverts commit 13fefacfd7.
* Log OS info
* Remove tests that are no longer needed
* Oops, did need that one after all
* Fix the arm vs x86 discrepancy?
* Try another quantization
* Cleanup from review
* Add a test for the actual purpose of this PR
* Fix tile-join crash when trying to merge empty tilesets with --overzoom
* Add an option not to reduce tiny polygons to dust at maxzoom
* Add test for prevention of tiny polygon reduction at maxzoom
* Change version number
* Dropping a 0-length feature doesn't count as dropping-as-needed
* Add an option not to reduce tiny polygons to dust at maxzoom
* Add test for prevention of tiny polygon reduction at maxzoom
* Fix accounting for tiny polygons not to include degenerate geometries
* Revert "Add test for prevention of tiny polygon reduction at maxzoom"
This reverts commit f931bbd73e.
* Revert "Add an option not to reduce tiny polygons to dust at maxzoom"
This reverts commit 03f0882bb6.
* Fix tests
* Another test that no longer has any really tiny polygons
* Oops, that broke LineString simplification
* This time for sure!
* Update changelog and version
* Add an option to extend zooms if still dropping, but with a limit
* At least when to overzoom, even if not actually doing it yet
* Refactor to give tile-join access to overzoom()
* Didn't work, but *might* have worked
* OK, it did something now
* Ah, there's the bug!
* Hook up pmtiles and dirtiles as overzooming sources
* Add command line option to enable or disable overzooming
* Add (currently broken) test of overzooming in tile-join
* Slightly more abstraction for the tile-join readers
* Factor out duplicated code
* Move construction into a constructor
* More changing accessors to methods
* Reduce magic
* Start tracking a list of the tiles at maxzoom
* I think it worked?
* Add missing #include
* Fix sequence of overzoomed tiles (Y sorts backwards for TMS)
* Don't spend memory on overzooming when we aren't going to use it
* Diff rather than cmp, in the hope of figuring out this broken test
* Keep full coordinate precision if we might extend zooms
* Try a slightly different byte limit
* Make drop-densest more consistent across tile boundaries
* Also affects this test
* Does it behave any differently if it can extend forever?
* I think the discrepancy is a thread-safety problem here
* Revert "Does it behave any differently if it can extend forever?"
This reverts commit 0dff0a0acc.
* Lost this change to the test
* This time for sure!
* Revert "Also affects this test"
This reverts commit cd1f7c2e78.
* Revert "Make drop-densest more consistent across tile boundaries"
This reverts commit 563f7d2bc2.
* Revert "Try a slightly different byte limit"
This reverts commit 2e271213d6.
* Add some more explanatory comments
* Amend the join-test to detect my current bug
* Allow overzooming to complete the zoom if it ever starts
* Forgot to correct the test
* Update changelog and version
* Cleanups from code review
* Remove version number from fixture to fix test
* Starting work on overzooming
* Factor clipping out of geometry.cpp to simplify linkage
* Pull out more geometry functions into now-badly-named clip.cpp
* Not surprisingly, there is a bug
* Found the bug
* Make indent
* Pass attributes through
* Make formatting more consistent
* Fix typos in comments
* Fix the typos better
* Add tippecanoe-overzoom to the install list
* Give overzoom a predictable exit status
* Forgot to translate to and from polygon ring closepaths
* Fix geometry collapse at z21
* Add attribute stripping; don't generate layers if they have no features
* Add docs for tippecanoe-overzoom (as it will be, not as it is)
* Change overzoom to accept input and output files as arguments
* Working on overzooming tests
* Oops
* Hook up and test the detail and buffer options, and the empty-tile case
* Also test attribute stripping
* Fix error message
* Update changelog and version
* Starting on --set-attributes
* Add a JSON form to accumulate-attribute
* Fix formatting
* Set attributes, JSON form
* Also handle setting attributes to unquoted values
* Factoring out character-mangling in the names of tests
* Rename tests for new quoting convention
* Test for JSON form of attribute setting
* Test for JSON form of attribute accumulation
* Forgot to increment the version
* Turn off unit, which has bit-rotted
* Instrument line simplification
* Do more work in integer space
* Update tests with simplification done with more integers
* Clean up
* Update changelog
* Eradicate calls to deprecated sprintf()
* Revert "Turn off unit, which has bit-rotted"
This reverts commit 998779313f.
* Fix catch by switching to an unsigned type
* Calculate a new antimeridian-adjusted bounding box
* Add antimeridian bounding box to pmtiles, dirtiles, and tile-join
* Don't take out-of-bounds latitudes into account in the adjusted bbox
* Update changelog
* Forgot to adjust tests after the last change
* Allow passing a file with a list of input files to tile-join in response to Issue 69 (https://github.com/felt/tippecanoe/issues/69)
* Do not print usage if -r is specified without a commandline readfrom file
* Add test for -r option to tile-join
* Remove the concept of "separate metadata"
This was an extra level of attribute indirection (features point
to metadata records which point to key and value strings) which was
intended to reduce the size of temporary storage for features with
large numbers of attributes that were also spread across large numbers
of tiles at maxzoom.
For other kinds of features, the extra indirection slowed things down
instead, and, especially when maxzoom guessing was being used, many more
features were having their metadata externalized than could actually
benefit from it.
* Shave a few bytes off temporary files by using more unsigned integers
* Flush stderr after logging progress
* Revert "Shave a few bytes off temporary files by using more unsigned integers"
This reverts commit eef29084ec.
* Limit the size of the string pools and trees to fit in memory
* Add missing #include
* Move the string pool and search tree from mmap to allocated memory
* Sort in allocated rather than mapped memory too
* Also use pread instead of mapping to read in the data to sort
* When the pool gets too big, switch to just the file, not memory
* Switch string pool from memory to disk when memory is 10% full
* Add to-memory versions of the serialization functions
* Crashy work in progress toward compression
* Fix the pointer bug that was causing the crash
* Serialize features into memory rather than straight to disk
* Compress individual features in the temporary files
* Don't need to store the length of the geometry
* Remove per-feature compression; move minzoom back into the object
* Start adding a stream compressor object
* Track file position within fwrite_check()
* Add compressed stream writer functions
* Pull the writing of the serialized feature out to the callers
* Starting toward compression again from a different point
* Hook up more compression functions
* Remove unused code from the other day
* Make enough deflate calls to flush out all the buffered data
* Start on decompression
* Tile number is uncompressed, tile content is compressed
* Work on alternating compressed and uncompressed in decompression
* Closer, but still doesn't work
* Sort of works
* Works until we get to concatenated tiles
* More attempts that don't work
* One bug down
* It made a tileset!
* Handle nonzero initial zooms
* Fix seeking within compressed feature streams
* Tests pass!
* Remove debug spew
* Oops: remember to delete the temporary files so they don't hang around
* Test that fails with the current compression code
* Properly account for bytes read while closing the compressed stream
* Limit the number of warnings about bad label points
* A little more armor when closing decompression
* This time for sure
* A different, less fragile, test that failed previously with compression
* Move feature stream compression to its own file
* Remove now-unused code to deserialize from a file
* Forgot to add the new files
* Remove a little debugging logging
* Add a couple of comments on what it means to be within decompression
* Fix indentation
* Update changelog. Remove stray debugging comment.
* Back out the slow search for other small features when dropping features
* Actually, *do* look for small features still when coalescing
* Update changelog
* Avoid placing polygon labels in holes
* Traverse vertices in hilbert order to find potential label points
* Update test to reflect new label placements
* Sorting by Y coordinate is better and easier than Hilbert order
* Give a bonus for being near the center of mass
* Make label points a little more border-shy
* Tall places, not just wide places; border lines, not just border points
* Clean up
* Try diagonals through the features too
* Limit the search for label points to prevent slowing down too much
* At this point the center of mass bonus is doing more harm than good
* Forgot to update test
* Round coordinates instead of truncating them
* Update all the tests for coordinate rounding changes
* Curses, integer division still truncates
* Fix tests
* Also round instead of shifting when looking for no-op linetos
* Should I worry that the same change for moveto doesn't change any tests?
* Also round instead of shifting when scaling down to maxzoom resolution
* Replace another explicit shift, for origin point
* Round instead of shift when writing clipped geometries to the next zoom
* Fix low-zoom gridding and smaller-than-a-pixel checks
* Don't guess an excessively large maxzoom when there is only one feature
* Add a test for guessing the maxzoom of a single point
* Explicitly sort by index if no other order distinguishes features
* Another affected test
* Set base zoom if requested as part of maxzoom guessing
* Add a test for guessing the maxzoom of a single point
* Don't guess an excessively large maxzoom when there is only one feature
* Update changelog
* add pmtiles.hpp from github.com/protomaps/PMTiles [#10]
* tippecanoe main writes pmtiles output. [#10]
* detect output format using suffix
* after mbtiles is done writing, replace with pmtiles based on map/image tables.
* add method to write_json for writing json sub-object.
* tippecanoe-decode reads pmtiles input. [#10]
* tile-join reads and writes pmtiles. [#10]
* pmtiles test suite for decode and tile-join [#10]
* add base GitHub CI action for compiling and test suite.
* update pmtiles.hpp with z>15 fix
* Fix some ordering problems with pmtiles decode
* Pmtiles should also pass the raw tiles tests
* Eradicate spaces from tileset metadata JSON fields
* Eradicate spaces from more test fixtures
* Update more tests
* Pmtiles tests pass now too
* Remove unnecessary sort (and make indent)
* Update changelog
* The allow-existing test for pmtiles needs -o, not -e
* Declare --allow-existing to be unsupported for pmtiles.
It was always a bad idea even for mbtiles.
Co-authored-by: Brandon Liu <bdon@bdon.org>
* Add an option to limit geometry vertex count
* Update docs and changelog
* Track desired feature count and geometry size in strategies
* Remove dead code for a long-forgotten inaccessible option
* Fix the option name in changelog
* Refine coalesce-smallest to only coalesce onto other small features
* Clean polygons before coalescing-as-needed
* Don't accumulate tiny polygon holes as negative dust
* Add the option not to limit the feature count at maxzoom
* Add options to limit feature count more abruptly in each tile
* Revert "Add the option not to limit the feature count at maxzoom"
This reverts commit ace173ab7a.
* Revert "Fix the option name in changelog"
This reverts commit 38dbce8405.
* Revert "Add an option to limit geometry vertex count"
This reverts commit d379fdf06b.
* Remove test for reverted option
* Remove more leftovers from geometry size limiting
* Update changelog
* Add an option to drop the densest features by rate, not a fraction
* Restrict dot dropping just to dots, as it should be
* 50% mix of sparse and uniform point preservation
* Update version, changelog, test, and documentation
* 60% sparse, 40% dense
* Didn't update the test after the last change
* Make the drop-denser rate configurable without recompiling
* Change sqlite3 schema to deduplicate identical tiles
* Limit guessed maxzoom to avoid spending too many tiles on polygon fill
* Fix test.
These dust polygons now have their area calculated because their
maxzoom is being guessed, so the attributes from the largest one
rather than the last one are preserved.
* Increase polygon limit to a million tiles
* Two million tiles ought to be enough for anyone, right?
* Add explanatory comments for mysterious numbers
* Add an option to generate label points in place of polygons
* Change all these places where I said "extent" but really meant "area"
* Revert "Change all these places where I said "extent" but really meant "area""
This reverts commit 403828d2f7.
* Add --order-smallest-first and --order-largest-first options
* Use Turf's center-of-mass algorithm for polygon label points
* If the label point isn't within the polygon, find one that is
* Don't choose a label point that is too close to a border
* Try a little harder to find an optimal label point
* Checkerboard which tiles labels are generated in, to reduce adjacency
* Use a label point for the general representative point for polygons
(Skipping the iteration to find one that is as far as possible from
the borders)
This makes the labels look better in many cases (like France at z1)
but unfortunately ripples into changing the sequence of polygons in
many tests, so the diff is big.
* Revert "Use a label point for the general representative point for polygons"
This reverts commit 2261adf05e.
* Checkpoint work on spiral labels
* Clip label spirals to the feature bounds
* Fix label test
* Be careful not to place spiral labels too close to borders either
* For spiral anchors, only check tile scale, not feature size
* Update test
* Only try to find a central label point for the largest ring
* In tiny polygon dust, keep the attributes of the largest feature
* Improve precision of get_area by using long double
* Trying to get consistent polygon area results between ARM and x86
* Calculate polygon area closer to the origin for better precision
* Update changelog
* Also exercise tiny polygon dust in the ring area test
They previously behaved differently here between x86 and ARM
* On M1 Macs, long double is just double anyway, so don't use it
* Be more careful about overflow: scale the polygon ring down into range
* Fix the bug I just introduced in the scaled area calculation
* Use only the sign from the scaled-down area calculation
Co-authored-by: Roman Karavia <47303530+romankaravia@users.noreply.github.com>
* First attempt at porting Paul Mach's Visvalingam implementation
* Make indent
* Mostly working
* Approximate equivalence from Douglas-Peucker to Visvalingam levels
* Don't simplify away tile boundary crossing points
* Add a command-line option and test for Visvalingam simplification
* Update changelog
* Cleanup in response to review feedback
* Include <stdio.h> to fix compiler warnings
* Stop adding features to a tile if it can't possibly work
* Add --integer and --fraction options to tippecanoe-decode
* Carry the strategies field from tileset metadata through tile-join
* Update changelog
* Assign different codes to different kinds of error exits
* Reduce maxzooms being guessed a little:
* Use 1.5 standard deviations, not 2, as the minimum distinguishable
* Give overlapping polygons and linestrings more distinct indices
* Add another drop rate guessing options, from the same metrics -zg uses
* Guard against using -rp without -zg
* Add an option to retain extra coordinate precision at maxzoom
* Make sure not to shift away the extra detail from coordinates
* Add an option to convert double-precision attributes to single
* Sort attribute values in tiles to make them compress a little better
* Slightly improve polygon simplification
By choosing a point that would be retained after simplification
to be the start/end point that always gets retained
* I regret making all of these tests involve polygons
* Add an option to specify the size of tiny polygons
* Fix accidental requiring of argument for --single-precision
* Guard against duplicate points when generating "sizes" for them
* Restore the intended behavior that tiny polygons don't get simplified
* Make the extra detail settable rather than always maximizing it
* Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)"
This reverts commit fec5e8354c.
* Add an option to prevent choosing a base zoom higher than the maxzoom
* Keep the drop rate high enough when the basezoom gets constrained
* Revert "Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)""
This reverts commit db6bc27d9e.
* Add --order-by and --order-descending options
* Accept multiple --order-by and --order-descending-by sort keys
* Add --maximum-zoom-at-least to specify a minimum maxzoom to guess
* Check whether the specified minimum maxzoom is out of range
* Rename the option to --smallest-maximum-zoom-guess
* Update tests, changelog, and version
* Handle points too when dropping or coalescing the "smallest" features.
* Add statistics of tile size reduction strategies to tileset metadata
* Update changelog and version
* Update documentation
* Change JSON objects to a union type to use less memory
* Stop storing the string representation of JSON numbers
* Restore the ability to create features with large integer attributes
* Make sure large-integer feature IDs still behave as before
* Add missing #include
* Don't preallocate as much space for arrays and objects
* Treat inability to check free disk space as a warning, not an error
* Update changelog and version
Erica Fischer