main moved each tool's option table to file scope so that usage.cpp can
generate the usage message from it (#409), which conflicted with the
tables the MLT options had been added to. The three MLT options move to
the file-scope tables, under the same headings they were listed with
before: "Setting or disabling tile size limits" in tippecanoe and
tile-join, and "Output tile" in tippecanoe-overzoom.
Also combine the new radix-sort-test with the MLT tests in the test
target, add usage.o to the tools that gained MLT objects, keep both the
MLT=0 and the new docs CI jobs, and regenerate the man page for the
README changes, which the docs job now checks.
* Fix variable-length-array and uninitialized-union compiler warnings
Clang warns about every variable-length array in C++ (-Wvla-cxx-extension,
on by default), since VLAs are a compiler extension rather than standard
C++. Replace all 57 of them with std::vector, or with std::string for the
mkstemp() template buffers built from tmpdir. Add -Wvla to WARNING_FLAGS so
new ones don't creep back in.
Separately, mvt_value's numeric_value union is 16 bytes wide (the size of
string_value), but both constructors only wrote the 8 bytes of the member
they were setting, leaving the rest indeterminate. The implicit copy
constructor copies the union as a whole, so copying any non-string value
read uninitialized bytes, which GCC reports as
mvt.hpp:83:8: warning: 'v.mvt_value::numeric_value. ... .len' may be
used uninitialized [-Wmaybe-uninitialized]
Give string_value, the widest member, a default member initializer so the
union's full width is initialized however it is later used.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Fix remaining float-conversion and format-truncation warnings
Clang's -Wimplicit-const-int-float-conversion flagged two comparisons
against LLONG_MAX, which is not representable as a double and rounds up
to 2^63.
In serial.cpp this was a real latent overflow, not just noise: the guard
`extent <= LLONG_MAX` was really `extent <= 2^63`, so an extent of exactly
2^63 passed it and then hit `(long long) extent`, which is undefined for
that value and yields LLONG_MIN in practice -- the opposite of the clamp
the else branch intends. Make the bound exclusive so the conversion is
always in range. Requires a polygon area at the very top of the double
range to reach, but the clamp now behaves as written.
In mbtiles.cpp the value is only a stand-in for infinity on its way into
JSON, so cast explicitly; the emitted number is unchanged.
Separately, g++ at -O0 warned that `char abbrev[20]` can be truncated by
"%lld", which is correct: the most negative long long needs 21 bytes with
the NUL. That branch is only reached when point_count < 1000, so it cannot
happen today, but size the buffer to fit rather than rely on that, and
replace the garbled comment about how the size was derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Clamp the low end of extent before converting to long long too
The upper bound was fixed in the previous commit; the same overflow exists
on the negative side. get_area() returns a signed shoelace area, so inner
rings contribute negatively, and a polygon whose holes outweigh its rings
drives extent below zero. Far enough below and `(long long) extent` is
undefined again.
The bounds are asymmetric, so this is not simply the mirror of the upper
one: LLONG_MIN is exactly -2^63 and converts exactly, so unlike LLONG_MAX
it can be an inclusive bound.
Verified with -fsanitize=float-cast-overflow that the previous form traps
on 2^63 and on doubles just below -2^63, and that this one is clean across
both boundaries, the infinities, and NaN (which falls to LLONG_MAX, as it
did before).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Add CHANGELOG entries for 2.81.0 and bump the version
CHANGELOG.md was last updated for 2.80.0 (#361), and version.hpp has not
moved since. Twelve PRs have landed in the meantime with no entry: #365,
#368, #375, #382, #384, #385, #391, #395, #397, #399, #400, and #401.
Document all of them, plus this PR, under a single 2.81.0 heading. They are
not given separate version numbers because none of them was ever released
under one -- version.hpp read v2.80.0 throughout -- so assigning a version
per PR would invent release history. 2.81.0 is the version that will
actually carry them.
Where an unreleased PR was corrected by a later one (#384 by #385, #397 by
#399), the pair is described as the single behavior that ships, since the
intermediate behavior was never in a release.
Minor rather than patch bump: the batch adds command-line options.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Review feedback: enforce the union-width assumption, describe both clamp ends
The comment on mvt_value's union claimed string_value is the widest member.
That is true on LP64 (16 bytes against 8) but not on ILP32, where size_t is
4 and it ties with double and long long. The default member initializer still
covers the full union either way, so the fix held, but the justification did
not travel. Replace the claim with a static_assert that checks it on whatever
target is being built, so a platform where it stops holding is a compile
error rather than silently indeterminate bytes. Verified the assert is not
vacuous by widening the union in a scratch copy and watching it fail.
The changelog described only the upper end of the extent clamp. Describe both:
the old guard admitted everything below LLONG_MIN too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Add 2.81.0 changelog entries for the four PRs merged from main
#404, #408, #409, and #410 landed while this branch was open. None of them
bumped version.hpp, so they belong under the same 2.81.0 heading as the rest
of the unreleased work rather than getting versions of their own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
---------
Co-authored-by: Claude <noreply@anthropic.com>
Both tools could read MapLibre Tiles but only ever wrote Mapbox Vector
Tiles, so there was no way to convert a tileset into MLT, or to keep a
tileset in MLT once it had been through either of them.
Give them the same --output-format, --pretessellate, and
--no-mlt-feature-sort options that tippecanoe has. tile-join writes the
chosen format to mbtiles files, PMTiles archives, and tile directories,
naming directory tiles and the tileset metadata format accordingly, and
tippecanoe-overzoom writes it to its output tile. Either tool will read
whichever format its sources are in regardless of what it is writing.
The output format selection and the MLT encoder options now live in
mlt.cpp, shared by all three tools rather than defined in main.cpp for
tippecanoe alone, along with encode_tile() for encoding a tile in the
selected format. overzoom() takes the format as a parameter, since
tile-join uses it internally to rescale tiles that will be re-encoded
afterward, and those intermediate tiles should stay MVT.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxhELiLtpUFwrPSWwTYdHG
* Add a flag to use an H3 index for the feature index
* Give mvt_value and serial_val a double-with-count concept
* Switch mean over to internal accumulation state
* Get rid of the attribute accumulation map
* Change vectors of features to vectors of pointers to features
* Fix --coalesce
* Revert "Add a flag to use an H3 index for the feature index"
This reverts commit b9b48f42c9.
* Update version and changelog
* Add a tippecanoe-decode option to restrict which attributes to decode
* Plumb buffer and feature limit around
* Check the feature limit
* Clarifying cases where output detail can be unspecified
* Clip bins to the tile buffer instead of just passing them through
* Add missing include
* Missed some tests
* Add --no-tile-compression option to tippecanoe-overzoom
* Update version and changelog
* Add an all-mvt_value attribute accumulation path
* Only bin by ID, not geometrically
* A little cleanup; changelog and version; test
* Remove accidental double-conversion
* Replace duplicated code with template
* Update changelog
* Factoring out tilestats management from GeoJSON file reading
* Move code around so overzoom can link against parse_layers
* Read the file of bins
* Plumb the bins through to overzoom()
* Some zip code bins to test with
* (Currently non-functional) test of binning
* Starting to spell out the bin matching loop
* Can't flatten points, so don't flatten bins either
* More fleshing out bin traversal
* Bounding box of tile-relative mvt geometry
* Smallest enclosing tile from bbox
* Most of the bin scan
* Add point in polygon check. It crashes.
* Find the matching bins
* GDAL-style bounding boxes have eaten my brain
* Make some features to bin into
* Increment a count as features are found to be within the bins
* Fix longitude wraparound in overzoom bins
* Fix the tests
* Push off attribute copying until after bin assignment
* Carry sum of numeric attributes into the bins
* Also add mean, min, and max
* Add --calculate-feature-index since I keep needing it for testing
* Add an option to accumulate sum/mean/max/min/count of all numeric attrs
* Don't bake in tippecanoe:mean, since we redo it from sum and count
* Forgot to update this test fixture after removing tiled mean
* Update version and changelog
* Prep to track conditions other than just "dropped" or "kept"
* Count up instead of down
* Drop or retain whole multiplier clusters based on their first feature
* Calculate a global feature dropping sequence
* Switch over to using the drop sequence for drop-fraction
* Remove unused arguments for the old drop-fraction implementation
* Fix copy-and-paste bugs, update tests
* Properly incorporate feature_minzoom into the drop sequence, I hope
* Rename drop_by to drop_sequence
* See if sorting within clusters fixes filter stability between zooms
* Remove very chatty debug print
* Update changelog and version
* Use named constants instead of numbers for feature dropping/keeping
* Add comment to explain purpose and method of bit reversal
* Add a way to run tippecanoe single-threaded for profiling
* Do less work when the tilestats sample values list is already full
* Save a copy when retrieving the attribute key
* Fewer atomic operations
* Move string hashing from mbtiles to text
* Only do approximate attribute deduplication when writing tiles
* Feature dropping tests are sensitive to exact tile size
* All tile creators now create a string pool for the tile
* Features clipped away to nothing should not participate in that tile
* Revert "Only do approximate attribute deduplication when writing tiles"
This reverts commit c42b34b498.
* Also revert the related test changes
* Revert "Revert "Only do approximate attribute deduplication when writing tiles""
This reverts commit 18509876c3.
* Be more specific about the string hash function
* Use fnv1a instead of std::hash for everything
* Reduce the chance of hash collisions
* Stick a hash search on the front of the tree search in addpool
* Eliminate repeated hashing of the same string
* Switch instead of ifs in json parsing
* A few more cases to populate the hash in addpool
* Store the hash in the tree instead of recalculating
* Add explanatory comment for mysterious argument
* Fewer copies in attribute stringification
* Clean up ancient weirdness in JSON attribute stringification
* More serial_val cleanup
* Pass a serial_feature to rewrite instead of many broken-down arguments
* Get rid of the multiple geometries within `partial`
* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"
This reverts commit 6f4ab9b725.
* Goodbye, struct coalesce
* Revert "Features clipped away to nothing should not participate in that tile"
This reverts commit 124462fbdc.
* Migrating fields from partial to serial_feature
* Name reconciliation between serial_feature and partial
* Replace struct partial with an augmented serial_feature
* Fix some overzealous search-and-replace renaming
* Don't say struct so often
* Remove more of the former partial construction
* Commenting and cleaning up
* Trying again to avoid all these arguments to rewrite
* I swear I did this same thing before and it didn't work.
* More rewrite cleanup
* Exile --detect-shared-borders to its own file
* Add missing headers
* More commenting and cleanup
* More comments
* Sprinkle consts around
* Emplacing and std::moving
* More cleanup
* That shouldn't have worked after a std::move
* Don't need to allocate memory to compare keys
* Reduce use of the global string pool in tiling
* Another avoidable mvt_value construction
* Further reduction to explicit string pool passing
* These reverses are no longer optimizations
* These layernames can all be references
* Don't drag an unused layername string around with every feature
* Heed a compiler warning about potential buffer overflow
* Fix my confusion about which feature's string pool is relevant
* Avoid some unnecessary allocations in attribute accumulation
* Maybe faster serialization?
* Eliminate a comparison
* Do the same here
* Save a couple of allocations when parsing numbers in JSON
* Immediately assign features to layers instead of subdividing later
* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence
* Crunch out more duplicate attribute values when writing out the tile
* Do tilestats for tippecanoe:retain_points_multiplier_first too
* Shell filters need to be real threads, even if nothing else does
* Simplify tippecanoe_minzoom/maxzoom representation
* Update version and changelog
* Speed up mvt_value comparison
* Converting repetitive ifs to cases
* More conversions from ifs to cases
* Optimize the always-true filter case
* Don't convert types of attributes without accumulators
* Unordered map seems to be faster than map
* Add missing header
* Fix some warnings
* Fix the warnings better
* Avoid an int->string->int conversion
* Lazily initialize layer key and values maps when actually needed
* More switches from maps to unordered_maps
* Sure, I'll take the microoptimization
* More emplacement
* Save some copies
* Emplaces and moves
* Lazy linear scan of attributes instead of building a map
* Extra printfs, missing header
* Avoid clipping if the input and output tiles are the same
* But do clip if the tile extent is being reduced
* Make sure I'm not constructing std::strings here at runtime
* More worrying about runtime string construction
* A couple more std::moves
* Const references!
* More const references
* Another std::move
* Make the string_value of mvt_value std::optional
* Reserve storage when decoding
* Provision for different mvt_values to share a string pool
* Use the string pool when decoding
* Avoid another string construction
* Try limiting the depth of the search for duplicate attributes
* Revert "Try limiting the depth of the search for duplicate attributes"
This reverts commit 9ec94a15ff.
* Update changelog
* Fix typo noticed during code review
* Make feature ordering cooperate with --retain-points-multiplier
* Forgot to check in the actual code changes???
* Sort within each multiplier cluster as well as between clusters
* Correct description of behavior in changelog
* Drag original feature sequence along in megatiles for post-filter sort
* Plumb the preserve-input-order flag through overzoom
* Sort in overzoom if requested
* Use within-tile input sequence numbers, not global sequence numbers
* Documentation
* Reverse direction of search to prevent accidental skipping
* Add some comments about converting between attribute representations
* Remove the concept of "separate metadata"
This was an extra level of attribute indirection (features point
to metadata records which point to key and value strings) which was
intended to reduce the size of temporary storage for features with
large numbers of attributes that were also spread across large numbers
of tiles at maxzoom.
For other kinds of features, the extra indirection slowed things down
instead, and, especially when maxzoom guessing was being used, many more
features were having their metadata externalized than could actually
benefit from it.
* Shave a few bytes off temporary files by using more unsigned integers
* Flush stderr after logging progress
* Revert "Shave a few bytes off temporary files by using more unsigned integers"
This reverts commit eef29084ec.
* Limit the size of the string pools and trees to fit in memory
* Add missing #include
* Move the string pool and search tree from mmap to allocated memory
* Sort in allocated rather than mapped memory too
* Also use pread instead of mapping to read in the data to sort
* When the pool gets too big, switch to just the file, not memory
* Switch string pool from memory to disk when memory is 10% full
* Add to-memory versions of the serialization functions
* Crashy work in progress toward compression
* Fix the pointer bug that was causing the crash
* Serialize features into memory rather than straight to disk
* Compress individual features in the temporary files
* Don't need to store the length of the geometry
* Remove per-feature compression; move minzoom back into the object
* Start adding a stream compressor object
* Track file position within fwrite_check()
* Add compressed stream writer functions
* Pull the writing of the serialized feature out to the callers
* Starting toward compression again from a different point
* Hook up more compression functions
* Remove unused code from the other day
* Make enough deflate calls to flush out all the buffered data
* Start on decompression
* Tile number is uncompressed, tile content is compressed
* Work on alternating compressed and uncompressed in decompression
* Closer, but still doesn't work
* Sort of works
* Works until we get to concatenated tiles
* More attempts that don't work
* One bug down
* It made a tileset!
* Handle nonzero initial zooms
* Fix seeking within compressed feature streams
* Tests pass!
* Remove debug spew
* Oops: remember to delete the temporary files so they don't hang around
* Test that fails with the current compression code
* Properly account for bytes read while closing the compressed stream
* Limit the number of warnings about bad label points
* A little more armor when closing decompression
* This time for sure
* A different, less fragile, test that failed previously with compression
* Move feature stream compression to its own file
* Remove now-unused code to deserialize from a file
* Forgot to add the new files
* Remove a little debugging logging
* Add a couple of comments on what it means to be within decompression
* Fix indentation
* Update changelog. Remove stray debugging comment.
* Add an option to retain extra coordinate precision at maxzoom
* Make sure not to shift away the extra detail from coordinates
* Add an option to convert double-precision attributes to single
* Sort attribute values in tiles to make them compress a little better
* Slightly improve polygon simplification
By choosing a point that would be retained after simplification
to be the start/end point that always gets retained
* I regret making all of these tests involve polygons
* Add an option to specify the size of tiny polygons
* Fix accidental requiring of argument for --single-precision
* Guard against duplicate points when generating "sizes" for them
* Restore the intended behavior that tiny polygons don't get simplified
* Make the extra detail settable rather than always maximizing it
* Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)"
This reverts commit fec5e8354c.
* Add an option to prevent choosing a base zoom higher than the maxzoom
* Keep the drop rate high enough when the basezoom gets constrained
* Revert "Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)""
This reverts commit db6bc27d9e.
* Add --order-by and --order-descending options
* Accept multiple --order-by and --order-descending-by sort keys
The first feature in a tile can never be dropped, since there is
no previous feature to attach its properties to.
Remove the previous special case that reset the dropping counter
at the first feature within each tile proper (as opposed to the
first feature in each tile, including its buffer, which is now
the one that is guaranteed to be preserved).
Claude