The deduplicate-by-id case of overzoom-test writes six artifacts but its
cleanup line removes only five: tests/pbf/merged-dedup.pbf, created by the
--deduplicate-by-id overzoom run, is the one left out, while its decoded
.json.check counterpart is removed. So a full `make test` leaves an untracked
file in the working tree, which shows up as spurious `git status` noise and
can be committed by accident.
No CHANGELOG entry or version bump: this only affects the test harness, and
nothing about a released build changes. Keeping those files untouched also
means this does not collide with the 2.81.1 entry in the label-anchor fix,
so the two can merge in either order.
Verified by running `make overzoom-test` both ways: with the old line the
file is left behind as untracked, with the new one the tree is clean, and
the target passes either way.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NcSXLjtokxioaBCCwGTEv1
* Fix variable-length-array and uninitialized-union compiler warnings
Clang warns about every variable-length array in C++ (-Wvla-cxx-extension,
on by default), since VLAs are a compiler extension rather than standard
C++. Replace all 57 of them with std::vector, or with std::string for the
mkstemp() template buffers built from tmpdir. Add -Wvla to WARNING_FLAGS so
new ones don't creep back in.
Separately, mvt_value's numeric_value union is 16 bytes wide (the size of
string_value), but both constructors only wrote the 8 bytes of the member
they were setting, leaving the rest indeterminate. The implicit copy
constructor copies the union as a whole, so copying any non-string value
read uninitialized bytes, which GCC reports as
mvt.hpp:83:8: warning: 'v.mvt_value::numeric_value. ... .len' may be
used uninitialized [-Wmaybe-uninitialized]
Give string_value, the widest member, a default member initializer so the
union's full width is initialized however it is later used.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Fix remaining float-conversion and format-truncation warnings
Clang's -Wimplicit-const-int-float-conversion flagged two comparisons
against LLONG_MAX, which is not representable as a double and rounds up
to 2^63.
In serial.cpp this was a real latent overflow, not just noise: the guard
`extent <= LLONG_MAX` was really `extent <= 2^63`, so an extent of exactly
2^63 passed it and then hit `(long long) extent`, which is undefined for
that value and yields LLONG_MIN in practice -- the opposite of the clamp
the else branch intends. Make the bound exclusive so the conversion is
always in range. Requires a polygon area at the very top of the double
range to reach, but the clamp now behaves as written.
In mbtiles.cpp the value is only a stand-in for infinity on its way into
JSON, so cast explicitly; the emitted number is unchanged.
Separately, g++ at -O0 warned that `char abbrev[20]` can be truncated by
"%lld", which is correct: the most negative long long needs 21 bytes with
the NUL. That branch is only reached when point_count < 1000, so it cannot
happen today, but size the buffer to fit rather than rely on that, and
replace the garbled comment about how the size was derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Clamp the low end of extent before converting to long long too
The upper bound was fixed in the previous commit; the same overflow exists
on the negative side. get_area() returns a signed shoelace area, so inner
rings contribute negatively, and a polygon whose holes outweigh its rings
drives extent below zero. Far enough below and `(long long) extent` is
undefined again.
The bounds are asymmetric, so this is not simply the mirror of the upper
one: LLONG_MIN is exactly -2^63 and converts exactly, so unlike LLONG_MAX
it can be an inclusive bound.
Verified with -fsanitize=float-cast-overflow that the previous form traps
on 2^63 and on doubles just below -2^63, and that this one is clean across
both boundaries, the infinities, and NaN (which falls to LLONG_MAX, as it
did before).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Add CHANGELOG entries for 2.81.0 and bump the version
CHANGELOG.md was last updated for 2.80.0 (#361), and version.hpp has not
moved since. Twelve PRs have landed in the meantime with no entry: #365,
#368, #375, #382, #384, #385, #391, #395, #397, #399, #400, and #401.
Document all of them, plus this PR, under a single 2.81.0 heading. They are
not given separate version numbers because none of them was ever released
under one -- version.hpp read v2.80.0 throughout -- so assigning a version
per PR would invent release history. 2.81.0 is the version that will
actually carry them.
Where an unreleased PR was corrected by a later one (#384 by #385, #397 by
#399), the pair is described as the single behavior that ships, since the
intermediate behavior was never in a release.
Minor rather than patch bump: the batch adds command-line options.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Review feedback: enforce the union-width assumption, describe both clamp ends
The comment on mvt_value's union claimed string_value is the widest member.
That is true on LP64 (16 bytes against 8) but not on ILP32, where size_t is
4 and it ties with double and long long. The default member initializer still
covers the full union either way, so the fix held, but the justification did
not travel. Replace the claim with a static_assert that checks it on whatever
target is being built, so a platform where it stops holding is a compile
error rather than silently indeterminate bytes. Verified the assert is not
vacuous by widening the union in a scratch copy and watching it fail.
The changelog described only the upper end of the extent clamp. Describe both:
the old guard admitted everything below LLONG_MIN too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
* Add 2.81.0 changelog entries for the four PRs merged from main
#404, #408, #409, and #410 landed while this branch was open. None of them
bumped version.hpp, so they belong under the same 2.81.0 heading as the rest
of the unreleased work rather than getting versions of their own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Don't write an extra byte when the radix sort writes a bucket directly
radix1() writes out a sorted bucket in two places. merge() writes all but
the last byte of each serialized feature and then appends the byte for the
feature minzoom, since the minzoom is the last byte of the feature. The path
taken when a bucket holds only one feature, or when the recursion has
consumed every bit of the index, instead writes the feature's whole
serialized length and then appends another minzoom byte, which is one byte
more than the feature's length prefix says it is. Everything read from the
geometry afterward is then misaligned by a byte.
--prefer-radix-sort lowers the memory limit to 8K so that this code gets
exercised, and any bucket that has to be written directly is enough to
desynchronize the stream, so it fails on several of the existing test
inputs:
$ ./tippecanoe -q -f -o out.mbtiles -z4 -aR tests/ne_110m_ocean/in.json
wrong length decoding feature: used 10, len is 33
Write one byte less here too, as merge() does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011wLk2itWETPBAS9a9yE8zu
* Keep the radix sort from recursing forever when it runs out of files
radix1() subdivides a bucket by the next splitbits bits of the index, and
stops recursing once prefix + splitbits reaches the width of the index.
The number of buckets comes from the number of files still available, which
shrinks at every level, so deep enough recursion reaches availfiles / 4 == 1
and therefore splitbits == 0. At that point the recursion consumes no bits
of the index and availfiles stops shrinking, so prefix never advances and
the recursion has no way to terminate.
A splitbits of 0 also makes the shift that chooses a feature's bucket a
shift by the full width of the index, which is undefined. In practice it
leaves the shift count masked to zero, so the bucket number is the whole
index rather than 0, and writing to that bucket runs off the end of the
arrays of open files.
Require at least two buckets so that each subdivision always consumes at
least one bit of the index and the shift is always in range, and don't
recurse at all when the next level would not have enough files to split
with: sort that bucket in memory instead, even though it is larger than
the memory limit asked for, since that is the only way left to get it
sorted.
--prefer-radix-sort, which lowers the memory limit to 8K so that this code
gets exercised, segfaults on tests/feature-filter/in.json without this:
$ ./tippecanoe -q -f -o out.mbtiles -z0 -aR tests/feature-filter/in.json
Segmentation fault
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011wLk2itWETPBAS9a9yE8zu
* Check that the radix sort and the in-memory sort agree
The result of a sort shouldn't depend on how the sort was performed, so
rather than checking the sorted output against a committed copy of it,
check that --prefer-radix-sort, which lowers the memory limit to 8K to
force the radix subdivision to recurse, produces the same tiles as sorting
in memory. Nothing new has to be kept up to date, and the comparison holds
regardless of how deeply the subdivision recurses on a given machine, which
depends on how many files it will let us open at once.
What sends the sort down the paths that are otherwise almost never taken is
the shape of the input rather than the size of it, so two small inputs are
generated for the purpose: several well-separated features that are each
too big to sort in memory, which are each written out as a bucket of their
own, and many features at one location, which have to be subdivided until
there are no index bits left. Between them and tests/feature-filter, all
three of radix1()'s branches are covered, including sorting in memory
because there are no files left to subdivide with.
Both of these inputs fail without the two preceding commits, and every
input here failed before them.
The whole target runs in about ten seconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011wLk2itWETPBAS9a9yE8zu
* Record what the shift by the full index width actually did
Say in the comment that masking the shift count to zero makes the bucket
number come out as the whole shifted index, so the writes go somewhere
past the end of the arrays of buckets, rather than only that the shift is
undefined.
Also correct the note on the test: --prefer-radix-sort sets the memory
limit to 8K, but radix() halves it again, so the subdivision is working
against 4K.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011wLk2itWETPBAS9a9yE8zu
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Generate the man page with go-md2man instead of md2man-roff
md2man-roff is distributed only as a Ruby gem -- it is in neither Homebrew
nor apt -- so in practice nobody has it installed and man/tippecanoe.1
drifts away from README.md. It was stale again as of #400: the man page
still had the dead All Streets link that commit fixed.
Switch to go-md2man, the maintained Go port of the same converter (it is
what Docker, podman and runc use). It is packaged as a single static
binary for Homebrew, apt, Fedora and Alpine, and it renders inline code as
bold the same way md2man-roff did, so the man page still reads the way it
used to.
It also emits valid roff, which md2man-roff did not. `mandoc -T lint` goes
from 621 errors and warnings to 1 (an empty .TH date, left empty on
purpose so that generation stays reproducible). 590 of those were
`invalid escape sequence: \fC`, from md2man-roff wrapping every inline
code span in `\fB\fC` -- `\fC` is not a font escape.
md2man-roff was losing content, too:
README: 1/(2^32) of the size of Earth
md2man-roff: 1/(2 of the size of Earth
go-md2man: 1/(2^32) of the size of Earth
README: '{"attr": "operation", "attr2": "operation2"}'
md2man-roff: '{"attr": "operation", "attr2", "operation2"}'
go-md2man: '{"attr": "operation", "attr2": "operation2"}'
Prepend a title block and a NAME section during generation rather than
adding them to README.md, where they would render as noise on GitHub.
The man page had neither, so its header rendered as "tippecanoe()" with no
section, and `man -k tippecanoe` and `whatis tippecanoe` found nothing.
It now renders as TIPPECANOE(1) and is indexed.
Finally, add a CI job that regenerates the man page and fails if the
committed copy differs, so a README edit that needs `make docs` gets
caught rather than sitting stale until someone notices. This is what
makes the missing-tool problem stop mattering: contributors no longer
need go-md2man installed to keep the man page current, since CI will
tell them when it needs regenerating. The go-md2man version is pinned
there because different versions produce different roff for the same
input.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015B6PcMbNY779iaczo6S6Pu
* Strip the redundant blank lines go-md2man puts between paragraphs
go-md2man separates paragraphs with a blank line as well as a .PP macro.
A blank line is itself a break in roff, so the two together double-space
the page: every paragraph was followed by two blank lines rather than one.
md2man-roff did not do this, so it showed up as a regression -- the source
went from 13 blank lines to 200.
Filter them out after generation. Blank lines inside .EX and .TS blocks
are kept, since there they are part of the example or the table rather
than spacing around it; that is all 13 of the ones md2man-roff emitted.
The rendered page loses 186 blank lines and the source loses 187, with
byte-identical non-blank output under both groff -t -man and mandoc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015B6PcMbNY779iaczo6S6Pu
* Decouple the man page from version.hpp, and name the first section
Review feedback on #408.
Making man/tippecanoe.1 depend on version.hpp turned the docs job into a
hard CI failure on any release commit that bumps the version without
regenerating -- #406, which is open and moves version.hpp to v2.81.0
without touching the man page, would have tripped it as soon as either
merged. The only thing the dependency bought was the version in the page
footer, so every release would have had to regenerate the whole file to
rewrite that one line, gated by CI. Drop it: the source field is now just
"tippecanoe", and the page depends on README.md alone.
Separately, README.md's own title heading became the second .SH, directly
below the NAME section this branch adds, so the page opened with a stray
"tippecanoe" section. Rename it to DESCRIPTION, which is where that text
belongs and what a reader expects after NAME.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015B6PcMbNY779iaczo6S6Pu
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Generate the usage message of each tool from its long_options
The usage messages of tile-join, tippecanoe-overzoom,
tippecanoe-json-tool, tippecanoe-decode, and tippecanoe-enumerate were
hand-written lists of options that had drifted years out of date, since
nothing tied them to the options that are really accepted. Move the
option-list printing that tippecanoe already does into a shared
print_usage(), and use it in all the tools, so that the message is
derived from the same long_options table that getopt_long() gets and
can't fall behind it again.
The tables now carry section headings, as tippecanoe's does, and the
options that were only reachable by their short names (tile-join's -O,
-b, -R, and -r among them) are listed for the first time.
Also state the non-option arguments the way each tool really treats
them: tile-join takes source tilesets unless --read-from names a file to
read them from, tippecanoe-decode takes a tileset either alone or with a
zoom/x/y, tippecanoe-json-tool reads standard input when no files are
named, and tippecanoe-overzoom's two forms are the ones its argument
parsing recognizes. tippecanoe-overzoom now reports the missing -o
instead of passing NULL to fopen(), and tippecanoe-enumerate goes
through getopt_long() so that it will pick up any options added later.
The shared getopt_string() replaces the identical loop that four of the
tools each had for building the short option string, and strip_usage_headings()
the one for dropping the headings before getopt_long() sees them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016frkRY1xXtiWjxYuCJ8vZY
* Print the usage message when tippecanoe is run with no arguments
Running `tippecanoe` with nothing at all reported the missing output
file, which is true but is not what someone who typed the bare command
needs to know. Check for the empty command line before parsing and print
the general usage message instead, and leave the specific complaint for
the case where an input file was named but an output file wasn't.
To make the message reachable from there, the options table and the
usage printing move out of main() into a usage() function, as in the
other tools.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016frkRY1xXtiWjxYuCJ8vZY
* Address review: alternation, the dead tile-join option, and --version
Four fixes from review of the generated usage messages:
* `--output` and `--output-to-directory` are one-of, not one required and
one optional, in both tippecanoe and tile-join. A `usage_required_option`
can now name an alternation that it belongs to, and the options in one
are listed together as `(--output=... | --output-to-directory=...)`,
which is what the runtime check enforces.
* tile-join's `--use-attribute-for-id` has had no implementation since
533e000 removed it; only the table entry was left behind, so the option
parsed and then exited with "Unrecognized option". Generating the usage
message from the table turned that into a documented option that doesn't
work, so remove the leftover entry too.
* `--version` was grouped under "Progress indicator", in the options table
and in the README both. Give it a heading of its own now that the
headings are something users see.
* print_usage() left `width` holding the length of the last synopsis line,
and only got away with it because every table so far begins with a
heading, which resets it. Start the option list on a line of its own
instead of depending on that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016frkRY1xXtiWjxYuCJ8vZY
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Sketching out sqlite options for tile-join
* Enable sqlite3 serialized multithreading
* Fix some unnecessary round trips from std::string to char * and back
* Gathering join keys for sql query
* Make a query
* Actually open the gpkg. Fix the query quoting.
* Actually do the query and get results back
* Join the attributes onto the feature
* Set matched if the sql join matches
* Add a flag to get the feature ID from the query
* Observe attribute exclusion when joining from sql queries
* Add a flag to exclude all attributes from the tile side of the join
* Make tile-join bounding boxes reflect feature bounds, not tile bounds
* More tests
* An empty tileset has empty bounds at null island
* Gather the results from each thread *after* the thread finishes
* Missed a test
* Fix accidental inclusion of the top left of the tile in the bbox
* Case smashing and prefix trimming in the select
* Add test of sql join
* Make the join column option a join expression option
* Fix antimeridian adjustment. Z0 can't wait until the end of the tile
* Checkpoint on accepting multiple joined rows per tiled feature
* Adding the joined attribute should be per-feature, not per-attribute
* Forgot to update the test. Order of joined attributes has changed.
* Get the attributes back in the right order
* Add test of sql join with limit
* Allow multiple tile features to have the same join key
* Update tests for a country name with two distinct geometries
* Update version and changelog
* Forgot to mention the bounding box improvements
* Make tippecanoe-overzoom accept filters from a file
* Accept clip polygons from a file too
* Add test of clipping by polygon from file
* Add a test of reading an overzoom filter from a file
* Update version and changelog
* Plumb a clip bounding box around through overzoom
* Actually do some clipping
* Add a test
* Fix post-binning clipping
* Factoring out geometry parsing from feature parsing
* Accept a clip polygon argument to tippecanoe-overzoom
* Progress in the direction of polygon clipping
* Fix the wagyu flags. We need intersection, not union
* Remove debug spew
* Clip points to polygon bounds too
* Copy the geometric binning code to serve as intersection-finding code
* Add clipper2 for linestring clipping
* Compiles, but does not actually seem to clip. Hmm.
* Oh, it helps if I actually call the function
* Add clipping tests
* Add missing fixture, and don't crash if it is missing
* Remember to do polygon clipping after binning too
* Fix scaling before post-binning clipping. Add test.
* Remove unused parts of clipper
* Rename for consistency
* Revert accidentally added line
* Clip the clip regions to the tile bounds to reduce their complexity
* Add a test of clipping the clip region down to the tile boundary
* Update version and changelog
* Strip out unwanted attributes earlier in the process
* Skip aggregations whose attributes have been excluded
* Forgot one
* Add some more tests
* Update version and changelog
* Add a tippecanoe-decode option to restrict which attributes to decode
* Plumb buffer and feature limit around
* Check the feature limit
* Clarifying cases where output detail can be unspecified
* Clip bins to the tile buffer instead of just passing them through
* Add missing include
* Missed some tests
* Add --no-tile-compression option to tippecanoe-overzoom
* Update version and changelog
* Choose the megatile features from those that will be in the next N zooms
* Take fractional zooms into account in multiplier feature choices
* Fix more tests
* Add a flag to retain multiplier features by minimum distance
* Limit feature expansion from multiplier density to 2x
* The multiplier cap was a bad idea
* Revert "The multiplier cap was a bad idea"
This reverts commit 6f8273a4c8.
* Revert "Limit feature expansion from multiplier density to 2x"
This reverts commit a26e41309d.
* Revert "Add a flag to retain multiplier features by minimum distance"
This reverts commit 01f14a4255.
* Remove the multiplier sequence, which should no longer matter
* Revert "Revert "Add a flag to retain multiplier features by minimum distance""
This reverts commit 776da4a1b8.
* Revert "Revert "Limit feature expansion from multiplier density to 2x""
This reverts commit 44a683d808.
* Revert "Revert "The multiplier cap was a bad idea""
This reverts commit 80f7cb1c0e.
* Track two kinds of previous index for next_feature
* Fix multiplier density threshold, I think
* Oh, I didn't git add the code changes
* Update version and changelog
* Try to install sqlite3 to fix the automated build
* Deleted too much
* Only let --preserve-point-density-threshold shift density around
* Remove the density debt concept, since it doesn't help
* Make the drop states a vector instead of an array
* Revert "Make the drop states a vector instead of an array"
This reverts commit 66c7abb6fa.
* Revert "Remove the density debt concept, since it doesn't help"
This reverts commit 707bb0c562.
* Revert "Only let --preserve-point-density-threshold shift density around"
This reverts commit ecf01f2231.
* Plumb bounding boxes through potential intersections
* Quick bbox reject for bins that can't possibly intersect
* Inching toward attribute accumulation in megatile handling
* Some sort of test for how all these things interact with each other.
Automatic numeric attribute accumulation does *not* apply to attributes
that have an explicit attribute accumulator set, because the order of
operations is too messy and weird
* More sketching
* More sketching
* Actually do some accumulation
* Put all that behind an --accumulate-numeric flag
* Use the same attribute accumulation logic in binning as in megatiles
* Fix backwards conditional
* Add means, but somehow I have some counts of 0
* Handle aggregated attributes with no base attribute in the feature
* Checkpoint before I break everything
* Found a flaw, now to debug
* Fix a typo that broke accumulation
* Add binning tests
* Make sure IDs make it through on the bins
* Fix count/mean accumulation
* Make the numeric accumulation prefix configurable
* Make sure the accumulate test still works with a different prefix
* Forgot to update this test
* More testing to make sure cluster sizes make it all the way through
* Fix neglected --accumulate-attribute when binning
* Mark unexercised attribute accumulation cases as "can't happen"
* Factor out numeric preservation
* Attrs with the accumulation prefix are just preserved, not accumulated
* Test behavior of prefixed attributes
* Plumbing for exclude and exclude-prefix
* Implement and test attribute prefix stripping in overzoom
* Update version and changelog
* For debugging, make an attribute list of source feature IDs
* Revert "For debugging, make an attribute list of source feature IDs"
This reverts commit 65fc99c9d1.
* Another try at fixing longitude wraparound for bins
* Gonna get it right this time
* Forgot to update the comment
* The filter case was not supposed to reinterpret geometry
* Copy antimeridian-crossing geometries to the other side too
* Getting closer to getting antimeridian-crossing polygons right
* Update changelog and version
* Factoring out tilestats management from GeoJSON file reading
* Move code around so overzoom can link against parse_layers
* Read the file of bins
* Plumb the bins through to overzoom()
* Some zip code bins to test with
* (Currently non-functional) test of binning
* Starting to spell out the bin matching loop
* Can't flatten points, so don't flatten bins either
* More fleshing out bin traversal
* Bounding box of tile-relative mvt geometry
* Smallest enclosing tile from bbox
* Most of the bin scan
* Add point in polygon check. It crashes.
* Find the matching bins
* GDAL-style bounding boxes have eaten my brain
* Make some features to bin into
* Increment a count as features are found to be within the bins
* Fix longitude wraparound in overzoom bins
* Fix the tests
* Push off attribute copying until after bin assignment
* Carry sum of numeric attributes into the bins
* Also add mean, min, and max
* Add --calculate-feature-index since I keep needing it for testing
* Add an option to accumulate sum/mean/max/min/count of all numeric attrs
* Don't bake in tippecanoe:mean, since we redo it from sum and count
* Forgot to update this test fixture after removing tiled mean
* Update version and changelog
* Clip before dealing with multiplier or filters in overzoom
* Be more careful to retry when the feature count is exceeded
* Adjust the estimated total feature count for the multiplier too
* Fix the feature count estimates, I think
* Pass build info into the version string
* Report the actual max zoom of any tiles as the metadata maxzoom
* Revert unneeded renaming to make the diff more readable
* Clean up the adjustments to tile sizes and feature counts
* Update version and changelog
* Dropping a feature into a multiplier cluster still effectively drops it
* Update changelog
* Rethink the changelog description
* Don't try to truncate zooms if we are still tiling at z18
* Track output position at the file level instead of within each tile
* Track file position where the child tile data begins
* Add option and document its intended behavior
* Changing the detail loop to account for stopping early
* I forgot I already added an option for this
* Stop early if we can make a complete tile
* Add a test of zoom truncation with limited feature count
* Forgot to commit the actual code change
* Make room for a vertex count in the header of each serialized tile
* Estimate tile complexity; don't try truncating when unlikely to work
* Be more conservative, because ever retrying a tile is a big speed hit
* If stopping early, don't simplify or clean; leave that to overzoom
* Add tiny polygon reduction / dust to overzoom
* Don't try to stop early in the children if we dropped anything by rate
* Fflush here too before pwriting
* Don't stop early if we ended up dropping any features.
Rework the can-the-next-zoom-stop-early logic to avoid going
one zoom further than needed.
* Fix warning
* Fix warnings
* Oops, checking for the wrong expected return value
* Cleanup from adding line simplification in overzoom
* Current (wrong) behavior when combining coalescing and truncating
* Keep a list of parent tiles to skip rather than truncating
* Now the coalesced tiles in z12 get children in z13
* Don't double-count feature dropping when the zoom level is retried
* Correct README description
* Remove todo about special case below basezoom, which is accounted for
* Be a little more aggressive in drop-densest determination
* Scale tile feature limit for megatiles in the same way as byte limit
* Fully deprecate -detect-shared-borders into an alias
* Track the distances found in the douglas-peucker recursion
* Serialize and deserialize the distance with the vertices
* Revert "Serialize and deserialize the distance with the vertices"
This reverts commit 753f1b7909.
* Revert "Track the distances found in the douglas-peucker recursion"
This reverts commit e5361f8c22.
* Revert "Fully deprecate -detect-shared-borders into an alias"
This reverts commit 0698aeb766.
* Better tracking of whether we failed to make a full-detail tile
* Put a bloom filter in front of the binary search for shared nodes
* Forgot to take out this printf
* Improve dispatch of tiling tasks
* Still dispatch the biggest tasks first
* Track zoom truncation in the strategies list in the tileset metadata
* Prescan for small deltas before doing proper simplification
* Revert "Prescan for small deltas before doing proper simplification"
This reverts commit d1d8238b83.
* Update version and changelog
* Rename to --generate-variable-depth-tile-pyramid
* Reviving multi-source-tile overzoom: the clip.cpp side
* Reviving multi-source-tile overzoom: the overzoom.cpp side
* Update readme
* Update version and changelog
* Stabilize feature order in overzoom
* I want my sorts to be stable, please
* Revert "[ci] test in debug mode (#202)"
This reverts commit 853ada87b5.
* No need to reinitialize here
* Add a way to run tippecanoe single-threaded for profiling
* Do less work when the tilestats sample values list is already full
* Save a copy when retrieving the attribute key
* Fewer atomic operations
* Move string hashing from mbtiles to text
* Only do approximate attribute deduplication when writing tiles
* Feature dropping tests are sensitive to exact tile size
* All tile creators now create a string pool for the tile
* Features clipped away to nothing should not participate in that tile
* Revert "Only do approximate attribute deduplication when writing tiles"
This reverts commit c42b34b498.
* Also revert the related test changes
* Revert "Revert "Only do approximate attribute deduplication when writing tiles""
This reverts commit 18509876c3.
* Be more specific about the string hash function
* Use fnv1a instead of std::hash for everything
* Reduce the chance of hash collisions
* Stick a hash search on the front of the tree search in addpool
* Eliminate repeated hashing of the same string
* Switch instead of ifs in json parsing
* A few more cases to populate the hash in addpool
* Store the hash in the tree instead of recalculating
* Add explanatory comment for mysterious argument
* Fewer copies in attribute stringification
* Clean up ancient weirdness in JSON attribute stringification
* More serial_val cleanup
* Pass a serial_feature to rewrite instead of many broken-down arguments
* Get rid of the multiple geometries within `partial`
* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"
This reverts commit 6f4ab9b725.
* Goodbye, struct coalesce
* Revert "Features clipped away to nothing should not participate in that tile"
This reverts commit 124462fbdc.
* Migrating fields from partial to serial_feature
* Name reconciliation between serial_feature and partial
* Replace struct partial with an augmented serial_feature
* Fix some overzealous search-and-replace renaming
* Don't say struct so often
* Remove more of the former partial construction
* Commenting and cleaning up
* Trying again to avoid all these arguments to rewrite
* I swear I did this same thing before and it didn't work.
* More rewrite cleanup
* Exile --detect-shared-borders to its own file
* Add missing headers
* More commenting and cleanup
* More comments
* Sprinkle consts around
* Emplacing and std::moving
* More cleanup
* That shouldn't have worked after a std::move
* Don't need to allocate memory to compare keys
* Reduce use of the global string pool in tiling
* Another avoidable mvt_value construction
* Further reduction to explicit string pool passing
* These reverses are no longer optimizations
* These layernames can all be references
* Don't drag an unused layername string around with every feature
* Heed a compiler warning about potential buffer overflow
* Fix my confusion about which feature's string pool is relevant
* Avoid some unnecessary allocations in attribute accumulation
* Maybe faster serialization?
* Eliminate a comparison
* Do the same here
* Save a couple of allocations when parsing numbers in JSON
* Immediately assign features to layers instead of subdividing later
* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence
* Crunch out more duplicate attribute values when writing out the tile
* Do tilestats for tippecanoe:retain_points_multiplier_first too
* Shell filters need to be real threads, even if nothing else does
* Simplify tippecanoe_minzoom/maxzoom representation
* Update version and changelog
* Starting to factor out attribute accumulation into its own file
* Continuing to factor out attribute accumulation
* Reduce duplicate code
* Plumbing the accumulate-attribute option around
* Call the attribute accumulator
* Test that accumulation works
* Add missing #includes
* Don't sort within individual multiplier clusters
Doing so throws off the spatial distribution of the low zooms
* Docs and changelog
* Add comments
* Make feature ordering cooperate with --retain-points-multiplier
* Forgot to check in the actual code changes???
* Sort within each multiplier cluster as well as between clusters
* Correct description of behavior in changelog
* Drag original feature sequence along in megatiles for post-filter sort
* Plumb the preserve-input-order flag through overzoom
* Sort in overzoom if requested
* Use within-tile input sequence numbers, not global sequence numbers
* Documentation
* Reverse direction of search to prevent accidental skipping
* Add some comments about converting between attribute representations
* Add an option to retain N times as many points as usual at each zoom
* Tests for point multipler with specified and guessed maxzooms
* Work in progress on inverse spatial ordering
* Fix inverse spatial feature order
* --reorder was depending on a feature index that wasn't being preserved
* Separate ordering by feature_minzoom from ordering inverse-spatially
* Add a test for the inverse spatial ordering
* Store the basezoom/droprate/multiplier decisions in tileset metadata
* Progress on adding filters to tippecanoe-overzoom
* Type promotion for comparison
* Look up the attribute value for ordering
* Add test of thinning and ordering features
* Plumb tippecanoe_decisions metadata through pmtiles
* Be careful not to put infinities in JSON
* Fix accidental dropping in what is meant to preserve sparse points
* Start distinguishing true, false, and null in expressions
* Most of the type conversions
* Add boolean conversions
* Literals and conjunctions
* Add filtering to tippecanoe-overzoom
* Add a test of filtering in overzoom
* Fix boolean conjunctions
* Handle the combination of cluster size and filtering
* Rework dot dropping to reconcile density threshold and multiplier
* Revert "Rework dot dropping to reconcile density threshold and multiplier"
This reverts commit f253a66382.
* Retain points by multiplier within each tile, not in global probability
* Test that intends to verify that the multiplier is reversible
* Get the test to detect the discrepancy
* Mark the start of multiplier clusters with a magic attribute
* Add string-contains
* Add in and ni operators
* Revert "Look up the attribute value for ordering"
This reverts commit 56bc73e49a.
* Revert "Type promotion for comparison"
This reverts commit 6f3256f5af.
* Make number formatting in tippecanoe_decisions consistent
* Revert "Add a test for the inverse spatial ordering"
This reverts commit c8047de9ab.
* Revert "Separate ordering by feature_minzoom from ordering inverse-spatially"
This reverts commit 35b19a223c.
* Revert "Fix inverse spatial feature order"
This reverts commit 5978ecdb44.
* Revert "Work in progress on inverse spatial ordering"
This reverts commit fdf230f632.
* Somehow missed the tests associated with that last revert
* Round-robin assign attributes to partials from across the multiplier
* Count the multiplier separately in each layer
* Fix distribution of accumulated attribute across multiplier features
* Update changelog, version, and docs
* Add "is null" and "isnt null" expressions
* Update interpretation of FSL expressions to pass the tests
* Test to assert that polygons are unaffected by the multiplier
* Clean up and comment
* Remove accidental unused case
* Clip away entire features by bbox. Avoid unnecessary recompression.
* Move parent tile decoding in tile-join out of overzoom proper
* An ever-growing cache of parent tiles
* Limit the size of the cache
* Remove the current reader *before* checking if we can run the queue
* Clean up
* Add missing #include
* Add comment
* When the tile-join cache fills up, evict the least recently used
* Fix microsecond math
* Factoring out tile-join's cache for testing
* Add unit tests for tile-join cache
* Update changelog and version
* Start of externalizing polygon shard detection
* Completely untested external quicksort
* Add unit test for external quicksort
* Remember to clean up temporary files
* Sort and scan the vertices
* Bring over more vertex logic
* Make nodes from vertices
* Checkpoint on switching over to global shared nodes
* Do the thing
* Revert unintended change to coalesced linestring behavior
* Take shared nodes into account in early simplification
* Let it do more sorting in memory
* Fix overnoding of collinear linestrings
* Remove duplicate nodes, since only duplicate vertices now matter
* Remember to delete temporary files
* Fix out of bounds memory access below, apparently
* Fix the actual undefined behavior
* Still running out of memory in one case. Find out where.
* Forgot the conditional
* Try again to make it not run out of memory
* Update version and changelog
When running "make test" in a pristine working copy, the "test" target
calls the "raw-tiles-test" target, which tries to execute ./tile-join.
But that binary has not been built, yet, at this point. The target fails
with an error.
Add the required binaries as dependencies to the test targets in the
Makefile. This makes sure that they are built before they are attempted
to be executed.
* Put all of this back in geometry.cpp for conflict resolution
* Stabilize line simplification to behave the same regardless of winding
* Round instead of truncating when clipping lines
* Restore non-Wagyu polygon clipping from prior to 2fdec7d2
* Make it round, not truncate, which reverts the last commit's test diffs
* Clip in floating point, not integers, which makes no difference
* Track nodes added at tile edges during clipping
* Scale geometry up before wagyu to prevent changes from precision loss
* Actually do the shared edge detection
* Fix cases where nodes were not being added at the tile boundary
* One more place I should have rounded
* Narrow down where the discrepancy comes in
* Revert "Narrow down where the discrepancy comes in"
This reverts commit 221c4c5fc0ac9a6567e091c6a94b3d87dc8ade83.
* Another attempt to narrow it down
* The discrepancy seems to be introduced in reordering. No obvious bug
* Was still truncating instead of rounding in projection
* Also makes no difference...
* Just forget that line reversal exists for a minute
* Just reversal no coalescing
* Try clipping in integers instead of floating point
* Don't simplify after coalescing if they said no simplification
* Check whether behavior is consistent with intentional simplification
* Replace more floating point with integer
* Are these three features enough to demonstrate the problem?
* Add a few more nearby borders
* All the features that touch tile 6/16/23
* Stay in integers in line simplification
* More attempts to solve failures to simplify consistently
* Fix most of the overflow errors
* Fix known cases of integer overflow
* All the tests change again
* Pull clipping and scaling code back out into clip.cpp
* Resolve the test conflicts
* Stabilize choice of which three points to keep with different windings
* Almost right, I think!
* Fix collapse of islands to shards
* Self-intersections in the same feature don't count
* Revert "Self-intersections in the same feature don't count"
This reverts commit e04b19916e.
* Don't scale down geometry if we are going to look for shared nodes
* Fix the missing multiply that was keeping simplification from happening
* Fix one more opportunity for overflow
* Somehow I deleted this test?
* Lost this test too
* Clean up debugging printfs
* Restore code sequence from main to make it reviewable
* Remove unneeded rounding
* Update documentation
* This test is no longer useful
* Back to floating point Douglas-Peucker to fix undersimplification
* Try an older ubuntu
* Revert "Try an older ubuntu"
This reverts commit 13fefacfd7.
* Log OS info
* Remove tests that are no longer needed
* Oops, did need that one after all
* Fix the arm vs x86 discrepancy?
* Try another quantization
* Cleanup from review
* Add a test for the actual purpose of this PR
* Fix tile-join crash when trying to merge empty tilesets with --overzoom
* Add an option not to reduce tiny polygons to dust at maxzoom
* Add test for prevention of tiny polygon reduction at maxzoom
* Change version number
* Add an option to extend zooms if still dropping, but with a limit
* At least when to overzoom, even if not actually doing it yet
* Refactor to give tile-join access to overzoom()
* Didn't work, but *might* have worked
* OK, it did something now
* Ah, there's the bug!
* Hook up pmtiles and dirtiles as overzooming sources
* Add command line option to enable or disable overzooming
* Add (currently broken) test of overzooming in tile-join
* Slightly more abstraction for the tile-join readers
* Factor out duplicated code
* Move construction into a constructor
* More changing accessors to methods
* Reduce magic
* Start tracking a list of the tiles at maxzoom
* I think it worked?
* Add missing #include
* Fix sequence of overzoomed tiles (Y sorts backwards for TMS)
* Don't spend memory on overzooming when we aren't going to use it
* Diff rather than cmp, in the hope of figuring out this broken test
* Keep full coordinate precision if we might extend zooms
* Try a slightly different byte limit
* Make drop-densest more consistent across tile boundaries
* Also affects this test
* Does it behave any differently if it can extend forever?
* I think the discrepancy is a thread-safety problem here
* Revert "Does it behave any differently if it can extend forever?"
This reverts commit 0dff0a0acc.
* Lost this change to the test
* This time for sure!
* Revert "Also affects this test"
This reverts commit cd1f7c2e78.
* Revert "Make drop-densest more consistent across tile boundaries"
This reverts commit 563f7d2bc2.
* Revert "Try a slightly different byte limit"
This reverts commit 2e271213d6.
* Add some more explanatory comments
* Amend the join-test to detect my current bug
* Allow overzooming to complete the zoom if it ever starts
* Forgot to correct the test
* Update changelog and version
* Cleanups from code review
* Remove version number from fixture to fix test
* Starting work on overzooming
* Factor clipping out of geometry.cpp to simplify linkage
* Pull out more geometry functions into now-badly-named clip.cpp
* Not surprisingly, there is a bug
* Found the bug
* Make indent
* Pass attributes through
* Make formatting more consistent
* Fix typos in comments
* Fix the typos better
* Add tippecanoe-overzoom to the install list
* Give overzoom a predictable exit status
* Forgot to translate to and from polygon ring closepaths
* Fix geometry collapse at z21
* Add attribute stripping; don't generate layers if they have no features
* Add docs for tippecanoe-overzoom (as it will be, not as it is)
* Change overzoom to accept input and output files as arguments
* Working on overzooming tests
* Oops
* Hook up and test the detail and buffer options, and the empty-tile case
* Also test attribute stripping
* Fix error message
* Update changelog and version
* Starting on --set-attributes
* Add a JSON form to accumulate-attribute
* Fix formatting
* Set attributes, JSON form
* Also handle setting attributes to unquoted values
* Factoring out character-mangling in the names of tests
* Rename tests for new quoting convention
* Test for JSON form of attribute setting
* Test for JSON form of attribute accumulation
* Forgot to increment the version
* Added -fPIE as g++ 9.4.0 complained and would not link
* Prevent crash by checking for properties before dereferencing.
---------
Co-authored-by: Tim Finer <finert@pixia.com>
* Allow passing a file with a list of input files to tile-join in response to Issue 69 (https://github.com/felt/tippecanoe/issues/69)
* Do not print usage if -r is specified without a commandline readfrom file
* Add test for -r option to tile-join
* Remove the concept of "separate metadata"
This was an extra level of attribute indirection (features point
to metadata records which point to key and value strings) which was
intended to reduce the size of temporary storage for features with
large numbers of attributes that were also spread across large numbers
of tiles at maxzoom.
For other kinds of features, the extra indirection slowed things down
instead, and, especially when maxzoom guessing was being used, many more
features were having their metadata externalized than could actually
benefit from it.
* Shave a few bytes off temporary files by using more unsigned integers
* Flush stderr after logging progress
* Revert "Shave a few bytes off temporary files by using more unsigned integers"
This reverts commit eef29084ec.
* Limit the size of the string pools and trees to fit in memory
* Add missing #include
* Move the string pool and search tree from mmap to allocated memory
* Sort in allocated rather than mapped memory too
* Also use pread instead of mapping to read in the data to sort
* When the pool gets too big, switch to just the file, not memory
* Switch string pool from memory to disk when memory is 10% full
* Add to-memory versions of the serialization functions
* Crashy work in progress toward compression
* Fix the pointer bug that was causing the crash
* Serialize features into memory rather than straight to disk
* Compress individual features in the temporary files
* Don't need to store the length of the geometry
* Remove per-feature compression; move minzoom back into the object
* Start adding a stream compressor object
* Track file position within fwrite_check()
* Add compressed stream writer functions
* Pull the writing of the serialized feature out to the callers
* Starting toward compression again from a different point
* Hook up more compression functions
* Remove unused code from the other day
* Make enough deflate calls to flush out all the buffered data
* Start on decompression
* Tile number is uncompressed, tile content is compressed
* Work on alternating compressed and uncompressed in decompression
* Closer, but still doesn't work
* Sort of works
* Works until we get to concatenated tiles
* More attempts that don't work
* One bug down
* It made a tileset!
* Handle nonzero initial zooms
* Fix seeking within compressed feature streams
* Tests pass!
* Remove debug spew
* Oops: remember to delete the temporary files so they don't hang around
* Test that fails with the current compression code
* Properly account for bytes read while closing the compressed stream
* Limit the number of warnings about bad label points
* A little more armor when closing decompression
* This time for sure
* A different, less fragile, test that failed previously with compression
* Move feature stream compression to its own file
* Remove now-unused code to deserialize from a file
* Forgot to add the new files
* Remove a little debugging logging
* Add a couple of comments on what it means to be within decompression
* Fix indentation
* Update changelog. Remove stray debugging comment.
* Round coordinates instead of truncating them
* Update all the tests for coordinate rounding changes
* Curses, integer division still truncates
* Fix tests
* Also round instead of shifting when looking for no-op linetos
* Should I worry that the same change for moveto doesn't change any tests?
* Also round instead of shifting when scaling down to maxzoom resolution
* Replace another explicit shift, for origin point
* Round instead of shift when writing clipped geometries to the next zoom
* Fix low-zoom gridding and smaller-than-a-pixel checks
* Don't guess an excessively large maxzoom when there is only one feature
* Add a test for guessing the maxzoom of a single point
* Explicitly sort by index if no other order distinguishes features
* Another affected test
* add pmtiles.hpp from github.com/protomaps/PMTiles [#10]
* tippecanoe main writes pmtiles output. [#10]
* detect output format using suffix
* after mbtiles is done writing, replace with pmtiles based on map/image tables.
* add method to write_json for writing json sub-object.
* tippecanoe-decode reads pmtiles input. [#10]
* tile-join reads and writes pmtiles. [#10]
* pmtiles test suite for decode and tile-join [#10]
* add base GitHub CI action for compiling and test suite.
* update pmtiles.hpp with z>15 fix
* Fix some ordering problems with pmtiles decode
* Pmtiles should also pass the raw tiles tests
* Eradicate spaces from tileset metadata JSON fields
* Eradicate spaces from more test fixtures
* Update more tests
* Pmtiles tests pass now too
* Remove unnecessary sort (and make indent)
* Update changelog
* The allow-existing test for pmtiles needs -o, not -e
* Declare --allow-existing to be unsupported for pmtiles.
It was always a bad idea even for mbtiles.
Co-authored-by: Brandon Liu <bdon@bdon.org>
Claude