Commit Graph
316 Commits
Author SHA1 Message Date
Claude 691c8f2251 Implement tile-join's --use-attribute-for-id
The option had been listed in tile-join's option table since 533e000 (#361)
removed the code behind it, so it parsed and then exited with "Unrecognized
option." #409 dropped the leftover entry rather than ship a usage message
advertising an option that didn't work. Implement it instead.

The ID handling that #361 removed only ever applied to attributes that came
back from a SQLite join, and that join went away in the same commit, so there
is no longer anywhere in this tool to hang it on. Instead the option takes the
ID from an attribute of either kind that tile-join does have: one already
present in the source tiles, or one joined from a CSV with -c, with a joined
value superseding a tile value of the same name just as it does for ordinary
attributes. The attribute is consumed rather than copied through, and it is
checked before -x, -y, -X, and --exclude-all-tile-attributes, so the ID can
come from an attribute that isn't kept -- which is the useful case, as in
`-x GEOID10 --use-attribute-for-id=GEOID10`.

The conversion from an attribute value to an ID, along with the warnings for
the values that can't be represented as one, moves out of serialize_feature()
into attribute_to_feature_id() in mvt.cpp, which both tools link, so that
tile-join reports the same problems the same way instead of silently
truncating. Its one difference is that tile-join converts stringified numbers
without any additional option, since it has no equivalent of tippecanoe's -aI.

Along the way, an attribute whose value is 0 is now usable as a feature ID.
The check that an ID survives a round trip through its string form compared
std::to_string() against the value with its leading zeros stripped, and
stripping the only digit of "0" left "" to compare against "0", so a zero was
rejected as too large to represent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DRXWaDXVEZV4t1ddR6ZRcw
2026-08-07 19:50:01 +00:00
Erica FischerandClaude Opus 5 734bba7c78 Fix three latent defects exposed by compiler warnings, and clear the rest (#406)
* Fix variable-length-array and uninitialized-union compiler warnings

Clang warns about every variable-length array in C++ (-Wvla-cxx-extension,
on by default), since VLAs are a compiler extension rather than standard
C++. Replace all 57 of them with std::vector, or with std::string for the
mkstemp() template buffers built from tmpdir. Add -Wvla to WARNING_FLAGS so
new ones don't creep back in.

Separately, mvt_value's numeric_value union is 16 bytes wide (the size of
string_value), but both constructors only wrote the 8 bytes of the member
they were setting, leaving the rest indeterminate. The implicit copy
constructor copies the union as a whole, so copying any non-string value
read uninitialized bytes, which GCC reports as

  mvt.hpp:83:8: warning: 'v.mvt_value::numeric_value. ... .len' may be
  used uninitialized [-Wmaybe-uninitialized]

Give string_value, the widest member, a default member initializer so the
union's full width is initialized however it is later used.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR

* Fix remaining float-conversion and format-truncation warnings

Clang's -Wimplicit-const-int-float-conversion flagged two comparisons
against LLONG_MAX, which is not representable as a double and rounds up
to 2^63.

In serial.cpp this was a real latent overflow, not just noise: the guard
`extent <= LLONG_MAX` was really `extent <= 2^63`, so an extent of exactly
2^63 passed it and then hit `(long long) extent`, which is undefined for
that value and yields LLONG_MIN in practice -- the opposite of the clamp
the else branch intends. Make the bound exclusive so the conversion is
always in range. Requires a polygon area at the very top of the double
range to reach, but the clamp now behaves as written.

In mbtiles.cpp the value is only a stand-in for infinity on its way into
JSON, so cast explicitly; the emitted number is unchanged.

Separately, g++ at -O0 warned that `char abbrev[20]` can be truncated by
"%lld", which is correct: the most negative long long needs 21 bytes with
the NUL. That branch is only reached when point_count < 1000, so it cannot
happen today, but size the buffer to fit rather than rely on that, and
replace the garbled comment about how the size was derived.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR

* Clamp the low end of extent before converting to long long too

The upper bound was fixed in the previous commit; the same overflow exists
on the negative side. get_area() returns a signed shoelace area, so inner
rings contribute negatively, and a polygon whose holes outweigh its rings
drives extent below zero. Far enough below and `(long long) extent` is
undefined again.

The bounds are asymmetric, so this is not simply the mirror of the upper
one: LLONG_MIN is exactly -2^63 and converts exactly, so unlike LLONG_MAX
it can be an inclusive bound.

Verified with -fsanitize=float-cast-overflow that the previous form traps
on 2^63 and on doubles just below -2^63, and that this one is clean across
both boundaries, the infinities, and NaN (which falls to LLONG_MAX, as it
did before).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR

* Add CHANGELOG entries for 2.81.0 and bump the version

CHANGELOG.md was last updated for 2.80.0 (#361), and version.hpp has not
moved since. Twelve PRs have landed in the meantime with no entry: #365,
#368, #375, #382, #384, #385, #391, #395, #397, #399, #400, and #401.

Document all of them, plus this PR, under a single 2.81.0 heading. They are
not given separate version numbers because none of them was ever released
under one -- version.hpp read v2.80.0 throughout -- so assigning a version
per PR would invent release history. 2.81.0 is the version that will
actually carry them.

Where an unreleased PR was corrected by a later one (#384 by #385, #397 by
#399), the pair is described as the single behavior that ships, since the
intermediate behavior was never in a release.

Minor rather than patch bump: the batch adds command-line options.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR

* Review feedback: enforce the union-width assumption, describe both clamp ends

The comment on mvt_value's union claimed string_value is the widest member.
That is true on LP64 (16 bytes against 8) but not on ILP32, where size_t is
4 and it ties with double and long long. The default member initializer still
covers the full union either way, so the fix held, but the justification did
not travel. Replace the claim with a static_assert that checks it on whatever
target is being built, so a platform where it stops holding is a compile
error rather than silently indeterminate bytes. Verified the assert is not
vacuous by widening the union in a scratch copy and watching it fail.

The changelog described only the upper end of the extent clamp. Describe both:
the old guard admitted everything below LLONG_MIN too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR

* Add 2.81.0 changelog entries for the four PRs merged from main

#404, #408, #409, and #410 landed while this branch was open. None of them
bumped version.hpp, so they belong under the same 2.81.0 heading as the rest
of the unreleased work rather than getting versions of their own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-06 17:06:07 -07:00
Erica Fischer 533e000faa Remove undocumented command-line options (#361)
* Remove --accumulate-numeric-attributes

* Remove join-sqlite, etc.

* Remove --accumulate-numeric-attributes from overzoom

* Remove --assign-to-bins and --bin-by-id-list

* Remove --clip-polygon and --clip-bounding-box

* Remove FSL expressions

* Update version and changelog
2025-07-31 17:03:01 -07:00
Erica Fischer 68ab8dcc22 Deduplicate in tippecanoe-overzoom even when the duplicate is clipped away (#353)
* Deduplicate by ID even when the duplicate is clipped away

* Test that deduplication works across tile boundaries

* Update version and changelog
2025-07-24 13:21:10 -07:00
Erica Fischer 2d548bed06 Infinite loop fixes, minimizing changes to behavior (#345)
* Divide-and-conquer polygon cleaning

* Catch the case where the gap can't be increased further

* Catch the case where we try to keep impossibly many features

* Make label points earlier in the tiling process

* Another case where it could try to drop even after already limiting.

* And do not coalesce on impossibly small geometries

* Add missing return

* Update version and changelog
2025-05-09 09:08:28 -07:00
Erica Fischer 94929b048c Add --deduplicate-by-id option to tippecanoe-overzoom (#331)
* Add option to deduplicate by feature ID in overzoom

* Add test, fix default

* Update version and changelog
2025-04-03 12:48:29 -07:00
Erica Fischer bfb62ee2db Add missing case for accumulating the mean of attributes that are inconsistently present (#329)
* Add missing case for accumulating the mean of attributes that are inconsistently present

* Add more specific test
2025-03-20 14:53:04 -07:00
Erica Fischer 423fea1544 Reduce memory consumption during tiling and tile encoding (#319)
* Improve the memory spike during tile construction

* Remove `need_tilestats`. Add a bunch of debug logging

* Remove debug logging

* Update version and changelog
2025-01-31 12:57:41 -08:00
Erica Fischer 583fc3744a Reduce attribute accumulation memory consumption (#318)
* Add a flag to use an H3 index for the feature index

* Give mvt_value and serial_val a double-with-count concept

* Switch mean over to internal accumulation state

* Get rid of the attribute accumulation map

* Change vectors of features to vectors of pointers to features

* Fix --coalesce

* Revert "Add a flag to use an H3 index for the feature index"

This reverts commit b9b48f42c9.

* Update version and changelog
2025-01-30 21:58:35 -08:00
Erica Fischer 10f7f0a3c2 Support joins from sqlite tables in tile-join (#308)
* Sketching out sqlite options for tile-join

* Enable sqlite3 serialized multithreading

* Fix some unnecessary round trips from std::string to char * and back

* Gathering join keys for sql query

* Make a query

* Actually open the gpkg. Fix the query quoting.

* Actually do the query and get results back

* Join the attributes onto the feature

* Set matched if the sql join matches

* Add a flag to get the feature ID from the query

* Observe attribute exclusion when joining from sql queries

* Add a flag to exclude all attributes from the tile side of the join

* Make tile-join bounding boxes reflect feature bounds, not tile bounds

* More tests

* An empty tileset has empty bounds at null island

* Gather the results from each thread *after* the thread finishes

* Missed a test

* Fix accidental inclusion of the top left of the tile in the bbox

* Case smashing and prefix trimming in the select

* Add test of sql join

* Make the join column option a join expression option

* Fix antimeridian adjustment. Z0 can't wait until the end of the tile

* Checkpoint on accepting multiple joined rows per tiled feature

* Adding the joined attribute should be per-feature, not per-attribute

* Forgot to update the test. Order of joined attributes has changed.

* Get the attributes back in the right order

* Add test of sql join with limit

* Allow multiple tile features to have the same join key

* Update tests for a country name with two distinct geometries

* Update version and changelog

* Forgot to mention the bounding box improvements
2025-01-13 12:25:00 -08:00
Erica Fischer 7165ae6999 Fix clipping bug when the clip region doesn't intersect the tile (#312) 2024-12-11 10:24:12 -08:00
Erica Fischer bdfb06cd6a Make tippecanoe-overzoom accept filters from a file (#307)
* Make tippecanoe-overzoom accept filters from a file

* Accept clip polygons from a file too

* Add test of clipping by polygon from file

* Add a test of reading an overzoom filter from a file

* Update version and changelog
2024-12-05 12:05:50 -08:00
Erica Fischer dcc616d3d4 Adding optional clipping to tippecanoe-overzoom (#298)
* Plumb a clip bounding box around through overzoom

* Actually do some clipping

* Add a test

* Fix post-binning clipping

* Factoring out geometry parsing from feature parsing

* Accept a clip polygon argument to tippecanoe-overzoom

* Progress in the direction of polygon clipping

* Fix the wagyu flags. We need intersection, not union

* Remove debug spew

* Clip points to polygon bounds too

* Copy the geometric binning code to serve as intersection-finding code

* Add clipper2 for linestring clipping

* Compiles, but does not actually seem to clip. Hmm.

* Oh, it helps if I actually call the function

* Add clipping tests

* Add missing fixture, and don't crash if it is missing

* Remember to do polygon clipping after binning too

* Fix scaling before post-binning clipping. Add test.

* Remove unused parts of clipper

* Rename for consistency

* Revert accidentally added line

* Clip the clip regions to the tile bounds to reduce their complexity

* Add a test of clipping the clip region down to the tile boundary

* Update version and changelog
2024-11-27 13:43:40 -08:00
Erica Fischer 11e3196c9a Raise tippecanoe-decode tile size limit to 250 MB (#299)
* Raise tippecanoe-decode tile size limit to 250 MB

* Update version and changelog
2024-11-21 09:17:22 -08:00
Erica Fischer 905b58cdf4 Reducing attribute tagging within tippecanoe-overzoom (#296)
* Strip out unwanted attributes earlier in the process

* Skip aggregations whose attributes have been excluded

* Forgot one

* Add some more tests

* Update version and changelog
2024-11-15 13:14:30 -08:00
Erica Fischer 6d46578ad8 Avoid crash if the first bin gets clipped away (#294)
* Avoid crash if the first bin gets clipped away

* Add test

* Update version and changelog
2024-11-13 10:48:33 -08:00
Erica Fischer 794ae2b1eb Tippecanoe-decode and tippecanoe-overzoom changes to improve binning performance (#292)
* Add a tippecanoe-decode option to restrict which attributes to decode

* Plumb buffer and feature limit around

* Check the feature limit

* Clarifying cases where output detail can be unspecified

* Clip bins to the tile buffer instead of just passing them through

* Add missing include

* Missed some tests

* Add --no-tile-compression option to tippecanoe-overzoom

* Update version and changelog
2024-11-08 15:46:04 -08:00
Erica Fischer 23667bb8eb Reduce memory consumption from attribute accumulation (#290)
* Progress on plumbing a string pool for full_keys through

* More plumbing for key_pool

* Don't keep features with identical locations as multiplier features

* Revert "Don't keep features with identical locations as multiplier features"

This reverts commit 413f0c8024.

* Adjust calculated maxzoom to account for duplicate feature locations

* Update changelog and version

* Add a test affected by the maxzoom change with duplicate locations

* Round the drop rate a little for cross-platform test consistency
2024-11-05 14:39:41 -08:00
Erica Fischer b3b89e1e07 Performance optimizations for binning (#283)
* Add an all-mvt_value attribute accumulation path

* Only bin by ID, not geometrically

* A little cleanup; changelog and version; test

* Remove accidental double-conversion

* Replace duplicated code with template

* Update changelog
2024-11-01 15:27:13 -07:00
Erica Fischer 28efc40e6e Choose the megatile features from those that will be in the next N zooms (#280)
* Choose the megatile features from those that will be in the next N zooms

* Take fractional zooms into account in multiplier feature choices

* Fix more tests

* Add a flag to retain multiplier features by minimum distance

* Limit feature expansion from multiplier density to 2x

* The multiplier cap was a bad idea

* Revert "The multiplier cap was a bad idea"

This reverts commit 6f8273a4c8.

* Revert "Limit feature expansion from multiplier density to 2x"

This reverts commit a26e41309d.

* Revert "Add a flag to retain multiplier features by minimum distance"

This reverts commit 01f14a4255.

* Remove the multiplier sequence, which should no longer matter

* Revert "Revert "Add a flag to retain multiplier features by minimum distance""

This reverts commit 776da4a1b8.

* Revert "Revert "Limit feature expansion from multiplier density to 2x""

This reverts commit 44a683d808.

* Revert "Revert "The multiplier cap was a bad idea""

This reverts commit 80f7cb1c0e.

* Track two kinds of previous index for next_feature

* Fix multiplier density threshold, I think

* Oh, I didn't git add the code changes

* Update version and changelog

* Try to install sqlite3 to fix the automated build

* Deleted too much

* Only let --preserve-point-density-threshold shift density around

* Remove the density debt concept, since it doesn't help

* Make the drop states a vector instead of an array

* Revert "Make the drop states a vector instead of an array"

This reverts commit 66c7abb6fa.

* Revert "Remove the density debt concept, since it doesn't help"

This reverts commit 707bb0c562.

* Revert "Only let --preserve-point-density-threshold shift density around"

This reverts commit ecf01f2231.
2024-10-17 15:59:44 -07:00
Erica Fischer 78661f1cb2 Binning by ID (#276)
* Binning by ID

* Add test of binning by ID
2024-10-03 10:56:58 -07:00
Erica Fischer 8409bf9ce4 Top-level null filter now evaluates to true (#277) 2024-10-02 15:19:30 -07:00
Erica Fischer 23c6559b03 Remove buggy optimization to avoid reclipping in overzoom (#275)
* Remove buggy optimization to avoid reclipping in overzoom

* Add clarifying comment
2024-10-01 10:22:41 -07:00
Erica Fischer 66e5d66300 Another round of binning fixes (#274)
* Bin more aggressively if a point doesn't meet the pnpoly test

* Don't clip the points if we are binning

* Fix output of features added to the bin after its closure

* Update tests

* Update version and changelog
2024-09-30 08:57:13 -07:00
Erica Fischer 4822d02f24 Fix count accumulation in overzoom (#272)
* Fix count accumulation in overzoom

* Add test

* Increment version and changelog
2024-09-24 23:55:43 -07:00
Erica Fischer c5f2f0da34 More work on plumbing attribute accumulation through (#263)
* Plumb bounding boxes through potential intersections

* Quick bbox reject for bins that can't possibly intersect

* Inching toward attribute accumulation in megatile handling

* Some sort of test for how all these things interact with each other.

Automatic numeric attribute accumulation does *not* apply to attributes
that have an explicit attribute accumulator set, because the order of
operations is too messy and weird

* More sketching

* More sketching

* Actually do some accumulation

* Put all that behind an --accumulate-numeric flag

* Use the same attribute accumulation logic in binning as in megatiles

* Fix backwards conditional

* Add means, but somehow I have some counts of 0

* Handle aggregated attributes with no base attribute in the feature

* Checkpoint before I break everything

* Found a flaw, now to debug

* Fix a typo that broke accumulation

* Add binning tests

* Make sure IDs make it through on the bins

* Fix count/mean accumulation

* Make the numeric accumulation prefix configurable

* Make sure the accumulate test still works with a different prefix

* Forgot to update this test

* More testing to make sure cluster sizes make it all the way through

* Fix neglected --accumulate-attribute when binning

* Mark unexercised attribute accumulation cases as "can't happen"

* Factor out numeric preservation

* Attrs with the accumulation prefix are just preserved, not accumulated

* Test behavior of prefixed attributes

* Plumbing for exclude and exclude-prefix

* Implement and test attribute prefix stripping in overzoom

* Update version and changelog

* For debugging, make an attribute list of source feature IDs

* Revert "For debugging, make an attribute list of source feature IDs"

This reverts commit 65fc99c9d1.
2024-09-20 15:30:29 -07:00
Erica Fischer 534242887d Pass bin IDs through to the output (#266)
* Pass bin IDs through to the output

* Update version and changelog
2024-09-16 17:08:56 -07:00
Erica Fischer 84f6e887a7 Another try at fixing longitude wraparound for bins (#261)
* Another try at fixing longitude wraparound for bins

* Gonna get it right this time

* Forgot to update the comment

* The filter case was not supposed to reinterpret geometry

* Copy antimeridian-crossing geometries to the other side too

* Getting closer to getting antimeridian-crossing polygons right

* Update changelog and version
2024-09-10 14:25:57 -07:00
Erica Fischer 51fcf142df Fix bad interaction between dynamic dropping and limiting by truncation (#260)
* Fix bad interaction between dynamic dropping and limiting by truncation

* Don't enforce the tile size limit here if they said not to
2024-09-05 13:39:53 -07:00
Erica Fischer 5b18eea673 Work in progress on binning features in overzoom (#258)
* Factoring out tilestats management from GeoJSON file reading

* Move code around so overzoom can link against parse_layers

* Read the file of bins

* Plumb the bins through to overzoom()

* Some zip code bins to test with

* (Currently non-functional) test of binning

* Starting to spell out the bin matching loop

* Can't flatten points, so don't flatten bins either

* More fleshing out bin traversal

* Bounding box of tile-relative mvt geometry

* Smallest enclosing tile from bbox

* Most of the bin scan

* Add point in polygon check. It crashes.

* Find the matching bins

* GDAL-style bounding boxes have eaten my brain

* Make some features to bin into

* Increment a count as features are found to be within the bins

* Fix longitude wraparound in overzoom bins

* Fix the tests

* Push off attribute copying until after bin assignment

* Carry sum of numeric attributes into the bins

* Also add mean, min, and max

* Add --calculate-feature-index since I keep needing it for testing

* Add an option to accumulate sum/mean/max/min/count of all numeric attrs

* Don't bake in tippecanoe:mean, since we redo it from sum and count

* Forgot to update this test fixture after removing tiled mean

* Update version and changelog
2024-09-05 12:06:51 -07:00
Erica Fischer 40bb4ff732 Be more careful to retry when the feature count is exceeded (#257)
* Clip before dealing with multiplier or filters in overzoom

* Be more careful to retry when the feature count is exceeded

* Adjust the estimated total feature count for the multiplier too

* Fix the feature count estimates, I think

* Pass build info into the version string

* Report the actual max zoom of any tiles as the metadata maxzoom

* Revert unneeded renaming to make the diff more readable

* Clean up the adjustments to tile sizes and feature counts

* Update version and changelog

* Dropping a feature into a multiplier cluster still effectively drops it

* Update changelog

* Rethink the changelog description

* Don't try to truncate zooms if we are still tiling at z18
2024-08-20 10:54:16 -07:00
Erica Fischer d891ca7289 Fix latitude bboxes for features that extend beyond the mercator plane (#254)
* Fix latitude bboxes for features that extend beyond the mercator plane

* Update some more test expectations

* Update version and changelog
2024-08-08 15:30:52 -07:00
Erica Fischer bc3ef87c3f Add --generate-variable-depth-tile-pyramid option (#251)
* Track output position at the file level instead of within each tile

* Track file position where the child tile data begins

* Add option and document its intended behavior

* Changing the detail loop to account for stopping early

* I forgot I already added an option for this

* Stop early if we can make a complete tile

* Add a test of zoom truncation with limited feature count

* Forgot to commit the actual code change

* Make room for a vertex count in the header of each serialized tile

* Estimate tile complexity; don't try truncating when unlikely to work

* Be more conservative, because ever retrying a tile is a big speed hit

* If stopping early, don't simplify or clean; leave that to overzoom

* Add tiny polygon reduction / dust to overzoom

* Don't try to stop early in the children if we dropped anything by rate

* Fflush here too before pwriting

* Don't stop early if we ended up dropping any features.

Rework the can-the-next-zoom-stop-early logic to avoid going
one zoom further than needed.

* Fix warning

* Fix warnings

* Oops, checking for the wrong expected return value

* Cleanup from adding line simplification in overzoom

* Current (wrong) behavior when combining coalescing and truncating

* Keep a list of parent tiles to skip rather than truncating

* Now the coalesced tiles in z12 get children in z13

* Don't double-count feature dropping when the zoom level is retried

* Correct README description

* Remove todo about special case below basezoom, which is accounted for

* Be a little more aggressive in drop-densest determination

* Scale tile feature limit for megatiles in the same way as byte limit

* Fully deprecate -detect-shared-borders into an alias

* Track the distances found in the douglas-peucker recursion

* Serialize and deserialize the distance with the vertices

* Revert "Serialize and deserialize the distance with the vertices"

This reverts commit 753f1b7909.

* Revert "Track the distances found in the douglas-peucker recursion"

This reverts commit e5361f8c22.

* Revert "Fully deprecate -detect-shared-borders into an alias"

This reverts commit 0698aeb766.

* Better tracking of whether we failed to make a full-detail tile

* Put a bloom filter in front of the binary search for shared nodes

* Forgot to take out this printf

* Improve dispatch of tiling tasks

* Still dispatch the biggest tasks first

* Track zoom truncation in the strategies list in the tileset metadata

* Prescan for small deltas before doing proper simplification

* Revert "Prescan for small deltas before doing proper simplification"

This reverts commit d1d8238b83.

* Update version and changelog

* Rename to --generate-variable-depth-tile-pyramid
2024-08-06 16:05:52 -07:00
Erica Fischer 50deb9ce63 Add multi-tile input to tippecanoe-overzoom (#249)
* Reviving multi-source-tile overzoom: the clip.cpp side

* Reviving multi-source-tile overzoom: the overzoom.cpp side

* Update readme

* Update version and changelog
2024-07-23 13:21:10 -07:00
Erica Fischer 47e774adc4 Improve the appearance of coalesce-densest-as-needed tiles (#247)
* Start to distinguish fixed cluster density setting from as-needed density

* Make consistent {drop,coalesce}-densest decisions between zooms

* Actually track the previous index instead of just intending to

* Clean up collinearities in coalesced features

* To determine densest, look at actual physical distance, not just index

* Don't actually need the previous index in serial_feature now

* Center of mass of one feature to most distant point of the next

* Add apologetic comment

* Wait, how did the tests pass before?

* Revert "Wait, how did the tests pass before?"

This reverts commit f73c8ee543.

* Add --maximum-string-attribute-length option

* Update version and changelog

* A little more testing to make sure
2024-07-16 13:23:23 -07:00
Erica Fischer e11583df1c Fix hash collision in the string pool (#239)
* Current broken behavior

* More blatant test

* Fix hash collision in string pool

* Update version and changelog
2024-06-07 15:01:14 -07:00
Erica Fischer bb4f220678 Reduce tiling memory (#227)
* Trying to reduce memory in tiling

* Let the simplification workers go out of scope earlier

* Bail out quickly once the maximum feature count is reached

* Update changelog and version
2024-04-03 07:12:26 -07:00
Erica Fischer bd48ba8ea1 Fix accidental loss (at all zooms) of features with an explicit minzoom (#221)
* Fix accidental loss (at all zooms) of features with an explicit minzoom

* Try to stabilize centers and bounding boxes between architectures
2024-03-22 10:20:10 -07:00
Erica Fischer e796400377 Fix null behavior in in/ni expressions (#214)
* Fix null behavior in in/ni expressions

* Add test for null attributes in "ni" filters

* Remove unidecode from "in" expression evaluation again
2024-03-08 13:01:12 -08:00
Erica Fischer 655eccfcc3 Restore the use of unidecode for in/ni operators (#212)
* Restore the use of unidecode for in/ni operators

* Update changelog and version
2024-03-08 09:22:55 -08:00
Erica Fischer 021bb96aeb Allow non-string types to be used in "in" expressions (#211)
* Allow non-string types to be used in "in" expressions

* No unidecode smashing in "in" expressions, though!

* Update version and changelog
2024-03-04 10:16:48 -08:00
Erica Fischer 6a8f1b83d8 Fix some undefined behavior (#209)
* Fix some undefined behavior

* Avoid overflow in line simplification calculations

* Oops, missed a test

* Didn't mean to add that to the Makefile

* Revert "Revert "[ci] test in debug mode (#202)""

This reverts commit c95c328e47.

* Fix reference to out-of-scope pointer

* Fix invalid shift and out-of-bounds vector element reference

* Update changelog and version
2024-03-01 10:11:41 -08:00
Erica Fischer 312e1560a3 Stabilize feature order in overzoom (#210)
* Stabilize feature order in overzoom

* I want my sorts to be stable, please

* Revert "[ci] test in debug mode (#202)"

This reverts commit 853ada87b5.

* No need to reinitialize here
2024-02-29 10:39:10 -08:00
Erica Fischer a987197eed Don't swap attributes when reducing tiny polygon dust (#207)
* Don't swap attributes when reducing tiny polygon dust

Because the dust placeholder may be misleadingly far from the feature
that contributed the most area to it

* Remove unused arguments; update changelog
2024-02-26 15:41:26 -08:00
Erica Fischer 2b6630c42c Allow features that would be dropped dynamically to become multiplier features (#199)
* Postpone tagging features as being the first of a multiplier cluster

* Upgrade some dynamically dropped features to multiplier features

* Still don't let it put more features in a cluster than is allowed

* Update tests

* Make the current multiplier cluster size per-layer

* Make sure the first non-empty-geometry in the layer is marked as primary

* Improve comments

* Update changelog and version

* Remove commented out debugging printf

* Factor out duplicated code
2024-02-15 11:34:09 -08:00
Erica Fischer 96f126dd59 FSL-style expressions can use unidecode data to smash case and diacritics (#197)
* Read unidecode data, do some plumbing of it

* More unidecode plumbing

* Do the unidecode smashing, but it doesn't seem to be working

* Ah, that's better!

* Add missing header

* And reorder the includes too

* Shortcut when there is no unidecode data to work with

* Update version and changelog

* Avoid repeated unidecode smashing of the same constant string
2024-02-13 14:18:30 -08:00
Erica Fischer 4e52cbd957 Drop or retain whole multiplier clusters when dropping as needed (#198)
* Prep to track conditions other than just "dropped" or "kept"

* Count up instead of down

* Drop or retain whole multiplier clusters based on their first feature

* Calculate a global feature dropping sequence

* Switch over to using the drop sequence for drop-fraction

* Remove unused arguments for the old drop-fraction implementation

* Fix copy-and-paste bugs, update tests

* Properly incorporate feature_minzoom into the drop sequence, I hope

* Rename drop_by to drop_sequence

* See if sorting within clusters fixes filter stability between zooms

* Remove very chatty debug print

* Update changelog and version

* Use named constants instead of numbers for feature dropping/keeping

* Add comment to explain purpose and method of bit reversal
2024-02-12 10:58:49 -08:00
Erica Fischer e2a7a409c7 Improve tiling speed (#195)
* Add a way to run tippecanoe single-threaded for profiling

* Do less work when the tilestats sample values list is already full

* Save a copy when retrieving the attribute key

* Fewer atomic operations

* Move string hashing from mbtiles to text

* Only do approximate attribute deduplication when writing tiles

* Feature dropping tests are sensitive to exact tile size

* All tile creators now create a string pool for the tile

* Features clipped away to nothing should not participate in that tile

* Revert "Only do approximate attribute deduplication when writing tiles"

This reverts commit c42b34b498.

* Also revert the related test changes

* Revert "Revert "Only do approximate attribute deduplication when writing tiles""

This reverts commit 18509876c3.

* Be more specific about the string hash function

* Use fnv1a instead of std::hash for everything

* Reduce the chance of hash collisions

* Stick a hash search on the front of the tree search in addpool

* Eliminate repeated hashing of the same string

* Switch instead of ifs in json parsing

* A few more cases to populate the hash in addpool

* Store the hash in the tree instead of recalculating

* Add explanatory comment for mysterious argument

* Fewer copies in attribute stringification

* Clean up ancient weirdness in JSON attribute stringification

* More serial_val cleanup

* Pass a serial_feature to rewrite instead of many broken-down arguments

* Get rid of the multiple geometries within `partial`

* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"

This reverts commit 6f4ab9b725.

* Goodbye, struct coalesce

* Revert "Features clipped away to nothing should not participate in that tile"

This reverts commit 124462fbdc.

* Migrating fields from partial to serial_feature

* Name reconciliation between serial_feature and partial

* Replace struct partial with an augmented serial_feature

* Fix some overzealous search-and-replace renaming

* Don't say struct so often

* Remove more of the former partial construction

* Commenting and cleaning up

* Trying again to avoid all these arguments to rewrite

* I swear I did this same thing before and it didn't work.

* More rewrite cleanup

* Exile --detect-shared-borders to its own file

* Add missing headers

* More commenting and cleanup

* More comments

* Sprinkle consts around

* Emplacing and std::moving

* More cleanup

* That shouldn't have worked after a std::move

* Don't need to allocate memory to compare keys

* Reduce use of the global string pool in tiling

* Another avoidable mvt_value construction

* Further reduction to explicit string pool passing

* These reverses are no longer optimizations

* These layernames can all be references

* Don't drag an unused layername string around with every feature

* Heed a compiler warning about potential buffer overflow

* Fix my confusion about which feature's string pool is relevant

* Avoid some unnecessary allocations in attribute accumulation

* Maybe faster serialization?

* Eliminate a comparison

* Do the same here

* Save a couple of allocations when parsing numbers in JSON

* Immediately assign features to layers instead of subdividing later

* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence

* Crunch out more duplicate attribute values when writing out the tile

* Do tilestats for tippecanoe:retain_points_multiplier_first too

* Shell filters need to be real threads, even if nothing else does

* Simplify tippecanoe_minzoom/maxzoom representation

* Update version and changelog
2024-02-07 17:50:35 -08:00
Erica Fischer 6a2bce8164 Scale the tile size limit up with the multiplier at low zooms (#192)
* Scale the tile size limit up with the multiplier at low zooms

* Add a test to demonstrate that high zoom tiles can't be extra large

* Update changelog and version

* Fail more cleanly when a tile can't be made small enough

* Guard against a cluster where the start marker has been dropped

* Look harder for a working feature interval instead of giving up

* That change to the drop-smallest logic changed a test output

* Update changelog

* Add explanatory comment
2024-01-31 10:23:05 -08:00
Erica Fischer 7d4d264d50 Speeding up tippecanoe-overzoom (#191)
* Speed up mvt_value comparison

* Converting repetitive ifs to cases

* More conversions from ifs to cases

* Optimize the always-true filter case

* Don't convert types of attributes without accumulators

* Unordered map seems to be faster than map

* Add missing header

* Fix some warnings

* Fix the warnings better

* Avoid an int->string->int conversion

* Lazily initialize layer key and values maps when actually needed

* More switches from maps to unordered_maps

* Sure, I'll take the microoptimization

* More emplacement

* Save some copies

* Emplaces and moves

* Lazy linear scan of attributes instead of building a map

* Extra printfs, missing header

* Avoid clipping if the input and output tiles are the same

* But do clip if the tile extent is being reduced

* Make sure I'm not constructing std::strings here at runtime

* More worrying about runtime string construction

* A couple more std::moves

* Const references!

* More const references

* Another std::move

* Make the string_value of mvt_value std::optional

* Reserve storage when decoding

* Provision for different mvt_values to share a string pool

* Use the string pool when decoding

* Avoid another string construction

* Try limiting the depth of the search for duplicate attributes

* Revert "Try limiting the depth of the search for duplicate attributes"

This reverts commit 9ec94a15ff.

* Update changelog

* Fix typo noticed during code review
2024-01-29 11:33:56 -08:00