Commit Graph
1984 Commits
Author SHA1 Message Date
Erica Fischer dda13b3c97 Case smashing and prefix trimming in the select test-dda13b3c 2025-01-03 10:26:54 -08:00
Erica Fischer b87d8e4560 Fix accidental inclusion of the top left of the tile in the bbox test-b87d8e45 2024-12-20 10:41:53 -08:00
Erica Fischer 632d031caa Missed a test 2024-12-20 09:46:22 -08:00
Erica Fischer 4788b67a54 Gather the results from each thread *after* the thread finishes 2024-12-20 09:42:09 -08:00
Erica Fischer 7338c4b6b0 An empty tileset has empty bounds at null island 2024-12-19 16:24:01 -08:00
Erica Fischer acffe23b97 More tests 2024-12-19 16:18:58 -08:00
Erica Fischer d6ce92422f Make tile-join bounding boxes reflect feature bounds, not tile bounds 2024-12-19 16:16:52 -08:00
Erica Fischer 5c8f1545a5 Add a flag to exclude all attributes from the tile side of the join test-5c8f154 2024-12-17 15:43:34 -08:00
Erica Fischer 638a7617d0 Merge remote-tracking branch 'origin/main' into tile-join-sqlite 2024-12-17 15:23:01 -08:00
Erica Fischer fcc611adc4 Observe attribute exclusion when joining from sql queries 2024-12-17 15:22:34 -08:00
Erica Fischer 03daaefd8e Add a flag to get the feature ID from the query 2024-12-17 15:01:06 -08:00
Erica Fischer c88f8c07e7 Set matched if the sql join matches 2024-12-17 14:43:04 -08:00
Erica Fischer 7165ae6999 Fix clipping bug when the clip region doesn't intersect the tile (#312) 2.73.0 2024-12-11 10:24:12 -08:00
Ryan Mast d70e3326da Consolidate code for getting platform details into one place (#311)
Consolidate code for getting platform details into a single place
2024-12-09 15:21:50 -08:00
Ryan Mast a8bd7ac723 Add macOS CI jobs (#310) 2024-12-06 09:01:55 -08:00
Erica Fischer a4b866833a Join the attributes onto the feature test-a4b8668 2024-12-05 16:19:54 -08:00
Erica Fischer cbcdbc5516 Actually do the query and get results back 2024-12-05 16:01:27 -08:00
Erica Fischer ad47b93a81 Actually open the gpkg. Fix the query quoting. 2024-12-05 14:42:48 -08:00
Erica Fischer 264665b0ce Make a query 2024-12-05 13:52:28 -08:00
Erica Fischer eab74d4700 Merge remote-tracking branch 'origin/main' into tile-join-sqlite 2024-12-05 12:39:51 -08:00
Erica Fischer bdfb06cd6a Make tippecanoe-overzoom accept filters from a file (#307)
* Make tippecanoe-overzoom accept filters from a file

* Accept clip polygons from a file too

* Add test of clipping by polygon from file

* Add a test of reading an overzoom filter from a file

* Update version and changelog
2.72.0
2024-12-05 12:05:50 -08:00
Erica Fischer 0ae73c1143 Gathering join keys for sql query 2024-12-03 16:16:41 -08:00
Erica Fischer 36b207c16e Fix some unnecessary round trips from std::string to char * and back 2024-12-03 14:43:02 -08:00
Erica Fischer a07aad9db7 Enable sqlite3 serialized multithreading 2024-12-02 14:21:53 -08:00
Erica Fischer f2d149059e Merge remote-tracking branch 'origin/main' into tile-join-sqlite 2024-12-02 11:35:28 -08:00
Sergey Fedorov 1d81935cd5 mbtiles.cpp: isinf to std::isinf (#303) 2024-11-30 22:33:56 -08:00
Erica Fischer dcc616d3d4 Adding optional clipping to tippecanoe-overzoom (#298)
* Plumb a clip bounding box around through overzoom

* Actually do some clipping

* Add a test

* Fix post-binning clipping

* Factoring out geometry parsing from feature parsing

* Accept a clip polygon argument to tippecanoe-overzoom

* Progress in the direction of polygon clipping

* Fix the wagyu flags. We need intersection, not union

* Remove debug spew

* Clip points to polygon bounds too

* Copy the geometric binning code to serve as intersection-finding code

* Add clipper2 for linestring clipping

* Compiles, but does not actually seem to clip. Hmm.

* Oh, it helps if I actually call the function

* Add clipping tests

* Add missing fixture, and don't crash if it is missing

* Remember to do polygon clipping after binning too

* Fix scaling before post-binning clipping. Add test.

* Remove unused parts of clipper

* Rename for consistency

* Revert accidentally added line

* Clip the clip regions to the tile bounds to reduce their complexity

* Add a test of clipping the clip region down to the tile boundary

* Update version and changelog
2.71.0
2024-11-27 13:43:40 -08:00
Erica Fischer 11e3196c9a Raise tippecanoe-decode tile size limit to 250 MB (#299)
* Raise tippecanoe-decode tile size limit to 250 MB

* Update version and changelog
2.70.1
2024-11-21 09:17:22 -08:00
Erica Fischer 905b58cdf4 Reducing attribute tagging within tippecanoe-overzoom (#296)
* Strip out unwanted attributes earlier in the process

* Skip aggregations whose attributes have been excluded

* Forgot one

* Add some more tests

* Update version and changelog
2.70.0
2024-11-15 13:14:30 -08:00
Erica Fischer 6d46578ad8 Avoid crash if the first bin gets clipped away (#294)
* Avoid crash if the first bin gets clipped away

* Add test

* Update version and changelog
2.69.0
2024-11-13 10:48:33 -08:00
Erica Fischer 794ae2b1eb Tippecanoe-decode and tippecanoe-overzoom changes to improve binning performance (#292)
* Add a tippecanoe-decode option to restrict which attributes to decode

* Plumb buffer and feature limit around

* Check the feature limit

* Clarifying cases where output detail can be unspecified

* Clip bins to the tile buffer instead of just passing them through

* Add missing include

* Missed some tests

* Add --no-tile-compression option to tippecanoe-overzoom

* Update version and changelog
2.68.0
2024-11-08 15:46:04 -08:00
Erica Fischer 23667bb8eb Reduce memory consumption from attribute accumulation (#290)
* Progress on plumbing a string pool for full_keys through

* More plumbing for key_pool

* Don't keep features with identical locations as multiplier features

* Revert "Don't keep features with identical locations as multiplier features"

This reverts commit 413f0c8024.

* Adjust calculated maxzoom to account for duplicate feature locations

* Update changelog and version

* Add a test affected by the maxzoom change with duplicate locations

* Round the drop rate a little for cross-platform test consistency
2.67.0
2024-11-05 14:39:41 -08:00
Erica Fischer 5c57d5aef2 Sketching out sqlite options for tile-join 2024-11-04 12:17:25 -08:00
Erica Fischer b3b89e1e07 Performance optimizations for binning (#283)
* Add an all-mvt_value attribute accumulation path

* Only bin by ID, not geometrically

* A little cleanup; changelog and version; test

* Remove accidental double-conversion

* Replace duplicated code with template

* Update changelog
2.66.0
2024-11-01 15:27:13 -07:00
Patrick M YoungandPatrick Young 120b8f4e41 Fixes bool atrributes coming from FGB files (#289)
Fixes https://github.com/felt/tippecanoe/issues/180.

Co-authored-by: Patrick Young <pyoung@snapchat.com>
2024-11-01 12:27:05 -07:00
Erica Fischer 28efc40e6e Choose the megatile features from those that will be in the next N zooms (#280)
* Choose the megatile features from those that will be in the next N zooms

* Take fractional zooms into account in multiplier feature choices

* Fix more tests

* Add a flag to retain multiplier features by minimum distance

* Limit feature expansion from multiplier density to 2x

* The multiplier cap was a bad idea

* Revert "The multiplier cap was a bad idea"

This reverts commit 6f8273a4c8.

* Revert "Limit feature expansion from multiplier density to 2x"

This reverts commit a26e41309d.

* Revert "Add a flag to retain multiplier features by minimum distance"

This reverts commit 01f14a4255.

* Remove the multiplier sequence, which should no longer matter

* Revert "Revert "Add a flag to retain multiplier features by minimum distance""

This reverts commit 776da4a1b8.

* Revert "Revert "Limit feature expansion from multiplier density to 2x""

This reverts commit 44a683d808.

* Revert "Revert "The multiplier cap was a bad idea""

This reverts commit 80f7cb1c0e.

* Track two kinds of previous index for next_feature

* Fix multiplier density threshold, I think

* Oh, I didn't git add the code changes

* Update version and changelog

* Try to install sqlite3 to fix the automated build

* Deleted too much

* Only let --preserve-point-density-threshold shift density around

* Remove the density debt concept, since it doesn't help

* Make the drop states a vector instead of an array

* Revert "Make the drop states a vector instead of an array"

This reverts commit 66c7abb6fa.

* Revert "Remove the density debt concept, since it doesn't help"

This reverts commit 707bb0c562.

* Revert "Only let --preserve-point-density-threshold shift density around"

This reverts commit ecf01f2231.
2.65.0
2024-10-17 15:59:44 -07:00
Erica Fischer 78661f1cb2 Binning by ID (#276)
* Binning by ID

* Add test of binning by ID
2.64.0
2024-10-03 10:56:58 -07:00
Erica Fischer 8409bf9ce4 Top-level null filter now evaluates to true (#277) 2.63.0 2024-10-02 15:19:30 -07:00
Erica Fischer 23c6559b03 Remove buggy optimization to avoid reclipping in overzoom (#275)
* Remove buggy optimization to avoid reclipping in overzoom

* Add clarifying comment
2024-10-01 10:22:41 -07:00
Erica Fischer 66e5d66300 Another round of binning fixes (#274)
* Bin more aggressively if a point doesn't meet the pnpoly test

* Don't clip the points if we are binning

* Fix output of features added to the bin after its closure

* Update tests

* Update version and changelog
2.62.5
2024-09-30 08:57:13 -07:00
Erica Fischer 4822d02f24 Fix count accumulation in overzoom (#272)
* Fix count accumulation in overzoom

* Add test

* Increment version and changelog
2.62.4
2024-09-24 23:55:43 -07:00
Erica Fischer c5f2f0da34 More work on plumbing attribute accumulation through (#263)
* Plumb bounding boxes through potential intersections

* Quick bbox reject for bins that can't possibly intersect

* Inching toward attribute accumulation in megatile handling

* Some sort of test for how all these things interact with each other.

Automatic numeric attribute accumulation does *not* apply to attributes
that have an explicit attribute accumulator set, because the order of
operations is too messy and weird

* More sketching

* More sketching

* Actually do some accumulation

* Put all that behind an --accumulate-numeric flag

* Use the same attribute accumulation logic in binning as in megatiles

* Fix backwards conditional

* Add means, but somehow I have some counts of 0

* Handle aggregated attributes with no base attribute in the feature

* Checkpoint before I break everything

* Found a flaw, now to debug

* Fix a typo that broke accumulation

* Add binning tests

* Make sure IDs make it through on the bins

* Fix count/mean accumulation

* Make the numeric accumulation prefix configurable

* Make sure the accumulate test still works with a different prefix

* Forgot to update this test

* More testing to make sure cluster sizes make it all the way through

* Fix neglected --accumulate-attribute when binning

* Mark unexercised attribute accumulation cases as "can't happen"

* Factor out numeric preservation

* Attrs with the accumulation prefix are just preserved, not accumulated

* Test behavior of prefixed attributes

* Plumbing for exclude and exclude-prefix

* Implement and test attribute prefix stripping in overzoom

* Update version and changelog

* For debugging, make an attribute list of source feature IDs

* Revert "For debugging, make an attribute list of source feature IDs"

This reverts commit 65fc99c9d1.
2.62.3
2024-09-20 15:30:29 -07:00
Erica Fischer 534242887d Pass bin IDs through to the output (#266)
* Pass bin IDs through to the output

* Update version and changelog
2.62.2
2024-09-16 17:08:56 -07:00
Erica Fischer 84f6e887a7 Another try at fixing longitude wraparound for bins (#261)
* Another try at fixing longitude wraparound for bins

* Gonna get it right this time

* Forgot to update the comment

* The filter case was not supposed to reinterpret geometry

* Copy antimeridian-crossing geometries to the other side too

* Getting closer to getting antimeridian-crossing polygons right

* Update changelog and version
2.62.1
2024-09-10 14:25:57 -07:00
Erica Fischer 51fcf142df Fix bad interaction between dynamic dropping and limiting by truncation (#260)
* Fix bad interaction between dynamic dropping and limiting by truncation

* Don't enforce the tile size limit here if they said not to
2.62.0
2024-09-05 13:39:53 -07:00
Erica Fischer 5b18eea673 Work in progress on binning features in overzoom (#258)
* Factoring out tilestats management from GeoJSON file reading

* Move code around so overzoom can link against parse_layers

* Read the file of bins

* Plumb the bins through to overzoom()

* Some zip code bins to test with

* (Currently non-functional) test of binning

* Starting to spell out the bin matching loop

* Can't flatten points, so don't flatten bins either

* More fleshing out bin traversal

* Bounding box of tile-relative mvt geometry

* Smallest enclosing tile from bbox

* Most of the bin scan

* Add point in polygon check. It crashes.

* Find the matching bins

* GDAL-style bounding boxes have eaten my brain

* Make some features to bin into

* Increment a count as features are found to be within the bins

* Fix longitude wraparound in overzoom bins

* Fix the tests

* Push off attribute copying until after bin assignment

* Carry sum of numeric attributes into the bins

* Also add mean, min, and max

* Add --calculate-feature-index since I keep needing it for testing

* Add an option to accumulate sum/mean/max/min/count of all numeric attrs

* Don't bake in tippecanoe:mean, since we redo it from sum and count

* Forgot to update this test fixture after removing tiled mean

* Update version and changelog
2024-09-05 12:06:51 -07:00
Monty Taylor f3b575ead4 Minor dockerfile optimizations (#240)
* Replace build-essential in Docker builder image

build-essential is overkill, it pulls in all of the tools one needs to
make to build debian packages, which we are not doing. Even though this
is a builder image, it still takes time to download all the extra crud.

Replace build-essential with make gcc and g++, which are all that are
needed.

* Replace dev packages in final Docker image

The dev packages are only needed in the builder image. They add header
files and docs and things that are not needed in the final image.

Removing them and replacing them with just the runtime libraries reduces
the final image size by about 25% (160M->124M) in a local build.

Additionally, just for simplicity, zlib1g is already part of base
ubuntu minimal, so we don't even have to list it.

* Remove build-essential mention from the README

Just like the in Dockerfile, people don't need all of those packages
to build directly on their system.
2024-08-28 10:35:21 -07:00
Rizki Alhamid f44230f5f5 Fix a bug when reading MultiLineString from .fgb file (#241)
fix logic
2024-08-28 10:34:33 -07:00
Erica Fischer 40bb4ff732 Be more careful to retry when the feature count is exceeded (#257)
* Clip before dealing with multiplier or filters in overzoom

* Be more careful to retry when the feature count is exceeded

* Adjust the estimated total feature count for the multiplier too

* Fix the feature count estimates, I think

* Pass build info into the version string

* Report the actual max zoom of any tiles as the metadata maxzoom

* Revert unneeded renaming to make the diff more readable

* Clean up the adjustments to tile sizes and feature counts

* Update version and changelog

* Dropping a feature into a multiplier cluster still effectively drops it

* Update changelog

* Rethink the changelog description

* Don't try to truncate zooms if we are still tiling at z18
2.60.0
2024-08-20 10:54:16 -07:00
Erica Fischer d891ca7289 Fix latitude bboxes for features that extend beyond the mercator plane (#254)
* Fix latitude bboxes for features that extend beyond the mercator plane

* Update some more test expectations

* Update version and changelog
2.59.0
2024-08-08 15:30:52 -07:00