Commit Graph
1957 Commits
Author SHA1 Message Date
Erica Fischer 8bd955ae5b Carry sum of numeric attributes into the bins test-8bd955ae 2024-08-29 13:24:12 -07:00
Erica Fischer 2ee982c54f Push off attribute copying until after bin assignment 2024-08-29 12:05:04 -07:00
Erica Fischer 5e98b63c06 Fix the tests test-5e98b63c 2024-08-28 20:20:02 -07:00
Erica Fischer 7b3f0246a6 Fix longitude wraparound in overzoom bins 2024-08-28 14:23:10 -07:00
Erica Fischer 29934401a1 Increment a count as features are found to be within the bins test-29934401 2024-08-23 16:21:45 -07:00
Erica Fischer 1521c376dd Make some features to bin into 2024-08-23 15:59:53 -07:00
Erica Fischer aae1ab9085 GDAL-style bounding boxes have eaten my brain 2024-08-23 15:42:14 -07:00
Erica Fischer bc10e57a81 Find the matching bins 2024-08-23 15:01:25 -07:00
Erica Fischer 2b489cf338 Add point in polygon check. It crashes. 2024-08-23 14:48:24 -07:00
Erica Fischer 419a116168 Most of the bin scan 2024-08-23 14:32:06 -07:00
Erica Fischer 7243ae76c4 Smallest enclosing tile from bbox 2024-08-23 14:10:24 -07:00
Erica Fischer afa6a73a79 Bounding box of tile-relative mvt geometry 2024-08-23 13:12:43 -07:00
Erica Fischer 0514802f66 More fleshing out bin traversal 2024-08-23 12:18:46 -07:00
Erica Fischer ff9b59dacb Can't flatten points, so don't flatten bins either 2024-08-21 16:16:43 -07:00
Erica Fischer 14be3d2505 Starting to spell out the bin matching loop 2024-08-21 15:55:59 -07:00
Erica Fischer 7917d8b132 (Currently non-functional) test of binning 2024-08-21 15:16:44 -07:00
Erica Fischer 513c810c1b Some zip code bins to test with 2024-08-21 14:56:10 -07:00
Erica Fischer f1e5e4f109 Plumb the bins through to overzoom() 2024-08-21 14:30:04 -07:00
Erica Fischer f1112f6a12 Read the file of bins 2024-08-21 14:00:42 -07:00
Erica Fischer 58143b4815 Move code around so overzoom can link against parse_layers 2024-08-21 13:45:38 -07:00
Erica Fischer c91e07aaa6 Factoring out tilestats management from GeoJSON file reading 2024-08-21 12:46:21 -07:00
Erica Fischer 40bb4ff732 Be more careful to retry when the feature count is exceeded (#257)
* Clip before dealing with multiplier or filters in overzoom

* Be more careful to retry when the feature count is exceeded

* Adjust the estimated total feature count for the multiplier too

* Fix the feature count estimates, I think

* Pass build info into the version string

* Report the actual max zoom of any tiles as the metadata maxzoom

* Revert unneeded renaming to make the diff more readable

* Clean up the adjustments to tile sizes and feature counts

* Update version and changelog

* Dropping a feature into a multiplier cluster still effectively drops it

* Update changelog

* Rethink the changelog description

* Don't try to truncate zooms if we are still tiling at z18
2.60.0
2024-08-20 10:54:16 -07:00
Erica Fischer d891ca7289 Fix latitude bboxes for features that extend beyond the mercator plane (#254)
* Fix latitude bboxes for features that extend beyond the mercator plane

* Update some more test expectations

* Update version and changelog
2.59.0
2024-08-08 15:30:52 -07:00
Erica Fischer bc3ef87c3f Add --generate-variable-depth-tile-pyramid option (#251)
* Track output position at the file level instead of within each tile

* Track file position where the child tile data begins

* Add option and document its intended behavior

* Changing the detail loop to account for stopping early

* I forgot I already added an option for this

* Stop early if we can make a complete tile

* Add a test of zoom truncation with limited feature count

* Forgot to commit the actual code change

* Make room for a vertex count in the header of each serialized tile

* Estimate tile complexity; don't try truncating when unlikely to work

* Be more conservative, because ever retrying a tile is a big speed hit

* If stopping early, don't simplify or clean; leave that to overzoom

* Add tiny polygon reduction / dust to overzoom

* Don't try to stop early in the children if we dropped anything by rate

* Fflush here too before pwriting

* Don't stop early if we ended up dropping any features.

Rework the can-the-next-zoom-stop-early logic to avoid going
one zoom further than needed.

* Fix warning

* Fix warnings

* Oops, checking for the wrong expected return value

* Cleanup from adding line simplification in overzoom

* Current (wrong) behavior when combining coalescing and truncating

* Keep a list of parent tiles to skip rather than truncating

* Now the coalesced tiles in z12 get children in z13

* Don't double-count feature dropping when the zoom level is retried

* Correct README description

* Remove todo about special case below basezoom, which is accounted for

* Be a little more aggressive in drop-densest determination

* Scale tile feature limit for megatiles in the same way as byte limit

* Fully deprecate -detect-shared-borders into an alias

* Track the distances found in the douglas-peucker recursion

* Serialize and deserialize the distance with the vertices

* Revert "Serialize and deserialize the distance with the vertices"

This reverts commit 753f1b7909.

* Revert "Track the distances found in the douglas-peucker recursion"

This reverts commit e5361f8c22.

* Revert "Fully deprecate -detect-shared-borders into an alias"

This reverts commit 0698aeb766.

* Better tracking of whether we failed to make a full-detail tile

* Put a bloom filter in front of the binary search for shared nodes

* Forgot to take out this printf

* Improve dispatch of tiling tasks

* Still dispatch the biggest tasks first

* Track zoom truncation in the strategies list in the tileset metadata

* Prescan for small deltas before doing proper simplification

* Revert "Prescan for small deltas before doing proper simplification"

This reverts commit d1d8238b83.

* Update version and changelog

* Rename to --generate-variable-depth-tile-pyramid
2.58.0
2024-08-06 16:05:52 -07:00
Erica Fischer 50deb9ce63 Add multi-tile input to tippecanoe-overzoom (#249)
* Reviving multi-source-tile overzoom: the clip.cpp side

* Reviving multi-source-tile overzoom: the overzoom.cpp side

* Update readme

* Update version and changelog
2.57.0
2024-07-23 13:21:10 -07:00
Erica Fischer 47e774adc4 Improve the appearance of coalesce-densest-as-needed tiles (#247)
* Start to distinguish fixed cluster density setting from as-needed density

* Make consistent {drop,coalesce}-densest decisions between zooms

* Actually track the previous index instead of just intending to

* Clean up collinearities in coalesced features

* To determine densest, look at actual physical distance, not just index

* Don't actually need the previous index in serial_feature now

* Center of mass of one feature to most distant point of the next

* Add apologetic comment

* Wait, how did the tests pass before?

* Revert "Wait, how did the tests pass before?"

This reverts commit f73c8ee543.

* Add --maximum-string-attribute-length option

* Update version and changelog

* A little more testing to make sure
2.56.0
2024-07-16 13:23:23 -07:00
Drew Good 939579e5a2 Update README (#244) 2024-07-03 11:10:35 -07:00
Erica Fischer e11583df1c Fix hash collision in the string pool (#239)
* Current broken behavior

* More blatant test

* Fix hash collision in string pool

* Update version and changelog
2.55.0
2024-06-07 15:01:14 -07:00
Erica Fischer bb4f220678 Reduce tiling memory (#227)
* Trying to reduce memory in tiling

* Let the simplification workers go out of scope earlier

* Bail out quickly once the maximum feature count is reached

* Update changelog and version
2.53.0
2024-04-03 07:12:26 -07:00
Dane Springmeyer 266ef3c286 Add tippecanoe-overzoom to .gitignore (#224)
add tippecanoe-overzoom to .gitignore
2024-03-25 11:00:18 -07:00
Dane Springmeyer 1519b50791 Upgrade catch from v1.5.7 to v2.13.10 (#223)
upgrade catch from v1.5.7 to v2.13.10
2024-03-25 10:39:11 -07:00
Erica Fischer bd48ba8ea1 Fix accidental loss (at all zooms) of features with an explicit minzoom (#221)
* Fix accidental loss (at all zooms) of features with an explicit minzoom

* Try to stabilize centers and bounding boxes between architectures
2.52.0
2024-03-22 10:20:10 -07:00
Erica Fischer e796400377 Fix null behavior in in/ni expressions (#214)
* Fix null behavior in in/ni expressions

* Add test for null attributes in "ni" filters

* Remove unidecode from "in" expression evaluation again
2.51.0
2024-03-08 13:01:12 -08:00
Erica Fischer 655eccfcc3 Restore the use of unidecode for in/ni operators (#212)
* Restore the use of unidecode for in/ni operators

* Update changelog and version
2.50.0
2024-03-08 09:22:55 -08:00
Erica Fischer 021bb96aeb Allow non-string types to be used in "in" expressions (#211)
* Allow non-string types to be used in "in" expressions

* No unidecode smashing in "in" expressions, though!

* Update version and changelog
2.49.0
2024-03-04 10:16:48 -08:00
Erica Fischer 6a8f1b83d8 Fix some undefined behavior (#209)
* Fix some undefined behavior

* Avoid overflow in line simplification calculations

* Oops, missed a test

* Didn't mean to add that to the Makefile

* Revert "Revert "[ci] test in debug mode (#202)""

This reverts commit c95c328e47.

* Fix reference to out-of-scope pointer

* Fix invalid shift and out-of-bounds vector element reference

* Update changelog and version
2.48.0
2024-03-01 10:11:41 -08:00
Erica Fischer 312e1560a3 Stabilize feature order in overzoom (#210)
* Stabilize feature order in overzoom

* I want my sorts to be stable, please

* Revert "[ci] test in debug mode (#202)"

This reverts commit 853ada87b5.

* No need to reinitialize here
2.47.0
2024-02-29 10:39:10 -08:00
Pal Szabo d9a5c2eed7 highz precision values (#208) 2024-02-28 14:08:34 -08:00
Dane Springmeyer 853ada87b5 [ci] test in debug mode (#202) 2024-02-28 12:54:22 -08:00
Erica Fischer a987197eed Don't swap attributes when reducing tiny polygon dust (#207)
* Don't swap attributes when reducing tiny polygon dust

Because the dust placeholder may be misleadingly far from the feature
that contributed the most area to it

* Remove unused arguments; update changelog
2.46.0
2024-02-26 15:41:26 -08:00
Erica Fischer 2b6630c42c Allow features that would be dropped dynamically to become multiplier features (#199)
* Postpone tagging features as being the first of a multiplier cluster

* Upgrade some dynamically dropped features to multiplier features

* Still don't let it put more features in a cluster than is allowed

* Update tests

* Make the current multiplier cluster size per-layer

* Make sure the first non-empty-geometry in the layer is marked as primary

* Improve comments

* Update changelog and version

* Remove commented out debugging printf

* Factor out duplicated code
2.45.0
2024-02-15 11:34:09 -08:00
Erica Fischer 96f126dd59 FSL-style expressions can use unidecode data to smash case and diacritics (#197)
* Read unidecode data, do some plumbing of it

* More unidecode plumbing

* Do the unidecode smashing, but it doesn't seem to be working

* Ah, that's better!

* Add missing header

* And reorder the includes too

* Shortcut when there is no unidecode data to work with

* Update version and changelog

* Avoid repeated unidecode smashing of the same constant string
2.44.0
2024-02-13 14:18:30 -08:00
Erica Fischer 4e52cbd957 Drop or retain whole multiplier clusters when dropping as needed (#198)
* Prep to track conditions other than just "dropped" or "kept"

* Count up instead of down

* Drop or retain whole multiplier clusters based on their first feature

* Calculate a global feature dropping sequence

* Switch over to using the drop sequence for drop-fraction

* Remove unused arguments for the old drop-fraction implementation

* Fix copy-and-paste bugs, update tests

* Properly incorporate feature_minzoom into the drop sequence, I hope

* Rename drop_by to drop_sequence

* See if sorting within clusters fixes filter stability between zooms

* Remove very chatty debug print

* Update changelog and version

* Use named constants instead of numbers for feature dropping/keeping

* Add comment to explain purpose and method of bit reversal
2.43.0
2024-02-12 10:58:49 -08:00
Erica Fischer e2a7a409c7 Improve tiling speed (#195)
* Add a way to run tippecanoe single-threaded for profiling

* Do less work when the tilestats sample values list is already full

* Save a copy when retrieving the attribute key

* Fewer atomic operations

* Move string hashing from mbtiles to text

* Only do approximate attribute deduplication when writing tiles

* Feature dropping tests are sensitive to exact tile size

* All tile creators now create a string pool for the tile

* Features clipped away to nothing should not participate in that tile

* Revert "Only do approximate attribute deduplication when writing tiles"

This reverts commit c42b34b498.

* Also revert the related test changes

* Revert "Revert "Only do approximate attribute deduplication when writing tiles""

This reverts commit 18509876c3.

* Be more specific about the string hash function

* Use fnv1a instead of std::hash for everything

* Reduce the chance of hash collisions

* Stick a hash search on the front of the tree search in addpool

* Eliminate repeated hashing of the same string

* Switch instead of ifs in json parsing

* A few more cases to populate the hash in addpool

* Store the hash in the tree instead of recalculating

* Add explanatory comment for mysterious argument

* Fewer copies in attribute stringification

* Clean up ancient weirdness in JSON attribute stringification

* More serial_val cleanup

* Pass a serial_feature to rewrite instead of many broken-down arguments

* Get rid of the multiple geometries within `partial`

* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"

This reverts commit 6f4ab9b725.

* Goodbye, struct coalesce

* Revert "Features clipped away to nothing should not participate in that tile"

This reverts commit 124462fbdc.

* Migrating fields from partial to serial_feature

* Name reconciliation between serial_feature and partial

* Replace struct partial with an augmented serial_feature

* Fix some overzealous search-and-replace renaming

* Don't say struct so often

* Remove more of the former partial construction

* Commenting and cleaning up

* Trying again to avoid all these arguments to rewrite

* I swear I did this same thing before and it didn't work.

* More rewrite cleanup

* Exile --detect-shared-borders to its own file

* Add missing headers

* More commenting and cleanup

* More comments

* Sprinkle consts around

* Emplacing and std::moving

* More cleanup

* That shouldn't have worked after a std::move

* Don't need to allocate memory to compare keys

* Reduce use of the global string pool in tiling

* Another avoidable mvt_value construction

* Further reduction to explicit string pool passing

* These reverses are no longer optimizations

* These layernames can all be references

* Don't drag an unused layername string around with every feature

* Heed a compiler warning about potential buffer overflow

* Fix my confusion about which feature's string pool is relevant

* Avoid some unnecessary allocations in attribute accumulation

* Maybe faster serialization?

* Eliminate a comparison

* Do the same here

* Save a couple of allocations when parsing numbers in JSON

* Immediately assign features to layers instead of subdividing later

* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence

* Crunch out more duplicate attribute values when writing out the tile

* Do tilestats for tippecanoe:retain_points_multiplier_first too

* Shell filters need to be real threads, even if nothing else does

* Simplify tippecanoe_minzoom/maxzoom representation

* Update version and changelog
2.42.0
2024-02-07 17:50:35 -08:00
Can Durukandsnyk-bot c5c4698161 [Snyk] Security upgrade centos from 7 to centos8 (#196)
fix: Dockerfile.centos7 to reduce vulnerabilities

The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-CENTOS7-EXPAT-2408807
- https://snyk.io/vuln/SNYK-CENTOS7-EXPAT-3035862
- https://snyk.io/vuln/SNYK-CENTOS7-NSS-2308137
- https://snyk.io/vuln/SNYK-CENTOS7-NSSSYSINIT-2308141
- https://snyk.io/vuln/SNYK-CENTOS7-NSSTOOLS-2308139

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-02-01 14:38:43 -08:00
Erica Fischer 6a2bce8164 Scale the tile size limit up with the multiplier at low zooms (#192)
* Scale the tile size limit up with the multiplier at low zooms

* Add a test to demonstrate that high zoom tiles can't be extra large

* Update changelog and version

* Fail more cleanly when a tile can't be made small enough

* Guard against a cluster where the start marker has been dropped

* Look harder for a working feature interval instead of giving up

* That change to the drop-smallest logic changed a test output

* Update changelog

* Add explanatory comment
2.41.3
2024-01-31 10:23:05 -08:00
David Haberthür ca1aa58a11 Prettify one code line (#187) 2024-01-29 11:34:47 -08:00
Erica Fischer 7d4d264d50 Speeding up tippecanoe-overzoom (#191)
* Speed up mvt_value comparison

* Converting repetitive ifs to cases

* More conversions from ifs to cases

* Optimize the always-true filter case

* Don't convert types of attributes without accumulators

* Unordered map seems to be faster than map

* Add missing header

* Fix some warnings

* Fix the warnings better

* Avoid an int->string->int conversion

* Lazily initialize layer key and values maps when actually needed

* More switches from maps to unordered_maps

* Sure, I'll take the microoptimization

* More emplacement

* Save some copies

* Emplaces and moves

* Lazy linear scan of attributes instead of building a map

* Extra printfs, missing header

* Avoid clipping if the input and output tiles are the same

* But do clip if the tile extent is being reduced

* Make sure I'm not constructing std::strings here at runtime

* More worrying about runtime string construction

* A couple more std::moves

* Const references!

* More const references

* Another std::move

* Make the string_value of mvt_value std::optional

* Reserve storage when decoding

* Provision for different mvt_values to share a string pool

* Use the string pool when decoding

* Avoid another string construction

* Try limiting the depth of the search for duplicate attributes

* Revert "Try limiting the depth of the search for duplicate attributes"

This reverts commit 9ec94a15ff.

* Update changelog

* Fix typo noticed during code review
2024-01-29 11:33:56 -08:00
Erica Fischer cbf222754b Add --accumulate-attribute to tippecanoe-overzoom (#189)
* Starting to factor out attribute accumulation into its own file

* Continuing to factor out attribute accumulation

* Reduce duplicate code

* Plumbing the accumulate-attribute option around

* Call the attribute accumulator

* Test that accumulation works

* Add missing #includes

* Don't sort within individual multiplier clusters

Doing so throws off the spatial distribution of the low zooms

* Docs and changelog

* Add comments
2.41.2
2024-01-23 15:41:36 -08:00
Erica Fischer f957f30f90 Clean up internal naming related to tilestats (#190)
* Get rid of the type_and_string near-synonym for serial_val

* Rename file_keys to the more familiar tilestats

* "tas" (type_and_string) => "sv" (serial_val)

* "fk" (tile_keys) => "ts" (tilestats)

* Revert ""tas" (type_and_string) => "sv" (serial_val)"

This reverts commit 4854c57e22.

* More carefully this time: "tas" (type_and_string) => "sv" (serial_val)
2024-01-23 14:17:19 -08:00