Commit Graph
1967 Commits
Author SHA1 Message Date
Erica Fischer aeba5d604c Intersecting only at endpoints is fine and doesn't require any work 2024-02-23 09:32:25 -08:00
Erica Fischer 9279c8e034 Debugging 2024-02-22 16:25:36 -08:00
Erica Fischer 9b2c5baddf Merge remote-tracking branch 'origin/main' into clean-polygons 2024-02-22 15:44:29 -08:00
Erica Fischer e88bdde3f0 Trying to remember where I left off 2024-02-22 15:39:44 -08:00
Erica Fischer 2b6630c42c Allow features that would be dropped dynamically to become multiplier features (#199)
* Postpone tagging features as being the first of a multiplier cluster

* Upgrade some dynamically dropped features to multiplier features

* Still don't let it put more features in a cluster than is allowed

* Update tests

* Make the current multiplier cluster size per-layer

* Make sure the first non-empty-geometry in the layer is marked as primary

* Improve comments

* Update changelog and version

* Remove commented out debugging printf

* Factor out duplicated code
2.45.0
2024-02-15 11:34:09 -08:00
Erica Fischer 96f126dd59 FSL-style expressions can use unidecode data to smash case and diacritics (#197)
* Read unidecode data, do some plumbing of it

* More unidecode plumbing

* Do the unidecode smashing, but it doesn't seem to be working

* Ah, that's better!

* Add missing header

* And reorder the includes too

* Shortcut when there is no unidecode data to work with

* Update version and changelog

* Avoid repeated unidecode smashing of the same constant string
2.44.0
2024-02-13 14:18:30 -08:00
Erica Fischer 4e52cbd957 Drop or retain whole multiplier clusters when dropping as needed (#198)
* Prep to track conditions other than just "dropped" or "kept"

* Count up instead of down

* Drop or retain whole multiplier clusters based on their first feature

* Calculate a global feature dropping sequence

* Switch over to using the drop sequence for drop-fraction

* Remove unused arguments for the old drop-fraction implementation

* Fix copy-and-paste bugs, update tests

* Properly incorporate feature_minzoom into the drop sequence, I hope

* Rename drop_by to drop_sequence

* See if sorting within clusters fixes filter stability between zooms

* Remove very chatty debug print

* Update changelog and version

* Use named constants instead of numbers for feature dropping/keeping

* Add comment to explain purpose and method of bit reversal
2.43.0
2024-02-12 10:58:49 -08:00
Erica Fischer e2a7a409c7 Improve tiling speed (#195)
* Add a way to run tippecanoe single-threaded for profiling

* Do less work when the tilestats sample values list is already full

* Save a copy when retrieving the attribute key

* Fewer atomic operations

* Move string hashing from mbtiles to text

* Only do approximate attribute deduplication when writing tiles

* Feature dropping tests are sensitive to exact tile size

* All tile creators now create a string pool for the tile

* Features clipped away to nothing should not participate in that tile

* Revert "Only do approximate attribute deduplication when writing tiles"

This reverts commit c42b34b498.

* Also revert the related test changes

* Revert "Revert "Only do approximate attribute deduplication when writing tiles""

This reverts commit 18509876c3.

* Be more specific about the string hash function

* Use fnv1a instead of std::hash for everything

* Reduce the chance of hash collisions

* Stick a hash search on the front of the tree search in addpool

* Eliminate repeated hashing of the same string

* Switch instead of ifs in json parsing

* A few more cases to populate the hash in addpool

* Store the hash in the tree instead of recalculating

* Add explanatory comment for mysterious argument

* Fewer copies in attribute stringification

* Clean up ancient weirdness in JSON attribute stringification

* More serial_val cleanup

* Pass a serial_feature to rewrite instead of many broken-down arguments

* Get rid of the multiple geometries within `partial`

* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"

This reverts commit 6f4ab9b725.

* Goodbye, struct coalesce

* Revert "Features clipped away to nothing should not participate in that tile"

This reverts commit 124462fbdc.

* Migrating fields from partial to serial_feature

* Name reconciliation between serial_feature and partial

* Replace struct partial with an augmented serial_feature

* Fix some overzealous search-and-replace renaming

* Don't say struct so often

* Remove more of the former partial construction

* Commenting and cleaning up

* Trying again to avoid all these arguments to rewrite

* I swear I did this same thing before and it didn't work.

* More rewrite cleanup

* Exile --detect-shared-borders to its own file

* Add missing headers

* More commenting and cleanup

* More comments

* Sprinkle consts around

* Emplacing and std::moving

* More cleanup

* That shouldn't have worked after a std::move

* Don't need to allocate memory to compare keys

* Reduce use of the global string pool in tiling

* Another avoidable mvt_value construction

* Further reduction to explicit string pool passing

* These reverses are no longer optimizations

* These layernames can all be references

* Don't drag an unused layername string around with every feature

* Heed a compiler warning about potential buffer overflow

* Fix my confusion about which feature's string pool is relevant

* Avoid some unnecessary allocations in attribute accumulation

* Maybe faster serialization?

* Eliminate a comparison

* Do the same here

* Save a couple of allocations when parsing numbers in JSON

* Immediately assign features to layers instead of subdividing later

* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence

* Crunch out more duplicate attribute values when writing out the tile

* Do tilestats for tippecanoe:retain_points_multiplier_first too

* Shell filters need to be real threads, even if nothing else does

* Simplify tippecanoe_minzoom/maxzoom representation

* Update version and changelog
2.42.0
2024-02-07 17:50:35 -08:00
Can Durukandsnyk-bot c5c4698161 [Snyk] Security upgrade centos from 7 to centos8 (#196)
fix: Dockerfile.centos7 to reduce vulnerabilities

The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-CENTOS7-EXPAT-2408807
- https://snyk.io/vuln/SNYK-CENTOS7-EXPAT-3035862
- https://snyk.io/vuln/SNYK-CENTOS7-NSS-2308137
- https://snyk.io/vuln/SNYK-CENTOS7-NSSSYSINIT-2308141
- https://snyk.io/vuln/SNYK-CENTOS7-NSSTOOLS-2308139

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-02-01 14:38:43 -08:00
Erica Fischer 6a2bce8164 Scale the tile size limit up with the multiplier at low zooms (#192)
* Scale the tile size limit up with the multiplier at low zooms

* Add a test to demonstrate that high zoom tiles can't be extra large

* Update changelog and version

* Fail more cleanly when a tile can't be made small enough

* Guard against a cluster where the start marker has been dropped

* Look harder for a working feature interval instead of giving up

* That change to the drop-smallest logic changed a test output

* Update changelog

* Add explanatory comment
2.41.3
2024-01-31 10:23:05 -08:00
David Haberthür ca1aa58a11 Prettify one code line (#187) 2024-01-29 11:34:47 -08:00
Erica Fischer 7d4d264d50 Speeding up tippecanoe-overzoom (#191)
* Speed up mvt_value comparison

* Converting repetitive ifs to cases

* More conversions from ifs to cases

* Optimize the always-true filter case

* Don't convert types of attributes without accumulators

* Unordered map seems to be faster than map

* Add missing header

* Fix some warnings

* Fix the warnings better

* Avoid an int->string->int conversion

* Lazily initialize layer key and values maps when actually needed

* More switches from maps to unordered_maps

* Sure, I'll take the microoptimization

* More emplacement

* Save some copies

* Emplaces and moves

* Lazy linear scan of attributes instead of building a map

* Extra printfs, missing header

* Avoid clipping if the input and output tiles are the same

* But do clip if the tile extent is being reduced

* Make sure I'm not constructing std::strings here at runtime

* More worrying about runtime string construction

* A couple more std::moves

* Const references!

* More const references

* Another std::move

* Make the string_value of mvt_value std::optional

* Reserve storage when decoding

* Provision for different mvt_values to share a string pool

* Use the string pool when decoding

* Avoid another string construction

* Try limiting the depth of the search for duplicate attributes

* Revert "Try limiting the depth of the search for duplicate attributes"

This reverts commit 9ec94a15ff.

* Update changelog

* Fix typo noticed during code review
2024-01-29 11:33:56 -08:00
Erica Fischer cbf222754b Add --accumulate-attribute to tippecanoe-overzoom (#189)
* Starting to factor out attribute accumulation into its own file

* Continuing to factor out attribute accumulation

* Reduce duplicate code

* Plumbing the accumulate-attribute option around

* Call the attribute accumulator

* Test that accumulation works

* Add missing #includes

* Don't sort within individual multiplier clusters

Doing so throws off the spatial distribution of the low zooms

* Docs and changelog

* Add comments
2.41.2
2024-01-23 15:41:36 -08:00
Erica Fischer f957f30f90 Clean up internal naming related to tilestats (#190)
* Get rid of the type_and_string near-synonym for serial_val

* Rename file_keys to the more familiar tilestats

* "tas" (type_and_string) => "sv" (serial_val)

* "fk" (tile_keys) => "ts" (tilestats)

* Revert ""tas" (type_and_string) => "sv" (serial_val)"

This reverts commit 4854c57e22.

* More carefully this time: "tas" (type_and_string) => "sv" (serial_val)
2024-01-23 14:17:19 -08:00
Erica Fischer 679a0d62f2 Make feature ordering cooperate with --retain-points-multiplier (#188)
* Make feature ordering cooperate with --retain-points-multiplier

* Forgot to check in the actual code changes???

* Sort within each multiplier cluster as well as between clusters

* Correct description of behavior in changelog

* Drag original feature sequence along in megatiles for post-filter sort

* Plumb the preserve-input-order flag through overzoom

* Sort in overzoom if requested

* Use within-tile input sequence numbers, not global sequence numbers

* Documentation

* Reverse direction of search to prevent accidental skipping

* Add some comments about converting between attribute representations
2.41.1
2024-01-21 12:09:05 -08:00
Erica Fischer 5d92a17193 Add point retention multiplier (#179)
* Add an option to retain N times as many points as usual at each zoom

* Tests for point multipler with specified and guessed maxzooms

* Work in progress on inverse spatial ordering

* Fix inverse spatial feature order

* --reorder was depending on a feature index that wasn't being preserved

* Separate ordering by feature_minzoom from ordering inverse-spatially

* Add a test for the inverse spatial ordering

* Store the basezoom/droprate/multiplier decisions in tileset metadata

* Progress on adding filters to tippecanoe-overzoom

* Type promotion for comparison

* Look up the attribute value for ordering

* Add test of thinning and ordering features

* Plumb tippecanoe_decisions metadata through pmtiles

* Be careful not to put infinities in JSON

* Fix accidental dropping in what is meant to preserve sparse points

* Start distinguishing true, false, and null in expressions

* Most of the type conversions

* Add boolean conversions

* Literals and conjunctions

* Add filtering to tippecanoe-overzoom

* Add a test of filtering in overzoom

* Fix boolean conjunctions

* Handle the combination of cluster size and filtering

* Rework dot dropping to reconcile density threshold and multiplier

* Revert "Rework dot dropping to reconcile density threshold and multiplier"

This reverts commit f253a66382.

* Retain points by multiplier within each tile, not in global probability

* Test that intends to verify that the multiplier is reversible

* Get the test to detect the discrepancy

* Mark the start of multiplier clusters with a magic attribute

* Add string-contains

* Add in and ni operators

* Revert "Look up the attribute value for ordering"

This reverts commit 56bc73e49a.

* Revert "Type promotion for comparison"

This reverts commit 6f3256f5af.

* Make number formatting in tippecanoe_decisions consistent

* Revert "Add a test for the inverse spatial ordering"

This reverts commit c8047de9ab.

* Revert "Separate ordering by feature_minzoom from ordering inverse-spatially"

This reverts commit 35b19a223c.

* Revert "Fix inverse spatial feature order"

This reverts commit 5978ecdb44.

* Revert "Work in progress on inverse spatial ordering"

This reverts commit fdf230f632.

* Somehow missed the tests associated with that last revert

* Round-robin assign attributes to partials from across the multiplier

* Count the multiplier separately in each layer

* Fix distribution of accumulated attribute across multiplier features

* Update changelog, version, and docs

* Add "is null" and "isnt null" expressions

* Update interpretation of FSL expressions to pass the tests

* Test to assert that polygons are unaffected by the multiplier

* Clean up and comment

* Remove accidental unused case
2.41.0
2024-01-18 15:56:09 -08:00
Erica Fischer e8ca6c6de3 Slightly less compression makes as-needed dropping twice as fast (#182)
* Slightly less compression makes as-needed dropping twice as fast

* Update changelog
2.40.0
2024-01-05 11:07:14 -08:00
Erica Fischer e58892ccfd Replace a pair with a named structure 2023-12-22 14:55:29 -08:00
Erica Fischer 7130d21bf5 Merge remote-tracking branch 'origin/main' into clean-polygons 2023-12-22 14:38:02 -08:00
Erica Fischer 02e3bac2c0 Reduce memory consumption during tiling (#177)
* Drop duplicate geometries sooner in coalescing-as-needed

* Concatenate geometries before partial cleaning

* Spend less time with two feature representations in memory

* Further reduce in-memory duplication

* Remove another copy

* Don't keep duplicates in memory while coalescing

* Also clear out the mvt layer once it is no longer needed

* Update changelog
2.39.0
2023-12-22 14:20:21 -08:00
Erica Fischer 262ab8eb65 Actually, handle collinear verticals and diagonals together here 2023-12-22 13:17:41 -08:00
Erica Fischer 8d9f1e6163 Add special case for collinear verticals 2023-12-22 13:08:35 -08:00
Erica Fischer bfb0104bd1 Sketch of finding intersections by bubbling 2023-12-22 12:19:04 -08:00
Erica Fischer 0cb09dbca4 Merge remote-tracking branch 'origin/main' into clean-polygons 2023-12-22 10:34:48 -08:00
Erica Fischer 15a3e313ff Tolerate polygon rings with insufficiently many points in input (#175)
* Tolerate polygon rings with insufficiently many points in input

* Fix changelog typo
2.38.0
2023-12-11 11:14:26 -08:00
Erica Fischer d7bdbe363b Reduce maximum memory used for vertex sorting (#170)
* Add logging for out-of-memory debugging

* Use less memory for sub-sorting

* Reduce maximum memory used for vertex sorting
2.37.1
2023-11-30 13:48:50 -08:00
Erica Fischer 21f02049da Make polygon cleaning output idempotent 2023-11-16 13:42:41 -08:00
Erica Fischer ef271ad0bb Switch other calls to Wagyu over. But now coalesce is slow. 2023-11-14 18:39:09 -08:00
Erica Fischer 33052aa60b Make polygon scaling its own function 2023-11-14 18:00:03 -08:00
Erica Fischer 9e46ac8c0e Tiebreaker for snap-rounding examination order 2023-11-14 13:24:36 -08:00
Erica Fischer 8a996b9057 Tiebreaker for comparing rings of the same area 2023-11-14 13:19:59 -08:00
Erica Fischer 68d7d907d4 Clean up 2023-11-14 13:15:01 -08:00
Erica Fischer 52e6e7479a So many affected tests 2023-11-14 12:26:32 -08:00
Erica Fischer 78a964776f Don't revive spikes that are beyond the tile edge 2023-11-14 12:19:18 -08:00
Erica Fischer b22d6c8ba1 Remove debug spew 2023-11-14 10:06:42 -08:00
Erica Fischer 624c2b697a Try to fix winding reversals introduced by scaling 2023-11-14 09:56:31 -08:00
Erica Fischer bee87bdab2 Limit decay into polygon dust to reasonably compact polygons 2023-11-13 22:16:37 -08:00
Erica Fischer dd8762da1f Remove dead code 2023-11-13 16:03:52 -08:00
Erica Fischer c05ae1bbff Compare segments to the current pool as they come into scope 2023-11-13 13:08:46 -08:00
Erica Fischer 1238711571 Only reinspect segments that changed in the previous pass 2023-11-13 12:12:58 -08:00
Erica Fischer d23f0a243b Speed improvements from doing less repetitive rounding 2023-11-13 11:31:14 -08:00
Erica Fischer e80d8786b8 Drop holes that are not inside an outer ring 2023-11-13 10:30:14 -08:00
Erica Fischer 546b5fdbd3 Remove collinear points 2023-11-13 10:24:07 -08:00
Erica Fischer bd24168416 Add missing header 2023-11-13 10:10:26 -08:00
Erica Fischer d06363e9f9 Turn polygon revival back on; turn off some debug spew 2023-11-12 23:17:26 -08:00
Erica Fischer c0866b9cde Maybe right now? 2023-11-12 22:51:28 -08:00
Erica Fischer ba9dbda6f6 Something is wrong with the traversal 2023-11-12 19:19:35 -08:00
Erica Fischer 3eddb81d81 Trying to straighten out ring nesting 2023-11-12 18:28:43 -08:00
Erica Fischer 5d6a54a235 Ring nesting isn't right yet, so just keep the outer rings 2023-11-11 22:57:47 -08:00
Erica Fischer 10290b1b1b Committing to integer coordinates to fix more accidental mirror segs 2023-11-10 15:45:25 -08:00