Commit Graph
221 Commits
Author SHA1 Message Date
Erica Fischer b89cc9f425 Make enough deflate calls to flush out all the buffered data 2023-01-27 13:43:25 -08:00
Erica Fischer 69ebf3e988 Merge remote-tracking branch 'origin/main' into compression 2023-01-27 13:24:40 -08:00
Erica Fischer b155b4671b It only needs to look for other small features when coalescing, not dropping (#64)
* Back out the slow search for other small features when dropping features

* Actually, *do* look for small features still when coalescing

* Update changelog
2023-01-27 12:04:36 -08:00
Erica Fischer c58a8e3b97 Round coordinates instead of truncating them (#60)
* Round coordinates instead of truncating them

* Update all the tests for coordinate rounding changes

* Curses, integer division still truncates

* Fix tests

* Also round instead of shifting when looking for no-op linetos

* Should I worry that the same change for moveto doesn't change any tests?

* Also round instead of shifting when scaling down to maxzoom resolution

* Replace another explicit shift, for origin point

* Round instead of shift when writing clipped geometries to the next zoom

* Fix low-zoom gridding and smaller-than-a-pixel checks

* Don't guess an excessively large maxzoom when there is only one feature

* Add a test for guessing the maxzoom of a single point

* Explicitly sort by index if no other order distinguishes features

* Another affected test
2023-01-27 10:57:22 -08:00
Erica Fischer 8b0490ba35 Hook up more compression functions 2023-01-26 22:07:44 -08:00
Erica Fischer a7bc8856fe Starting toward compression again from a different point 2023-01-26 21:33:52 -08:00
Erica Fischer 12e3af018b Pull the writing of the serialized feature out to the callers 2023-01-25 14:41:22 -08:00
Erica Fischer 484dc27c5e Compress individual features in the temporary files 2023-01-24 13:24:36 -08:00
Erica Fischer 21c946dd49 Serialize features into memory rather than straight to disk 2023-01-24 12:19:58 -08:00
Erica Fischer f59f8588eb Revert "Shave a few bytes off temporary files by using more unsigned integers"
This reverts commit eef29084ec.
2023-01-03 10:55:42 -08:00
Erica Fischer fc8007a2ab Flush stderr after logging progress 2022-12-30 12:38:37 -08:00
Erica Fischer eef29084ec Shave a few bytes off temporary files by using more unsigned integers 2022-12-30 11:46:03 -08:00
Erica Fischer d05f8f8e4c Remove the concept of "separate metadata"
This was an extra level of attribute indirection (features point
to metadata records which point to key and value strings) which was
intended to reduce the size of temporary storage for features with
large numbers of attributes that were also spread across large numbers
of tiles at maxzoom.

For other kinds of features, the extra indirection slowed things down
instead, and, especially when maxzoom guessing was being used, many more
features were having their metadata externalized than could actually
benefit from it.
2022-12-29 14:47:27 -08:00
Erica FischerandBrandon Liu 3b2599f587 Add support for pmtiles output format (#55)
* add pmtiles.hpp from github.com/protomaps/PMTiles [#10]

* tippecanoe main writes pmtiles output. [#10]

* detect output format using suffix
* after mbtiles is done writing, replace with pmtiles based on map/image tables.
* add method to write_json for writing json sub-object.

* tippecanoe-decode reads pmtiles input. [#10]

* tile-join reads and writes pmtiles. [#10]

* pmtiles test suite for decode and tile-join [#10]

* add base GitHub CI action for compiling and test suite.

* update pmtiles.hpp with z>15 fix

* Fix some ordering problems with pmtiles decode

* Pmtiles should also pass the raw tiles tests

* Eradicate spaces from tileset metadata JSON fields

* Eradicate spaces from more test fixtures

* Update more tests

* Pmtiles tests pass now too

* Remove unnecessary sort (and make indent)

* Update changelog

* The allow-existing test for pmtiles needs -o, not -e

* Declare --allow-existing to be unsupported for pmtiles.

It was always a bad idea even for mbtiles.

Co-authored-by: Brandon Liu <bdon@bdon.org>
2022-12-29 12:00:13 -08:00
Erica Fischer 079929717e Avoid spending gigabytes of memory on statistics for -as-needed dropping (#50)
* Downsample indices and areas during tiling if they get too big

* Cap the indices rather than downsampling them
2022-12-20 12:54:21 -08:00
Erica Fischer f1df09f147 Generate fewer duplicate label points at high zoom levels (#42) 2022-12-02 11:02:33 -08:00
Erica Fischer 5c647cdb83 Don't preflight zoom levels for potential as-needed dropping (#40)
* Working on eliminating preflighting

* Adjust for the map/images schema change

* Avoid generating duplicate tiles with the detail reduction strategy

* Do error checking if tiles in a directory can't be written

* Update changelog and version

* Revert unintentional code reordering

* Neglected to add the exit on error here
2022-11-29 12:42:30 -08:00
Erica Fischer 3095adb467 Simplify geometry earlier when the in-memory representation of a tile gets large, to reduce peak memory usage (#38)
* Simplify geometries earlier if the in-memory tile starts getting big

* Update changelog
2022-11-22 14:57:07 -08:00
Erica Fischer 8eec2be462 Improve quality of coalescing-as-needed; add options to limit feature count (#25)
* Add an option to limit geometry vertex count

* Update docs and changelog

* Track desired feature count and geometry size in strategies

* Remove dead code for a long-forgotten inaccessible option

* Fix the option name in changelog

* Refine coalesce-smallest to only coalesce onto other small features

* Clean polygons before coalescing-as-needed

* Don't accumulate tiny polygon holes as negative dust

* Add the option not to limit the feature count at maxzoom

* Add options to limit feature count more abruptly in each tile

* Revert "Add the option not to limit the feature count at maxzoom"

This reverts commit ace173ab7a.

* Revert "Fix the option name in changelog"

This reverts commit 38dbce8405.

* Revert "Add an option to limit geometry vertex count"

This reverts commit d379fdf06b.

* Remove test for reverted option

* Remove more leftovers from geometry size limiting

* Update changelog
2022-11-17 13:09:10 -08:00
Erica Fischer 5b03b18f31 Generate label points after simplification, not before. (#27)
* Generate labels points after simplification, not before.

Previously there were some cases where dropping the smallest
polygons would never reduce the number of labels.

* Also wait until after polygon cleaning to make labels

* Remove the extra newlines after the TILES ONLY COMPLETE message
2022-10-21 13:59:36 -07:00
Erica Fischer b763625862 Add an option to generate label points in place of polygons (#20)
* Add an option to generate label points in place of polygons

* Change all these places where I said "extent" but really meant "area"

* Revert "Change all these places where I said "extent" but really meant "area""

This reverts commit 403828d2f7.

* Add --order-smallest-first and --order-largest-first options

* Use Turf's center-of-mass algorithm for polygon label points

* If the label point isn't within the polygon, find one that is

* Don't choose a label point that is too close to a border

* Try a little harder to find an optimal label point

* Checkerboard which tiles labels are generated in, to reduce adjacency

* Use a label point for the general representative point for polygons

(Skipping the iteration to find one that is as far as possible from
the borders)

This makes the labels look better in many cases (like France at z1)
but unfortunately ripples into changing the sequence of polygons in
many tests, so the diff is big.

* Revert "Use a label point for the general representative point for polygons"

This reverts commit 2261adf05e.

* Checkpoint work on spiral labels

* Clip label spirals to the feature bounds

* Fix label test

* Be careful not to place spiral labels too close to borders either

* For spiral anchors, only check tile scale, not feature size

* Update test

* Only try to find a central label point for the largest ring

* In tiny polygon dust, keep the attributes of the largest feature
2022-10-13 14:21:36 -07:00
Erica Fischer a6abb0bc30 Add the option to use a different simplification level at maxzoom (#17)
* Add an option to specify a different simplification at maxzoom

* Add test
2022-10-03 12:18:02 -07:00
Erica Fischer af1a7ed7ae Once features can't possibly fit in a tile, stop trying (#9)
* Stop adding features to a tile if it can't possibly work

* Add --integer and --fraction options to tippecanoe-decode

* Carry the strategies field from tileset metadata through tile-join

* Update changelog

* Assign different codes to different kinds of error exits
2022-09-23 20:01:13 -04:00
Erica Fischer a447dfc089 Extra coordinate precision; feature ordering; compression improvements
* Add an option to retain extra coordinate precision at maxzoom

* Make sure not to shift away the extra detail from coordinates

* Add an option to convert double-precision attributes to single

* Sort attribute values in tiles to make them compress a little better

* Slightly improve polygon simplification

By choosing a point that would be retained after simplification
to be the start/end point that always gets retained

* I regret making all of these tests involve polygons

* Add an option to specify the size of tiny polygons

* Fix accidental requiring of argument for --single-precision

* Guard against duplicate points when generating "sizes" for them

* Restore the intended behavior that tiny polygons don't get simplified

* Make the extra detail settable rather than always maximizing it

* Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)"

This reverts commit fec5e8354c.

* Add an option to prevent choosing a base zoom higher than the maxzoom

* Keep the drop rate high enough when the basezoom gets constrained

* Revert "Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)""

This reverts commit db6bc27d9e.

* Add --order-by and --order-descending options

* Accept multiple --order-by and --order-descending-by sort keys
2022-09-06 13:08:11 -07:00
Erica Fischer 86f3e98c02 Track the desired tile size for each zoom level in tileset metadata (#7) 2022-08-11 09:58:00 -07:00
Erica Fischer c65d6938a4 Drop or coalesce points too when dropping or coalescing smallest (#6)
* Handle points too when dropping or coalescing the "smallest" features.

* Add statistics of tile size reduction strategies to tileset metadata

* Update changelog and version

* Update documentation
2022-08-09 15:29:03 -07:00
Erica Fischer 6978a0e102 Reindent with clang-format (#3) 2022-07-28 11:19:30 -07:00
Brandon Liu f442e403c5 add logging struct with -u json options flag, log tiling progress [#13] 2022-07-05 15:32:11 +08:00
Eric Fischer ed9f844cef Fix calculation of mean when accumulating attributes in clusters 2020-02-06 15:28:10 -08:00
Eric Fischer fc335e2221 Be careful to avoid undefined behavior from shifting negative numbers 2019-06-17 17:50:25 -07:00
Eric Fischer 42ee195bc3 Add an option to keep intersection nodes from being simplified away 2019-05-13 17:39:07 -07:00
Eric Fischer 4d3e30773f Be more consistent about when longitudes beyond 180 are allowed 2019-04-12 12:12:14 -07:00
Eric Fischer ffab876002 Don't run shell filters if the current zoom is below the minzoom 2019-04-08 16:30:19 -07:00
Eric Fischer ca4aad0c64 Point features may now be coalesced into MultiPoint features with --coalesce 2018-12-11 17:09:13 -08:00
Eric Fischer 5229990523 Fix gridding of small squares; use less memory for small features 2018-11-21 15:34:43 -08:00
Eric Fischer 5dfbc4a281 Add --clip-bounding-box option to clip input geometry 2018-10-22 16:49:33 -07:00
Eric Fischer 1b75ca33b4 Bug fix: mismatched features could be coalesced if from a prefilter 2018-10-10 14:35:46 -07:00
Eric Fischer 759bceb986 Duplicate features within the buffer of the antimeridian on both sides 2018-07-27 16:14:49 -07:00
Eric Fischer 8d6bd4e10f Add a filter extension to allow filtering individual attributes 2018-06-18 11:26:22 -07:00
Eric Fischer b0229530ac Turn off progress indicators during tests
(For the sake of Travis's 4MB log limit)

Also fix a few places that logged progress even when told not to.
2018-06-07 13:27:49 -07:00
Eric Fischer c9eed0d457 Be careful to remove null attributes from prefilter/postfilter output 2018-06-06 11:48:57 -07:00
Eric Fischer 09565b4632 Fix arithmetic overflows 2018-05-11 17:29:29 -07:00
Eric Fischer 59dd095607 Make file positions and lengths thread-safe 2018-05-07 14:42:49 -07:00
Eric Fischer 8ba884530a Also take note of --force-feature-limit if extending zooms 2018-05-04 12:43:27 -07:00
Eric Fischer 1b26becad9 Clear up some confusion about attribute count and external references
Now the count is always adjacent to whereever the key/value pair is
stored, and is not kept in the serial feature object other than as
the length of the vectors of keys and values.
2018-04-05 15:40:14 -07:00
Eric Fischer d01d8177a0 Allow filtering features by zoom level in conditional expressions
This moves filtering from the serialization stage to the
tiling stage so that the zoom level can be known to the filter.

The side effect is to carry null attributes much further through
the pipeline than previously.
2018-04-05 14:15:19 -07:00
Eric Fischer 1c4aa57ffd Change the JSON writing functions to object methods 2018-03-16 13:43:21 -07:00
Eric Fischer 623e0c90fe More regularization of JSON writing 2018-03-15 17:07:26 -07:00
Eric Fischer fac0ebbf52 All the other places where I used volatile but really wanted atomic 2018-03-13 15:21:21 -07:00
Eric Fischer 834c05038c Add --progress-interval setting to reduce progress indicator frequency 2018-03-13 14:51:48 -07:00