Add --generate-variable-depth-tile-pyramid option (#251)

* Track output position at the file level instead of within each tile

* Track file position where the child tile data begins

* Add option and document its intended behavior

* Changing the detail loop to account for stopping early

* I forgot I already added an option for this

* Stop early if we can make a complete tile

* Add a test of zoom truncation with limited feature count

* Forgot to commit the actual code change

* Make room for a vertex count in the header of each serialized tile

* Estimate tile complexity; don't try truncating when unlikely to work

* Be more conservative, because ever retrying a tile is a big speed hit

* If stopping early, don't simplify or clean; leave that to overzoom

* Add tiny polygon reduction / dust to overzoom

* Don't try to stop early in the children if we dropped anything by rate

* Fflush here too before pwriting

* Don't stop early if we ended up dropping any features.

Rework the can-the-next-zoom-stop-early logic to avoid going
one zoom further than needed.

* Fix warning

* Fix warnings

* Oops, checking for the wrong expected return value

* Cleanup from adding line simplification in overzoom

* Current (wrong) behavior when combining coalescing and truncating

* Keep a list of parent tiles to skip rather than truncating

* Now the coalesced tiles in z12 get children in z13

* Don't double-count feature dropping when the zoom level is retried

* Correct README description

* Remove todo about special case below basezoom, which is accounted for

* Be a little more aggressive in drop-densest determination

* Scale tile feature limit for megatiles in the same way as byte limit

* Fully deprecate -detect-shared-borders into an alias

* Track the distances found in the douglas-peucker recursion

* Serialize and deserialize the distance with the vertices

* Revert "Serialize and deserialize the distance with the vertices"

This reverts commit 753f1b7909.

* Revert "Track the distances found in the douglas-peucker recursion"

This reverts commit e5361f8c22.

* Revert "Fully deprecate -detect-shared-borders into an alias"

This reverts commit 0698aeb766.

* Better tracking of whether we failed to make a full-detail tile

* Put a bloom filter in front of the binary search for shared nodes

* Forgot to take out this printf

* Improve dispatch of tiling tasks

* Still dispatch the biggest tasks first

* Track zoom truncation in the strategies list in the tileset metadata

* Prescan for small deltas before doing proper simplification

* Revert "Prescan for small deltas before doing proper simplification"

This reverts commit d1d8238b83.

* Update version and changelog

* Rename to --generate-variable-depth-tile-pyramid

tests/ne_110m_admin_0_countries/out/-ae_-zg_-M5000_--force-feature-limit.json

@@ -9,7 +9,7 @@
 "maxzoom": "5",
 "minzoom": "0",
 "name": "tests/ne_110m_admin_0_countries/out/-ae_-zg_-M5000_--force-feature-limit.json.check.mbtiles",
-"strategies": "[{\"dropped_as_needed\":324,\"tile_size_desired\":39239},{\"dropped_as_needed\":160,\"tiny_polygons\":2,\"tile_size_desired\":25163},{\"dropped_as_needed\":124,\"tiny_polygons\":2,\"tile_size_desired\":21214},{\"dropped_as_needed\":80,\"tile_size_desired\":10758},{\"dropped_as_needed\":12,\"tile_size_desired\":6601},{\"tiny_polygons\":2}]",
+"strategies": "[{\"dropped_as_needed\":165,\"tile_size_desired\":39239},{\"dropped_as_needed\":160,\"tiny_polygons\":1,\"tile_size_desired\":25163},{\"dropped_as_needed\":124,\"tiny_polygons\":1,\"tile_size_desired\":21214},{\"dropped_as_needed\":80,\"tile_size_desired\":10758},{\"dropped_as_needed\":12,\"tile_size_desired\":6601},{\"tiny_polygons\":2}]",
 "tippecanoe_decisions": "{\"basezoom\":0,\"droprate\":2.5,\"retain_points_multiplier\":1}",
 "type": "overlay",
 "version": "2"

tests/ne_110m_admin_0_countries/out/-z5_-M5000_--coalesce-fraction-as-needed.json

@@ -9,7 +9,7 @@
 "maxzoom": "5",
 "minzoom": "0",
 "name": "tests/ne_110m_admin_0_countries/out/-z5_-M5000_--coalesce-fraction-as-needed.json.check.mbtiles",
-"strategies": "[{\"coalesced_as_needed\":528,\"detail_reduced\":2,\"tiny_polygons\":2,\"tile_size_desired\":39240},{\"coalesced_as_needed\":216,\"tile_size_desired\":25160},{\"coalesced_as_needed\":213,\"tile_size_desired\":21223},{\"coalesced_as_needed\":204,\"tile_size_desired\":10749},{\"coalesced_as_needed\":126,\"tile_size_desired\":6591},{\"tiny_polygons\":1}]",
+"strategies": "[{\"coalesced_as_needed\":176,\"detail_reduced\":2,\"tiny_polygons\":1,\"tile_size_desired\":39240},{\"coalesced_as_needed\":216,\"tile_size_desired\":25160},{\"coalesced_as_needed\":213,\"tile_size_desired\":21223},{\"coalesced_as_needed\":204,\"tile_size_desired\":10749},{\"coalesced_as_needed\":126,\"tile_size_desired\":6591},{\"tiny_polygons\":1}]",
 "type": "overlay",
 "version": "2"
 }, "features": [

tests/ne_110m_admin_0_countries/out/-z5_-M5000_--coalesce-smallest-as-needed.json

@@ -9,7 +9,7 @@
 "maxzoom": "5",
 "minzoom": "0",
 "name": "tests/ne_110m_admin_0_countries/out/-z5_-M5000_--coalesce-smallest-as-needed.json.check.mbtiles",
-"strategies": "[{\"coalesced_as_needed\":7519,\"tiny_polygons\":2,\"tile_size_desired\":39240},{\"coalesced_as_needed\":442,\"tile_size_desired\":25160},{\"coalesced_as_needed\":214,\"tiny_polygons\":1,\"tile_size_desired\":21223},{\"coalesced_as_needed\":179,\"tile_size_desired\":10749},{\"coalesced_as_needed\":73,\"tile_size_desired\":6591},{\"tiny_polygons\":1}]",
+"strategies": "[{\"coalesced_as_needed\":175,\"tiny_polygons\":1,\"tile_size_desired\":39240},{\"coalesced_as_needed\":212,\"tile_size_desired\":25160},{\"coalesced_as_needed\":214,\"tiny_polygons\":1,\"tile_size_desired\":21223},{\"coalesced_as_needed\":179,\"tile_size_desired\":10749},{\"coalesced_as_needed\":73,\"tile_size_desired\":6591},{\"tiny_polygons\":1}]",
 "type": "overlay",
 "version": "2"
 }, "features": [