Commit Graph
36 Commits
Author SHA1 Message Date
Erica Fischer 563f7d2bc2 Make drop-densest more consistent across tile boundaries 2023-08-23 13:20:25 -07:00
Erica Fischer 523ca55243 Fix bugs in --no-simplification-of-shared-nodes (#99)
* Fix bugs in --no-simplification-of-shared-nodes

* Update changelog
2023-05-08 15:31:46 -07:00
Erica Fischer afc7ca01a2 Calculate an antimeridian-adjusted bounding box in tileset metadata (#82)
* Calculate a new antimeridian-adjusted bounding box

* Add antimeridian bounding box to pmtiles, dirtiles, and tile-join

* Don't take out-of-bounds latitudes into account in the adjusted bbox

* Update changelog

* Forgot to adjust tests after the last change
2023-03-09 20:26:48 -08:00
Erica Fischer f2127fec97 Reduce thrashing during feature ingestion and tiling (#56)
* Remove the concept of "separate metadata"

This was an extra level of attribute indirection (features point
to metadata records which point to key and value strings) which was
intended to reduce the size of temporary storage for features with
large numbers of attributes that were also spread across large numbers
of tiles at maxzoom.

For other kinds of features, the extra indirection slowed things down
instead, and, especially when maxzoom guessing was being used, many more
features were having their metadata externalized than could actually
benefit from it.

* Shave a few bytes off temporary files by using more unsigned integers

* Flush stderr after logging progress

* Revert "Shave a few bytes off temporary files by using more unsigned integers"

This reverts commit eef29084ec.

* Limit the size of the string pools and trees to fit in memory

* Add missing #include

* Move the string pool and search tree from mmap to allocated memory

* Sort in allocated rather than mapped memory too

* Also use pread instead of mapping to read in the data to sort

* When the pool gets too big, switch to just the file, not memory

* Switch string pool from memory to disk when memory is 10% full

* Add to-memory versions of the serialization functions

* Crashy work in progress toward compression

* Fix the pointer bug that was causing the crash

* Serialize features into memory rather than straight to disk

* Compress individual features in the temporary files

* Don't need to store the length of the geometry

* Remove per-feature compression; move minzoom back into the object

* Start adding a stream compressor object

* Track file position within fwrite_check()

* Add compressed stream writer functions

* Pull the writing of the serialized feature out to the callers

* Starting toward compression again from a different point

* Hook up more compression functions

* Remove unused code from the other day

* Make enough deflate calls to flush out all the buffered data

* Start on decompression

* Tile number is uncompressed, tile content is compressed

* Work on alternating compressed and uncompressed in decompression

* Closer, but still doesn't work

* Sort of works

* Works until we get to concatenated tiles

* More attempts that don't work

* One bug down

* It made a tileset!

* Handle nonzero initial zooms

* Fix seeking within compressed feature streams

* Tests pass!

* Remove debug spew

* Oops: remember to delete the temporary files so they don't hang around

* Test that fails with the current compression code

* Properly account for bytes read while closing the compressed stream

* Limit the number of warnings about bad label points

* A little more armor when closing decompression

* This time for sure

* A different, less fragile, test that failed previously with compression

* Move feature stream compression to its own file

* Remove now-unused code to deserialize from a file

* Forgot to add the new files

* Remove a little debugging logging

* Add a couple of comments on what it means to be within decompression

* Fix indentation

* Update changelog. Remove stray debugging comment.
2023-02-14 12:47:40 -08:00
Erica Fischer 622084a507 Limit guessed maxzoom to avoid spending too many tiles on polygon fill (#23)
* Change sqlite3 schema to deduplicate identical tiles

* Limit guessed maxzoom to avoid spending too many tiles on polygon fill

* Fix test.

These dust polygons now have their area calculated because their
maxzoom is being guessed, so the attributes from the largest one
rather than the last one are preserved.

* Increase polygon limit to a million tiles

* Two million tiles ought to be enough for anyone, right?

* Add explanatory comments for mysterious numbers
2022-11-07 10:14:17 -08:00
Erica Fischer b763625862 Add an option to generate label points in place of polygons (#20)
* Add an option to generate label points in place of polygons

* Change all these places where I said "extent" but really meant "area"

* Revert "Change all these places where I said "extent" but really meant "area""

This reverts commit 403828d2f7.

* Add --order-smallest-first and --order-largest-first options

* Use Turf's center-of-mass algorithm for polygon label points

* If the label point isn't within the polygon, find one that is

* Don't choose a label point that is too close to a border

* Try a little harder to find an optimal label point

* Checkerboard which tiles labels are generated in, to reduce adjacency

* Use a label point for the general representative point for polygons

(Skipping the iteration to find one that is as far as possible from
the borders)

This makes the labels look better in many cases (like France at z1)
but unfortunately ripples into changing the sequence of polygons in
many tests, so the diff is big.

* Revert "Use a label point for the general representative point for polygons"

This reverts commit 2261adf05e.

* Checkpoint work on spiral labels

* Clip label spirals to the feature bounds

* Fix label test

* Be careful not to place spiral labels too close to borders either

* For spiral anchors, only check tile scale, not feature size

* Update test

* Only try to find a central label point for the largest ring

* In tiny polygon dust, keep the attributes of the largest feature
2022-10-13 14:21:36 -07:00
Eric Fischer e805d113d0 Remove now-unused filter argument during parsing 2019-01-22 14:26:06 -08:00
Eric Fischer f070c7464b Add missing #include 2018-05-07 15:10:18 -07:00
Eric Fischer 59dd095607 Make file positions and lengths thread-safe 2018-05-07 14:42:49 -07:00
Eric Fischer 1b26becad9 Clear up some confusion about attribute count and external references
Now the count is always adjacent to whereever the key/value pair is
stored, and is not kept in the serial feature object other than as
the length of the vectors of keys and values.
2018-04-05 15:40:14 -07:00
Eric Fischer fac0ebbf52 All the other places where I used volatile but really wanted atomic 2018-03-13 15:21:21 -07:00
Eric Fischer a8a342f701 Send dot-dropping through the same pipeline.
The first feature in a tile can never be dropped, since there is
no previous feature to attach its properties to.

Remove the previous special case that reset the dropping counter
at the first feature within each tile proper (as opposed to the
first feature in each tile, including its buffer, which is now
the one that is guaranteed to be preserved).
2018-02-23 17:19:54 -08:00
Eric Fischer 0152db4a20 More initializers 2017-11-07 15:57:56 -08:00
Eric Fischer 4f974b3dc6 Less verbose initializer syntax 2017-11-07 15:25:54 -08:00
Eric Fischer ba62ab8596 More structure initializers 2017-11-07 15:20:17 -08:00
Eric Fischer 0fd4454129 Allow filter expressions during tippecanoe as well as during tile-join 2017-09-01 11:51:12 -07:00
Eric Fischer 5943c82457 Move file-format-neutral code out of JSON-specific source file 2017-08-28 11:10:57 -07:00
Eric Fischer b98bf6e8c7 Get attribute value decoding working 2017-08-25 15:46:32 -07:00
Eric Fischer e7ee83f27b Move attribute type coercion out of parsing and into serialization 2017-08-24 17:27:30 -07:00
Eric Fischer f4818ffb07 Move attribute include/exclude logic into serialization 2017-08-24 17:10:15 -07:00
Eric Fischer 34b1b215f4 Move tilestats management out of parsing and into serialization 2017-08-24 16:30:01 -07:00
Eric Fischer ed8fbd0236 Split more serialization details out from being parsing parameters 2017-08-24 15:57:33 -07:00
Eric Fischer 6caf20b9c8 Put the pieces back together 2017-08-23 11:43:48 -07:00
Eric Fischer 6cea2d5db6 Progress on factoring out serialization state into a single object 2017-08-22 18:10:52 -07:00
Eric Fischer c79f19e3ca Merge branch 'master' into plugins 2017-08-08 11:08:10 -07:00
Eric Fischer 65c095cc2b Clean up #includes and add fields for counting attributes 2017-07-14 16:56:23 -07:00
Eric Fischer 6a5461763c Fix reordering of attributes and failure to update layer name table 2016-12-20 16:41:23 -08:00
Eric Fischer 0e5b513637 Start getting features (just geometry so far) back from the prefilter 2016-12-09 15:35:57 -08:00
Eric Fischer 569825324a Factor out feature deserialization 2016-12-08 17:11:37 -08:00
Eric Fischer 013e6512b4 Add an option to drop the smallest features to make tiles small enough 2016-11-09 17:09:05 -08:00
Eric Fischer 2e3ba8f374 Retain original feature index rather than recalculating
For better density calculation of clipped features
2016-11-02 15:11:22 -07:00
Eric Fischer c8a1b082e0 Don't serialize the per-feature minzoom until geometry merging time 2016-10-10 15:31:09 -07:00
Eric Fischer 475ce9dd23 Fix g++ compiler warnings 2016-08-08 17:14:48 -07:00
Eric Fischer bf571571a9 Factor out (initial) feature serialization 2016-08-08 15:36:49 -07:00
Eric Fischer 488dff0efb Encode the "id" field of GeoJSON objects as vector tile feature ID 2016-07-15 15:00:56 -07:00
Eric Fischer f3b9e15267 Move serialization code to its own file 2016-04-27 14:19:10 -07:00