Commit Graph
26 Commits
Author SHA1 Message Date
Erica Fischer 1e2813ed65 Generalize overzooming functions to multiple tiles 2023-12-07 11:25:06 -08:00
Erica Fischer c1051232ad Do more of line simplification with integer coordinates (#113)
* Turn off unit, which has bit-rotted

* Instrument line simplification

* Do more work in integer space

* Update tests with simplification done with more integers

* Clean up

* Update changelog

* Eradicate calls to deprecated sprintf()

* Revert "Turn off unit, which has bit-rotted"

This reverts commit 998779313f.

* Fix catch by switching to an unsigned type
2023-07-10 15:54:21 -07:00
Erica Fischer f2127fec97 Reduce thrashing during feature ingestion and tiling (#56)
* Remove the concept of "separate metadata"

This was an extra level of attribute indirection (features point
to metadata records which point to key and value strings) which was
intended to reduce the size of temporary storage for features with
large numbers of attributes that were also spread across large numbers
of tiles at maxzoom.

For other kinds of features, the extra indirection slowed things down
instead, and, especially when maxzoom guessing was being used, many more
features were having their metadata externalized than could actually
benefit from it.

* Shave a few bytes off temporary files by using more unsigned integers

* Flush stderr after logging progress

* Revert "Shave a few bytes off temporary files by using more unsigned integers"

This reverts commit eef29084ec.

* Limit the size of the string pools and trees to fit in memory

* Add missing #include

* Move the string pool and search tree from mmap to allocated memory

* Sort in allocated rather than mapped memory too

* Also use pread instead of mapping to read in the data to sort

* When the pool gets too big, switch to just the file, not memory

* Switch string pool from memory to disk when memory is 10% full

* Add to-memory versions of the serialization functions

* Crashy work in progress toward compression

* Fix the pointer bug that was causing the crash

* Serialize features into memory rather than straight to disk

* Compress individual features in the temporary files

* Don't need to store the length of the geometry

* Remove per-feature compression; move minzoom back into the object

* Start adding a stream compressor object

* Track file position within fwrite_check()

* Add compressed stream writer functions

* Pull the writing of the serialized feature out to the callers

* Starting toward compression again from a different point

* Hook up more compression functions

* Remove unused code from the other day

* Make enough deflate calls to flush out all the buffered data

* Start on decompression

* Tile number is uncompressed, tile content is compressed

* Work on alternating compressed and uncompressed in decompression

* Closer, but still doesn't work

* Sort of works

* Works until we get to concatenated tiles

* More attempts that don't work

* One bug down

* It made a tileset!

* Handle nonzero initial zooms

* Fix seeking within compressed feature streams

* Tests pass!

* Remove debug spew

* Oops: remember to delete the temporary files so they don't hang around

* Test that fails with the current compression code

* Properly account for bytes read while closing the compressed stream

* Limit the number of warnings about bad label points

* A little more armor when closing decompression

* This time for sure

* A different, less fragile, test that failed previously with compression

* Move feature stream compression to its own file

* Remove now-unused code to deserialize from a file

* Forgot to add the new files

* Remove a little debugging logging

* Add a couple of comments on what it means to be within decompression

* Fix indentation

* Update changelog. Remove stray debugging comment.
2023-02-14 12:47:40 -08:00
Erica Fischer af1a7ed7ae Once features can't possibly fit in a tile, stop trying (#9)
* Stop adding features to a tile if it can't possibly work

* Add --integer and --fraction options to tippecanoe-decode

* Carry the strategies field from tileset metadata through tile-join

* Update changelog

* Assign different codes to different kinds of error exits
2022-09-23 20:01:13 -04:00
Erica Fischer a447dfc089 Extra coordinate precision; feature ordering; compression improvements
* Add an option to retain extra coordinate precision at maxzoom

* Make sure not to shift away the extra detail from coordinates

* Add an option to convert double-precision attributes to single

* Sort attribute values in tiles to make them compress a little better

* Slightly improve polygon simplification

By choosing a point that would be retained after simplification
to be the start/end point that always gets retained

* I regret making all of these tests involve polygons

* Add an option to specify the size of tiny polygons

* Fix accidental requiring of argument for --single-precision

* Guard against duplicate points when generating "sizes" for them

* Restore the intended behavior that tiny polygons don't get simplified

* Make the extra detail settable rather than always maximizing it

* Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)"

This reverts commit fec5e8354c.

* Add an option to prevent choosing a base zoom higher than the maxzoom

* Keep the drop rate high enough when the basezoom gets constrained

* Revert "Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)""

This reverts commit db6bc27d9e.

* Add --order-by and --order-descending options

* Accept multiple --order-by and --order-descending-by sort keys
2022-09-06 13:08:11 -07:00
Erica Fischer 6978a0e102 Reindent with clang-format (#3) 2022-07-28 11:19:30 -07:00
Eric Fischer 32b14b1269 Limit detail to 30 and buffer 127 to prevent coordinate delta overflow 2018-12-18 15:53:11 -08:00
Eric Fischer 07ab900393 Report decompression errors in tippecanoe-decode 2018-07-13 17:02:49 -07:00
Eric Fischer 2318673108 Be more systematic about nulls and undefined attribute values
It is an error to write a null attribute to a tile, but
null attributes found within existing tiles will be decoded
into JSON as null.
2018-06-06 11:48:01 -07:00
Eric Fischer fda0e1f28a Fix more cases of loss of precision for large magnitude integers 2017-11-09 13:56:07 -08:00
Eric Fischer aa7191b1ee Also test large integers. Work around an apparent bug in strtoull. 2017-11-09 12:52:38 -08:00
Eric Fischer 87ce5b9310 Be more careful about checking for overflow when parsing numbers 2017-11-09 12:11:07 -08:00
Eric Fischer c232e565db Working on testing against Mapbox GL JS filter tests 2017-09-01 16:27:27 -07:00
Eric Fischer 5665d08745 Support per-feature minzoom and maxzoom in geobuf 2017-08-30 11:44:57 -07:00
Eric Fischer b301512860 Merge branch 'master' into plugins 2017-05-12 17:20:30 -07:00
Eric Fischer 30d54ff50d Tile-join had not been compressing its tiles. 2017-05-11 12:08:47 -07:00
Eric Fischer 94a15eaf85 Merge branch 'master' into plugins 2017-04-07 14:28:58 -07:00
Shan-Chun Kuo aee12ac1fe Add flag --raw-tiles or -pC to get raw protobuf 2017-04-05 00:34:54 +01:00
Eric Fischer 7d47226444 Merge branch 'master' into plugins 2017-03-28 16:42:22 -07:00
Eric Fischer 8cf81483b1 Add missing #include 2016-12-08 10:31:27 -08:00
Eric Fischer 3f14a0dd55 Factor out conversion from JSON types to vector tile attribute types 2016-12-07 16:17:17 -08:00
Eric Fischer 488dff0efb Encode the "id" field of GeoJSON objects as vector tile feature ID 2016-07-15 15:00:56 -07:00
Eric Fischer 2bc1b9bd91 Support feature IDs for decoding 2016-07-15 13:58:15 -07:00
Eric Fischer c40ec6c194 Fix the tile layer version number in tile-join output 2016-06-16 12:31:48 -07:00
Eric Fischer f1b3f6d231 Fix warnings about shadowed variables 2016-05-03 15:48:42 -07:00
Eric Fischer b12413eddb Rename everything from .cc to .cpp to match other projects 2016-04-27 12:22:47 -07:00