Commit Graph
106 Commits
Author SHA1 Message Date
Erica Fischer e2a7a409c7 Improve tiling speed (#195)
* Add a way to run tippecanoe single-threaded for profiling

* Do less work when the tilestats sample values list is already full

* Save a copy when retrieving the attribute key

* Fewer atomic operations

* Move string hashing from mbtiles to text

* Only do approximate attribute deduplication when writing tiles

* Feature dropping tests are sensitive to exact tile size

* All tile creators now create a string pool for the tile

* Features clipped away to nothing should not participate in that tile

* Revert "Only do approximate attribute deduplication when writing tiles"

This reverts commit c42b34b498.

* Also revert the related test changes

* Revert "Revert "Only do approximate attribute deduplication when writing tiles""

This reverts commit 18509876c3.

* Be more specific about the string hash function

* Use fnv1a instead of std::hash for everything

* Reduce the chance of hash collisions

* Stick a hash search on the front of the tree search in addpool

* Eliminate repeated hashing of the same string

* Switch instead of ifs in json parsing

* A few more cases to populate the hash in addpool

* Store the hash in the tree instead of recalculating

* Add explanatory comment for mysterious argument

* Fewer copies in attribute stringification

* Clean up ancient weirdness in JSON attribute stringification

* More serial_val cleanup

* Pass a serial_feature to rewrite instead of many broken-down arguments

* Get rid of the multiple geometries within `partial`

* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"

This reverts commit 6f4ab9b725.

* Goodbye, struct coalesce

* Revert "Features clipped away to nothing should not participate in that tile"

This reverts commit 124462fbdc.

* Migrating fields from partial to serial_feature

* Name reconciliation between serial_feature and partial

* Replace struct partial with an augmented serial_feature

* Fix some overzealous search-and-replace renaming

* Don't say struct so often

* Remove more of the former partial construction

* Commenting and cleaning up

* Trying again to avoid all these arguments to rewrite

* I swear I did this same thing before and it didn't work.

* More rewrite cleanup

* Exile --detect-shared-borders to its own file

* Add missing headers

* More commenting and cleanup

* More comments

* Sprinkle consts around

* Emplacing and std::moving

* More cleanup

* That shouldn't have worked after a std::move

* Don't need to allocate memory to compare keys

* Reduce use of the global string pool in tiling

* Another avoidable mvt_value construction

* Further reduction to explicit string pool passing

* These reverses are no longer optimizations

* These layernames can all be references

* Don't drag an unused layername string around with every feature

* Heed a compiler warning about potential buffer overflow

* Fix my confusion about which feature's string pool is relevant

* Avoid some unnecessary allocations in attribute accumulation

* Maybe faster serialization?

* Eliminate a comparison

* Do the same here

* Save a couple of allocations when parsing numbers in JSON

* Immediately assign features to layers instead of subdividing later

* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence

* Crunch out more duplicate attribute values when writing out the tile

* Do tilestats for tippecanoe:retain_points_multiplier_first too

* Shell filters need to be real threads, even if nothing else does

* Simplify tippecanoe_minzoom/maxzoom representation

* Update version and changelog
2024-02-07 17:50:35 -08:00
Erica Fischer 2b7bbbdbca Report JSON object context on the same line as the error message (#49)
* Report JSON object context on the same line as the error message

* Update changelog
2022-12-14 12:34:21 -08:00
Erica Fischer af1a7ed7ae Once features can't possibly fit in a tile, stop trying (#9)
* Stop adding features to a tile if it can't possibly work

* Add --integer and --fraction options to tippecanoe-decode

* Carry the strategies field from tileset metadata through tile-join

* Update changelog

* Assign different codes to different kinds of error exits
2022-09-23 20:01:13 -04:00
Erica Fischer 67cd9d8d85 Reduce tippecanoe memory usage (#5)
* Change JSON objects to a union type to use less memory

* Stop storing the string representation of JSON numbers

* Restore the ability to create features with large integer attributes

* Make sure large-integer feature IDs still behave as before

* Add missing #include

* Don't preallocate as much space for arrays and objects

* Treat inability to check free disk space as a warning, not an error

* Update changelog and version
2022-08-09 15:18:51 -07:00
Erica Fischer 6978a0e102 Reindent with clang-format (#3) 2022-07-28 11:19:30 -07:00
Eric Fischer fc810d4156 Fix line numbers in GeoJSON feature parsing error messages 2019-08-30 15:34:24 -07:00
Eric Fischer 53de8a2014 Add an option to convert stringified number feature IDs to numbers 2018-11-01 17:07:33 -07:00
Eric Fischer 0a3e4ad561 Fix missing filename in warning messages 2018-08-16 13:46:37 -07:00
Eric Fischer ba8966a4ae Use the same GeoJSON parsing loop in tippecanoe-json-tool 2018-08-08 13:11:25 -07:00
Eric Fischer 79363e397b Factor out the GeoJSON parsing loop from feature serialization 2018-08-08 13:11:10 -07:00
Eric Fischer d1a0e495ce Require that per-feature maxzoom and minzoom be small integers
Also fixes a place in the Makefile that accidentally still used
CFLAGS instead of CXXFLAGS
2018-07-12 11:24:30 -07:00
Eric Fischer 262fbf26f0 Fix missing null pointer check 2018-06-19 13:10:58 -07:00
Eric Fischer 760a0c9209 Add missing JSON type check; bump version number. 2018-06-06 11:54:23 -07:00
Eric Fischer 1b26becad9 Clear up some confusion about attribute count and external references
Now the count is always adjacent to whereever the key/value pair is
stored, and is not kept in the serial feature object other than as
the length of the vectors of keys and values.
2018-04-05 15:40:14 -07:00
Eric Fischer 9964784e77 Add null pointer check when testing for bare geometry in GeoJSON 2017-11-21 15:01:33 -08:00
Eric Fischer dca1ebbbfb Get rid of some C99-style variable-length arrays 2017-11-09 16:54:06 -08:00
Eric Fischer 50910b211b Remove more unused parameters 2017-11-07 11:09:22 -08:00
Eric Fischer 0801a9324b Silence the projection warning if you asked for --quiet 2017-11-03 15:25:14 -07:00
Eric Fischer c232e565db Working on testing against Mapbox GL JS filter tests 2017-09-01 16:27:27 -07:00
Eric Fischer 5943c82457 Move file-format-neutral code out of JSON-specific source file 2017-08-28 11:10:57 -07:00
Eric Fischer e7ee83f27b Move attribute type coercion out of parsing and into serialization 2017-08-24 17:27:30 -07:00
Eric Fischer f4818ffb07 Move attribute include/exclude logic into serialization 2017-08-24 17:10:15 -07:00
Eric Fischer 34b1b215f4 Move tilestats management out of parsing and into serialization 2017-08-24 16:30:01 -07:00
Eric Fischer ed8fbd0236 Split more serialization details out from being parsing parameters 2017-08-24 15:57:33 -07:00
Eric Fischer b114e22d39 Factor out geometry fixups 2017-08-23 12:48:45 -07:00
Eric Fischer 6caf20b9c8 Put the pieces back together 2017-08-23 11:43:48 -07:00
Eric Fischer 6cea2d5db6 Progress on factoring out serialization state into a single object 2017-08-22 18:10:52 -07:00
Eric Fischer 235dbf57af More progress on splitting apart parsing and serialization 2017-08-22 16:51:11 -07:00
Eric Fischer b1620f6c59 Work on factoring out serial_feature 2017-08-15 10:25:11 -07:00
Eric Fischer 200f6777ba Fix layer list in tilestats when filtering and tile-joining.
(The feature count when filtering will be the sum of features
across tiles instead of filters from the original input, since
the filter reader doesn't know what the original input feature
set was.)
2017-08-08 16:41:20 -07:00
Eric Fischer 635429cd87 Fix dangling pointer. Defer tilestats generation until tiling if filtering. 2017-08-08 13:38:48 -07:00
Eric Fischer c79f19e3ca Merge branch 'master' into plugins 2017-08-08 11:08:10 -07:00
Eric Fischer 62f135a97d Exclude long attribute values from tilestats output, as specified 2017-07-21 12:28:14 -07:00
Eric Fischer 3cafef89f1 Factor out duplicated tilestats code 2017-07-19 14:45:15 -07:00
Eric Fischer 7b03e1ee87 Include tilestats in the metadata table. (Not all tests are passing) 2017-07-19 13:09:47 -07:00
Eric Fischer 9078098511 Only insert a new tilestats record if there isn't already one 2017-07-18 17:28:30 -07:00
Eric Fischer 71f57793fd Derive tilejson attribute types from sample attribute values 2017-07-17 17:31:46 -07:00
Eric Fischer 77bf14bfb0 Track sample values for each attribute in each layer 2017-07-17 13:28:46 -07:00
Eric Fischer b1771a3365 Make layer attributes a map to sample values 2017-07-17 11:58:37 -07:00
Eric Fischer 7445feb845 Work in the direction of tracking representative attribute values 2017-07-14 17:59:24 -07:00
Eric Fischer 62ee53992b Track how many features of each type are in each layer 2017-07-14 17:23:41 -07:00
Eric Fischer 65c095cc2b Clean up #includes and add fields for counting attributes 2017-07-14 16:56:23 -07:00
Eric Fischer 04d0cc6fa1 Merge branch 'master' into plugins 2017-07-06 22:28:35 -07:00
Eric Fischer c7d146ea48 Don't do work for guessing maxzoom unless needed. Add a better test. 2017-05-30 16:58:56 -07:00
Eric Fischer ab8ecb7e00 Add missing #include 2017-05-30 13:41:24 -07:00
Eric Fischer 331707f88a Better maxzoom guessing by considering resolution within features 2017-05-30 13:28:25 -07:00
Eric Fischer 5a8bfa23a4 Merge branch 'master' into plugins 2017-04-21 16:18:00 -07:00
Eric Fischer d19680e392 Try to be clearer in the warning message about projections 2017-04-18 13:24:16 -07:00
Eric Fischer 7dc586bc84 Merge branch 'master' into plugins 2017-04-17 16:01:12 -07:00
Eric Fischer 178b5d0054 Add -T option to coerce the types of feature attributes 2017-04-17 15:20:03 -07:00