Commit Graph
1857 Commits
Author SHA1 Message Date
Brandon Liu 1ac6afa543 update pmtiles.hpp with z>15 fix 2022-12-22 07:57:12 +08:00
Brandon Liu f8319ee5f1 add base GitHub CI action for compiling and test suite. 2022-12-21 15:03:08 +08:00
Brandon Liu f1c0bf50b1 pmtiles test suite for decode and tile-join [#10] 2022-12-21 15:02:44 +08:00
Brandon Liu e958f1adf9 tile-join reads and writes pmtiles. [#10] 2022-12-21 14:58:49 +08:00
Brandon Liu 64a5a684b0 tippecanoe-decode reads pmtiles input. [#10] 2022-12-21 14:57:54 +08:00
Brandon Liu fedebec55a tippecanoe main writes pmtiles output. [#10]
* detect output format using suffix
* after mbtiles is done writing, replace with pmtiles based on map/image tables.
* add method to write_json for writing json sub-object.
2022-12-21 14:56:46 +08:00
Brandon Liu f6332f302a add pmtiles.hpp from github.com/protomaps/PMTiles [#10] 2022-12-21 14:53:34 +08:00
Erica Fischer 079929717e Avoid spending gigabytes of memory on statistics for -as-needed dropping (#50)
* Downsample indices and areas during tiling if they get too big

* Cap the indices rather than downsampling them
2.16.0
2022-12-20 12:54:21 -08:00
Erica Fischer 2b7bbbdbca Report JSON object context on the same line as the error message (#49)
* Report JSON object context on the same line as the error message

* Update changelog
2022-12-14 12:34:21 -08:00
Brandon Liu bfabf8325f change hash function to fnv1a (#48)
change mbtiles hash function from hashCode to fnv1a.
2022-12-12 13:33:55 -08:00
Erica Fischer 9c77029b63 Add a data structure for tileset metadata (#44)
* Progress toward making a tileset metadata structure

* Write metadata from structure to mbtiles

* Write dirtiles metadata.json from metadata structure

* Update changelog
2022-12-12 13:32:02 -08:00
Brandon Liu b9af922650 Change mbtiles tile_id inserts from blob to text (#41)
Change mbtiles tile_id inserts from blob to text to enable querying by text.

* previous blob type worked for internal join, but made querying by text impossible without casting.
2022-12-12 13:28:51 -08:00
Erica Fischer f1df09f147 Generate fewer duplicate label points at high zoom levels (#42) 2.15.0 2022-12-02 11:02:33 -08:00
Erica Fischer 5c647cdb83 Don't preflight zoom levels for potential as-needed dropping (#40)
* Working on eliminating preflighting

* Adjust for the map/images schema change

* Avoid generating duplicate tiles with the detail reduction strategy

* Do error checking if tiles in a directory can't be written

* Update changelog and version

* Revert unintentional code reordering

* Neglected to add the exit on error here
2.14.0
2022-11-29 12:42:30 -08:00
Erica Fischer 3095adb467 Simplify geometry earlier when the in-memory representation of a tile gets large, to reduce peak memory usage (#38)
* Simplify geometries earlier if the in-memory tile starts getting big

* Update changelog
2.13.1
2022-11-22 14:57:07 -08:00
Erica Fischer 8eec2be462 Improve quality of coalescing-as-needed; add options to limit feature count (#25)
* Add an option to limit geometry vertex count

* Update docs and changelog

* Track desired feature count and geometry size in strategies

* Remove dead code for a long-forgotten inaccessible option

* Fix the option name in changelog

* Refine coalesce-smallest to only coalesce onto other small features

* Clean polygons before coalescing-as-needed

* Don't accumulate tiny polygon holes as negative dust

* Add the option not to limit the feature count at maxzoom

* Add options to limit feature count more abruptly in each tile

* Revert "Add the option not to limit the feature count at maxzoom"

This reverts commit ace173ab7a.

* Revert "Fix the option name in changelog"

This reverts commit 38dbce8405.

* Revert "Add an option to limit geometry vertex count"

This reverts commit d379fdf06b.

* Remove test for reverted option

* Remove more leftovers from geometry size limiting

* Update changelog
2.13.0
2022-11-17 13:09:10 -08:00
Erica Fischer 7733cced43 Add an option to drop more points in dense areas than in sparse areas (#35)
* Add an option to drop the densest features by rate, not a fraction

* Restrict dot dropping just to dots, as it should be

* 50% mix of sparse and uniform point preservation

* Update version, changelog, test, and documentation

* 60% sparse, 40% dense

* Didn't update the test after the last change

* Make the drop-denser rate configurable without recompiling
2.12.0
2022-11-16 10:55:07 -08:00
Erica Fischer 622084a507 Limit guessed maxzoom to avoid spending too many tiles on polygon fill (#23)
* Change sqlite3 schema to deduplicate identical tiles

* Limit guessed maxzoom to avoid spending too many tiles on polygon fill

* Fix test.

These dust polygons now have their area calculated because their
maxzoom is being guessed, so the attributes from the largest one
rather than the last one are preserved.

* Increase polygon limit to a million tiles

* Two million tiles ought to be enough for anyone, right?

* Add explanatory comments for mysterious numbers
2.11.0
2022-11-07 10:14:17 -08:00
Erica Fischer 1584543224 Fix the flatgeobuf build (#30) 2022-11-03 14:43:00 -07:00
zstadler 4317c826a9 Fix docker build compilation (#29)
Fixes #28
2022-11-03 14:24:31 -07:00
James Chevalier c2214631f2 Update flatbuffers to v22.9.29 (#26)
* update flatbuffers files

How:
- Copied files in from the latest release (v22.9.29) of flatbuffers

Why:
- The files in place blocked compilation within Alpine Linux

Links:
- https://github.com/felt/tippecanoe/issues/24
- https://github.com/google/flatbuffers/releases/tag/v22.9.29

* update flatgeobuf generate files

How
- Created `feature_generated.h` & `header_generated.h` using the `src/fbs` files from the latest release (3.23.1) of flatgeobuf
    - Downloaded the `flatc` binary from the v22.9.29 release of flatbuffers
    - Copied `flatc` into the flatgeobuf repository
    - Created a new directory named `out` to collect the generated files
    - `./flatc --cpp -o out -I include src/fbs/feature.fbs`
    - `./flatc --cpp -o out -I include src/fbs/header.fbs`
    - Moved the generate files from `out` into this repository's `flatgeobuf` directory

Why
- This is a required step in the process of updating flatbuffers files

Links
- https://github.com/flatgeobuf/flatgeobuf/releases/tag/3.23.1
- https://github.com/google/flatbuffers/releases/tag/v22.9.29
- https://github.com/felt/tippecanoe/issues/24#issuecomment-1286476550
2022-11-03 14:24:02 -07:00
Erica Fischer 5b03b18f31 Generate label points after simplification, not before. (#27)
* Generate labels points after simplification, not before.

Previously there were some cases where dropping the smallest
polygons would never reduce the number of labels.

* Also wait until after polygon cleaning to make labels

* Remove the extra newlines after the TILES ONLY COMPLETE message
2.9.1
2022-10-21 13:59:36 -07:00
Erica Fischer b763625862 Add an option to generate label points in place of polygons (#20)
* Add an option to generate label points in place of polygons

* Change all these places where I said "extent" but really meant "area"

* Revert "Change all these places where I said "extent" but really meant "area""

This reverts commit 403828d2f7.

* Add --order-smallest-first and --order-largest-first options

* Use Turf's center-of-mass algorithm for polygon label points

* If the label point isn't within the polygon, find one that is

* Don't choose a label point that is too close to a border

* Try a little harder to find an optimal label point

* Checkerboard which tiles labels are generated in, to reduce adjacency

* Use a label point for the general representative point for polygons

(Skipping the iteration to find one that is as far as possible from
the borders)

This makes the labels look better in many cases (like France at z1)
but unfortunately ripples into changing the sequence of polygons in
many tests, so the diff is big.

* Revert "Use a label point for the general representative point for polygons"

This reverts commit 2261adf05e.

* Checkpoint work on spiral labels

* Clip label spirals to the feature bounds

* Fix label test

* Be careful not to place spiral labels too close to borders either

* For spiral anchors, only check tile scale, not feature size

* Update test

* Only try to find a central label point for the largest ring

* In tiny polygon dust, keep the attributes of the largest feature
2.9.0
2022-10-13 14:21:36 -07:00
Erica FischerandRoman Karavia 182093bdc7 Improve precision of polygon area calculations (#19)
* Improve precision of get_area by using long double

* Trying to get consistent polygon area results between ARM and x86

* Calculate polygon area closer to the origin for better precision

* Update changelog

* Also exercise tiny polygon dust in the ring area test

They previously behaved differently here between x86 and ARM

* On M1 Macs, long double is just double anyway, so don't use it

* Be more careful about overflow: scale the polygon ring down into range

* Fix the bug I just introduced in the scaled area calculation

* Use only the sign from the scaled-down area calculation

Co-authored-by: Roman Karavia <47303530+romankaravia@users.noreply.github.com>
2.8.1
2022-10-04 10:04:34 -07:00
Erica Fischer a6abb0bc30 Add the option to use a different simplification level at maxzoom (#17)
* Add an option to specify a different simplification at maxzoom

* Add test
2022-10-03 12:18:02 -07:00
Erica Fischer 4ef430b913 Add an option for the Visvalingam simplification algorithm (#16)
* First attempt at porting Paul Mach's Visvalingam implementation

* Make indent

* Mostly working

* Approximate equivalence from Douglas-Peucker to Visvalingam levels

* Don't simplify away tile boundary crossing points

* Add a command-line option and test for Visvalingam simplification

* Update changelog

* Cleanup in response to review feedback

* Include <stdio.h> to fix compiler warnings
2022-09-30 12:00:07 -07:00
Erica Fischer 6de00c15ec Fix tests that I missed updating in 2.6.2 (#15) 2022-09-27 10:37:02 -07:00
Erica Fischer 3086838a10 Fix wrong-way comparison that was causing buffer overflow crash in tile-join (#14)
* Fix wrong-way comparison that was causing buffer overflow in tile-join

* Update changelog
2.6.3
2022-09-27 09:15:54 -07:00
Erica Fischer af1a7ed7ae Once features can't possibly fit in a tile, stop trying (#9)
* Stop adding features to a tile if it can't possibly work

* Add --integer and --fraction options to tippecanoe-decode

* Carry the strategies field from tileset metadata through tile-join

* Update changelog

* Assign different codes to different kinds of error exits
2.6.2
2022-09-23 20:01:13 -04:00
Vincent Sarago ed31b9ac84 Update README.md (#13)
protomaps -> felt
2022-09-23 19:54:57 -04:00
Erica Fischer 8e46650da7 Upgrade to protozero 1.7.1 (#5)
* Upgrade to protozero 1.7.1

* Bump version number
2022-09-13 11:24:01 -07:00
Erica Fischer 073700aa38 Reduce maxzooms being guessed a little: (#2)
* Reduce maxzooms being guessed a little:

* Use 1.5 standard deviations, not 2, as the minimum distinguishable
* Give overlapping polygons and linestrings more distinct indices

* Add another drop rate guessing options, from the same metrics -zg uses

* Guard against using -rp without -zg
2.6.0
2022-09-08 15:47:32 -07:00
Erica Fischer a447dfc089 Extra coordinate precision; feature ordering; compression improvements
* Add an option to retain extra coordinate precision at maxzoom

* Make sure not to shift away the extra detail from coordinates

* Add an option to convert double-precision attributes to single

* Sort attribute values in tiles to make them compress a little better

* Slightly improve polygon simplification

By choosing a point that would be retained after simplification
to be the start/end point that always gets retained

* I regret making all of these tests involve polygons

* Add an option to specify the size of tiny polygons

* Fix accidental requiring of argument for --single-precision

* Guard against duplicate points when generating "sizes" for them

* Restore the intended behavior that tiny polygons don't get simplified

* Make the extra detail settable rather than always maximizing it

* Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)"

This reverts commit fec5e8354c.

* Add an option to prevent choosing a base zoom higher than the maxzoom

* Keep the drop rate high enough when the basezoom gets constrained

* Revert "Revert "Improve maxzoom guessing for tightly-clustered point data sources (#4)""

This reverts commit db6bc27d9e.

* Add --order-by and --order-descending options

* Accept multiple --order-by and --order-descending-by sort keys
2.5.0
2022-09-06 13:08:11 -07:00
Erica Fischer 4ea8a37611 Accept tilestats limiting options in tile-join, not just tippecanoe (#8) 2.4.1 2022-08-12 13:57:04 -07:00
Erica Fischer fec5e8354c Improve maxzoom guessing for tightly-clustered point data sources (#4)
* Improve maxzoom guessing for tightly-clustered point data sources

* Go back to the old distance estimate, since it is less mysterious

* Update changelog and version
2022-08-11 10:13:54 -07:00
Erica Fischer a9bf6cacd4 Add --smallest-maximum-zoom-guess to specify a minimum maxzoom to guess (#2)
* Add --maximum-zoom-at-least to specify a minimum maxzoom to guess

* Check whether the specified minimum maxzoom is out of range

* Rename the option to --smallest-maximum-zoom-guess

* Update tests, changelog, and version
2022-08-11 10:04:03 -07:00
Erica Fischer 86f3e98c02 Track the desired tile size for each zoom level in tileset metadata (#7) 2022-08-11 09:58:00 -07:00
Erica Fischer c65d6938a4 Drop or coalesce points too when dropping or coalescing smallest (#6)
* Handle points too when dropping or coalescing the "smallest" features.

* Add statistics of tile size reduction strategies to tileset metadata

* Update changelog and version

* Update documentation
2022-08-09 15:29:03 -07:00
Erica Fischer 67cd9d8d85 Reduce tippecanoe memory usage (#5)
* Change JSON objects to a union type to use less memory

* Stop storing the string representation of JSON numbers

* Restore the ability to create features with large integer attributes

* Make sure large-integer feature IDs still behave as before

* Add missing #include

* Don't preallocate as much space for arrays and objects

* Treat inability to check free disk space as a warning, not an error

* Update changelog and version
2022-08-09 15:18:51 -07:00
Erica Fischer 6978a0e102 Reindent with clang-format (#3) 2022-07-28 11:19:30 -07:00
Brandon Liu e6ab02ddd3 lambda default progress is public [#13] 2022-07-06 12:21:12 +08:00
Brandon Liu fbc7b02123 improvements to lambda wrapper [#13] 2022-07-05 21:26:13 +08:00
Brandon Liu f442e403c5 add logging struct with -u json options flag, log tiling progress [#13] 2022-07-05 15:32:11 +08:00
Brandon Liu 4adb22b74e suppress flatgeobuf message 2022-07-05 15:32:00 +08:00
Brandon Liu 9875ae97a8 add lambda info 2022-05-17 15:29:38 +08:00
Brandon Liu d04bd09d11 fix struct name collision [#12] 2022-04-27 22:14:22 +08:00
Brandon Liu 7c726e4d2c add missing column types [#2] 2022-04-23 13:26:26 +08:00
Brandon Liu 170ddf73ff comments cleanup 2022-04-02 12:28:28 +08:00
Brandon Liu 7f678502af FlatGeobuf: read files with heterogeneous geometry types [#8] 2022-04-02 12:07:22 +08:00
Brandon Liu 19241f8373 FlatGeobuf: if fgb has index, assign feature IDs based on sequence number [#2] 2022-03-30 12:37:55 +08:00