Commit Graph
1890 Commits
Author SHA1 Message Date
Erica Fischer 98bca0b742 Refactor to give tile-join access to overzoom() 2023-08-18 10:46:59 -07:00
Erica Fischer faeb328e84 At least when to overzoom, even if not actually doing it yet 2023-08-17 16:27:24 -07:00
Erica Fischer 989cd76403 Add an option to extend zooms if still dropping, but with a limit 2023-08-17 12:44:14 -07:00
Bert Temme 2b133a1def Update Dockerfile typo tippacanoe -> tippecanoe (#129) 2023-08-14 14:07:15 -07:00
Brandon Liu f00eeb486a Fix narrowing of index_size (#126)
Fix narrowing of index_size, affecting indexed flatgeobufs with lots of features [#125]
2023-08-14 14:06:38 -07:00
Can Duruk 9e9e21bcc0 Remove Can from the codeowners (#128)
I am not immediately qualified to be a reviewer on this repo right now. I am still watching updates on Slack.
2023-08-14 14:05:59 -07:00
Erica Fischer 430d8edd17 Tool for overzooming individual tiles (#121)
* Starting work on overzooming

* Factor clipping out of geometry.cpp to simplify linkage

* Pull out more geometry functions into now-badly-named clip.cpp

* Not surprisingly, there is a bug

* Found the bug

* Make indent

* Pass attributes through

* Make formatting more consistent

* Fix typos in comments

* Fix the typos better

* Add tippecanoe-overzoom to the install list

* Give overzoom a predictable exit status

* Forgot to translate to and from polygon ring closepaths

* Fix geometry collapse at z21

* Add attribute stripping; don't generate layers if they have no features

* Add docs for tippecanoe-overzoom (as it will be, not as it is)

* Change overzoom to accept input and output files as arguments

* Working on overzooming tests

* Oops

* Hook up and test the detail and buffer options, and the empty-tile case

* Also test attribute stripping

* Fix error message

* Update changelog and version
2.29.0
2023-08-09 11:48:16 -07:00
Philippe Latulippe 5cba91981a Include errno.h in memfile.cpp (#122)
Include errno.h in memfile.cpp to fix compilation errors on some platforms
2023-07-27 18:34:22 -07:00
Erica Fischer 55b9aa501c Allow --set-attribute to override existing attribute values (#123)
* Allow --set-attribute to override existing attribute values

* Bump version number
2.28.1
2023-07-26 11:05:35 -07:00
Erica Fischer 4318e964e3 Add maximum pseudocluster size option for low-zoom points (#119)
* Trying to improve cluster positions

* Fix centroid calculation

* Just move points to cluster centroids; add point gap threshold

* Add maximum-point-gap option

* Fix formatting

* Rename options for clarity; add tests; update changelog

* Add missing break to switch

* Revert unneeded constructor cleanup that broke a test somehow

* Clean up comments

* Remove unused --move-points-to-cluster-centroids

* Mention tile-join change in changelog
2.28.0
2023-07-17 15:33:07 -07:00
Erica Fischer 1c3bd5352c Add a new option to set an initial value for an attribute in each feature (#115)
* Starting on --set-attributes

* Add a JSON form to accumulate-attribute

* Fix formatting

* Set attributes, JSON form

* Also handle setting attributes to unquoted values

* Factoring out character-mangling in the names of tests

* Rename tests for new quoting convention

* Test for JSON form of attribute setting

* Test for JSON form of attribute accumulation

* Forgot to increment the version
2023-07-14 10:47:55 -07:00
Erica Fischer 48e769c94c More codeowners (#118)
* Add more code owners

* Fix typo
2023-07-12 10:34:53 -07:00
John SeekinsandJohn Seekins 58bfd121a7 fix build on fedora 38 (#117)
Signed-off-by: John Seekins <john@pluralpolicy.com>
Co-authored-by: John Seekins <john@pluralpolicy.com>
2023-07-12 10:34:26 -07:00
Erica Fischer b5ec3c46f0 Reduce excessive progress logging during pmtiles conversion (#111)
* Don't log progress so often during pmtiles conversion

And turn off the `catch` tests, which have bit-rotted

* Fix accidental double-multiplication-by-100

* Reenable catch

* Update changelog
2023-07-10 15:59:52 -07:00
Erica Fischer c1051232ad Do more of line simplification with integer coordinates (#113)
* Turn off unit, which has bit-rotted

* Instrument line simplification

* Do more work in integer space

* Update tests with simplification done with more integers

* Clean up

* Update changelog

* Eradicate calls to deprecated sprintf()

* Revert "Turn off unit, which has bit-rotted"

This reverts commit 998779313f.

* Fix catch by switching to an unsigned type
2023-07-10 15:54:21 -07:00
Erica Fischer b09a880822 Add CODEOWNERS (#114)
* Add CODEOWNERS

* Narrower list?
2023-07-10 15:53:13 -07:00
Mary M ba3e42568c Try catch exception (#108)
* catch exception when parsing invalid pbf - fixes #817

* updating exit error
2023-07-05 14:03:02 -07:00
James Scott-Brown 87ed674a30 Add backticks around an option in README (#109) 2023-07-05 14:02:14 -07:00
Tim FinerandTim Finer 0422d77bbe Check for nullptr in call to properties, fixes segfault. (#106)
* Added -fPIE as g++ 9.4.0 complained and would not link

* Prevent crash by checking for properties before dereferencing.

---------

Co-authored-by: Tim Finer <finert@pixia.com>
2023-05-23 09:47:45 -07:00
Erica Fischer 20eb9c8dae Avoid crashing if there is a polygon ring containing only one point (#102) 2.26.1 2023-05-11 12:20:00 -07:00
Erica Fischer 523ca55243 Fix bugs in --no-simplification-of-shared-nodes (#99)
* Fix bugs in --no-simplification-of-shared-nodes

* Update changelog
2.26.0
2023-05-08 15:31:46 -07:00
Erica Fischer 605640e138 Add --include/-y to tile-join (#86)
* Add --include/-y to tile-join

* Update documentation and changelog
2023-05-04 11:55:36 -07:00
Jt Miclat 5bd46e7ed5 Update dockerfile (#79)
Update docker image to use newer version of ubuntu and reduce image size
2023-05-04 11:54:27 -07:00
James 434817bde9 Update makefile to use c++17 standard (#94)
The C++ standard didn't match the utilization inside the code and would fail to compile using the original makefile
2023-05-04 10:49:00 -07:00
James Scott-Brown da7bbbd6ea Change fpclassify() to std::fpclassify() (#92)
Change fpclassify() std::fpclassify()

This fixes the compiler errors "'fpclassify' was not declared in this scope"
2023-05-04 10:47:46 -07:00
Brandon Liu e43d6dacdd update pmtiles.hpp from upstream (#98)
* remove unused code [#93]

* fix OOB caused by specific entry set lengths

* add more checks for valid tile IDs
2023-05-04 10:45:22 -07:00
Erica Fischer afc7ca01a2 Calculate an antimeridian-adjusted bounding box in tileset metadata (#82)
* Calculate a new antimeridian-adjusted bounding box

* Add antimeridian bounding box to pmtiles, dirtiles, and tile-join

* Don't take out-of-bounds latitudes into account in the adjusted bbox

* Update changelog

* Forgot to adjust tests after the last change
2.24.0
2023-03-09 20:26:48 -08:00
Erica FischerandAurele Nitoref 329dd86bfd Add --cluster-maxzoom option (#75)
* Add `--cluster-maxzoom` option

* Add tests, update changelog

* Update manpage

* Update tests

---------

Co-authored-by: Aurele Nitoref <aurele.nitoref@icloud.com>
2023-02-24 16:05:45 -08:00
Erica FischerandAurele Nitoref 3d714e32b0 Add point_count_abbreviated property to clustered features (#74)
* Add `point_count_abbreviated` property to clustered features

* Update tests and changelog

---------

Co-authored-by: Aurele Nitoref <aurele.nitoref@icloud.com>
2023-02-24 15:59:00 -08:00
DarcyB 7371072e45 Make Tippecanoe build on FreeBSD in response to issue 18 (#72)
* Make tippecanoe build on FreeBSD in responce to issue 18 (https://github.com/felt/tippecanoe/issues/18)

* Make conditional FreeBSD specific, instead of otherway around.
2023-02-24 11:32:56 -08:00
DarcyB 0fa758c1e3 Allow passing a file with a list of input files to tile-join in response to issue 69 (#73)
* Allow passing a file with a list of input files to tile-join in response to Issue 69 (https://github.com/felt/tippecanoe/issues/69)

* Do not print usage if -r is specified without a commandline readfrom file

* Add test for -r option to tile-join
2023-02-24 11:31:46 -08:00
Erica Fischer f2127fec97 Reduce thrashing during feature ingestion and tiling (#56)
* Remove the concept of "separate metadata"

This was an extra level of attribute indirection (features point
to metadata records which point to key and value strings) which was
intended to reduce the size of temporary storage for features with
large numbers of attributes that were also spread across large numbers
of tiles at maxzoom.

For other kinds of features, the extra indirection slowed things down
instead, and, especially when maxzoom guessing was being used, many more
features were having their metadata externalized than could actually
benefit from it.

* Shave a few bytes off temporary files by using more unsigned integers

* Flush stderr after logging progress

* Revert "Shave a few bytes off temporary files by using more unsigned integers"

This reverts commit eef29084ec.

* Limit the size of the string pools and trees to fit in memory

* Add missing #include

* Move the string pool and search tree from mmap to allocated memory

* Sort in allocated rather than mapped memory too

* Also use pread instead of mapping to read in the data to sort

* When the pool gets too big, switch to just the file, not memory

* Switch string pool from memory to disk when memory is 10% full

* Add to-memory versions of the serialization functions

* Crashy work in progress toward compression

* Fix the pointer bug that was causing the crash

* Serialize features into memory rather than straight to disk

* Compress individual features in the temporary files

* Don't need to store the length of the geometry

* Remove per-feature compression; move minzoom back into the object

* Start adding a stream compressor object

* Track file position within fwrite_check()

* Add compressed stream writer functions

* Pull the writing of the serialized feature out to the callers

* Starting toward compression again from a different point

* Hook up more compression functions

* Remove unused code from the other day

* Make enough deflate calls to flush out all the buffered data

* Start on decompression

* Tile number is uncompressed, tile content is compressed

* Work on alternating compressed and uncompressed in decompression

* Closer, but still doesn't work

* Sort of works

* Works until we get to concatenated tiles

* More attempts that don't work

* One bug down

* It made a tileset!

* Handle nonzero initial zooms

* Fix seeking within compressed feature streams

* Tests pass!

* Remove debug spew

* Oops: remember to delete the temporary files so they don't hang around

* Test that fails with the current compression code

* Properly account for bytes read while closing the compressed stream

* Limit the number of warnings about bad label points

* A little more armor when closing decompression

* This time for sure

* A different, less fragile, test that failed previously with compression

* Move feature stream compression to its own file

* Remove now-unused code to deserialize from a file

* Forgot to add the new files

* Remove a little debugging logging

* Add a couple of comments on what it means to be within decompression

* Fix indentation

* Update changelog. Remove stray debugging comment.
2.23.0
2023-02-14 12:47:40 -08:00
Erica Fischer b155b4671b It only needs to look for other small features when coalescing, not dropping (#64)
* Back out the slow search for other small features when dropping features

* Actually, *do* look for small features still when coalescing

* Update changelog
2.22.0
2023-01-27 12:04:36 -08:00
Erica Fischer e615668475 Avoid placing polygon labels in holes (#62)
* Avoid placing polygon labels in holes

* Traverse vertices in hilbert order to find potential label points

* Update test to reflect new label placements

* Sorting by Y coordinate is better and easier than Hilbert order

* Give a bonus for being near the center of mass

* Make label points a little more border-shy

* Tall places, not just wide places; border lines, not just border points

* Clean up

* Try diagonals through the features too

* Limit the search for label points to prevent slowing down too much

* At this point the center of mass bonus is doing more harm than good

* Forgot to update test
2.21.0
2023-01-27 11:31:57 -08:00
Erica Fischer c58a8e3b97 Round coordinates instead of truncating them (#60)
* Round coordinates instead of truncating them

* Update all the tests for coordinate rounding changes

* Curses, integer division still truncates

* Fix tests

* Also round instead of shifting when looking for no-op linetos

* Should I worry that the same change for moveto doesn't change any tests?

* Also round instead of shifting when scaling down to maxzoom resolution

* Replace another explicit shift, for origin point

* Round instead of shift when writing clipped geometries to the next zoom

* Fix low-zoom gridding and smaller-than-a-pixel checks

* Don't guess an excessively large maxzoom when there is only one feature

* Add a test for guessing the maxzoom of a single point

* Explicitly sort by index if no other order distinguishes features

* Another affected test
2.20.0
2023-01-27 10:57:22 -08:00
Brandon Liu 1c576c2dc1 Add a few PMTiles details to README; regenerate man page (#58)
* Add details on pmtiles input/output to a few parts of the README. [#57]

* regenerate man page [#57]
2023-01-27 10:44:54 -08:00
Erica Fischer 3cb6e4dd5e Set base zoom if requested as part of maxzoom guessing (#63)
* Set base zoom if requested as part of maxzoom guessing

* Add a test for guessing the maxzoom of a single point

* Don't guess an excessively large maxzoom when there is only one feature

* Update changelog
2.19.0
2023-01-26 15:27:06 -08:00
Erica Fischer 54b47a6757 Fix crash when using tile-join to copy an empty pmtiles tileset (#61)
* Fix crash when using tile-join to copy an empty pmtiles tileset

* Update changelog
2.18.0
2023-01-17 15:16:22 -08:00
Erica FischerandBrandon Liu 3b2599f587 Add support for pmtiles output format (#55)
* add pmtiles.hpp from github.com/protomaps/PMTiles [#10]

* tippecanoe main writes pmtiles output. [#10]

* detect output format using suffix
* after mbtiles is done writing, replace with pmtiles based on map/image tables.
* add method to write_json for writing json sub-object.

* tippecanoe-decode reads pmtiles input. [#10]

* tile-join reads and writes pmtiles. [#10]

* pmtiles test suite for decode and tile-join [#10]

* add base GitHub CI action for compiling and test suite.

* update pmtiles.hpp with z>15 fix

* Fix some ordering problems with pmtiles decode

* Pmtiles should also pass the raw tiles tests

* Eradicate spaces from tileset metadata JSON fields

* Eradicate spaces from more test fixtures

* Update more tests

* Pmtiles tests pass now too

* Remove unnecessary sort (and make indent)

* Update changelog

* The allow-existing test for pmtiles needs -o, not -e

* Declare --allow-existing to be unsupported for pmtiles.

It was always a bad idea even for mbtiles.

Co-authored-by: Brandon Liu <bdon@bdon.org>
2.17.0
2022-12-29 12:00:13 -08:00
Sam Hashemi a374263466 Update README (#54) 2022-12-28 14:06:42 -08:00
Erica Fischer 079929717e Avoid spending gigabytes of memory on statistics for -as-needed dropping (#50)
* Downsample indices and areas during tiling if they get too big

* Cap the indices rather than downsampling them
2.16.0
2022-12-20 12:54:21 -08:00
Erica Fischer 2b7bbbdbca Report JSON object context on the same line as the error message (#49)
* Report JSON object context on the same line as the error message

* Update changelog
2022-12-14 12:34:21 -08:00
Brandon Liu bfabf8325f change hash function to fnv1a (#48)
change mbtiles hash function from hashCode to fnv1a.
2022-12-12 13:33:55 -08:00
Erica Fischer 9c77029b63 Add a data structure for tileset metadata (#44)
* Progress toward making a tileset metadata structure

* Write metadata from structure to mbtiles

* Write dirtiles metadata.json from metadata structure

* Update changelog
2022-12-12 13:32:02 -08:00
Brandon Liu b9af922650 Change mbtiles tile_id inserts from blob to text (#41)
Change mbtiles tile_id inserts from blob to text to enable querying by text.

* previous blob type worked for internal join, but made querying by text impossible without casting.
2022-12-12 13:28:51 -08:00
Erica Fischer f1df09f147 Generate fewer duplicate label points at high zoom levels (#42) 2.15.0 2022-12-02 11:02:33 -08:00
Erica Fischer 5c647cdb83 Don't preflight zoom levels for potential as-needed dropping (#40)
* Working on eliminating preflighting

* Adjust for the map/images schema change

* Avoid generating duplicate tiles with the detail reduction strategy

* Do error checking if tiles in a directory can't be written

* Update changelog and version

* Revert unintentional code reordering

* Neglected to add the exit on error here
2.14.0
2022-11-29 12:42:30 -08:00
Erica Fischer 3095adb467 Simplify geometry earlier when the in-memory representation of a tile gets large, to reduce peak memory usage (#38)
* Simplify geometries earlier if the in-memory tile starts getting big

* Update changelog
2.13.1
2022-11-22 14:57:07 -08:00
Erica Fischer 8eec2be462 Improve quality of coalescing-as-needed; add options to limit feature count (#25)
* Add an option to limit geometry vertex count

* Update docs and changelog

* Track desired feature count and geometry size in strategies

* Remove dead code for a long-forgotten inaccessible option

* Fix the option name in changelog

* Refine coalesce-smallest to only coalesce onto other small features

* Clean polygons before coalescing-as-needed

* Don't accumulate tiny polygon holes as negative dust

* Add the option not to limit the feature count at maxzoom

* Add options to limit feature count more abruptly in each tile

* Revert "Add the option not to limit the feature count at maxzoom"

This reverts commit ace173ab7a.

* Revert "Fix the option name in changelog"

This reverts commit 38dbce8405.

* Revert "Add an option to limit geometry vertex count"

This reverts commit d379fdf06b.

* Remove test for reverted option

* Remove more leftovers from geometry size limiting

* Update changelog
2.13.0
2022-11-17 13:09:10 -08:00
Erica Fischer 7733cced43 Add an option to drop more points in dense areas than in sparse areas (#35)
* Add an option to drop the densest features by rate, not a fraction

* Restrict dot dropping just to dots, as it should be

* 50% mix of sparse and uniform point preservation

* Update version, changelog, test, and documentation

* 60% sparse, 40% dense

* Didn't update the test after the last change

* Make the drop-denser rate configurable without recompiling
2.12.0
2022-11-16 10:55:07 -08:00