mirror of
https://github.com/felt/tippecanoe.git
synced 2026-10-02 08:25:40 +02:00
Fix three latent defects exposed by compiler warnings, and clear the rest (#406)
* Fix variable-length-array and uninitialized-union compiler warnings Clang warns about every variable-length array in C++ (-Wvla-cxx-extension, on by default), since VLAs are a compiler extension rather than standard C++. Replace all 57 of them with std::vector, or with std::string for the mkstemp() template buffers built from tmpdir. Add -Wvla to WARNING_FLAGS so new ones don't creep back in. Separately, mvt_value's numeric_value union is 16 bytes wide (the size of string_value), but both constructors only wrote the 8 bytes of the member they were setting, leaving the rest indeterminate. The implicit copy constructor copies the union as a whole, so copying any non-string value read uninitialized bytes, which GCC reports as mvt.hpp:83:8: warning: 'v.mvt_value::numeric_value. ... .len' may be used uninitialized [-Wmaybe-uninitialized] Give string_value, the widest member, a default member initializer so the union's full width is initialized however it is later used. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR * Fix remaining float-conversion and format-truncation warnings Clang's -Wimplicit-const-int-float-conversion flagged two comparisons against LLONG_MAX, which is not representable as a double and rounds up to 2^63. In serial.cpp this was a real latent overflow, not just noise: the guard `extent <= LLONG_MAX` was really `extent <= 2^63`, so an extent of exactly 2^63 passed it and then hit `(long long) extent`, which is undefined for that value and yields LLONG_MIN in practice -- the opposite of the clamp the else branch intends. Make the bound exclusive so the conversion is always in range. Requires a polygon area at the very top of the double range to reach, but the clamp now behaves as written. In mbtiles.cpp the value is only a stand-in for infinity on its way into JSON, so cast explicitly; the emitted number is unchanged. Separately, g++ at -O0 warned that `char abbrev[20]` can be truncated by "%lld", which is correct: the most negative long long needs 21 bytes with the NUL. That branch is only reached when point_count < 1000, so it cannot happen today, but size the buffer to fit rather than rely on that, and replace the garbled comment about how the size was derived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR * Clamp the low end of extent before converting to long long too The upper bound was fixed in the previous commit; the same overflow exists on the negative side. get_area() returns a signed shoelace area, so inner rings contribute negatively, and a polygon whose holes outweigh its rings drives extent below zero. Far enough below and `(long long) extent` is undefined again. The bounds are asymmetric, so this is not simply the mirror of the upper one: LLONG_MIN is exactly -2^63 and converts exactly, so unlike LLONG_MAX it can be an inclusive bound. Verified with -fsanitize=float-cast-overflow that the previous form traps on 2^63 and on doubles just below -2^63, and that this one is clean across both boundaries, the infinities, and NaN (which falls to LLONG_MAX, as it did before). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR * Add CHANGELOG entries for 2.81.0 and bump the version CHANGELOG.md was last updated for 2.80.0 (#361), and version.hpp has not moved since. Twelve PRs have landed in the meantime with no entry: #365, #368, #375, #382, #384, #385, #391, #395, #397, #399, #400, and #401. Document all of them, plus this PR, under a single 2.81.0 heading. They are not given separate version numbers because none of them was ever released under one -- version.hpp read v2.80.0 throughout -- so assigning a version per PR would invent release history. 2.81.0 is the version that will actually carry them. Where an unreleased PR was corrected by a later one (#384 by #385, #397 by #399), the pair is described as the single behavior that ships, since the intermediate behavior was never in a release. Minor rather than patch bump: the batch adds command-line options. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR * Review feedback: enforce the union-width assumption, describe both clamp ends The comment on mvt_value's union claimed string_value is the widest member. That is true on LP64 (16 bytes against 8) but not on ILP32, where size_t is 4 and it ties with double and long long. The default member initializer still covers the full union either way, so the fix held, but the justification did not travel. Replace the claim with a static_assert that checks it on whatever target is being built, so a platform where it stops holding is a compile error rather than silently indeterminate bytes. Verified the assert is not vacuous by widening the union in a scratch copy and watching it fail. The changelog described only the upper end of the extent clamp. Describe both: the old guard admitted everything below LLONG_MIN too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR * Add 2.81.0 changelog entries for the four PRs merged from main #404, #408, #409, and #410 landed while this branch was open. None of them bumped version.hpp, so they belong under the same 2.81.0 heading as the rest of the unreleased work rather than getting versions of their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8gsGMjK78TQiCGKTZ2PyR --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1820630392
commit
734bba7c78
@@ -1,3 +1,71 @@
|
|||||||
|
# 2.81.0
|
||||||
|
|
||||||
|
* Add `--drop-by-attribute-as-needed=`*attribute* to drop the features with
|
||||||
|
the lowest values of a numeric attribute from oversized tiles, and
|
||||||
|
`--drop-by-attribute-order=desc` to drop the highest values instead.
|
||||||
|
Features exactly at the threshold are kept rather than dropped. (#384, #385)
|
||||||
|
* Add `--exclude-all-tile-geometries` to tile-join, to produce tiles that
|
||||||
|
carry only attributes. (#382)
|
||||||
|
* Generate each tool's usage message from the same option table that
|
||||||
|
`getopt_long()` reads, so the hand-written lists in tile-join,
|
||||||
|
tippecanoe-overzoom, tippecanoe-json-tool, tippecanoe-decode, and
|
||||||
|
tippecanoe-enumerate can no longer fall behind the options actually
|
||||||
|
accepted. Options previously reachable only by their short names are now
|
||||||
|
listed. tippecanoe-overzoom reports a missing `-o` instead of passing a
|
||||||
|
null pointer to `fopen()`, and tippecanoe prints its usage when run with
|
||||||
|
no arguments. (#409)
|
||||||
|
* Fix the radix sort used by `--prefer-radix-sort`. A bucket written out
|
||||||
|
directly rather than through the merge was written one byte longer than
|
||||||
|
its length prefix claimed, desynchronizing everything read from the
|
||||||
|
geometry after it. Subdividing could also recurse forever once it ran out
|
||||||
|
of files to split with, shifting by the full width of the index and
|
||||||
|
writing past the end of the arrays of buckets. Radix-sorted output is now
|
||||||
|
checked against the in-memory sort rather than against a stored copy. (#404)
|
||||||
|
* Read FlatGeobuf integer and float properties as numbers. They were tagged
|
||||||
|
with types that the rest of tippecanoe does not treat as numeric, so they
|
||||||
|
were reported in tilestats as "mixed", with quoted values and no min or
|
||||||
|
max, and warned when used as a feature ID. ULong properties are now also
|
||||||
|
read as unsigned rather than signed. (#395)
|
||||||
|
* Respect the `-t` temporary directory option in sorting operations, which
|
||||||
|
previously always used the system temporary directory. (#368)
|
||||||
|
* Keep `--generate-variable-depth-tile-pyramid` from silently dropping
|
||||||
|
features whose explicit per-feature `minzoom` is deeper than the zoom at
|
||||||
|
which their region becomes a leaf. Such a feature was excluded from the
|
||||||
|
leaf tile while its children were never generated, so it appeared at no
|
||||||
|
zoom at all. (#397, #399)
|
||||||
|
* Drop a polygon hole that no remaining ring can parent, instead of failing
|
||||||
|
the whole run. Degenerate input could abort tiling over a single
|
||||||
|
unrepresentable sliver. (#401)
|
||||||
|
* Clamp the feature extent to the `long long` range before converting it,
|
||||||
|
at both ends. The previous `extent <= LLONG_MAX` guard was doubly wrong:
|
||||||
|
`LLONG_MAX` is not representable as a double and rounds up, so an extent
|
||||||
|
at the very top of the range overflowed the conversion and came out as the
|
||||||
|
most negative value rather than the largest, and the guard admitted
|
||||||
|
everything below `LLONG_MIN` as well, which overflowed the other way. Areas
|
||||||
|
are signed, so holes that outweigh their rings can reach the low end. (#406)
|
||||||
|
* Initialize the full width of the `mvt_value` numeric union, which left the
|
||||||
|
bytes of the wider unused member indeterminate even though the implicit
|
||||||
|
copy constructor copies the union as a whole. (#406)
|
||||||
|
* Replace all variable-length arrays with `std::vector` and `std::string`,
|
||||||
|
and build with `-Wvla`. VLAs are a compiler extension rather than standard
|
||||||
|
C++, and clang warns about every one of them by default. (#406)
|
||||||
|
* Remove the unused Dockerfiles, Travis configuration, and lambda
|
||||||
|
directory. (#365)
|
||||||
|
* Correct README statements that disagreed with the code. Among them, `-aD`
|
||||||
|
and `-aS` were documented the wrong way round,
|
||||||
|
`--limit-base-zoom-to-maximum-zoom` was given as `-Pb` rather than `-pb`,
|
||||||
|
and the dot-dropping description had both the fraction and the zoom
|
||||||
|
direction backwards: tippecanoe keeps 1/2.5 of the dots at zooms below the
|
||||||
|
base zoom, rather than dropping that share above it. (#410)
|
||||||
|
* Generate `man/tippecanoe.1` with go-md2man rather than md2man-roff, which
|
||||||
|
is packaged only as a Ruby gem and so had let the page drift out of date.
|
||||||
|
The page now has a proper header and a NAME section, so `man -k` and
|
||||||
|
`whatis` can find it, and no longer silently drops or mangles text the old
|
||||||
|
converter mishandled. CI checks it against README.md. (#408)
|
||||||
|
* Documentation fixes: correct three misspellings in the README and man
|
||||||
|
page, repair the dead All Streets link, and tag more README code blocks
|
||||||
|
with their language. (#375, #391, #400)
|
||||||
|
|
||||||
# 2.80.0
|
# 2.80.0
|
||||||
|
|
||||||
* Remove undocumented command-line options
|
* Remove undocumented command-line options
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ CXX := $(CXX)
|
|||||||
CFLAGS := $(CFLAGS) -fPIE -DBUILD_INFO=$(BUILD_INFO)
|
CFLAGS := $(CFLAGS) -fPIE -DBUILD_INFO=$(BUILD_INFO)
|
||||||
CXXFLAGS := $(CXXFLAGS) -std=c++17 -fPIE -DBUILD_INFO=$(BUILD_INFO)
|
CXXFLAGS := $(CXXFLAGS) -std=c++17 -fPIE -DBUILD_INFO=$(BUILD_INFO)
|
||||||
LDFLAGS := $(LDFLAGS)
|
LDFLAGS := $(LDFLAGS)
|
||||||
WARNING_FLAGS := -Wall -Wshadow -Wsign-compare -Wextra -Wunreachable-code -Wuninitialized -Wshadow
|
WARNING_FLAGS := -Wall -Wshadow -Wsign-compare -Wextra -Wunreachable-code -Wuninitialized -Wshadow -Wvla
|
||||||
RELEASE_FLAGS := -O3 -DNDEBUG
|
RELEASE_FLAGS := -O3 -DNDEBUG
|
||||||
DEBUG_FLAGS := -O0 -DDEBUG -fno-inline-functions -fno-omit-frame-pointer
|
DEBUG_FLAGS := -O0 -DDEBUG -fno-inline-functions -fno-omit-frame-pointer
|
||||||
|
|
||||||
|
|||||||
@@ -215,7 +215,7 @@ void init_cpus() {
|
|||||||
// MacOS can run out of system file descriptors
|
// MacOS can run out of system file descriptors
|
||||||
// even if we stay under the rlimit, so try to
|
// even if we stay under the rlimit, so try to
|
||||||
// find out the real limit.
|
// find out the real limit.
|
||||||
long long fds[MAX_FILES];
|
std::vector<long long> fds(MAX_FILES);
|
||||||
long long i;
|
long long i;
|
||||||
for (i = 0; i < MAX_FILES; i++) {
|
for (i = 0; i < MAX_FILES; i++) {
|
||||||
fds[i] = open(get_null_device(), O_RDONLY | O_CLOEXEC);
|
fds[i] = open(get_null_device(), O_RDONLY | O_CLOEXEC);
|
||||||
@@ -449,7 +449,7 @@ void *run_sort(void *v) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::unordered_map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
|
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::unordered_map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
|
||||||
long long segs[CPUS + 1];
|
std::vector<long long> segs(CPUS + 1);
|
||||||
segs[0] = 0;
|
segs[0] = 0;
|
||||||
segs[CPUS] = len;
|
segs[CPUS] = len;
|
||||||
|
|
||||||
@@ -461,11 +461,11 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
double dist_sums[CPUS];
|
std::vector<double> dist_sums(CPUS);
|
||||||
size_t dist_counts[CPUS];
|
std::vector<size_t> dist_counts(CPUS);
|
||||||
double area_sums[CPUS];
|
std::vector<double> area_sums(CPUS);
|
||||||
|
|
||||||
std::atomic<long long> layer_seq[CPUS];
|
std::vector<std::atomic<long long> > layer_seq(CPUS);
|
||||||
for (size_t i = 0; i < CPUS; i++) {
|
for (size_t i = 0; i < CPUS; i++) {
|
||||||
// To preserve feature ordering, unique id for each segment
|
// To preserve feature ordering, unique id for each segment
|
||||||
// begins with that segment's offset into the input
|
// begins with that segment's offset into the input
|
||||||
@@ -479,7 +479,7 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
|
|||||||
std::vector<serialization_state> sst;
|
std::vector<serialization_state> sst;
|
||||||
sst.resize(CPUS);
|
sst.resize(CPUS);
|
||||||
|
|
||||||
pthread_t pthreads[CPUS];
|
std::vector<pthread_t> pthreads(CPUS);
|
||||||
std::vector<std::set<serial_val> > file_subkeys;
|
std::vector<std::set<serial_val> > file_subkeys;
|
||||||
|
|
||||||
for (size_t i = 0; i < CPUS; i++) {
|
for (size_t i = 0; i < CPUS; i++) {
|
||||||
@@ -759,47 +759,45 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
|||||||
}
|
}
|
||||||
splits = 1 << splitbits;
|
splits = 1 << splitbits;
|
||||||
|
|
||||||
FILE *geomfiles[splits];
|
std::vector<FILE *> geomfiles(splits);
|
||||||
FILE *indexfiles[splits];
|
std::vector<FILE *> indexfiles(splits);
|
||||||
int geomfds[splits];
|
std::vector<int> geomfds(splits);
|
||||||
int indexfds[splits];
|
std::vector<int> indexfds(splits);
|
||||||
std::atomic<long long> sub_geompos[splits];
|
std::vector<std::atomic<long long> > sub_geompos(splits);
|
||||||
|
|
||||||
int i;
|
int i;
|
||||||
for (i = 0; i < splits; i++) {
|
for (i = 0; i < splits; i++) {
|
||||||
sub_geompos[i] = 0;
|
sub_geompos[i] = 0;
|
||||||
|
|
||||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
std::string geomname = std::string(tmpdir) + "/geom.XXXXXXXX";
|
||||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
std::string indexname = std::string(tmpdir) + "/index.XXXXXXXX";
|
||||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
|
||||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
|
||||||
|
|
||||||
geomfds[i] = mkstemp_cloexec(geomname);
|
geomfds[i] = mkstemp_cloexec(&geomname[0]);
|
||||||
if (geomfds[i] < 0) {
|
if (geomfds[i] < 0) {
|
||||||
perror(geomname);
|
perror(geomname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
indexfds[i] = mkstemp_cloexec(indexname);
|
indexfds[i] = mkstemp_cloexec(&indexname[0]);
|
||||||
if (indexfds[i] < 0) {
|
if (indexfds[i] < 0) {
|
||||||
perror(indexname);
|
perror(indexname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
|
|
||||||
geomfiles[i] = fopen_oflag(geomname, "wb", O_WRONLY | O_CLOEXEC);
|
geomfiles[i] = fopen_oflag(geomname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||||
if (geomfiles[i] == NULL) {
|
if (geomfiles[i] == NULL) {
|
||||||
perror(geomname);
|
perror(geomname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
indexfiles[i] = fopen_oflag(indexname, "wb", O_WRONLY | O_CLOEXEC);
|
indexfiles[i] = fopen_oflag(indexname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||||
if (indexfiles[i] == NULL) {
|
if (indexfiles[i] == NULL) {
|
||||||
perror(indexname);
|
perror(indexname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
|
|
||||||
*availfiles -= 4;
|
*availfiles -= 4;
|
||||||
|
|
||||||
unlink(geomname);
|
unlink(geomname.c_str());
|
||||||
unlink(indexname);
|
unlink(indexname.c_str());
|
||||||
}
|
}
|
||||||
|
|
||||||
for (i = 0; i < inputs; i++) {
|
for (i = 0; i < inputs; i++) {
|
||||||
@@ -928,13 +926,13 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
|||||||
}
|
}
|
||||||
|
|
||||||
size_t nmerges = (indexpos + unit - 1) / unit;
|
size_t nmerges = (indexpos + unit - 1) / unit;
|
||||||
struct mergelist merges[nmerges];
|
std::vector<struct mergelist> merges(nmerges);
|
||||||
|
|
||||||
for (size_t a = 0; a < nmerges; a++) {
|
for (size_t a = 0; a < nmerges; a++) {
|
||||||
merges[a].start = merges[a].end = 0;
|
merges[a].start = merges[a].end = 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
pthread_t pthreads[CPUS];
|
std::vector<pthread_t> pthreads(CPUS);
|
||||||
std::vector<sort_arg> args;
|
std::vector<sort_arg> args;
|
||||||
|
|
||||||
for (size_t a = 0; a < CPUS; a++) {
|
for (size_t a = 0; a < CPUS; a++) {
|
||||||
@@ -942,7 +940,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
|||||||
a,
|
a,
|
||||||
CPUS,
|
CPUS,
|
||||||
indexpos,
|
indexpos,
|
||||||
merges,
|
merges.data(),
|
||||||
indexfds[i],
|
indexfds[i],
|
||||||
nmerges,
|
nmerges,
|
||||||
unit,
|
unit,
|
||||||
@@ -980,7 +978,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
|
|||||||
madvise(geommap, geomst.st_size, MADV_RANDOM);
|
madvise(geommap, geomst.st_size, MADV_RANDOM);
|
||||||
madvise(geommap, geomst.st_size, MADV_WILLNEED);
|
madvise(geommap, geomst.st_size, MADV_WILLNEED);
|
||||||
|
|
||||||
merge(merges, nmerges, (unsigned char *) indexmap, indexfile, bytes, geommap, geomfile, geompos_out, progress, progress_max, progress_reported, maxzoom, gamma, ds);
|
merge(merges.data(), nmerges, (unsigned char *) indexmap, indexfile, bytes, geommap, geomfile, geompos_out, progress, progress_max, progress_reported, maxzoom, gamma, ds);
|
||||||
|
|
||||||
madvise(indexmap, indexst.st_size, MADV_DONTNEED);
|
madvise(indexmap, indexst.st_size, MADV_DONTNEED);
|
||||||
if (munmap(indexmap, indexst.st_size) < 0) {
|
if (munmap(indexmap, indexst.st_size) < 0) {
|
||||||
@@ -1119,8 +1117,8 @@ void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FI
|
|||||||
mem /= 2;
|
mem /= 2;
|
||||||
|
|
||||||
long long geom_total = 0;
|
long long geom_total = 0;
|
||||||
int geomfds[nreaders];
|
std::vector<int> geomfds(nreaders);
|
||||||
int indexfds[nreaders];
|
std::vector<int> indexfds(nreaders);
|
||||||
for (int i = 0; i < nreaders; i++) {
|
for (int i = 0; i < nreaders; i++) {
|
||||||
geomfds[i] = readers[i].geomfd;
|
geomfds[i] = readers[i].geomfd;
|
||||||
indexfds[i] = readers[i].indexfd;
|
indexfds[i] = readers[i].indexfd;
|
||||||
@@ -1133,12 +1131,12 @@ void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FI
|
|||||||
geom_total += geomst.st_size;
|
geom_total += geomst.st_size;
|
||||||
}
|
}
|
||||||
|
|
||||||
struct drop_state ds[maxzoom + 1];
|
std::vector<struct drop_state> ds(maxzoom + 1);
|
||||||
prep_drop_states(ds, maxzoom, basezoom, droprate);
|
prep_drop_states(ds.data(), maxzoom, basezoom, droprate);
|
||||||
|
|
||||||
long long progress = 0, progress_max = geom_total, progress_reported = -1;
|
long long progress = 0, progress_max = geom_total, progress_reported = -1;
|
||||||
long long availfiles_before = availfiles;
|
long long availfiles_before = availfiles;
|
||||||
radix1(geomfds, indexfds, nreaders, 0, splits, mem, tmpdir, &availfiles, geomfile, indexfile, geompos, &progress, &progress_max, &progress_reported, maxzoom, basezoom, droprate, gamma, ds);
|
radix1(geomfds.data(), indexfds.data(), nreaders, 0, splits, mem, tmpdir, &availfiles, geomfile, indexfile, geompos, &progress, &progress_max, &progress_reported, maxzoom, basezoom, droprate, gamma, ds.data());
|
||||||
|
|
||||||
if (availfiles - 2 * nreaders != availfiles_before) {
|
if (availfiles - 2 * nreaders != availfiles_before) {
|
||||||
fprintf(stderr, "Internal error: miscounted available file descriptors: %lld vs %lld\n", availfiles - 2 * nreaders, availfiles);
|
fprintf(stderr, "Internal error: miscounted available file descriptors: %lld vs %lld\n", availfiles - 2 * nreaders, availfiles);
|
||||||
@@ -1247,79 +1245,72 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
for (size_t i = 0; i < CPUS; i++) {
|
for (size_t i = 0; i < CPUS; i++) {
|
||||||
struct reader *r = &readers[i];
|
struct reader *r = &readers[i];
|
||||||
|
|
||||||
char poolname[strlen(tmpdir) + strlen("/pool.XXXXXXXX") + 1];
|
std::string poolname = std::string(tmpdir) + "/pool.XXXXXXXX";
|
||||||
char treename[strlen(tmpdir) + strlen("/tree.XXXXXXXX") + 1];
|
std::string treename = std::string(tmpdir) + "/tree.XXXXXXXX";
|
||||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
std::string geomname = std::string(tmpdir) + "/geom.XXXXXXXX";
|
||||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
std::string indexname = std::string(tmpdir) + "/index.XXXXXXXX";
|
||||||
char vertexname[strlen(tmpdir) + strlen("/vertex.XXXXXXXX") + 1];
|
std::string vertexname = std::string(tmpdir) + "/vertex.XXXXXXXX";
|
||||||
char nodename[strlen(tmpdir) + strlen("/node.XXXXXXXX") + 1];
|
std::string nodename = std::string(tmpdir) + "/node.XXXXXXXX";
|
||||||
|
|
||||||
snprintf(poolname, sizeof(poolname), "%s%s", tmpdir, "/pool.XXXXXXXX");
|
r->poolfd = mkstemp_cloexec(&poolname[0]);
|
||||||
snprintf(treename, sizeof(treename), "%s%s", tmpdir, "/tree.XXXXXXXX");
|
|
||||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
|
||||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
|
||||||
snprintf(vertexname, sizeof(vertexname), "%s%s", tmpdir, "/vertex.XXXXXXXX");
|
|
||||||
snprintf(nodename, sizeof(nodename), "%s%s", tmpdir, "/node.XXXXXXXX");
|
|
||||||
|
|
||||||
r->poolfd = mkstemp_cloexec(poolname);
|
|
||||||
if (r->poolfd < 0) {
|
if (r->poolfd < 0) {
|
||||||
perror(poolname);
|
perror(poolname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->treefd = mkstemp_cloexec(treename);
|
r->treefd = mkstemp_cloexec(&treename[0]);
|
||||||
if (r->treefd < 0) {
|
if (r->treefd < 0) {
|
||||||
perror(treename);
|
perror(treename.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->geomfd = mkstemp_cloexec(geomname);
|
r->geomfd = mkstemp_cloexec(&geomname[0]);
|
||||||
if (r->geomfd < 0) {
|
if (r->geomfd < 0) {
|
||||||
perror(geomname);
|
perror(geomname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->indexfd = mkstemp_cloexec(indexname);
|
r->indexfd = mkstemp_cloexec(&indexname[0]);
|
||||||
if (r->indexfd < 0) {
|
if (r->indexfd < 0) {
|
||||||
perror(indexname);
|
perror(indexname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->vertexfd = mkstemp_cloexec(vertexname);
|
r->vertexfd = mkstemp_cloexec(&vertexname[0]);
|
||||||
if (r->vertexfd < 0) {
|
if (r->vertexfd < 0) {
|
||||||
perror(vertexname);
|
perror(vertexname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->nodefd = mkstemp_cloexec(nodename);
|
r->nodefd = mkstemp_cloexec(&nodename[0]);
|
||||||
if (r->nodefd < 0) {
|
if (r->nodefd < 0) {
|
||||||
perror(nodename);
|
perror(nodename.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
|
|
||||||
r->poolfile = memfile_open(r->poolfd);
|
r->poolfile = memfile_open(r->poolfd);
|
||||||
if (r->poolfile == NULL) {
|
if (r->poolfile == NULL) {
|
||||||
perror(poolname);
|
perror(poolname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->treefile = memfile_open(r->treefd);
|
r->treefile = memfile_open(r->treefd);
|
||||||
if (r->treefile == NULL) {
|
if (r->treefile == NULL) {
|
||||||
perror(treename);
|
perror(treename.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->geomfile = fopen_oflag(geomname, "wb", O_WRONLY | O_CLOEXEC);
|
r->geomfile = fopen_oflag(geomname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||||
if (r->geomfile == NULL) {
|
if (r->geomfile == NULL) {
|
||||||
perror(geomname);
|
perror(geomname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->indexfile = fopen_oflag(indexname, "wb", O_WRONLY | O_CLOEXEC);
|
r->indexfile = fopen_oflag(indexname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||||
if (r->indexfile == NULL) {
|
if (r->indexfile == NULL) {
|
||||||
perror(indexname);
|
perror(indexname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->vertexfile = fopen_oflag(vertexname, "w+b", O_RDWR | O_CLOEXEC);
|
r->vertexfile = fopen_oflag(vertexname.c_str(), "w+b", O_RDWR | O_CLOEXEC);
|
||||||
if (r->vertexfile == NULL) {
|
if (r->vertexfile == NULL) {
|
||||||
perror(("open vertexfile " + std::string(vertexname)).c_str());
|
perror(("open vertexfile " + vertexname).c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->nodefile = fopen_oflag(nodename, "w+b", O_RDWR | O_CLOEXEC);
|
r->nodefile = fopen_oflag(nodename.c_str(), "w+b", O_RDWR | O_CLOEXEC);
|
||||||
if (r->nodefile == NULL) {
|
if (r->nodefile == NULL) {
|
||||||
perror(nodename);
|
perror(nodename.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
r->geompos = 0;
|
r->geompos = 0;
|
||||||
@@ -1327,12 +1318,12 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
r->vertexpos = 0;
|
r->vertexpos = 0;
|
||||||
r->nodepos = 0;
|
r->nodepos = 0;
|
||||||
|
|
||||||
unlink(poolname);
|
unlink(poolname.c_str());
|
||||||
unlink(treename);
|
unlink(treename.c_str());
|
||||||
unlink(geomname);
|
unlink(geomname.c_str());
|
||||||
unlink(indexname);
|
unlink(indexname.c_str());
|
||||||
unlink(vertexname);
|
unlink(vertexname.c_str());
|
||||||
unlink(nodename);
|
unlink(nodename.c_str());
|
||||||
|
|
||||||
// To distinguish a null value
|
// To distinguish a null value
|
||||||
{
|
{
|
||||||
@@ -1357,8 +1348,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
std::atomic<long long> progress_seq(0);
|
std::atomic<long long> progress_seq(0);
|
||||||
|
|
||||||
// 2 * CPUS: One per reader thread, one per tiling thread
|
// 2 * CPUS: One per reader thread, one per tiling thread
|
||||||
int initialized[2 * CPUS];
|
std::vector<int> initialized(2 * CPUS);
|
||||||
unsigned initial_x[2 * CPUS], initial_y[2 * CPUS];
|
std::vector<unsigned> initial_x(2 * CPUS), initial_y(2 * CPUS);
|
||||||
for (size_t i = 0; i < 2 * CPUS; i++) {
|
for (size_t i = 0; i < 2 * CPUS; i++) {
|
||||||
initialized[i] = initial_x[i] = initial_y[i] = 0;
|
initialized[i] = initial_x[i] = initial_y[i] = 0;
|
||||||
}
|
}
|
||||||
@@ -1489,10 +1480,10 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
exit(EXIT_MEMORY);
|
exit(EXIT_MEMORY);
|
||||||
}
|
}
|
||||||
|
|
||||||
std::atomic<long long> layer_seq[CPUS];
|
std::vector<std::atomic<long long> > layer_seq(CPUS);
|
||||||
double dist_sums[CPUS];
|
std::vector<double> dist_sums(CPUS);
|
||||||
size_t dist_counts[CPUS];
|
std::vector<size_t> dist_counts(CPUS);
|
||||||
double area_sums[CPUS];
|
std::vector<double> area_sums(CPUS);
|
||||||
std::vector<struct serialization_state> sst;
|
std::vector<struct serialization_state> sst;
|
||||||
sst.resize(CPUS);
|
sst.resize(CPUS);
|
||||||
|
|
||||||
@@ -1562,10 +1553,10 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
exit(EXIT_MEMORY);
|
exit(EXIT_MEMORY);
|
||||||
}
|
}
|
||||||
|
|
||||||
std::atomic<long long> layer_seq[CPUS];
|
std::vector<std::atomic<long long> > layer_seq(CPUS);
|
||||||
double dist_sums[CPUS];
|
std::vector<double> dist_sums(CPUS);
|
||||||
size_t dist_counts[CPUS];
|
std::vector<size_t> dist_counts(CPUS);
|
||||||
double area_sums[CPUS];
|
std::vector<double> area_sums(CPUS);
|
||||||
std::vector<struct serialization_state> sst;
|
std::vector<struct serialization_state> sst;
|
||||||
sst.resize(CPUS);
|
sst.resize(CPUS);
|
||||||
|
|
||||||
@@ -1622,10 +1613,10 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (sources[source].format == "csv" || (sources[source].file.size() > 4 && sources[source].file.substr(sources[source].file.size() - 4) == std::string(".csv"))) {
|
if (sources[source].format == "csv" || (sources[source].file.size() > 4 && sources[source].file.substr(sources[source].file.size() - 4) == std::string(".csv"))) {
|
||||||
std::atomic<long long> layer_seq[CPUS];
|
std::vector<std::atomic<long long> > layer_seq(CPUS);
|
||||||
double dist_sums[CPUS];
|
std::vector<double> dist_sums(CPUS);
|
||||||
size_t dist_counts[CPUS];
|
std::vector<size_t> dist_counts(CPUS);
|
||||||
double area_sums[CPUS];
|
std::vector<double> area_sums(CPUS);
|
||||||
|
|
||||||
std::vector<struct serialization_state> sst;
|
std::vector<struct serialization_state> sst;
|
||||||
sst.resize(CPUS);
|
sst.resize(CPUS);
|
||||||
@@ -1710,7 +1701,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (map != NULL && map != MAP_FAILED && read_parallel_this) {
|
if (map != NULL && map != MAP_FAILED && read_parallel_this) {
|
||||||
do_read_parallel(map, st.st_size - off, overall_offset, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, &layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, uses_gamma, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
do_read_parallel(map, st.st_size - off, overall_offset, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, &layermaps, initialized.data(), initial_x.data(), initial_y.data(), maxzoom, sources[layer].layer, uses_gamma, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||||
overall_offset += st.st_size - off;
|
overall_offset += st.st_size - off;
|
||||||
checkdisk(&readers);
|
checkdisk(&readers);
|
||||||
|
|
||||||
@@ -1742,19 +1733,18 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
if (read_parallel_this) {
|
if (read_parallel_this) {
|
||||||
// Serial reading of chunks that are then parsed in parallel
|
// Serial reading of chunks that are then parsed in parallel
|
||||||
|
|
||||||
char readname[strlen(tmpdir) + strlen("/read.XXXXXXXX") + 1];
|
std::string readname = std::string(tmpdir) + "/read.XXXXXXXX";
|
||||||
snprintf(readname, sizeof(readname), "%s%s", tmpdir, "/read.XXXXXXXX");
|
int readfd = mkstemp_cloexec(&readname[0]);
|
||||||
int readfd = mkstemp_cloexec(readname);
|
|
||||||
if (readfd < 0) {
|
if (readfd < 0) {
|
||||||
perror(readname);
|
perror(readname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
FILE *readfp = fdopen(readfd, "w");
|
FILE *readfp = fdopen(readfd, "w");
|
||||||
if (readfp == NULL) {
|
if (readfp == NULL) {
|
||||||
perror(readname);
|
perror(readname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
unlink(readname);
|
unlink(readname.c_str());
|
||||||
|
|
||||||
std::atomic<int> is_parsing(0);
|
std::atomic<int> is_parsing(0);
|
||||||
long long ahead = 0;
|
long long ahead = 0;
|
||||||
@@ -1789,25 +1779,25 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
}
|
}
|
||||||
|
|
||||||
fflush(readfp);
|
fflush(readfp);
|
||||||
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, ¶llel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, ¶llel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized.data(), initial_x.data(), initial_y.data(), maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||||
|
|
||||||
initial_offset += ahead;
|
initial_offset += ahead;
|
||||||
overall_offset += ahead;
|
overall_offset += ahead;
|
||||||
checkdisk(&readers);
|
checkdisk(&readers);
|
||||||
ahead = 0;
|
ahead = 0;
|
||||||
|
|
||||||
snprintf(readname, sizeof(readname), "%s%s", tmpdir, "/read.XXXXXXXX");
|
readname = std::string(tmpdir) + "/read.XXXXXXXX";
|
||||||
readfd = mkstemp_cloexec(readname);
|
readfd = mkstemp_cloexec(&readname[0]);
|
||||||
if (readfd < 0) {
|
if (readfd < 0) {
|
||||||
perror(readname);
|
perror(readname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
readfp = fdopen(readfd, "w");
|
readfp = fdopen(readfd, "w");
|
||||||
if (readfp == NULL) {
|
if (readfp == NULL) {
|
||||||
perror(readname);
|
perror(readname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
unlink(readname);
|
unlink(readname.c_str());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1826,7 +1816,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
fflush(readfp);
|
fflush(readfp);
|
||||||
|
|
||||||
if (ahead > 0) {
|
if (ahead > 0) {
|
||||||
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, ¶llel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, ¶llel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized.data(), initial_x.data(), initial_y.data(), maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
|
||||||
|
|
||||||
if (parser_created) {
|
if (parser_created) {
|
||||||
if (pthread_join(parallel_parser, NULL) != 0) {
|
if (pthread_join(parallel_parser, NULL) != 0) {
|
||||||
@@ -1935,27 +1925,26 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
// segment+offset to find the data.
|
// segment+offset to find the data.
|
||||||
|
|
||||||
// 2 * CPUS: One per input thread, one per tiling thread
|
// 2 * CPUS: One per input thread, one per tiling thread
|
||||||
long long pool_off[2 * CPUS];
|
std::vector<long long> pool_off(2 * CPUS);
|
||||||
for (size_t i = 0; i < 2 * CPUS; i++) {
|
for (size_t i = 0; i < 2 * CPUS; i++) {
|
||||||
pool_off[i] = 0;
|
pool_off[i] = 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
char poolname[strlen(tmpdir) + strlen("/pool.XXXXXXXX") + 1];
|
std::string poolname = std::string(tmpdir) + "/pool.XXXXXXXX";
|
||||||
snprintf(poolname, sizeof(poolname), "%s%s", tmpdir, "/pool.XXXXXXXX");
|
|
||||||
|
|
||||||
int poolfd = mkstemp_cloexec(poolname);
|
int poolfd = mkstemp_cloexec(&poolname[0]);
|
||||||
if (poolfd < 0) {
|
if (poolfd < 0) {
|
||||||
perror(poolname);
|
perror(poolname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
|
|
||||||
FILE *poolfile = fopen_oflag(poolname, "wb", O_WRONLY | O_CLOEXEC);
|
FILE *poolfile = fopen_oflag(poolname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||||
if (poolfile == NULL) {
|
if (poolfile == NULL) {
|
||||||
perror(poolname);
|
perror(poolname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
|
|
||||||
unlink(poolname);
|
unlink(poolname.c_str());
|
||||||
std::atomic<long long> poolpos(0);
|
std::atomic<long long> poolpos(0);
|
||||||
|
|
||||||
for (size_t i = 0; i < CPUS; i++) {
|
for (size_t i = 0; i < CPUS; i++) {
|
||||||
@@ -2183,36 +2172,34 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
fprintf(stderr, "Merging index \r");
|
fprintf(stderr, "Merging index \r");
|
||||||
}
|
}
|
||||||
|
|
||||||
char indexname[strlen(tmpdir) + strlen("/index.XXXXXXXX") + 1];
|
std::string indexname = std::string(tmpdir) + "/index.XXXXXXXX";
|
||||||
snprintf(indexname, sizeof(indexname), "%s%s", tmpdir, "/index.XXXXXXXX");
|
|
||||||
|
|
||||||
int indexfd = mkstemp_cloexec(indexname);
|
int indexfd = mkstemp_cloexec(&indexname[0]);
|
||||||
if (indexfd < 0) {
|
if (indexfd < 0) {
|
||||||
perror(indexname);
|
perror(indexname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
FILE *indexfile = fopen_oflag(indexname, "wb", O_WRONLY | O_CLOEXEC);
|
FILE *indexfile = fopen_oflag(indexname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||||
if (indexfile == NULL) {
|
if (indexfile == NULL) {
|
||||||
perror(indexname);
|
perror(indexname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
|
|
||||||
unlink(indexname);
|
unlink(indexname.c_str());
|
||||||
|
|
||||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX") + 1];
|
std::string geomname = std::string(tmpdir) + "/geom.XXXXXXXX";
|
||||||
snprintf(geomname, sizeof(geomname), "%s%s", tmpdir, "/geom.XXXXXXXX");
|
|
||||||
|
|
||||||
int geomfd = mkstemp_cloexec(geomname);
|
int geomfd = mkstemp_cloexec(&geomname[0]);
|
||||||
if (geomfd < 0) {
|
if (geomfd < 0) {
|
||||||
perror(geomname);
|
perror(geomname.c_str());
|
||||||
exit(EXIT_CLOSE);
|
exit(EXIT_CLOSE);
|
||||||
}
|
}
|
||||||
FILE *geomfile = fopen_oflag(geomname, "wb", O_WRONLY | O_CLOEXEC);
|
FILE *geomfile = fopen_oflag(geomname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||||
if (geomfile == NULL) {
|
if (geomfile == NULL) {
|
||||||
perror(geomname);
|
perror(geomname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
unlink(geomname);
|
unlink(geomname.c_str());
|
||||||
|
|
||||||
unsigned iz = 0, ix = 0, iy = 0;
|
unsigned iz = 0, ix = 0, iy = 0;
|
||||||
choose_first_zoom(file_bbox, file_bbox1, file_bbox2, readers, &iz, &ix, &iy, minzoom, buffer);
|
choose_first_zoom(file_bbox, file_bbox1, file_bbox2, readers, &iz, &ix, &iy, minzoom, buffer);
|
||||||
@@ -2699,8 +2686,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
madvise(geom, indexpos, MADV_SEQUENTIAL);
|
madvise(geom, indexpos, MADV_SEQUENTIAL);
|
||||||
madvise(geom, indexpos, MADV_WILLNEED);
|
madvise(geom, indexpos, MADV_WILLNEED);
|
||||||
|
|
||||||
struct drop_state ds[maxzoom + 1];
|
std::vector<struct drop_state> ds(maxzoom + 1);
|
||||||
prep_drop_states(ds, maxzoom, basezoom, droprate);
|
prep_drop_states(ds.data(), maxzoom, basezoom, droprate);
|
||||||
|
|
||||||
if (drop_denser > 0) {
|
if (drop_denser > 0) {
|
||||||
std::vector<drop_densest> ddv;
|
std::vector<drop_densest> ddv;
|
||||||
@@ -2718,7 +2705,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
|
|
||||||
previndex = map[ip].ix;
|
previndex = map[ip].ix;
|
||||||
} else {
|
} else {
|
||||||
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom, gamma);
|
int feature_minzoom = calc_feature_minzoom(&map[ip], ds.data(), maxzoom, gamma);
|
||||||
geom[map[ip].end - 1] = feature_minzoom;
|
geom[map[ip].end - 1] = feature_minzoom;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -2743,7 +2730,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
if (ip > 0 && map[ip].start != map[ip - 1].end) {
|
if (ip > 0 && map[ip].start != map[ip - 1].end) {
|
||||||
fprintf(stderr, "Mismatched index at %lld: %lld vs %lld\n", ip, map[ip].start, map[ip].end);
|
fprintf(stderr, "Mismatched index at %lld: %lld vs %lld\n", ip, map[ip].start, map[ip].end);
|
||||||
}
|
}
|
||||||
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom, gamma);
|
int feature_minzoom = calc_feature_minzoom(&map[ip], ds.data(), maxzoom, gamma);
|
||||||
geom[map[ip].end - 1] = feature_minzoom;
|
geom[map[ip].end - 1] = feature_minzoom;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -2766,8 +2753,8 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
exit(EXIT_STAT);
|
exit(EXIT_STAT);
|
||||||
}
|
}
|
||||||
|
|
||||||
int fd[TEMP_FILES];
|
std::vector<int> fd(TEMP_FILES);
|
||||||
off_t size[TEMP_FILES];
|
std::vector<off_t> size(TEMP_FILES);
|
||||||
|
|
||||||
fd[0] = geomfd;
|
fd[0] = geomfd;
|
||||||
size[0] = geomst.st_size;
|
size[0] = geomst.st_size;
|
||||||
@@ -2780,7 +2767,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
|
|||||||
std::atomic<unsigned> midx(0);
|
std::atomic<unsigned> midx(0);
|
||||||
std::atomic<unsigned> midy(0);
|
std::atomic<unsigned> midy(0);
|
||||||
std::vector<strategy> strategies;
|
std::vector<strategy> strategies;
|
||||||
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos, shared_nodes_bloom, basezoom, droprate, unidecode_data, &drop_by_attribute_as_needed_attribute, drop_by_attribute_descending);
|
int written = traverse_zooms(fd.data(), size.data(), stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off.data(), initial_x.data(), initial_y.data(), simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos, shared_nodes_bloom, basezoom, droprate, unidecode_data, &drop_by_attribute_as_needed_attribute, drop_by_attribute_descending);
|
||||||
|
|
||||||
if (maxzoom != written) {
|
if (maxzoom != written) {
|
||||||
if (written > minzoom) {
|
if (written > minzoom) {
|
||||||
|
|||||||
+4
-1
@@ -701,7 +701,10 @@ metadata make_metadata(const char *fname, int minzoom, int maxzoom, double minla
|
|||||||
m.strategies_json = stringify_strategies(strategies);
|
m.strategies_json = stringify_strategies(strategies);
|
||||||
|
|
||||||
if (std::isinf(droprate)) {
|
if (std::isinf(droprate)) {
|
||||||
droprate = LLONG_MAX;
|
// JSON has no representation for infinity, so substitute a huge
|
||||||
|
// finite value. The cast is explicit because LLONG_MAX itself is
|
||||||
|
// not representable as a double and rounds up to 2^63.
|
||||||
|
droprate = (double) LLONG_MAX;
|
||||||
}
|
}
|
||||||
if (basezoom != maxzoom || droprate != 2.5 || retain_points_multiplier != 1) {
|
if (basezoom != maxzoom || droprate != 2.5 || retain_points_multiplier != 1) {
|
||||||
m.decisions_json = std::string("{") +
|
m.decisions_json = std::string("{") +
|
||||||
|
|||||||
@@ -93,12 +93,21 @@ struct mvt_value {
|
|||||||
long long sint_value;
|
long long sint_value;
|
||||||
bool bool_value;
|
bool bool_value;
|
||||||
int null_value;
|
int null_value;
|
||||||
|
// Initializing string_value initializes the union's full width, which
|
||||||
|
// the static_assert below checks. Setting only a narrower member (a
|
||||||
|
// double, say) would leave the remaining bytes indeterminate, and the
|
||||||
|
// implicit copy constructor copies the union as a whole, so those
|
||||||
|
// bytes get read even when they aren't the member in use.
|
||||||
struct {
|
struct {
|
||||||
size_t off;
|
size_t off;
|
||||||
size_t len;
|
size_t len;
|
||||||
} string_value;
|
} string_value = {0, 0};
|
||||||
} numeric_value;
|
} numeric_value;
|
||||||
|
|
||||||
|
static_assert(sizeof(numeric_value) == sizeof(numeric_value.string_value),
|
||||||
|
"string_value must span the whole union, since its default member "
|
||||||
|
"initializer is what initializes the union");
|
||||||
|
|
||||||
std::string get_string_value() const {
|
std::string get_string_value() const {
|
||||||
if (type == mvt_string) {
|
if (type == mvt_string) {
|
||||||
return std::string(*s, numeric_value.string_value.off, numeric_value.string_value.len);
|
return std::string(*s, numeric_value.string_value.off, numeric_value.string_value.len);
|
||||||
|
|||||||
+10
-2
@@ -665,10 +665,18 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf, std::
|
|||||||
// VT_POINT extent will be calculated in write_tile from the distance between adjacent features.
|
// VT_POINT extent will be calculated in write_tile from the distance between adjacent features.
|
||||||
}
|
}
|
||||||
|
|
||||||
if (extent <= LLONG_MAX) {
|
// Clamp before converting, since converting a double that is out of range
|
||||||
|
// for a long long is undefined. The bounds are asymmetric: LLONG_MAX is not
|
||||||
|
// representable as a double and rounds up to 2^63, so the upper bound has to
|
||||||
|
// be exclusive, while LLONG_MIN is exactly -2^63 and so can be included.
|
||||||
|
// Areas are signed, so holes that outweigh their rings can make this
|
||||||
|
// negative.
|
||||||
|
if (extent >= (double) LLONG_MIN && extent < (double) LLONG_MAX) {
|
||||||
sf.extent = (long long) extent;
|
sf.extent = (long long) extent;
|
||||||
|
} else if (extent < 0) {
|
||||||
|
sf.extent = LLONG_MIN;
|
||||||
} else {
|
} else {
|
||||||
sf.extent = LLONG_MAX;
|
sf.extent = LLONG_MAX; // also the NaN case
|
||||||
}
|
}
|
||||||
|
|
||||||
if (sst->want_dist && sf.t == VT_POLYGON) {
|
if (sst->want_dist && sf.t == VT_POLYGON) {
|
||||||
|
|||||||
+1
-1
@@ -893,7 +893,7 @@ void *join_worker(void *v) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
void dispatch_tasks(std::map<zxy, std::vector<std::string>> &tasks, std::vector<std::map<std::string, layermap_entry>> &layermaps, sqlite3 *outdb, const char *outdir, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, sqlite3 *db, std::set<std::string> &exclude, std::set<std::string> &include, int ifmatched, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, json_object *filter, struct tileset_reader *readers, double *minlat, double *minlon, double *maxlat, double *maxlon, double *minlon2, double *maxlon2) {
|
void dispatch_tasks(std::map<zxy, std::vector<std::string>> &tasks, std::vector<std::map<std::string, layermap_entry>> &layermaps, sqlite3 *outdb, const char *outdir, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, sqlite3 *db, std::set<std::string> &exclude, std::set<std::string> &include, int ifmatched, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, json_object *filter, struct tileset_reader *readers, double *minlat, double *minlon, double *maxlat, double *maxlon, double *minlon2, double *maxlon2) {
|
||||||
pthread_t pthreads[CPUS];
|
std::vector<pthread_t> pthreads(CPUS);
|
||||||
std::vector<arg> args;
|
std::vector<arg> args;
|
||||||
|
|
||||||
for (size_t i = 0; i < CPUS; i++) {
|
for (size_t i = 0; i < CPUS; i++) {
|
||||||
|
|||||||
@@ -61,9 +61,6 @@ extern "C" {
|
|||||||
#define COORD_OFFSET (4LL << 32)
|
#define COORD_OFFSET (4LL << 32)
|
||||||
#define SHIFT_RIGHT(a) ((long long) std::round((double) (a) / (1LL << geometry_scale)))
|
#define SHIFT_RIGHT(a) ((long long) std::round((double) (a) / (1LL << geometry_scale)))
|
||||||
|
|
||||||
#define XSTRINGIFY(s) STRINGIFY(s)
|
|
||||||
#define STRINGIFY(s) #s
|
|
||||||
|
|
||||||
pthread_mutex_t db_lock = PTHREAD_MUTEX_INITIALIZER;
|
pthread_mutex_t db_lock = PTHREAD_MUTEX_INITIALIZER;
|
||||||
pthread_mutex_t var_lock = PTHREAD_MUTEX_INITIALIZER;
|
pthread_mutex_t var_lock = PTHREAD_MUTEX_INITIALIZER;
|
||||||
pthread_mutex_t task_lock = PTHREAD_MUTEX_INITIALIZER;
|
pthread_mutex_t task_lock = PTHREAD_MUTEX_INITIALIZER;
|
||||||
@@ -1747,8 +1744,8 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
|||||||
|
|
||||||
key_pool key_pool;
|
key_pool key_pool;
|
||||||
|
|
||||||
std::atomic<bool> within[child_shards];
|
std::vector<std::atomic<bool> > within(child_shards);
|
||||||
long long start_geompos[child_shards];
|
std::vector<long long> start_geompos(child_shards);
|
||||||
for (size_t i = 0; i < (size_t) child_shards; i++) {
|
for (size_t i = 0; i < (size_t) child_shards; i++) {
|
||||||
within[i] = false;
|
within[i] = false;
|
||||||
start_geompos[i] = -1;
|
start_geompos[i] = -1;
|
||||||
@@ -1813,10 +1810,10 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
|||||||
rpa.along = along;
|
rpa.along = along;
|
||||||
rpa.alongminus = alongminus;
|
rpa.alongminus = alongminus;
|
||||||
rpa.buffer = buffer;
|
rpa.buffer = buffer;
|
||||||
rpa.within = within;
|
rpa.within = within.data();
|
||||||
rpa.geomfile = geomfile;
|
rpa.geomfile = geomfile;
|
||||||
rpa.geompos = geompos;
|
rpa.geompos = geompos;
|
||||||
rpa.start_geompos = start_geompos;
|
rpa.start_geompos = start_geompos.data();
|
||||||
rpa.oprogress = &oprogress;
|
rpa.oprogress = &oprogress;
|
||||||
rpa.todo = todo;
|
rpa.todo = todo;
|
||||||
rpa.fname = fname;
|
rpa.fname = fname;
|
||||||
@@ -1861,7 +1858,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
|||||||
ssize_t which_serial_feature = -1;
|
ssize_t which_serial_feature = -1;
|
||||||
|
|
||||||
if (prefilter == NULL) {
|
if (prefilter == NULL) {
|
||||||
sf = next_feature(geoms, geompos_in, z, tx, ty, initial_x, initial_y, &original_features, &unclipped_features, nextzoom, maxzoom, minzoom, max_zoom_increment, pass, along, alongminus, buffer, within, geomfile, geompos, start_geompos, &oprogress, todo, fname, child_shards, filter, global_stringpool, pool_off, layer_unmaps, first_time, compressed_input, &multiplier_state, tile_stringpool, unidecode_data, next_feature_state, arg->droprate);
|
sf = next_feature(geoms, geompos_in, z, tx, ty, initial_x, initial_y, &original_features, &unclipped_features, nextzoom, maxzoom, minzoom, max_zoom_increment, pass, along, alongminus, buffer, within.data(), geomfile, geompos, start_geompos.data(), &oprogress, todo, fname, child_shards, filter, global_stringpool, pool_off, layer_unmaps, first_time, compressed_input, &multiplier_state, tile_stringpool, unidecode_data, next_feature_state, arg->droprate);
|
||||||
} else {
|
} else {
|
||||||
sf = parse_feature(prefilter_jp, z, tx, ty, layermaps, tiling_seg, layer_unmaps, postfilter != NULL, key_pool);
|
sf = parse_feature(prefilter_jp, z, tx, ty, layermaps, tiling_seg, layer_unmaps, postfilter != NULL, key_pool);
|
||||||
}
|
}
|
||||||
@@ -2398,7 +2395,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
|||||||
if (p.clustered > 0) {
|
if (p.clustered > 0) {
|
||||||
serial_val sv, sv2, sv3, sv4;
|
serial_val sv, sv2, sv3, sv4;
|
||||||
long long point_count = p.clustered + 1;
|
long long point_count = p.clustered + 1;
|
||||||
char abbrev[20]; // to_string(LLONG_MAX).length() / 1000 + 1;
|
char abbrev[24]; // fits "%lld" of any long long, including the sign and the NUL
|
||||||
|
|
||||||
p.full_keys.push_back(key_pool.pool("clustered"));
|
p.full_keys.push_back(key_pool.pool("clustered"));
|
||||||
sv.type = mvt_bool;
|
sv.type = mvt_bool;
|
||||||
@@ -2448,7 +2445,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
|
|||||||
}
|
}
|
||||||
|
|
||||||
{
|
{
|
||||||
pthread_t pthreads[tasks];
|
std::vector<pthread_t> pthreads(tasks);
|
||||||
std::vector<simplification_worker_arg> args;
|
std::vector<simplification_worker_arg> args;
|
||||||
args.resize(tasks);
|
args.resize(tasks);
|
||||||
for (int i = 0; i < tasks; i++) {
|
for (int i = 0; i < tasks; i++) {
|
||||||
@@ -3252,28 +3249,27 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *global_stringpool, std::
|
|||||||
for (z = iz; z <= maxzoom; z++) {
|
for (z = iz; z <= maxzoom; z++) {
|
||||||
std::atomic<long long> most(0);
|
std::atomic<long long> most(0);
|
||||||
|
|
||||||
compressor compressors[TEMP_FILES];
|
std::vector<compressor> compressors(TEMP_FILES);
|
||||||
compressor *sub[TEMP_FILES];
|
std::vector<compressor *> sub(TEMP_FILES);
|
||||||
std::atomic<long long> subpos[TEMP_FILES];
|
std::vector<std::atomic<long long> > subpos(TEMP_FILES);
|
||||||
int subfd[TEMP_FILES];
|
std::vector<int> subfd(TEMP_FILES);
|
||||||
for (size_t j = 0; j < TEMP_FILES; j++) {
|
for (size_t j = 0; j < TEMP_FILES; j++) {
|
||||||
char geomname[strlen(tmpdir) + strlen("/geom.XXXXXXXX" XSTRINGIFY(INT_MAX)) + 1];
|
std::string geomname = std::string(tmpdir) + "/geom" + std::to_string(j) + ".XXXXXXXX";
|
||||||
snprintf(geomname, sizeof(geomname), "%s/geom%zu.XXXXXXXX", tmpdir, j);
|
subfd[j] = mkstemp_cloexec(&geomname[0]);
|
||||||
subfd[j] = mkstemp_cloexec(geomname);
|
// printf("%s\n", geomname.c_str());
|
||||||
// printf("%s\n", geomname);
|
|
||||||
if (subfd[j] < 0) {
|
if (subfd[j] < 0) {
|
||||||
perror(geomname);
|
perror(geomname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
FILE *fp = fopen_oflag(geomname, "wb", O_WRONLY | O_CLOEXEC);
|
FILE *fp = fopen_oflag(geomname.c_str(), "wb", O_WRONLY | O_CLOEXEC);
|
||||||
if (fp == NULL) {
|
if (fp == NULL) {
|
||||||
perror(geomname);
|
perror(geomname.c_str());
|
||||||
exit(EXIT_OPEN);
|
exit(EXIT_OPEN);
|
||||||
}
|
}
|
||||||
compressors[j] = compressor(fp);
|
compressors[j] = compressor(fp);
|
||||||
sub[j] = &compressors[j];
|
sub[j] = &compressors[j];
|
||||||
subpos[j] = 0;
|
subpos[j] = 0;
|
||||||
unlink(geomname);
|
unlink(geomname.c_str());
|
||||||
}
|
}
|
||||||
|
|
||||||
size_t useful_threads = 0;
|
size_t useful_threads = 0;
|
||||||
@@ -3341,7 +3337,7 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *global_stringpool, std::
|
|||||||
std::set<zxy> skip_children_out;
|
std::set<zxy> skip_children_out;
|
||||||
|
|
||||||
for (size_t pass = 0;; pass++) {
|
for (size_t pass = 0;; pass++) {
|
||||||
pthread_t pthreads[threads];
|
std::vector<pthread_t> pthreads(threads);
|
||||||
std::vector<write_tile_args> args;
|
std::vector<write_tile_args> args;
|
||||||
args.resize(threads);
|
args.resize(threads);
|
||||||
std::atomic<int> running(threads);
|
std::atomic<int> running(threads);
|
||||||
@@ -3360,8 +3356,8 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *global_stringpool, std::
|
|||||||
args[thread].outdir = outdir;
|
args[thread].outdir = outdir;
|
||||||
args[thread].buffer = buffer;
|
args[thread].buffer = buffer;
|
||||||
args[thread].fname = fname;
|
args[thread].fname = fname;
|
||||||
args[thread].geomfile = sub + thread * (TEMP_FILES / threads);
|
args[thread].geomfile = sub.data() + thread * (TEMP_FILES / threads);
|
||||||
args[thread].geompos = subpos + thread * (TEMP_FILES / threads);
|
args[thread].geompos = subpos.data() + thread * (TEMP_FILES / threads);
|
||||||
args[thread].todo = todo;
|
args[thread].todo = todo;
|
||||||
args[thread].along = &along; // locked with var_lock
|
args[thread].along = &along; // locked with var_lock
|
||||||
args[thread].gamma = zoom_gamma;
|
args[thread].gamma = zoom_gamma;
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
#ifndef VERSION_HPP
|
#ifndef VERSION_HPP
|
||||||
#define VERSION_HPP
|
#define VERSION_HPP
|
||||||
|
|
||||||
#define VERSION "v2.80.0"
|
#define VERSION "v2.81.0"
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|||||||
Reference in New Issue
Block a user