Compare commits

...
5 Commits
16 changed files with 64 additions and 68892 deletions
+1 -14
View File
@@ -417,7 +417,6 @@ be reduced to the maximum that can be used with the specified _maxzoom_.
* `-pe` or `--empty-csv-columns-are-null`: Treat empty CSV columns as nulls rather than as empty strings.
* `-aI` or `--convert-stringified-ids-to-numbers`: If a feature ID is the string representation of a number, convert it to a plain number to use as the feature ID.
* `--use-attribute-for-id=`*name*: Use the attribute with the specified *name* as if it were specified as the feature ID. (If this attribute is a stringified number, you must also use `-aI` to convert it to a number.)
* `-pN` or `--single-precision`: Write double-precision numeric attribute values to tiles as single-precision to reduce tile size.
### Filtering features by attributes
@@ -460,8 +459,6 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
You can also specify a marker-width with `-Bg`*width* to allow fewer features in the densest tile to
compensate for the larger marker, or `-Bf`*number* to allow at most *number* features in the densest tile.
* `--retain-points-multiplier=`_multiple_: Retain the specified multiple of points instead of just the number of points that would ordinarily be retained by the drop rate. These can be thinned out later with the `-m` option to `tippecanoe-overzoom`. The start of each cluster is marked in the feature sequence by the `tippecanoe:retain_points_multiplier_first` attribute.
* `--drop-denser=`_percentage_: When dropping dots at zoom levels below the base zoom, give the specified _percentage_
preference to retaining points in sparse areas and dropping points in dense areas.
* `--limit-base-zoom-to-maximum-zoom` or `-Pb`: Limit the guessed base zoom not to exceed the maxzoom, even if this would put more than the requested number of features in a base zoom tile.
* `-al` or `--drop-lines`: Let "dot" dropping at lower zooms apply to lines too
* `-ap` or `--drop-polygons`: Let "dot" dropping at lower zooms apply to polygons too
@@ -478,14 +475,8 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
* `-aN` or `--coalesce-smallest-as-needed`: Dynamically combine the smallest features (physically smallest: the shortest lines or the smallest polygons or the densest points) from each zoom level into other nearby features to keep large tiles under the 500K size limit. This option will probably not help very much with LineStrings. It is mostly intended for polygons, to maintain the full original area covered by polygons while still reducing the feature count somehow. The attributes of the small polygons are *not* preserved into the combined features (except through `--accumulate-attribute`), only their geometry. Furthermore, the polygons to which nested polygons are coalesced may not necessarily be the immediately enclosing features.
* `-aD` or `--coalesce-densest-as-needed`: Dynamically combine the densest features from each zoom level into other nearby features to keep large tiles under the 500K size limit. (Again, mostly useful for polygons.)
* `-aS` or `--coalesce-fraction-as-needed`: Dynamically combine a fraction of features from each zoom level into other nearby features to keep large tiles under the 500K size limit. (Again, mostly useful for polygons.)
* `-pd` or `--force-feature-limit`: Dynamically drop some fraction of features from large tiles to keep them under the 500K size limit. It will probably look ugly at the tile boundaries. (This is like `-ad` but applies to each tile individually, not to the entire zoom level.) You probably don't want to use this.
* `-aC` or `--cluster-densest-as-needed`: If a tile is too large, try to reduce its size by increasing the minimum spacing between features, and leaving one placeholder feature from each group. The remaining feature will be given a `"clustered": true` attribute to indicate that it represents a cluster, a `"point_count"` attribute to indicate the number of features that were clustered into it, and a `"sqrt_point_count"` attribute to indicate the relative width of a feature to represent the cluster. If the features being clustered are points, the representative feature will be located at the average of the original points' locations; otherwise, one of the original features will be left as the representative.
### Dropping tightly overlapping features
* `-g` _gamma_ or `--gamma=_gamma`_: Rate at which especially dense dots are dropped (default 0, for no effect). A gamma of 2 reduces the number of dots less than a pixel apart to the square root of their original number.
* `-aG` or `--increase-gamma-as-needed`: If a tile is too large, try to reduce it to under 500K by increasing the `-g` gamma. The discovered gamma applies to the entire zoom level. You probably want to use `--drop-densest-as-needed` instead.
### Line and polygon simplification
* `-S` _scale_ or `--simplification=`_scale_: Multiply the tolerance for line and polygon simplification by _scale_. The standard tolerance tries to keep
@@ -576,7 +567,7 @@ as a series of newline-delimited GeoJSON objects on the standard input, and `tip
set of GeoJSON features from the filter's standard output.
The prefilter receives the features at the highest available resolution, before line simplification,
polygon topology repair, gamma calculation, dynamic feature dropping, or other internal processing.
polygon topology repair, dynamic feature dropping, or other internal processing.
The postfilter receives the features at tile resolution, after simplification, cleaning, and dropping.
The layer name is provided as part of the `tippecanoe` element of the feature and must be passed through
@@ -671,10 +662,6 @@ a different zoom specified with `-B` if you have precise but sparse data).
I don't know why 2.5 is the appropriate number, but the densities of many different
data sets fall off at about this same rate. You can use -r to specify a different rate.
You can use the gamma option to thin out especially dense clusters of points.
For any area where dots are closer than one pixel together (at whatever zoom level),
a gamma of 3, for example, will reduce these clusters to the cube root of their original density.
For line features, it drops any features that are too small to draw at all.
This still leaves the lower zooms too dark (and too dense for the 500K tile limit,
in some places), so I need to figure out an equitable way to throw features away.
+32 -141
View File
@@ -90,7 +90,6 @@ long justx = -1, justy = -1;
std::string attribute_for_id = "";
size_t limit_tile_feature_count = 0;
size_t limit_tile_feature_count_at_maxzoom = 0;
unsigned int drop_denser = 0;
std::map<std::string, serial_val> set_attributes;
unsigned long long preserve_point_density_threshold = 0;
long long extend_zooms_max = 0;
@@ -296,12 +295,12 @@ struct drop_densest {
}
};
int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, double gamma) {
int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom) {
int feature_minzoom = 0;
if (gamma >= 0 && (ix->t == VT_POINT ||
(additional[A_LINE_DROP] && ix->t == VT_LINE) ||
(additional[A_POLYGON_DROP] && ix->t == VT_POLYGON))) {
if ((ix->t == VT_POINT ||
(additional[A_LINE_DROP] && ix->t == VT_LINE) ||
(additional[A_POLYGON_DROP] && ix->t == VT_POLYGON))) {
for (ssize_t i = maxzoom; i >= 0; i--) {
ds[i].seq++;
}
@@ -342,14 +341,12 @@ int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, d
}
}
}
// XXX manage_gap
}
return feature_minzoom;
}
static void merge(struct mergelist *merges, size_t nmerges, unsigned char *map, FILE *indexfile, int bytes, char *geom_map, FILE *geom_out, std::atomic<long long> *geompos, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, double gamma, struct drop_state *ds) {
static void merge(struct mergelist *merges, size_t nmerges, unsigned char *map, FILE *indexfile, int bytes, char *geom_map, FILE *geom_out, std::atomic<long long> *geompos, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, struct drop_state *ds) {
struct mergelist *head = NULL;
for (size_t i = 0; i < nmerges; i++) {
@@ -368,7 +365,7 @@ static void merge(struct mergelist *merges, size_t nmerges, unsigned char *map,
// and is writing one byte less and then adding the byte for the minzoom.
fwrite_check(geom_map + ix.start, 1, ix.end - ix.start - 1, geom_out, geompos, "merge geometry");
int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom, gamma);
int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom);
serialize_byte(geom_out, feature_minzoom, geompos, "merge geometry");
// Count this as an 75%-accomplishment, since we already 25%-counted it
@@ -448,7 +445,7 @@ void *run_sort(void *v) {
return NULL;
}
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
long long segs[CPUS + 1];
segs[0] = 0;
segs[CPUS] = len;
@@ -501,7 +498,6 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
sst[i].dist_count = &(dist_counts[i]);
sst[i].want_dist = want_dist;
sst[i].maxzoom = maxzoom;
sst[i].uses_gamma = uses_gamma;
sst[i].filters = filters;
sst[i].layermap = &(*layermaps)[i];
sst[i].exclude = exclude;
@@ -648,7 +644,6 @@ struct read_parallel_arg {
unsigned *initial_x = NULL;
unsigned *initial_y = NULL;
std::string layername = "";
bool uses_gamma = false;
std::map<std::string, int> const *attribute_types = NULL;
double *dist_sum = NULL;
size_t *dist_count = NULL;
@@ -676,7 +671,7 @@ void *run_read_parallel(void *v) {
}
madvise(map, rpa->len, MADV_RANDOM); // sequential, but from several pointers at once
do_read_parallel(map, rpa->len, rpa->offset, rpa->reading, rpa->readers, rpa->progress_seq, rpa->exclude, rpa->include, rpa->exclude_all, rpa->basezoom, rpa->source, rpa->layermaps, rpa->initialized, rpa->initial_x, rpa->initial_y, rpa->maxzoom, rpa->layername, rpa->uses_gamma, rpa->attribute_types, rpa->separator, rpa->dist_sum, rpa->dist_count, rpa->area_sum, rpa->want_dist, rpa->filters);
do_read_parallel(map, rpa->len, rpa->offset, rpa->reading, rpa->readers, rpa->progress_seq, rpa->exclude, rpa->include, rpa->exclude_all, rpa->basezoom, rpa->source, rpa->layermaps, rpa->initialized, rpa->initial_x, rpa->initial_y, rpa->maxzoom, rpa->layername, rpa->attribute_types, rpa->separator, rpa->dist_sum, rpa->dist_count, rpa->area_sum, rpa->want_dist, rpa->filters);
madvise(map, rpa->len, MADV_DONTNEED);
if (munmap(map, rpa->len) != 0) {
@@ -693,7 +688,7 @@ void *run_read_parallel(void *v) {
return NULL;
}
void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::atomic<int> *is_parsing, pthread_t *parallel_parser, bool &parser_created, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > &layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::atomic<int> *is_parsing, pthread_t *parallel_parser, bool &parser_created, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > &layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
// This has to kick off an intermediate thread to start the parser threads,
// so the main thread can get back to reading the next input stage while
// the intermediate thread waits for the completion of the parser threads.
@@ -727,7 +722,6 @@ void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::ato
rpa->initial_y = initial_y;
rpa->maxzoom = maxzoom;
rpa->layername = layername;
rpa->uses_gamma = uses_gamma;
rpa->attribute_types = attribute_types;
rpa->dist_sum = dist_sum;
rpa->dist_count = dist_count;
@@ -742,7 +736,7 @@ void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::ato
parser_created = true;
}
void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int splits, long long mem, const char *tmpdir, long long *availfiles, FILE *geomfile, FILE *indexfile, std::atomic<long long> *geompos_out, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, int basezoom, double droprate, double gamma, struct drop_state *ds) {
void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int splits, long long mem, const char *tmpdir, long long *availfiles, FILE *geomfile, FILE *indexfile, std::atomic<long long> *geompos_out, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, int basezoom, double droprate, struct drop_state *ds) {
// Arranged as bits to facilitate subdividing again if a subdivided file is still huge
int splitbits = log(splits) / log(2);
splits = 1 << splitbits;
@@ -961,7 +955,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
madvise(geommap, geomst.st_size, MADV_RANDOM);
madvise(geommap, geomst.st_size, MADV_WILLNEED);
merge(merges, nmerges, (unsigned char *) indexmap, indexfile, bytes, geommap, geomfile, geompos_out, progress, progress_max, progress_reported, maxzoom, gamma, ds);
merge(merges, nmerges, (unsigned char *) indexmap, indexfile, bytes, geommap, geomfile, geompos_out, progress, progress_max, progress_reported, maxzoom, ds);
madvise(indexmap, indexst.st_size, MADV_DONTNEED);
if (munmap(indexmap, indexst.st_size) < 0) {
@@ -995,7 +989,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
long long pos = *geompos_out;
fwrite_check(geommap + ix.start, ix.end - ix.start, 1, geomfile, geompos_out, "geom");
int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom, gamma);
int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom);
serialize_byte(geomfile, feature_minzoom, geompos_out, "merge geometry");
// Count this as an 75%-accomplishment, since we already 25%-counted it
@@ -1030,7 +1024,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
// counter backward but will be an honest estimate of the work remaining.
*progress_max += geomst.st_size / 4;
radix1(&geomfds[i], &indexfds[i], 1, prefix + splitbits, *availfiles / 4, mem, tmpdir, availfiles, geomfile, indexfile, geompos_out, progress, progress_max, progress_reported, maxzoom, basezoom, droprate, gamma, ds);
radix1(&geomfds[i], &indexfds[i], 1, prefix + splitbits, *availfiles / 4, mem, tmpdir, availfiles, geomfile, indexfile, geompos_out, progress, progress_max, progress_reported, maxzoom, basezoom, droprate, ds);
already_closed = 1;
}
}
@@ -1090,7 +1084,7 @@ static size_t calc_memsize() {
return mem;
}
void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FILE *indexfile, const char *tmpdir, std::atomic<long long> *geompos, int maxzoom, int basezoom, double droprate, double gamma) {
void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FILE *indexfile, const char *tmpdir, std::atomic<long long> *geompos, int maxzoom, int basezoom, double droprate) {
// Run through the index and geometry for each reader,
// splitting the contents out by index into as many
// sub-files as we can write to simultaneously.
@@ -1140,7 +1134,7 @@ void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FI
long long progress = 0, progress_max = geom_total, progress_reported = -1;
long long availfiles_before = availfiles;
radix1(geomfds, indexfds, nreaders, 0, splits, mem, tmpdir, &availfiles, geomfile, indexfile, geompos, &progress, &progress_max, &progress_reported, maxzoom, basezoom, droprate, gamma, ds);
radix1(geomfds, indexfds, nreaders, 0, splits, mem, tmpdir, &availfiles, geomfile, indexfile, geompos, &progress, &progress_max, &progress_reported, maxzoom, basezoom, droprate, ds);
if (availfiles - 2 * nreaders != availfiles_before) {
fprintf(stderr, "Internal error: miscounted available file descriptors: %lld vs %lld\n", availfiles - 2 * nreaders, availfiles);
@@ -1237,7 +1231,7 @@ int vertexcmp(const void *void1, const void *void2) {
return 0;
}
std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzoom, int basezoom, double basezoom_marker_width, sqlite3 *outdb, const char *outdir, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, json_object *filter, double droprate, int buffer, const char *tmpdir, double gamma, int read_parallel, int forcetable, const char *attribution, bool uses_gamma, long long *file_bbox, long long *file_bbox1, long long *file_bbox2, const char *prefilter, const char *postfilter, const char *description, bool guess_maxzoom, bool guess_cluster_maxzoom, std::map<std::string, int> const *attribute_types, const char *pgm, std::map<std::string, attribute_op> const *attribute_accum, std::map<std::string, std::string> const &attribute_descriptions, std::string const &commandline, int minimum_maxzoom) {
std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzoom, int basezoom, double basezoom_marker_width, sqlite3 *outdb, const char *outdir, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, json_object *filter, double droprate, int buffer, const char *tmpdir, int read_parallel, int forcetable, const char *attribution, long long *file_bbox, long long *file_bbox1, long long *file_bbox2, const char *prefilter, const char *postfilter, const char *description, bool guess_maxzoom, bool guess_cluster_maxzoom, std::map<std::string, int> const *attribute_types, const char *pgm, std::map<std::string, attribute_op> const *attribute_accum, std::map<std::string, std::string> const &attribute_descriptions, std::string const &commandline, int minimum_maxzoom) {
int ret = EXIT_SUCCESS;
std::vector<struct reader> readers;
@@ -1514,7 +1508,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
sst[i].want_dist = guess_maxzoom;
sst[i].maxzoom = maxzoom;
sst[i].filters = prefilter != NULL || postfilter != NULL;
sst[i].uses_gamma = uses_gamma;
sst[i].layermap = &layermaps[i];
sst[i].exclude = exclude;
sst[i].include = include;
@@ -1587,7 +1580,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
sst[i].want_dist = guess_maxzoom;
sst[i].maxzoom = maxzoom;
sst[i].filters = prefilter != NULL || postfilter != NULL;
sst[i].uses_gamma = uses_gamma;
sst[i].layermap = &layermaps[i];
sst[i].exclude = exclude;
sst[i].include = include;
@@ -1649,7 +1641,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
sst[i].want_dist = guess_maxzoom;
sst[i].maxzoom = maxzoom;
sst[i].filters = prefilter != NULL || postfilter != NULL;
sst[i].uses_gamma = uses_gamma;
sst[i].layermap = &layermaps[i];
sst[i].exclude = exclude;
sst[i].include = include;
@@ -1707,7 +1698,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
}
if (map != NULL && map != MAP_FAILED && read_parallel_this) {
do_read_parallel(map, st.st_size - off, overall_offset, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, &layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, uses_gamma, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
do_read_parallel(map, st.st_size - off, overall_offset, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, &layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
overall_offset += st.st_size - off;
checkdisk(&readers);
@@ -1786,7 +1777,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
}
fflush(readfp);
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, &parallel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, &parallel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
initial_offset += ahead;
overall_offset += ahead;
@@ -1823,7 +1814,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
fflush(readfp);
if (ahead > 0) {
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, &parallel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, &parallel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
if (parser_created) {
if (pthread_join(parallel_parser, NULL) != 0) {
@@ -1857,7 +1848,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
sst.want_dist = guess_maxzoom;
sst.maxzoom = maxzoom;
sst.filters = prefilter != NULL || postfilter != NULL;
sst.uses_gamma = uses_gamma;
sst.layermap = &layermaps[0];
sst.exclude = exclude;
sst.include = include;
@@ -2219,7 +2209,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
serialize_uint(geomfile, ix, &geompos, fname);
serialize_uint(geomfile, iy, &geompos, fname);
radix(readers, CPUS, geomfile, indexfile, tmpdir, &geompos, maxzoom, basezoom, droprate, gamma);
radix(readers, CPUS, geomfile, indexfile, tmpdir, &geompos, maxzoom, basezoom, droprate);
/* end of tile */
serialize_ulong_long(geomfile, 0, &geompos, fname); // EOF
@@ -2521,8 +2511,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
yyy = yy >> (32 - z);
}
double scale = (double) (1LL << (64 - 2 * (z + 8)));
if (tile[z].x != xxx || tile[z].y != yyy) {
if (tile[z].count > max[z].count) {
max[z] = tile[z];
@@ -2537,11 +2525,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
}
tile[z].fullcount++;
if (manage_gap(map[ip].ix, &tile[z].previndex, scale, gamma, &tile[z].gap)) {
continue;
}
tile[z].count++;
}
}
@@ -2578,9 +2561,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
if (obasezoom < 0 && basezoom > maxzoom) {
fprintf(stderr, "Couldn't find a suitable base zoom. Working from the other direction.\n");
if (gamma == 0) {
fprintf(stderr, "You might want to try -g1 to limit near-duplicates.\n");
}
if (droprate < 0) {
if (maxzoom == 0) {
@@ -2622,44 +2602,10 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
}
}
if (gamma > 0) {
int effective = 0;
for (z = 0; z < maxzoom; z++) {
if (max[z].count < max[z].fullcount) {
effective = z + 1;
}
}
if (effective == 0) {
if (!quiet) {
fprintf(stderr, "With gamma, effective base zoom is 0, so no effective drop rate\n");
}
} else {
double interval_0 = exp(log(droprate) * (basezoom - 0));
double interval_eff = exp(log(droprate) * (basezoom - effective));
if (effective > basezoom) {
interval_eff = 1;
}
double scaled_0 = max[0].count / interval_0;
double scaled_eff = max[effective].count / interval_eff;
double rate_at_0 = scaled_0 / max[0].fullcount;
double rate_at_eff = scaled_eff / max[effective].fullcount;
double eff_drop = exp(log(rate_at_eff / rate_at_0) / (effective - 0));
if (!quiet) {
fprintf(stderr, "With gamma, effective base zoom of %d, effective drop rate of %f\n", effective, eff_drop);
}
}
}
fix_dropping = true;
}
if (fix_dropping || drop_denser > 0) {
if (fix_dropping) {
// Fix up the minzooms for features, now that we really know the base zoom
// and drop rate.
@@ -2679,48 +2625,12 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
struct drop_state ds[maxzoom + 1];
prep_drop_states(ds, maxzoom, basezoom, droprate);
if (drop_denser > 0) {
std::vector<drop_densest> ddv;
unsigned long long previndex = 0;
for (long long ip = 0; ip < indices; ip++) {
if (map[ip].t == VT_POINT ||
(additional[A_LINE_DROP] && map[ip].t == VT_LINE) ||
(additional[A_POLYGON_DROP] && map[ip].t == VT_POLYGON)) {
if (map[ip].ix % 100 < drop_denser) {
drop_densest dd;
dd.gap = map[ip].ix - previndex;
dd.seq = ip;
ddv.push_back(dd);
previndex = map[ip].ix;
} else {
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom, gamma);
geom[map[ip].end - 1] = feature_minzoom;
}
}
}
std::sort(ddv.begin(), ddv.end());
size_t i = 0;
for (int z = 0; z <= basezoom; z++) {
double keep_fraction = 1.0 / std::exp(std::log(droprate) * (basezoom - z));
size_t keep_count = ddv.size() * keep_fraction;
for (; i < keep_count && i < ddv.size(); i++) {
geom[map[ddv[i].seq].end - 1] = z;
}
}
for (; i < ddv.size(); i++) {
geom[map[ddv[i].seq].end - 1] = basezoom;
}
} else {
{
for (long long ip = 0; ip < indices; ip++) {
if (ip > 0 && map[ip].start != map[ip - 1].end) {
fprintf(stderr, "Mismatched index at %lld: %lld vs %lld\n", ip, map[ip].start, map[ip].end);
}
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom, gamma);
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom);
geom[map[ip].end - 1] = feature_minzoom;
}
}
@@ -2757,7 +2667,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
std::atomic<unsigned> midx(0);
std::atomic<unsigned> midy(0);
std::vector<strategy> strategies;
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos);
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos);
if (maxzoom != written) {
if (written > minzoom) {
@@ -2993,7 +2903,6 @@ int main(int argc, char **argv) {
int force = 0;
int forcetable = 0;
double droprate = 2.5;
double gamma = 0;
int buffer = 5;
const char *tmpdir = "/tmp";
const char *attribution = NULL;
@@ -3068,7 +2977,6 @@ int main(int argc, char **argv) {
{"empty-csv-columns-are-null", no_argument, &prevent[P_EMPTY_CSV_COLUMNS], 1},
{"convert-stringified-ids-to-numbers", no_argument, &additional[A_CONVERT_NUMERIC_IDS], 1},
{"use-attribute-for-id", required_argument, 0, '~'},
{"single-precision", no_argument, &prevent[P_SINGLE_PRECISION], 1},
{"set-attribute", required_argument, 0, '~'},
{"Filtering features by attributes", 0, 0, 0},
@@ -3079,7 +2987,6 @@ int main(int argc, char **argv) {
{"drop-rate", required_argument, 0, 'r'},
{"retain-points-multiplier", required_argument, 0, '~'},
{"base-zoom", required_argument, 0, 'B'},
{"drop-denser", required_argument, 0, '~'},
{"limit-base-zoom-to-maximum-zoom", no_argument, &prevent[P_BASEZOOM_ABOVE_MAXZOOM], 1},
{"drop-lines", no_argument, &additional[A_LINE_DROP], 1},
{"drop-polygons", no_argument, &additional[A_POLYGON_DROP], 1},
@@ -3094,13 +3001,8 @@ int main(int argc, char **argv) {
{"coalesce-densest-as-needed", no_argument, &additional[A_COALESCE_DENSEST_AS_NEEDED], 1},
{"coalesce-fraction-as-needed", no_argument, &additional[A_COALESCE_FRACTION_AS_NEEDED], 1},
{"coalesce-smallest-as-needed", no_argument, &additional[A_COALESCE_SMALLEST_AS_NEEDED], 1},
{"force-feature-limit", no_argument, &prevent[P_DYNAMIC_DROP], 1},
{"cluster-densest-as-needed", no_argument, &additional[A_CLUSTER_DENSEST_AS_NEEDED], 1},
{"Dropping tightly overlapping features", 0, 0, 0},
{"gamma", required_argument, 0, 'g'},
{"increase-gamma-as-needed", no_argument, &additional[A_INCREASE_GAMMA_AS_NEEDED], 1},
{"Line and polygon simplification", 0, 0, 0},
{"simplification", required_argument, 0, 'S'},
{"no-line-simplification", no_argument, &prevent[P_SIMPLIFY], 1},
@@ -3113,7 +3015,6 @@ int main(int argc, char **argv) {
{"visvalingam", no_argument, &additional[A_VISVALINGAM], 1},
{"Attempts to improve shared polygon boundaries", 0, 0, 0},
{"detect-shared-borders", no_argument, &additional[A_DETECT_SHARED_BORDERS], 1},
{"grid-low-zooms", no_argument, &additional[A_GRID_LOW_ZOOMS], 1},
{"Controlling clipping to tile boundaries", 0, 0, 0},
@@ -3178,6 +3079,9 @@ int main(int argc, char **argv) {
{"prefer-radix-sort", no_argument, &additional[A_PREFER_RADIX_SORT], 1},
{"help", no_argument, 0, 'H'},
{"Redundant", 0, 0, 0},
{"detect-shared-borders", no_argument, &prevent[P_SIMPLIFY_SHARED_NODES], 1},
{0, 0, 0, 0},
};
@@ -3204,6 +3108,10 @@ int main(int argc, char **argv) {
getopt_str[cout] = '\0';
for (size_t lo = 0; long_options[lo].name != NULL; lo++) {
if (strcmp(long_options[lo].name, "Redundant")) {
break;
}
if (long_options[lo].flag != NULL) {
if (*long_options[lo].flag != 0) {
fprintf(stderr, "Internal error: reused %s\n", long_options[lo].name);
@@ -3288,12 +3196,6 @@ int main(int argc, char **argv) {
limit_tile_feature_count = atoll_require(optarg, "Limit tile feature count");
} else if (strcmp(opt, "limit-tile-feature-count-at-maximum-zoom") == 0) {
limit_tile_feature_count_at_maxzoom = atoll_require(optarg, "Limit tile feature count at maxzoom");
} else if (strcmp(opt, "drop-denser") == 0) {
drop_denser = atoi_require(optarg, "Drop denser rate");
if (drop_denser > 100) {
fprintf(stderr, "%s: --drop-denser can be at most 100\n", argv[0]);
exit(EXIT_ARGS);
}
} else if (strcmp(opt, "preserve-point-density-threshold") == 0) {
preserve_point_density_threshold = atoll_require(optarg, "Preserve point density threshold");
} else if (strcmp(opt, "extend-zooms-if-still-dropping-maximum") == 0) {
@@ -3534,10 +3436,6 @@ int main(int argc, char **argv) {
}
break;
case 'g':
gamma = atof_require(optarg, "Gamma");
break;
case 'q':
quiet = 1;
break;
@@ -3760,13 +3658,6 @@ int main(int argc, char **argv) {
}
}
if ((basezoom < 0 || droprate < 0) && (gamma < 0)) {
// Can't use randomized (as opposed to evenly distributed) dot dropping
// if rate and base aren't known during feature reading.
gamma = 0;
fprintf(stderr, "Forcing -g0 since -B or -r is not known\n");
}
if (out_mbtiles == NULL && out_dir == NULL) {
fprintf(stderr, "%s: must specify -o out.mbtiles or -e directory\n", argv[0]);
exit(EXIT_ARGS);
@@ -3821,7 +3712,7 @@ int main(int argc, char **argv) {
auto input_ret = read_input(sources, name ? name : out_mbtiles ? out_mbtiles
: out_dir,
maxzoom, minzoom, basezoom, basezoom_marker_width, outdb, out_dir, &exclude, &include, exclude_all, filter, droprate, buffer, tmpdir, gamma, read_parallel, forcetable, attribution, gamma != 0, file_bbox, file_bbox1, file_bbox2, prefilter, postfilter, description, guess_maxzoom, guess_cluster_maxzoom, &attribute_types, argv[0], &attribute_accum, attribute_descriptions, commandline, minimum_maxzoom);
maxzoom, minzoom, basezoom, basezoom_marker_width, outdb, out_dir, &exclude, &include, exclude_all, filter, droprate, buffer, tmpdir, read_parallel, forcetable, attribution, file_bbox, file_bbox1, file_bbox2, prefilter, postfilter, description, guess_maxzoom, guess_cluster_maxzoom, &attribute_types, argv[0], &attribute_accum, attribute_descriptions, commandline, minimum_maxzoom);
ret = std::get<0>(input_ret);
+1 -19
View File
@@ -520,8 +520,6 @@ The attributes and values may also be specified as JSON keys and values: \fB\fC\
\fB\fC\-aI\fR or \fB\fC\-\-convert\-stringified\-ids\-to\-numbers\fR: If a feature ID is the string representation of a number, convert it to a plain number to use as the feature ID.
.IP \(bu 2
\fB\fC\-\-use\-attribute\-for\-id=\fR\fIname\fP: Use the attribute with the specified \fIname\fP as if it were specified as the feature ID. (If this attribute is a stringified number, you must also use \fB\fC\-aI\fR to convert it to a number.)
.IP \(bu 2
\fB\fC\-pN\fR or \fB\fC\-\-single\-precision\fR: Write double\-precision numeric attribute values to tiles as single\-precision to reduce tile size.
.RE
.SS Filtering features by attributes
.RS
@@ -576,9 +574,6 @@ compensate for the larger marker, or \fB\fC\-Bf\fR\fInumber\fP to allow at most
.IP \(bu 2
\fB\fC\-\-retain\-points\-multiplier=\fR\fImultiple\fP: Retain the specified multiple of points instead of just the number of points that would ordinarily be retained by the drop rate. These can be thinned out later with the \fB\fC\-m\fR option to \fB\fCtippecanoe\-overzoom\fR\&. The start of each cluster is marked in the feature sequence by the \fB\fCtippecanoe:retain_points_multiplier_first\fR attribute.
.IP \(bu 2
\fB\fC\-\-drop\-denser=\fR\fIpercentage\fP: When dropping dots at zoom levels below the base zoom, give the specified \fIpercentage\fP
preference to retaining points in sparse areas and dropping points in dense areas.
.IP \(bu 2
\fB\fC\-\-limit\-base\-zoom\-to\-maximum\-zoom\fR or \fB\fC\-Pb\fR: Limit the guessed base zoom not to exceed the maxzoom, even if this would put more than the requested number of features in a base zoom tile.
.IP \(bu 2
\fB\fC\-al\fR or \fB\fC\-\-drop\-lines\fR: Let "dot" dropping at lower zooms apply to lines too
@@ -608,17 +603,8 @@ preference to retaining points in sparse areas and dropping points in dense area
.IP \(bu 2
\fB\fC\-aS\fR or \fB\fC\-\-coalesce\-fraction\-as\-needed\fR: Dynamically combine a fraction of features from each zoom level into other nearby features to keep large tiles under the 500K size limit. (Again, mostly useful for polygons.)
.IP \(bu 2
\fB\fC\-pd\fR or \fB\fC\-\-force\-feature\-limit\fR: Dynamically drop some fraction of features from large tiles to keep them under the 500K size limit. It will probably look ugly at the tile boundaries. (This is like \fB\fC\-ad\fR but applies to each tile individually, not to the entire zoom level.) You probably don't want to use this.
.IP \(bu 2
\fB\fC\-aC\fR or \fB\fC\-\-cluster\-densest\-as\-needed\fR: If a tile is too large, try to reduce its size by increasing the minimum spacing between features, and leaving one placeholder feature from each group. The remaining feature will be given a \fB\fC"clustered": true\fR attribute to indicate that it represents a cluster, a \fB\fC"point_count"\fR attribute to indicate the number of features that were clustered into it, and a \fB\fC"sqrt_point_count"\fR attribute to indicate the relative width of a feature to represent the cluster. If the features being clustered are points, the representative feature will be located at the average of the original points' locations; otherwise, one of the original features will be left as the representative.
.RE
.SS Dropping tightly overlapping features
.RS
.IP \(bu 2
\fB\fC\-g\fR \fIgamma\fP or \fB\fC\-\-gamma=_gamma\fR_: Rate at which especially dense dots are dropped (default 0, for no effect). A gamma of 2 reduces the number of dots less than a pixel apart to the square root of their original number.
.IP \(bu 2
\fB\fC\-aG\fR or \fB\fC\-\-increase\-gamma\-as\-needed\fR: If a tile is too large, try to reduce it to under 500K by increasing the \fB\fC\-g\fR gamma. The discovered gamma applies to the entire zoom level. You probably want to use \fB\fC\-\-drop\-densest\-as\-needed\fR instead.
.RE
.SS Line and polygon simplification
.RS
.IP \(bu 2
@@ -759,7 +745,7 @@ as a series of newline\-delimited GeoJSON objects on the standard input, and \fB
set of GeoJSON features from the filter's standard output.
.PP
The prefilter receives the features at the highest available resolution, before line simplification,
polygon topology repair, gamma calculation, dynamic feature dropping, or other internal processing.
polygon topology repair, dynamic feature dropping, or other internal processing.
The postfilter receives the features at tile resolution, after simplification, cleaning, and dropping.
.PP
The layer name is provided as part of the \fB\fCtippecanoe\fR element of the feature and must be passed through
@@ -865,10 +851,6 @@ a different zoom specified with \fB\fC\-B\fR if you have precise but sparse data
I don't know why 2.5 is the appropriate number, but the densities of many different
data sets fall off at about this same rate. You can use \-r to specify a different rate.
.PP
You can use the gamma option to thin out especially dense clusters of points.
For any area where dots are closer than one pixel together (at whatever zoom level),
a gamma of 3, for example, will reduce these clusters to the cube root of their original density.
.PP
For line features, it drops any features that are too small to draw at all.
This still leaves the lower zooms too dark (and too dense for the 500K tile limit,
in some places), so I need to figure out an equitable way to throw features away.
-4
View File
@@ -7,10 +7,8 @@
#define A_LINE_DROP ((int) 'l')
#define A_DEBUG_POLYGON ((int) '@')
#define A_POLYGON_DROP ((int) 'p')
#define A_DETECT_SHARED_BORDERS ((int) 'b')
#define A_PREFER_RADIX_SORT ((int) 'R')
#define A_CALCULATE_FEATURE_DENSITY ((int) 'g')
#define A_INCREASE_GAMMA_AS_NEEDED ((int) 'G')
#define A_DROP_DENSEST_AS_NEEDED ((int) 's')
#define A_DROP_FRACTION_AS_NEEDED ((int) 'd')
#define A_DROP_SMALLEST_AS_NEEDED ((int) 'n')
@@ -30,10 +28,8 @@
#define P_SIMPLIFY ((int) 's')
#define P_SIMPLIFY_LOW ((int) 'S')
#define P_SIMPLIFY_SHARED_NODES ((int) 'n')
#define P_SINGLE_PRECISION ((int) 'N')
#define P_FEATURE_LIMIT ((int) 'f')
#define P_KILOBYTE_LIMIT ((int) 'k')
#define P_DYNAMIC_DROP ((int) 'd')
#define P_INPUT_ORDER ((int) 'i')
#define P_POLYGON_SPLIT ((int) 'p')
#define P_CLIPPING ((int) 'c')
+1 -10
View File
@@ -719,7 +719,7 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
}
}
if (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED] || additional[A_CALCULATE_FEATURE_DENSITY] || additional[A_DROP_SMALLEST_AS_NEEDED] || additional[A_COALESCE_SMALLEST_AS_NEEDED] || additional[A_INCREASE_GAMMA_AS_NEEDED] || additional[A_GENERATE_POLYGON_LABEL_POINTS] || sst->uses_gamma || cluster_distance != 0) {
if (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED] || additional[A_CALCULATE_FEATURE_DENSITY] || additional[A_DROP_SMALLEST_AS_NEEDED] || additional[A_COALESCE_SMALLEST_AS_NEEDED] || additional[A_GENERATE_POLYGON_LABEL_POINTS] || cluster_distance != 0) {
sf.index = bbox_index;
} else {
sf.index = 0;
@@ -766,15 +766,6 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
for (ssize_t i = (ssize_t) sf.full_keys.size() - 1; i >= 0; i--) {
coerce_value(sf.full_keys[i], sf.full_values[i].type, sf.full_values[i].s, sst->attribute_types);
if (prevent[P_SINGLE_PRECISION]) {
if (sf.full_values[i].type == mvt_double) {
// don't coerce integers to floats, since that is counterproductive
if (sf.full_values[i].s.find('.') != std::string::npos) {
sf.full_values[i].s = milo::dtoa_milo((float) atof(sf.full_values[i].s.c_str()));
}
}
}
if (sf.full_keys[i] == attribute_for_id) {
if (sf.full_values[i].type != mvt_double && !additional[A_CONVERT_NUMERIC_IDS]) {
static bool warned = false;
-1
View File
@@ -171,7 +171,6 @@ struct serialization_state {
int basezoom = 0;
bool filters = false;
bool uses_gamma = false;
std::map<std::string, layermap_entry> *layermap = NULL;
-96
View File
@@ -1,96 +0,0 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "0.000000,0.000000,0.000000,0.000000",
"bounds": "0.000000,0.000000,0.000000,0.000000",
"center": "0.000000,0.000000,0",
"description": "tests/attribute-type/out/-z0_-pN.json.check.mbtiles",
"format": "pbf",
"generator_options": "./tippecanoe -q -a@ -f -o tests/attribute-type/out/-z0_-pN.json.check.mbtiles -z0 -pN tests/attribute-type/in.json",
"json": "{\"vector_layers\":[{\"id\":\"in\",\"description\":\"\",\"minzoom\":0,\"maxzoom\":0,\"fields\":{\"booltype\":\"Mixed\",\"expect\":\"Mixed\",\"floattype\":\"Mixed\",\"inttype\":\"Mixed\",\"stringtype\":\"Mixed\"}}],\"tilestats\":{\"layerCount\":1,\"layers\":[{\"layer\":\"in\",\"count\":39,\"geometry\":\"Point\",\"attributeCount\":5,\"attributes\":[{\"attribute\":\"booltype\",\"count\":10,\"type\":\"mixed\",\"values\":[\"\",\"0\",0,1,\"[2,3]\",\"false\",false,\"null\",true,\"yes\"],\"min\":0,\"max\":1},{\"attribute\":\"expect\",\"count\":14,\"type\":\"mixed\",\"values\":[0,\"2\",\"2.5\",3,5,5.599999904632568,6,\"[2,3]\",\"false\",false,\"null\",\"something\",\"true\",true],\"min\":0,\"max\":6},{\"attribute\":\"floattype\",\"count\":9,\"type\":\"mixed\",\"values\":[\"\",\" 3\",\"5\",5.599999904632568,\"5.6\",\"[2,3]\",false,true,\"yes\"],\"min\":5.599999904632568,\"max\":5.599999904632568},{\"attribute\":\"inttype\",\"count\":9,\"type\":\"mixed\",\"values\":[\"\",\" 3\",\"5\",5.599999904632568,\"5.6\",\"[2,3]\",false,true,\"yes\"],\"min\":5.599999904632568,\"max\":5.599999904632568},{\"attribute\":\"stringtype\",\"count\":6,\"type\":\"mixed\",\"values\":[2,2.5,\"[2,3]\",false,\"something\",true],\"min\":2,\"max\":2.5}]}]}}",
"maxzoom": "0",
"minzoom": "0",
"name": "tests/attribute-type/out/-z0_-pN.json.check.mbtiles",
"type": "overlay",
"version": "2"
}, "features": [
{ "type": "FeatureCollection", "properties": { "zoom": 0, "x": 0, "y": 0 }, "features": [
{ "type": "FeatureCollection", "properties": { "layer": "in", "version": 2, "extent": 4096 }, "features": [
{ "type": "Feature", "properties": { "expect": false }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": "null", "expect": false }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": "", "expect": false }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": 0, "expect": false }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": "0", "expect": false }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": false, "expect": false }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": "false", "expect": false }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": true, "expect": true }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": 1, "expect": true }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": 0, "expect": false }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": "yes", "expect": true }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "booltype": "[2,3]", "expect": true }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "expect": "null" }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "stringtype": 2, "expect": "2" }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "stringtype": 2.5, "expect": "2.5" }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "stringtype": true, "expect": "true" }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "stringtype": false, "expect": "false" }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "stringtype": "something", "expect": "something" }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "stringtype": "[2,3]", "expect": "[2,3]" }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": "5", "expect": 5 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": "5.6", "expect": 6 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": 5.599999904632568, "expect": 6 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": true, "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": false, "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": "", "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": "yes", "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": " 3", "expect": 3 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "inttype": "[2,3]", "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": "5", "expect": 5 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": "5.6", "expect": 5.599999904632568 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": 5.599999904632568, "expect": 5.599999904632568 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": true, "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": false, "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": "", "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": "yes", "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": " 3", "expect": 3 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
,
{ "type": "Feature", "properties": { "floattype": "[2,3]", "expect": 0 }, "geometry": { "type": "Point", "coordinates": [ 0.000000, 0.000000 ] } }
] }
] }
] }
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large Load Diff
+10 -642
View File
@@ -514,7 +514,6 @@ struct partial {
std::vector<long long> values = std::vector<long long>();
std::vector<std::string> full_keys = std::vector<std::string>();
std::vector<serial_val> full_values = std::vector<serial_val>();
std::vector<ssize_t> arc_polygon = std::vector<ssize_t>();
long long layer = 0;
long long original_seq = 0;
unsigned long long index = 0;
@@ -631,9 +630,6 @@ double simplify_partial(partial *p, drawvec const &shared_nodes, node *shared_no
}
bool already_marked = false;
if (additional[A_DETECT_SHARED_BORDERS] && t == VT_POLYGON) {
already_marked = true;
}
if (!already_marked) {
if (p->coalesced && t == VT_POLYGON) {
@@ -748,575 +744,6 @@ int manage_gap(unsigned long long index, unsigned long long *previndex, double s
return 0;
}
// Does not fix up moveto/lineto
static drawvec reverse_subring(drawvec const &dv) {
drawvec out;
for (size_t i = dv.size(); i > 0; i--) {
out.push_back(dv[i - 1]);
}
return out;
}
struct edge {
unsigned x1 = 0;
unsigned y1 = 0;
unsigned x2 = 0;
unsigned y2 = 0;
unsigned ring = 0;
edge(unsigned _x1, unsigned _y1, unsigned _x2, unsigned _y2, unsigned _ring) {
x1 = _x1;
y1 = _y1;
x2 = _x2;
y2 = _y2;
ring = _ring;
}
bool operator<(const edge &s) const {
long long cmp = (long long) y1 - s.y1;
if (cmp == 0) {
cmp = (long long) x1 - s.x1;
}
if (cmp == 0) {
cmp = (long long) y2 - s.y2;
}
if (cmp == 0) {
cmp = (long long) x2 - s.x2;
}
return cmp < 0;
}
};
struct edgecmp_ring {
bool operator()(const edge &a, const edge &b) {
long long cmp = (long long) a.y1 - b.y1;
if (cmp == 0) {
cmp = (long long) a.x1 - b.x1;
}
if (cmp == 0) {
cmp = (long long) a.y2 - b.y2;
}
if (cmp == 0) {
cmp = (long long) a.x2 - b.x2;
}
if (cmp == 0) {
cmp = (long long) a.ring - b.ring;
}
return cmp < 0;
}
} edgecmp_ring;
bool edges_same(std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e1, std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e2) {
if ((e2.second - e2.first) != (e1.second - e1.first)) {
return false;
}
while (e1.first != e1.second) {
if (e1.first->ring != e2.first->ring) {
return false;
}
++e1.first;
++e2.first;
}
return true;
}
bool find_common_edges(std::vector<partial> &partials, int z, int line_detail, double simplification, int maxzoom, double merge_fraction) {
size_t merge_count = ceil((1 - merge_fraction) * partials.size());
for (size_t i = 0; i < partials.size(); i++) {
if (partials[i].t == VT_POLYGON) {
for (size_t j = 0; j < partials[i].geoms.size(); j++) {
drawvec &g = partials[i].geoms[j];
drawvec out;
for (size_t k = 0; k < g.size(); k++) {
if (g[k].op == VT_LINETO && k > 0 && g[k - 1] == g[k]) {
;
} else {
out.push_back(g[k]);
}
}
partials[i].geoms[j] = out;
}
}
}
// Construct a mapping from all polygon edges to the set of rings
// that each edge appears in. (The ring number is across all polygons;
// we don't need to look it back up, just to tell where it changes.)
std::vector<edge> edges;
size_t ring = 0;
for (size_t i = 0; i < partials.size(); i++) {
if (partials[i].t == VT_POLYGON) {
for (size_t j = 0; j < partials[i].geoms.size(); j++) {
for (size_t k = 0; k + 1 < partials[i].geoms[j].size(); k++) {
if (partials[i].geoms[j][k].op == VT_MOVETO) {
ring++;
}
if (partials[i].geoms[j][k + 1].op == VT_LINETO) {
drawvec dv;
if (partials[i].geoms[j][k] < partials[i].geoms[j][k + 1]) {
dv.push_back(partials[i].geoms[j][k]);
dv.push_back(partials[i].geoms[j][k + 1]);
} else {
dv.push_back(partials[i].geoms[j][k + 1]);
dv.push_back(partials[i].geoms[j][k]);
}
edges.push_back(edge(dv[0].x, dv[0].y, dv[1].x, dv[1].y, ring));
}
}
}
}
}
std::sort(edges.begin(), edges.end(), edgecmp_ring);
std::set<draw> necessaries;
// Now mark all the points where the set of rings using the edge on one side
// is not the same as the set of rings using the edge on the other side.
for (size_t i = 0; i < partials.size(); i++) {
if (partials[i].t == VT_POLYGON) {
for (size_t j = 0; j < partials[i].geoms.size(); j++) {
drawvec &g = partials[i].geoms[j];
for (size_t k = 0; k < g.size(); k++) {
g[k].necessary = 0;
}
for (size_t a = 0; a < g.size(); a++) {
if (g[a].op == VT_MOVETO) {
size_t b;
for (b = a + 1; b < g.size(); b++) {
if (g[b].op != VT_LINETO) {
break;
}
}
// -1 because of duplication at the end
size_t s = b - a - 1;
if (s > 0) {
drawvec left;
if (g[a + (s - 1) % s] < g[a]) {
left.push_back(g[a + (s - 1) % s]);
left.push_back(g[a]);
} else {
left.push_back(g[a]);
left.push_back(g[a + (s - 1) % s]);
}
if (left[1] < left[0]) {
fprintf(stderr, "left misordered\n");
}
std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e1 = std::equal_range(edges.begin(), edges.end(), edge(left[0].x, left[0].y, left[1].x, left[1].y, 0));
for (size_t k = 0; k < s; k++) {
drawvec right;
if (g[a + k] < g[a + k + 1]) {
right.push_back(g[a + k]);
right.push_back(g[a + k + 1]);
} else {
right.push_back(g[a + k + 1]);
right.push_back(g[a + k]);
}
std::pair<std::vector<edge>::iterator, std::vector<edge>::iterator> e2 = std::equal_range(edges.begin(), edges.end(), edge(right[0].x, right[0].y, right[1].x, right[1].y, 0));
if (right[1] < right[0]) {
fprintf(stderr, "left misordered\n");
}
if (e1.first == e1.second || e2.first == e2.second) {
fprintf(stderr, "Internal error: polygon edge lookup failed for %lld,%lld to %lld,%lld or %lld,%lld to %lld,%lld\n", left[0].x, left[0].y, left[1].x, left[1].y, right[0].x, right[0].y, right[1].x, right[1].y);
exit(EXIT_IMPOSSIBLE);
}
if (!edges_same(e1, e2)) {
g[a + k].necessary = 1;
necessaries.insert(g[a + k]);
}
e1 = e2;
}
}
a = b - 1;
}
}
}
}
}
edges.clear();
std::map<drawvec, size_t> arcs;
std::multimap<ssize_t, size_t> merge_candidates; // from arc to partial
// Roll rings that include a necessary point around so they start at one
for (size_t i = 0; i < partials.size(); i++) {
if (partials[i].t == VT_POLYGON) {
for (size_t j = 0; j < partials[i].geoms.size(); j++) {
drawvec &g = partials[i].geoms[j];
for (size_t k = 0; k < g.size(); k++) {
if (necessaries.count(g[k]) != 0) {
g[k].necessary = 1;
}
}
for (size_t k = 0; k < g.size(); k++) {
if (g[k].op == VT_MOVETO) {
ssize_t necessary = -1;
ssize_t lowest = k;
size_t l;
for (l = k + 1; l < g.size(); l++) {
if (g[l].op != VT_LINETO) {
break;
}
if (g[l].necessary) {
necessary = l;
}
if (g[l] < g[lowest]) {
lowest = l;
}
}
if (necessary < 0) {
necessary = lowest;
// Add a necessary marker if there was none in the ring,
// so the arc code below can find it.
g[lowest].necessary = 1;
}
{
drawvec tmp;
// l - 1 because the endpoint is duplicated
for (size_t m = necessary; m < l - 1; m++) {
tmp.push_back(g[m]);
}
for (ssize_t m = k; m < necessary; m++) {
tmp.push_back(g[m]);
}
// replace the endpoint
tmp.push_back(g[necessary]);
if (tmp.size() != l - k) {
fprintf(stderr, "internal error shifting ring\n");
exit(EXIT_IMPOSSIBLE);
}
for (size_t m = 0; m < tmp.size(); m++) {
if (m == 0) {
tmp[m].op = VT_MOVETO;
} else {
tmp[m].op = VT_LINETO;
}
g[k + m] = tmp[m];
}
}
// Now peel off each set of segments from one necessary point to the next
// into an "arc" as in TopoJSON
for (size_t m = k; m < l; m++) {
if (!g[m].necessary) {
fprintf(stderr, "internal error in arc building\n");
exit(EXIT_IMPOSSIBLE);
}
drawvec arc;
size_t n;
for (n = m; n < l; n++) {
arc.push_back(g[n]);
if (n > m && g[n].necessary) {
break;
}
}
auto f = arcs.find(arc);
if (f == arcs.end()) {
drawvec arc2 = reverse_subring(arc);
auto f2 = arcs.find(arc2);
if (f2 == arcs.end()) {
// Add new arc
size_t added = arcs.size() + 1;
arcs.insert(std::pair<drawvec, size_t>(arc, added));
partials[i].arc_polygon.push_back(added);
merge_candidates.insert(std::pair<ssize_t, size_t>(added, i));
} else {
partials[i].arc_polygon.push_back(-(ssize_t) f2->second);
merge_candidates.insert(std::pair<ssize_t, size_t>(-(ssize_t) f2->second, i));
}
} else {
partials[i].arc_polygon.push_back(f->second);
merge_candidates.insert(std::pair<ssize_t, size_t>(f->second, i));
}
m = n - 1;
}
partials[i].arc_polygon.push_back(0);
k = l - 1;
}
}
}
}
}
// Simplify each arc
std::vector<drawvec> simplified_arcs;
for (auto ai = arcs.begin(); ai != arcs.end(); ++ai) {
if (simplified_arcs.size() < ai->second + 1) {
simplified_arcs.resize(ai->second + 1);
}
drawvec dv = ai->first;
for (size_t i = 0; i < dv.size(); i++) {
if (i == 0) {
dv[i].op = VT_MOVETO;
} else {
dv[i].op = VT_LINETO;
}
}
if (!(prevent[P_SIMPLIFY] || (z == maxzoom && prevent[P_SIMPLIFY_LOW]) || (z < maxzoom && additional[A_GRID_LOW_ZOOMS]))) {
// tx and ty are 0 here because we aren't trying to do anything with the shared_nodes_map
simplified_arcs[ai->second] = simplify_lines(dv, z, 0, 0, line_detail, !(prevent[P_CLIPPING] || prevent[P_DUPLICATION]), simplification, 4, drawvec(), NULL, 0);
} else {
simplified_arcs[ai->second] = dv;
}
}
// If necessary, merge some adjacent polygons into some other polygons
struct merge_order {
ssize_t edge = 0;
unsigned long long gap = 0;
size_t p1 = 0;
size_t p2 = 0;
bool operator<(const merge_order &m) const {
return gap < m.gap;
}
};
std::vector<merge_order> order;
for (ssize_t i = 0; i < (ssize_t) simplified_arcs.size(); i++) {
auto r1 = merge_candidates.equal_range(i);
for (auto r1i = r1.first; r1i != r1.second; ++r1i) {
auto r2 = merge_candidates.equal_range(-i);
for (auto r2i = r2.first; r2i != r2.second; ++r2i) {
if (r1i->second != r2i->second) {
merge_order mo;
mo.edge = i;
if (partials[r1i->second].index > partials[r2i->second].index) {
mo.gap = partials[r1i->second].index - partials[r2i->second].index;
} else {
mo.gap = partials[r2i->second].index - partials[r1i->second].index;
}
mo.p1 = r1i->second;
mo.p2 = r2i->second;
order.push_back(mo);
}
}
}
}
std::sort(order.begin(), order.end());
size_t merged = 0;
for (size_t o = 0; o < order.size(); o++) {
if (merged >= merge_count) {
break;
}
size_t i = order[o].p1;
while (partials[i].renamed >= 0) {
i = partials[i].renamed;
}
size_t i2 = order[o].p2;
while (partials[i2].renamed >= 0) {
i2 = partials[i2].renamed;
}
for (size_t j = 0; j < partials[i].arc_polygon.size() && merged < merge_count; j++) {
if (partials[i].arc_polygon[j] == order[o].edge) {
{
// XXX snap links
if (partials[order[o].p2].arc_polygon.size() > 0) {
// This has to merge the ring that contains the anti-arc to this arc
// into the current ring, and then add whatever other rings were in
// that feature on to the end.
//
// This can't be good for keeping parent-child relationships among
// the rings in order, but Wagyu should sort that out later
std::vector<ssize_t> additions;
std::vector<ssize_t> &here = partials[i].arc_polygon;
std::vector<ssize_t> &other = partials[i2].arc_polygon;
#if 0
printf("seeking %zd\n", partials[i].arc_polygon[j]);
printf("before: ");
for (size_t k = 0; k < here.size(); k++) {
printf("%zd ", here[k]);
}
printf("\n");
printf("other: ");
for (size_t k = 0; k < other.size(); k++) {
printf("%zd ", other[k]);
}
printf("\n");
#endif
for (size_t k = 0; k < other.size(); k++) {
size_t l;
for (l = k; l < other.size(); l++) {
if (other[l] == 0) {
break;
}
}
if (l >= other.size()) {
l--;
}
#if 0
for (size_t m = k; m <= l; m++) {
printf("%zd ", other[m]);
}
printf("\n");
#endif
size_t m;
for (m = k; m <= l; m++) {
if (other[m] == -partials[i].arc_polygon[j]) {
break;
}
}
if (m <= l) {
// Found the shared arc
here.erase(here.begin() + j);
size_t off = 0;
for (size_t n = m + 1; n < l; n++) {
here.insert(here.begin() + j + off, other[n]);
off++;
}
for (size_t n = k; n < m; n++) {
here.insert(here.begin() + j + off, other[n]);
off++;
}
} else {
// Looking at some other ring
for (size_t n = k; n <= l; n++) {
additions.push_back(other[n]);
}
}
k = l;
}
partials[i2].arc_polygon.clear();
partials[i2].renamed = i;
merged++;
for (size_t k = 0; k < additions.size(); k++) {
partials[i].arc_polygon.push_back(additions[k]);
}
#if 0
printf("after: ");
for (size_t k = 0; k < here.size(); k++) {
printf("%zd ", here[k]);
}
printf("\n");
#endif
#if 0
for (size_t k = 0; k + 1 < here.size(); k++) {
if (here[k] != 0 && here[k + 1] != 0) {
if (simplified_arcs[here[k + 1]][0] != simplified_arcs[here[k]][simplified_arcs[here[k]].size() - 1]) {
printf("error from %zd to %zd\n", here[k], here[k + 1]);
}
}
}
#endif
}
}
}
}
}
// Turn the arc representations of the polygons back into standard polygon geometries
for (size_t i = 0; i < partials.size(); i++) {
if (partials[i].t == VT_POLYGON) {
partials[i].geoms.resize(0);
partials[i].geoms.push_back(drawvec());
bool at_start = true;
draw first(-1, 0, 0);
for (size_t j = 0; j < partials[i].arc_polygon.size(); j++) {
ssize_t p = partials[i].arc_polygon[j];
if (p == 0) {
if (first.op >= 0) {
partials[i].geoms[0].push_back(first);
first = draw(-1, 0, 0);
}
at_start = true;
} else if (p > 0) {
for (size_t k = 0; k + 1 < simplified_arcs[p].size(); k++) {
if (at_start) {
partials[i].geoms[0].push_back(draw(VT_MOVETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y));
first = draw(VT_LINETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y);
} else {
partials[i].geoms[0].push_back(draw(VT_LINETO, simplified_arcs[p][k].x, simplified_arcs[p][k].y));
}
at_start = 0;
}
} else { /* p < 0 */
for (ssize_t k = simplified_arcs[-p].size() - 1; k > 0; k--) {
if (at_start) {
partials[i].geoms[0].push_back(draw(VT_MOVETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y));
first = draw(VT_LINETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y);
} else {
partials[i].geoms[0].push_back(draw(VT_LINETO, simplified_arcs[-p][k].x, simplified_arcs[-p][k].y));
}
at_start = 0;
}
}
}
}
}
if (merged >= merge_count) {
return true;
} else {
return false;
}
}
unsigned long long choose_mingap(std::vector<unsigned long long> const &indices, double f) {
unsigned long long bot = ULLONG_MAX;
unsigned long long top = 0;
@@ -1373,8 +800,6 @@ struct write_tile_args {
compressor **geomfile = NULL;
double todo = 0;
std::atomic<long long> *along = NULL;
double gamma = 0;
double gamma_out = 0;
int child_shards = 0;
int *geomfd = NULL;
off_t *geom_size = NULL;
@@ -1954,8 +1379,7 @@ void coalesce_geometry(partial &p, serial_feature &sf) {
p.geoms.push_back(sf.geometry);
}
long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, char *stringpool, int z, const unsigned tx, const unsigned ty, const int detail, int min_detail, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, compressor **geomfile, int minzoom, int maxzoom, double todo, std::atomic<long long> *along, long long alongminus, double gamma, int child_shards, long long *pool_off, unsigned *initial_x, unsigned *initial_y, std::atomic<int> *running, double simplification, std::vector<std::map<std::string, layermap_entry>> *layermaps, std::vector<std::vector<std::string>> *layer_unmaps, size_t tiling_seg, size_t pass, unsigned long long mingap, long long minextent, double fraction, const char *prefilter, const char *postfilter, struct json_object *filter, write_tile_args *arg, atomic_strategy *strategy, bool compressed_input, struct node *shared_nodes_map, size_t nodepos) {
double merge_fraction = 1;
long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, char *stringpool, int z, const unsigned tx, const unsigned ty, const int detail, int min_detail, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, compressor **geomfile, int minzoom, int maxzoom, double todo, std::atomic<long long> *along, long long alongminus, int child_shards, long long *pool_off, unsigned *initial_x, unsigned *initial_y, std::atomic<int> *running, double simplification, std::vector<std::map<std::string, layermap_entry>> *layermaps, std::vector<std::vector<std::string>> *layer_unmaps, size_t tiling_seg, size_t pass, unsigned long long mingap, long long minextent, double fraction, const char *prefilter, const char *postfilter, struct json_object *filter, write_tile_args *arg, atomic_strategy *strategy, bool compressed_input, struct node *shared_nodes_map, size_t nodepos) {
double mingap_fraction = 1;
double minextent_fraction = 1;
@@ -1992,10 +1416,10 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
double fraction_accum = 0;
unsigned long long previndex = 0, density_previndex = 0, merge_previndex = 0;
unsigned long long density_previndex = 0, merge_previndex = 0;
unsigned long long extent_previndex = 0;
double scale = (double) (1LL << (64 - 2 * (z + 8)));
double gap = 0, density_gap = 0;
double density_gap = 0;
double spacing = 0;
long long original_features = 0;
@@ -2154,14 +1578,6 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
multiplier_seq = retain_points_multiplier - 1;
}
if (gamma > 0) {
if (manage_gap(sf.index, &previndex, scale, gamma, &gap) && find_partial(partials, sf, which_partial, layer_unmaps, LLONG_MAX, multiplier_seq)) {
preserve_attributes(arg->attribute_accum, sf, stringpool, pool_off, partials[which_partial]);
strategy->dropped_by_gamma++;
continue;
}
}
// Cap the indices, rather than sampling them like extents (areas),
// because choose_mingap cares about the distance between *surviving*
// features, not between *original* features, so we can't just store
@@ -2342,7 +1758,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
partials.push_back(p);
unsimplified_geometry_size += sf.geometry.size() * sizeof(draw);
if (unsimplified_geometry_size > 10 * 1024 * 1024 && !additional[A_DETECT_SHARED_BORDERS]) {
if (unsimplified_geometry_size > 10 * 1024 * 1024) {
// we should be safe to simplify here with P_SIMPLIFY_SHARED_NODES, since they will
// have been assembled globally, although that also means that simplification
// may not be very effective for reducing memory usage.
@@ -2487,10 +1903,6 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
first_time = false;
if (additional[A_DETECT_SHARED_BORDERS]) {
find_common_edges(partials, z, line_detail, simplification, maxzoom, merge_fraction);
}
int tasks = ceil((double) CPUS / *running);
if (tasks < 1) {
tasks = 1;
@@ -2777,24 +2189,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
fprintf(stderr, "tile %d/%u/%u has %zu features, >%zu \n", z, tx, ty, totalsize, max_tile_features);
}
if (additional[A_INCREASE_GAMMA_AS_NEEDED] && gamma < 10) {
if (gamma < 1) {
gamma = 1;
} else {
gamma = gamma * 1.25;
}
if (gamma > arg->gamma_out) {
arg->gamma_out = gamma;
arg->still_dropping = true;
}
if (!quiet) {
fprintf(stderr, "Going to try gamma of %0.3f to make it fit\n", gamma);
}
line_detail++; // to keep it the same when the loop decrements it
continue;
} else if (mingap < ULONG_MAX && (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED])) {
if (mingap < ULONG_MAX && (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED])) {
mingap_fraction = mingap_fraction * max_tile_features / totalsize * 0.90;
unsigned long long mg = choose_mingap(indices, mingap_fraction);
if (mg <= mingap) {
@@ -2829,7 +2224,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
line_detail++;
continue;
}
} else if (totalsize > layers.size() && (prevent[P_DYNAMIC_DROP] || additional[A_DROP_FRACTION_AS_NEEDED] || additional[A_COALESCE_FRACTION_AS_NEEDED])) {
} else if (totalsize > layers.size() && (additional[A_DROP_FRACTION_AS_NEEDED] || additional[A_COALESCE_FRACTION_AS_NEEDED])) {
// The 95% is a guess to avoid too many retries
// and probably actually varies based on how much duplicated metadata there is
@@ -2840,8 +2235,6 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
if ((additional[A_DROP_FRACTION_AS_NEEDED] || additional[A_COALESCE_FRACTION_AS_NEEDED]) && fraction < arg->fraction_out) {
arg->fraction_out = fraction;
arg->still_dropping = true;
} else if (prevent[P_DYNAMIC_DROP]) {
arg->still_dropping = true;
}
line_detail++; // to keep it the same when the loop decrements it
continue;
@@ -2881,23 +2274,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
}
}
if (additional[A_INCREASE_GAMMA_AS_NEEDED] && gamma < 10) {
if (gamma < 1) {
gamma = 1;
} else {
gamma = gamma * 1.25;
}
if (gamma > arg->gamma_out) {
arg->gamma_out = gamma;
arg->still_dropping = true;
}
if (!quiet) {
fprintf(stderr, "Going to try gamma of %0.3f to make it fit\n", gamma);
}
line_detail++; // to keep it the same when the loop decrements it
} else if (mingap < ULONG_MAX && (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED])) {
if (mingap < ULONG_MAX && (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED])) {
mingap_fraction = mingap_fraction * max_tile_size / (kept_adjust * compressed.size()) * 0.90;
unsigned long long mg = choose_mingap(indices, mingap_fraction);
if (mg <= mingap) {
@@ -2937,7 +2314,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
line_detail++;
continue;
}
} else if (totalsize > layers.size() && (prevent[P_DYNAMIC_DROP] || additional[A_DROP_FRACTION_AS_NEEDED] || additional[A_COALESCE_FRACTION_AS_NEEDED])) {
} else if (totalsize > layers.size() && (additional[A_DROP_FRACTION_AS_NEEDED] || additional[A_COALESCE_FRACTION_AS_NEEDED])) {
// The 95% is a guess to avoid too many retries
// and probably actually varies based on how much duplicated metadata there is
@@ -2948,8 +2325,6 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
if ((additional[A_DROP_FRACTION_AS_NEEDED] || additional[A_COALESCE_FRACTION_AS_NEEDED]) && fraction < arg->fraction_out) {
arg->fraction_out = fraction;
arg->still_dropping = true;
} else if (prevent[P_DYNAMIC_DROP]) {
arg->still_dropping = true;
}
line_detail++; // to keep it the same when the loop decrements it
} else {
@@ -3052,7 +2427,7 @@ void *run_thread(void *vargs) {
// fprintf(stderr, "%d/%u/%u\n", z, x, y);
long long len = write_tile(&dc, &geompos, arg->stringpool, z, x, y, z == arg->maxzoom ? arg->full_detail : arg->low_detail, arg->min_detail, arg->outdb, arg->outdir, arg->buffer, arg->fname, arg->geomfile, arg->minzoom, arg->maxzoom, arg->todo, arg->along, geompos, arg->gamma, arg->child_shards, arg->pool_off, arg->initial_x, arg->initial_y, arg->running, arg->simplification, arg->layermaps, arg->layer_unmaps, arg->tiling_seg, arg->pass, arg->mingap, arg->minextent, arg->fraction, arg->prefilter, arg->postfilter, arg->filter, arg, arg->strategy, arg->compressed, arg->shared_nodes_map, arg->nodepos);
long long len = write_tile(&dc, &geompos, arg->stringpool, z, x, y, z == arg->maxzoom ? arg->full_detail : arg->low_detail, arg->min_detail, arg->outdb, arg->outdir, arg->buffer, arg->fname, arg->geomfile, arg->minzoom, arg->maxzoom, arg->todo, arg->along, geompos, arg->child_shards, arg->pool_off, arg->initial_x, arg->initial_y, arg->running, arg->simplification, arg->layermaps, arg->layer_unmaps, arg->tiling_seg, arg->pass, arg->mingap, arg->minextent, arg->fraction, arg->prefilter, arg->postfilter, arg->filter, arg, arg->strategy, arg->compressed, arg->shared_nodes_map, arg->nodepos);
if (len < 0) {
int *err = &arg->err;
@@ -3117,7 +2492,7 @@ void *run_thread(void *vargs) {
return NULL;
}
int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<unsigned> *midx, std::atomic<unsigned> *midy, int &maxzoom, int minzoom, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, const char *tmpdir, double gamma, int full_detail, int low_detail, int min_detail, long long *pool_off, unsigned *initial_x, unsigned *initial_y, double simplification, double maxzoom_simplification, std::vector<std::map<std::string, layermap_entry>> &layermaps, const char *prefilter, const char *postfilter, std::map<std::string, attribute_op> const *attribute_accum, struct json_object *filter, std::vector<strategy> &strategies, int iz, struct node *shared_nodes_map, size_t nodepos) {
int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<unsigned> *midx, std::atomic<unsigned> *midy, int &maxzoom, int minzoom, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, const char *tmpdir, int full_detail, int low_detail, int min_detail, long long *pool_off, unsigned *initial_x, unsigned *initial_y, double simplification, double maxzoom_simplification, std::vector<std::map<std::string, layermap_entry>> &layermaps, const char *prefilter, const char *postfilter, std::map<std::string, attribute_op> const *attribute_accum, struct json_object *filter, std::vector<strategy> &strategies, int iz, struct node *shared_nodes_map, size_t nodepos) {
last_progress = 0;
// The existing layermaps are one table per input thread.
@@ -3251,7 +2626,6 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<
int err = INT_MAX;
double zoom_gamma = gamma;
unsigned long long zoom_mingap = ((1LL << (32 - z)) / 256 * cluster_distance) * ((1LL << (32 - z)) / 256 * cluster_distance);
long long zoom_minextent = 0;
double zoom_fraction = 1;
@@ -3276,8 +2650,6 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<
args[thread].geomfile = sub + thread * (TEMP_FILES / threads);
args[thread].todo = todo;
args[thread].along = &along; // locked with var_lock
args[thread].gamma = zoom_gamma;
args[thread].gamma_out = zoom_gamma;
args[thread].mingap = zoom_mingap;
args[thread].mingap_out = zoom_mingap;
args[thread].minextent = zoom_minextent;
@@ -3344,10 +2716,6 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<
err = *((int *) retval);
}
if (args[thread].gamma_out > zoom_gamma) {
zoom_gamma = args[thread].gamma_out;
again = true;
}
if (args[thread].mingap_out > zoom_mingap) {
zoom_mingap = args[thread].mingap_out;
again = true;
+3 -3
View File
@@ -53,10 +53,10 @@ struct strategy {
strategy() = default;
};
// long long write_tile(char **geom, char *stringpool, unsigned *file_bbox, int z, unsigned x, unsigned y, int detail, int min_detail, int basezoom, sqlite3 *outdb, const char *outdir, double droprate, int buffer, const char *fname, FILE **geomfile, int file_minzoom, int file_maxzoom, double todo, char *geomstart, long long along, double gamma, int nlayers, std::atomic<strategy> *strategy);
// long long write_tile(char **geom, char *stringpool, unsigned *file_bbox, int z, unsigned x, unsigned y, int detail, int min_detail, int basezoom, sqlite3 *outdb, const char *outdir, double droprate, int buffer, const char *fname, FILE **geomfile, int file_minzoom, int file_maxzoom, double todo, char *geomstart, long long along, int nlayers, std::atomic<strategy> *strategy);
int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<unsigned> *midx, std::atomic<unsigned> *midy, int &maxzoom, int minzoom, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, const char *tmpdir, double gamma, int full_detail, int low_detail, int min_detail, long long *pool_off, unsigned *initial_x, unsigned *initial_y, double simplification, double maxzoom_simplification, std::vector<std::map<std::string, layermap_entry> > &layermap, const char *prefilter, const char *postfilter, std::map<std::string, attribute_op> const *attribute_accum, struct json_object *filter, std::vector<strategy> &strategies, int iz, struct node *shared_nodes_map, size_t nodepos);
int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<unsigned> *midx, std::atomic<unsigned> *midy, int &maxzoom, int minzoom, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, const char *tmpdir, int full_detail, int low_detail, int min_detail, long long *pool_off, unsigned *initial_x, unsigned *initial_y, double simplification, double maxzoom_simplification, std::vector<std::map<std::string, layermap_entry> > &layermap, const char *prefilter, const char *postfilter, std::map<std::string, attribute_op> const *attribute_accum, struct json_object *filter, std::vector<strategy> &strategies, int iz, struct node *shared_nodes_map, size_t nodepos);
int manage_gap(unsigned long long index, unsigned long long *previndex, double scale, double gamma, double *gap);
int manage_gap(unsigned long long index, unsigned long long *previndex, double scale, double *gap);
#endif