diegoripley/tippecanoe
1
0
Fork 0
mirror of https://github.com/felt/tippecanoe.git synced 2026-10-06 10:25:41 +02:00
Code Issues Packages Projects Releases Wiki Activity

Get rid of gamma

Browse Source
This commit is contained in:
Erica Fischer
2024-01-26 16:09:27 -08:00
parent cbf222754b
commit 68388f6c32
11 changed files with 37 additions and 43370 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -10
View File
@@ -481,11 +481,6 @@ the same layer, enclose them in an `all` expression so they will all be evaluate
* `-pd` or `--force-feature-limit`: Dynamically drop some fraction of features from large tiles to keep them under the 500K size limit. It will probably look ugly at the tile boundaries. (This is like `-ad` but applies to each tile individually, not to the entire zoom level.) You probably don't want to use this. * `-pd` or `--force-feature-limit`: Dynamically drop some fraction of features from large tiles to keep them under the 500K size limit. It will probably look ugly at the tile boundaries. (This is like `-ad` but applies to each tile individually, not to the entire zoom level.) You probably don't want to use this.
* `-aC` or `--cluster-densest-as-needed`: If a tile is too large, try to reduce its size by increasing the minimum spacing between features, and leaving one placeholder feature from each group. The remaining feature will be given a `"clustered": true` attribute to indicate that it represents a cluster, a `"point_count"` attribute to indicate the number of features that were clustered into it, and a `"sqrt_point_count"` attribute to indicate the relative width of a feature to represent the cluster. If the features being clustered are points, the representative feature will be located at the average of the original points' locations; otherwise, one of the original features will be left as the representative. * `-aC` or `--cluster-densest-as-needed`: If a tile is too large, try to reduce its size by increasing the minimum spacing between features, and leaving one placeholder feature from each group. The remaining feature will be given a `"clustered": true` attribute to indicate that it represents a cluster, a `"point_count"` attribute to indicate the number of features that were clustered into it, and a `"sqrt_point_count"` attribute to indicate the relative width of a feature to represent the cluster. If the features being clustered are points, the representative feature will be located at the average of the original points' locations; otherwise, one of the original features will be left as the representative.
### Dropping tightly overlapping features
* `-g` _gamma_ or `--gamma=_gamma`_: Rate at which especially dense dots are dropped (default 0, for no effect). A gamma of 2 reduces the number of dots less than a pixel apart to the square root of their original number.
* `-aG` or `--increase-gamma-as-needed`: If a tile is too large, try to reduce it to under 500K by increasing the `-g` gamma. The discovered gamma applies to the entire zoom level. You probably want to use `--drop-densest-as-needed` instead.
### Line and polygon simplification ### Line and polygon simplification
* `-S` _scale_ or `--simplification=`_scale_: Multiply the tolerance for line and polygon simplification by _scale_. The standard tolerance tries to keep * `-S` _scale_ or `--simplification=`_scale_: Multiply the tolerance for line and polygon simplification by _scale_. The standard tolerance tries to keep
@@ -576,7 +571,7 @@ as a series of newline-delimited GeoJSON objects on the standard input, and `tip
set of GeoJSON features from the filter's standard output. set of GeoJSON features from the filter's standard output.
The prefilter receives the features at the highest available resolution, before line simplification, The prefilter receives the features at the highest available resolution, before line simplification,
polygon topology repair, gamma calculation, dynamic feature dropping, or other internal processing. polygon topology repair, dynamic feature dropping, or other internal processing.
The postfilter receives the features at tile resolution, after simplification, cleaning, and dropping. The postfilter receives the features at tile resolution, after simplification, cleaning, and dropping.
The layer name is provided as part of the `tippecanoe` element of the feature and must be passed through The layer name is provided as part of the `tippecanoe` element of the feature and must be passed through
@@ -671,10 +666,6 @@ a different zoom specified with `-B` if you have precise but sparse data).
I don't know why 2.5 is the appropriate number, but the densities of many different I don't know why 2.5 is the appropriate number, but the densities of many different
data sets fall off at about this same rate. You can use -r to specify a different rate. data sets fall off at about this same rate. You can use -r to specify a different rate.
You can use the gamma option to thin out especially dense clusters of points.
For any area where dots are closer than one pixel together (at whatever zoom level),
a gamma of 3, for example, will reduce these clusters to the cube root of their original density.
For line features, it drops any features that are too small to draw at all. For line features, it drops any features that are too small to draw at all.
This still leaves the lower zooms too dark (and too dense for the 500K tile limit, This still leaves the lower zooms too dark (and too dense for the 500K tile limit,
in some places), so I need to figure out an equitable way to throw features away. in some places), so I need to figure out an equitable way to throw features away.
main.cpp
+24 -93
View File
@@ -296,12 +296,12 @@ struct drop_densest {
} }
}; };
int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, double gamma) { int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom) {
int feature_minzoom = 0; int feature_minzoom = 0;
if (gamma >= 0 && (ix->t == VT_POINT || if ((ix->t == VT_POINT ||
(additional[A_LINE_DROP] && ix->t == VT_LINE) || (additional[A_LINE_DROP] && ix->t == VT_LINE) ||
(additional[A_POLYGON_DROP] && ix->t == VT_POLYGON))) { (additional[A_POLYGON_DROP] && ix->t == VT_POLYGON))) {
for (ssize_t i = maxzoom; i >= 0; i--) { for (ssize_t i = maxzoom; i >= 0; i--) {
ds[i].seq++; ds[i].seq++;
} }
@@ -342,14 +342,12 @@ int calc_feature_minzoom(struct index *ix, struct drop_state *ds, int maxzoom, d
} }
} }
} }
// XXX manage_gap
} }
return feature_minzoom; return feature_minzoom;
} }
static void merge(struct mergelist *merges, size_t nmerges, unsigned char *map, FILE *indexfile, int bytes, char *geom_map, FILE *geom_out, std::atomic<long long> *geompos, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, double gamma, struct drop_state *ds) { static void merge(struct mergelist *merges, size_t nmerges, unsigned char *map, FILE *indexfile, int bytes, char *geom_map, FILE *geom_out, std::atomic<long long> *geompos, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, struct drop_state *ds) {
struct mergelist *head = NULL; struct mergelist *head = NULL;
for (size_t i = 0; i < nmerges; i++) { for (size_t i = 0; i < nmerges; i++) {
@@ -368,7 +366,7 @@ static void merge(struct mergelist *merges, size_t nmerges, unsigned char *map,
// and is writing one byte less and then adding the byte for the minzoom. // and is writing one byte less and then adding the byte for the minzoom.
fwrite_check(geom_map + ix.start, 1, ix.end - ix.start - 1, geom_out, geompos, "merge geometry"); fwrite_check(geom_map + ix.start, 1, ix.end - ix.start - 1, geom_out, geompos, "merge geometry");
int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom, gamma); int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom);
serialize_byte(geom_out, feature_minzoom, geompos, "merge geometry"); serialize_byte(geom_out, feature_minzoom, geompos, "merge geometry");
// Count this as an 75%-accomplishment, since we already 25%-counted it // Count this as an 75%-accomplishment, since we already 25%-counted it
@@ -448,7 +446,7 @@ void *run_sort(void *v) {
return NULL; return NULL;
} }
void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) { void do_read_parallel(char *map, long long len, long long initial_offset, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > *layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
long long segs[CPUS + 1]; long long segs[CPUS + 1];
segs[0] = 0; segs[0] = 0;
segs[CPUS] = len; segs[CPUS] = len;
@@ -501,7 +499,6 @@ void do_read_parallel(char *map, long long len, long long initial_offset, const
sst[i].dist_count = &(dist_counts[i]); sst[i].dist_count = &(dist_counts[i]);
sst[i].want_dist = want_dist; sst[i].want_dist = want_dist;
sst[i].maxzoom = maxzoom; sst[i].maxzoom = maxzoom;
sst[i].uses_gamma = uses_gamma;
sst[i].filters = filters; sst[i].filters = filters;
sst[i].layermap = &(*layermaps)[i]; sst[i].layermap = &(*layermaps)[i];
sst[i].exclude = exclude; sst[i].exclude = exclude;
@@ -648,7 +645,6 @@ struct read_parallel_arg {
unsigned *initial_x = NULL; unsigned *initial_x = NULL;
unsigned *initial_y = NULL; unsigned *initial_y = NULL;
std::string layername = ""; std::string layername = "";
bool uses_gamma = false;
std::map<std::string, int> const *attribute_types = NULL; std::map<std::string, int> const *attribute_types = NULL;
double *dist_sum = NULL; double *dist_sum = NULL;
size_t *dist_count = NULL; size_t *dist_count = NULL;
@@ -676,7 +672,7 @@ void *run_read_parallel(void *v) {
} }
madvise(map, rpa->len, MADV_RANDOM); // sequential, but from several pointers at once madvise(map, rpa->len, MADV_RANDOM); // sequential, but from several pointers at once
do_read_parallel(map, rpa->len, rpa->offset, rpa->reading, rpa->readers, rpa->progress_seq, rpa->exclude, rpa->include, rpa->exclude_all, rpa->basezoom, rpa->source, rpa->layermaps, rpa->initialized, rpa->initial_x, rpa->initial_y, rpa->maxzoom, rpa->layername, rpa->uses_gamma, rpa->attribute_types, rpa->separator, rpa->dist_sum, rpa->dist_count, rpa->area_sum, rpa->want_dist, rpa->filters); do_read_parallel(map, rpa->len, rpa->offset, rpa->reading, rpa->readers, rpa->progress_seq, rpa->exclude, rpa->include, rpa->exclude_all, rpa->basezoom, rpa->source, rpa->layermaps, rpa->initialized, rpa->initial_x, rpa->initial_y, rpa->maxzoom, rpa->layername, rpa->attribute_types, rpa->separator, rpa->dist_sum, rpa->dist_count, rpa->area_sum, rpa->want_dist, rpa->filters);
madvise(map, rpa->len, MADV_DONTNEED); madvise(map, rpa->len, MADV_DONTNEED);
if (munmap(map, rpa->len) != 0) { if (munmap(map, rpa->len) != 0) {
@@ -693,7 +689,7 @@ void *run_read_parallel(void *v) {
return NULL; return NULL;
} }
void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::atomic<int> *is_parsing, pthread_t *parallel_parser, bool &parser_created, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > &layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, bool uses_gamma, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) { void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::atomic<int> *is_parsing, pthread_t *parallel_parser, bool &parser_created, const char *reading, std::vector<struct reader> *readers, std::atomic<long long> *progress_seq, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, int basezoom, int source, std::vector<std::map<std::string, layermap_entry> > &layermaps, int *initialized, unsigned *initial_x, unsigned *initial_y, int maxzoom, std::string layername, std::map<std::string, int> const *attribute_types, int separator, double *dist_sum, size_t *dist_count, double *area_sum, bool want_dist, bool filters) {
// This has to kick off an intermediate thread to start the parser threads, // This has to kick off an intermediate thread to start the parser threads,
// so the main thread can get back to reading the next input stage while // so the main thread can get back to reading the next input stage while
// the intermediate thread waits for the completion of the parser threads. // the intermediate thread waits for the completion of the parser threads.
@@ -727,7 +723,6 @@ void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::ato
rpa->initial_y = initial_y; rpa->initial_y = initial_y;
rpa->maxzoom = maxzoom; rpa->maxzoom = maxzoom;
rpa->layername = layername; rpa->layername = layername;
rpa->uses_gamma = uses_gamma;
rpa->attribute_types = attribute_types; rpa->attribute_types = attribute_types;
rpa->dist_sum = dist_sum; rpa->dist_sum = dist_sum;
rpa->dist_count = dist_count; rpa->dist_count = dist_count;
@@ -742,7 +737,7 @@ void start_parsing(int fd, STREAM *fp, long long offset, long long len, std::ato
parser_created = true; parser_created = true;
} }
void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int splits, long long mem, const char *tmpdir, long long *availfiles, FILE *geomfile, FILE *indexfile, std::atomic<long long> *geompos_out, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, int basezoom, double droprate, double gamma, struct drop_state *ds) { void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int splits, long long mem, const char *tmpdir, long long *availfiles, FILE *geomfile, FILE *indexfile, std::atomic<long long> *geompos_out, long long *progress, long long *progress_max, long long *progress_reported, int maxzoom, int basezoom, double droprate, struct drop_state *ds) {
// Arranged as bits to facilitate subdividing again if a subdivided file is still huge // Arranged as bits to facilitate subdividing again if a subdivided file is still huge
int splitbits = log(splits) / log(2); int splitbits = log(splits) / log(2);
splits = 1 << splitbits; splits = 1 << splitbits;
@@ -961,7 +956,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
madvise(geommap, geomst.st_size, MADV_RANDOM); madvise(geommap, geomst.st_size, MADV_RANDOM);
madvise(geommap, geomst.st_size, MADV_WILLNEED); madvise(geommap, geomst.st_size, MADV_WILLNEED);
merge(merges, nmerges, (unsigned char *) indexmap, indexfile, bytes, geommap, geomfile, geompos_out, progress, progress_max, progress_reported, maxzoom, gamma, ds); merge(merges, nmerges, (unsigned char *) indexmap, indexfile, bytes, geommap, geomfile, geompos_out, progress, progress_max, progress_reported, maxzoom, ds);
madvise(indexmap, indexst.st_size, MADV_DONTNEED); madvise(indexmap, indexst.st_size, MADV_DONTNEED);
if (munmap(indexmap, indexst.st_size) < 0) { if (munmap(indexmap, indexst.st_size) < 0) {
@@ -995,7 +990,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
long long pos = *geompos_out; long long pos = *geompos_out;
fwrite_check(geommap + ix.start, ix.end - ix.start, 1, geomfile, geompos_out, "geom"); fwrite_check(geommap + ix.start, ix.end - ix.start, 1, geomfile, geompos_out, "geom");
int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom, gamma); int feature_minzoom = calc_feature_minzoom(&ix, ds, maxzoom);
serialize_byte(geomfile, feature_minzoom, geompos_out, "merge geometry"); serialize_byte(geomfile, feature_minzoom, geompos_out, "merge geometry");
// Count this as an 75%-accomplishment, since we already 25%-counted it // Count this as an 75%-accomplishment, since we already 25%-counted it
@@ -1030,7 +1025,7 @@ void radix1(int *geomfds_in, int *indexfds_in, int inputs, int prefix, int split
// counter backward but will be an honest estimate of the work remaining. // counter backward but will be an honest estimate of the work remaining.
*progress_max += geomst.st_size / 4; *progress_max += geomst.st_size / 4;
radix1(&geomfds[i], &indexfds[i], 1, prefix + splitbits, *availfiles / 4, mem, tmpdir, availfiles, geomfile, indexfile, geompos_out, progress, progress_max, progress_reported, maxzoom, basezoom, droprate, gamma, ds); radix1(&geomfds[i], &indexfds[i], 1, prefix + splitbits, *availfiles / 4, mem, tmpdir, availfiles, geomfile, indexfile, geompos_out, progress, progress_max, progress_reported, maxzoom, basezoom, droprate, ds);
already_closed = 1; already_closed = 1;
} }
} }
@@ -1090,7 +1085,7 @@ static size_t calc_memsize() {
return mem; return mem;
} }
void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FILE *indexfile, const char *tmpdir, std::atomic<long long> *geompos, int maxzoom, int basezoom, double droprate, double gamma) { void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FILE *indexfile, const char *tmpdir, std::atomic<long long> *geompos, int maxzoom, int basezoom, double droprate) {
// Run through the index and geometry for each reader, // Run through the index and geometry for each reader,
// splitting the contents out by index into as many // splitting the contents out by index into as many
// sub-files as we can write to simultaneously. // sub-files as we can write to simultaneously.
@@ -1140,7 +1135,7 @@ void radix(std::vector<struct reader> &readers, int nreaders, FILE *geomfile, FI
long long progress = 0, progress_max = geom_total, progress_reported = -1; long long progress = 0, progress_max = geom_total, progress_reported = -1;
long long availfiles_before = availfiles; long long availfiles_before = availfiles;
radix1(geomfds, indexfds, nreaders, 0, splits, mem, tmpdir, &availfiles, geomfile, indexfile, geompos, &progress, &progress_max, &progress_reported, maxzoom, basezoom, droprate, gamma, ds); radix1(geomfds, indexfds, nreaders, 0, splits, mem, tmpdir, &availfiles, geomfile, indexfile, geompos, &progress, &progress_max, &progress_reported, maxzoom, basezoom, droprate, ds);
if (availfiles - 2 * nreaders != availfiles_before) { if (availfiles - 2 * nreaders != availfiles_before) {
fprintf(stderr, "Internal error: miscounted available file descriptors: %lld vs %lld\n", availfiles - 2 * nreaders, availfiles); fprintf(stderr, "Internal error: miscounted available file descriptors: %lld vs %lld\n", availfiles - 2 * nreaders, availfiles);
@@ -1237,7 +1232,7 @@ int vertexcmp(const void *void1, const void *void2) {
return 0; return 0;
} }
std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzoom, int basezoom, double basezoom_marker_width, sqlite3 *outdb, const char *outdir, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, json_object *filter, double droprate, int buffer, const char *tmpdir, double gamma, int read_parallel, int forcetable, const char *attribution, bool uses_gamma, long long *file_bbox, long long *file_bbox1, long long *file_bbox2, const char *prefilter, const char *postfilter, const char *description, bool guess_maxzoom, bool guess_cluster_maxzoom, std::map<std::string, int> const *attribute_types, const char *pgm, std::map<std::string, attribute_op> const *attribute_accum, std::map<std::string, std::string> const &attribute_descriptions, std::string const &commandline, int minimum_maxzoom) { std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, int maxzoom, int minzoom, int basezoom, double basezoom_marker_width, sqlite3 *outdb, const char *outdir, std::set<std::string> *exclude, std::set<std::string> *include, int exclude_all, json_object *filter, double droprate, int buffer, const char *tmpdir, int read_parallel, int forcetable, const char *attribution, long long *file_bbox, long long *file_bbox1, long long *file_bbox2, const char *prefilter, const char *postfilter, const char *description, bool guess_maxzoom, bool guess_cluster_maxzoom, std::map<std::string, int> const *attribute_types, const char *pgm, std::map<std::string, attribute_op> const *attribute_accum, std::map<std::string, std::string> const &attribute_descriptions, std::string const &commandline, int minimum_maxzoom) {
int ret = EXIT_SUCCESS; int ret = EXIT_SUCCESS;
std::vector<struct reader> readers; std::vector<struct reader> readers;
@@ -1514,7 +1509,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
sst[i].want_dist = guess_maxzoom; sst[i].want_dist = guess_maxzoom;
sst[i].maxzoom = maxzoom; sst[i].maxzoom = maxzoom;
sst[i].filters = prefilter != NULL || postfilter != NULL; sst[i].filters = prefilter != NULL || postfilter != NULL;
sst[i].uses_gamma = uses_gamma;
sst[i].layermap = &layermaps[i]; sst[i].layermap = &layermaps[i];
sst[i].exclude = exclude; sst[i].exclude = exclude;
sst[i].include = include; sst[i].include = include;
@@ -1587,7 +1581,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
sst[i].want_dist = guess_maxzoom; sst[i].want_dist = guess_maxzoom;
sst[i].maxzoom = maxzoom; sst[i].maxzoom = maxzoom;
sst[i].filters = prefilter != NULL || postfilter != NULL; sst[i].filters = prefilter != NULL || postfilter != NULL;
sst[i].uses_gamma = uses_gamma;
sst[i].layermap = &layermaps[i]; sst[i].layermap = &layermaps[i];
sst[i].exclude = exclude; sst[i].exclude = exclude;
sst[i].include = include; sst[i].include = include;
@@ -1649,7 +1642,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
sst[i].want_dist = guess_maxzoom; sst[i].want_dist = guess_maxzoom;
sst[i].maxzoom = maxzoom; sst[i].maxzoom = maxzoom;
sst[i].filters = prefilter != NULL || postfilter != NULL; sst[i].filters = prefilter != NULL || postfilter != NULL;
sst[i].uses_gamma = uses_gamma;
sst[i].layermap = &layermaps[i]; sst[i].layermap = &layermaps[i];
sst[i].exclude = exclude; sst[i].exclude = exclude;
sst[i].include = include; sst[i].include = include;
@@ -1707,7 +1699,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
} }
if (map != NULL && map != MAP_FAILED && read_parallel_this) { if (map != NULL && map != MAP_FAILED && read_parallel_this) {
do_read_parallel(map, st.st_size - off, overall_offset, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, &layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, uses_gamma, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL); do_read_parallel(map, st.st_size - off, overall_offset, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, &layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
overall_offset += st.st_size - off; overall_offset += st.st_size - off;
checkdisk(&readers); checkdisk(&readers);
@@ -1786,7 +1778,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
} }
fflush(readfp); fflush(readfp);
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, &parallel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL); start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, &parallel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
initial_offset += ahead; initial_offset += ahead;
overall_offset += ahead; overall_offset += ahead;
@@ -1823,7 +1815,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
fflush(readfp); fflush(readfp);
if (ahead > 0) { if (ahead > 0) {
start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, &parallel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, gamma != 0, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL); start_parsing(readfd, streamfpopen(readfp), initial_offset, ahead, &is_parsing, &parallel_parser, parser_created, reading.c_str(), &readers, &progress_seq, exclude, include, exclude_all, basezoom, layer, layermaps, initialized, initial_x, initial_y, maxzoom, sources[layer].layer, attribute_types, read_parallel_this, &dist_sum, &dist_count, &area_sum, guess_maxzoom, prefilter != NULL || postfilter != NULL);
if (parser_created) { if (parser_created) {
if (pthread_join(parallel_parser, NULL) != 0) { if (pthread_join(parallel_parser, NULL) != 0) {
@@ -1857,7 +1849,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
sst.want_dist = guess_maxzoom; sst.want_dist = guess_maxzoom;
sst.maxzoom = maxzoom; sst.maxzoom = maxzoom;
sst.filters = prefilter != NULL || postfilter != NULL; sst.filters = prefilter != NULL || postfilter != NULL;
sst.uses_gamma = uses_gamma;
sst.layermap = &layermaps[0]; sst.layermap = &layermaps[0];
sst.exclude = exclude; sst.exclude = exclude;
sst.include = include; sst.include = include;
@@ -2219,7 +2210,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
serialize_uint(geomfile, ix, &geompos, fname); serialize_uint(geomfile, ix, &geompos, fname);
serialize_uint(geomfile, iy, &geompos, fname); serialize_uint(geomfile, iy, &geompos, fname);
radix(readers, CPUS, geomfile, indexfile, tmpdir, &geompos, maxzoom, basezoom, droprate, gamma); radix(readers, CPUS, geomfile, indexfile, tmpdir, &geompos, maxzoom, basezoom, droprate);
/* end of tile */ /* end of tile */
serialize_ulong_long(geomfile, 0, &geompos, fname); // EOF serialize_ulong_long(geomfile, 0, &geompos, fname); // EOF
@@ -2521,8 +2512,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
yyy = yy >> (32 - z); yyy = yy >> (32 - z);
} }
double scale = (double) (1LL << (64 - 2 * (z + 8)));
if (tile[z].x != xxx || tile[z].y != yyy) { if (tile[z].x != xxx || tile[z].y != yyy) {
if (tile[z].count > max[z].count) { if (tile[z].count > max[z].count) {
max[z] = tile[z]; max[z] = tile[z];
@@ -2537,11 +2526,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
} }
tile[z].fullcount++; tile[z].fullcount++;
if (manage_gap(map[ip].ix, &tile[z].previndex, scale, gamma, &tile[z].gap)) {
continue;
}
tile[z].count++; tile[z].count++;
} }
} }
@@ -2578,9 +2562,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
if (obasezoom < 0 && basezoom > maxzoom) { if (obasezoom < 0 && basezoom > maxzoom) {
fprintf(stderr, "Couldn't find a suitable base zoom. Working from the other direction.\n"); fprintf(stderr, "Couldn't find a suitable base zoom. Working from the other direction.\n");
if (gamma == 0) {
fprintf(stderr, "You might want to try -g1 to limit near-duplicates.\n");
}
if (droprate < 0) { if (droprate < 0) {
if (maxzoom == 0) { if (maxzoom == 0) {
@@ -2622,40 +2603,6 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
} }
} }
if (gamma > 0) {
int effective = 0;
for (z = 0; z < maxzoom; z++) {
if (max[z].count < max[z].fullcount) {
effective = z + 1;
}
}
if (effective == 0) {
if (!quiet) {
fprintf(stderr, "With gamma, effective base zoom is 0, so no effective drop rate\n");
}
} else {
double interval_0 = exp(log(droprate) * (basezoom - 0));
double interval_eff = exp(log(droprate) * (basezoom - effective));
if (effective > basezoom) {
interval_eff = 1;
}
double scaled_0 = max[0].count / interval_0;
double scaled_eff = max[effective].count / interval_eff;
double rate_at_0 = scaled_0 / max[0].fullcount;
double rate_at_eff = scaled_eff / max[effective].fullcount;
double eff_drop = exp(log(rate_at_eff / rate_at_0) / (effective - 0));
if (!quiet) {
fprintf(stderr, "With gamma, effective base zoom of %d, effective drop rate of %f\n", effective, eff_drop);
}
}
}
fix_dropping = true; fix_dropping = true;
} }
@@ -2695,7 +2642,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
previndex = map[ip].ix; previndex = map[ip].ix;
} else { } else {
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom, gamma); int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom);
geom[map[ip].end - 1] = feature_minzoom; geom[map[ip].end - 1] = feature_minzoom;
} }
} }
@@ -2720,7 +2667,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
if (ip > 0 && map[ip].start != map[ip - 1].end) { if (ip > 0 && map[ip].start != map[ip - 1].end) {
fprintf(stderr, "Mismatched index at %lld: %lld vs %lld\n", ip, map[ip].start, map[ip].end); fprintf(stderr, "Mismatched index at %lld: %lld vs %lld\n", ip, map[ip].start, map[ip].end);
} }
int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom, gamma); int feature_minzoom = calc_feature_minzoom(&map[ip], ds, maxzoom);
geom[map[ip].end - 1] = feature_minzoom; geom[map[ip].end - 1] = feature_minzoom;
} }
} }
@@ -2757,7 +2704,7 @@ std::pair<int, metadata> read_input(std::vector<source> &sources, char *fname, i
std::atomic<unsigned> midx(0); std::atomic<unsigned> midx(0);
std::atomic<unsigned> midy(0); std::atomic<unsigned> midy(0);
std::vector<strategy> strategies; std::vector<strategy> strategies;
int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, gamma, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos); int written = traverse_zooms(fd, size, stringpool, &midx, &midy, maxzoom, minzoom, outdb, outdir, buffer, fname, tmpdir, full_detail, low_detail, min_detail, pool_off, initial_x, initial_y, simplification, maxzoom_simplification, layermaps, prefilter, postfilter, attribute_accum, filter, strategies, iz, shared_nodes_map, nodepos);
if (maxzoom != written) { if (maxzoom != written) {
if (written > minzoom) { if (written > minzoom) {
@@ -2993,7 +2940,6 @@ int main(int argc, char **argv) {
int force = 0; int force = 0;
int forcetable = 0; int forcetable = 0;
double droprate = 2.5; double droprate = 2.5;
double gamma = 0;
int buffer = 5; int buffer = 5;
const char *tmpdir = "/tmp"; const char *tmpdir = "/tmp";
const char *attribution = NULL; const char *attribution = NULL;
@@ -3097,10 +3043,6 @@ int main(int argc, char **argv) {
{"force-feature-limit", no_argument, &prevent[P_DYNAMIC_DROP], 1}, {"force-feature-limit", no_argument, &prevent[P_DYNAMIC_DROP], 1},
{"cluster-densest-as-needed", no_argument, &additional[A_CLUSTER_DENSEST_AS_NEEDED], 1}, {"cluster-densest-as-needed", no_argument, &additional[A_CLUSTER_DENSEST_AS_NEEDED], 1},
{"Dropping tightly overlapping features", 0, 0, 0},
{"gamma", required_argument, 0, 'g'},
{"increase-gamma-as-needed", no_argument, &additional[A_INCREASE_GAMMA_AS_NEEDED], 1},
{"Line and polygon simplification", 0, 0, 0}, {"Line and polygon simplification", 0, 0, 0},
{"simplification", required_argument, 0, 'S'}, {"simplification", required_argument, 0, 'S'},
{"no-line-simplification", no_argument, &prevent[P_SIMPLIFY], 1}, {"no-line-simplification", no_argument, &prevent[P_SIMPLIFY], 1},
@@ -3534,10 +3476,6 @@ int main(int argc, char **argv) {
} }
break; break;
case 'g':
gamma = atof_require(optarg, "Gamma");
break;
case 'q': case 'q':
quiet = 1; quiet = 1;
break; break;
@@ -3760,13 +3698,6 @@ int main(int argc, char **argv) {
} }
} }
if ((basezoom < 0 || droprate < 0) && (gamma < 0)) {
// Can't use randomized (as opposed to evenly distributed) dot dropping
// if rate and base aren't known during feature reading.
gamma = 0;
fprintf(stderr, "Forcing -g0 since -B or -r is not known\n");
}
if (out_mbtiles == NULL && out_dir == NULL) { if (out_mbtiles == NULL && out_dir == NULL) {
fprintf(stderr, "%s: must specify -o out.mbtiles or -e directory\n", argv[0]); fprintf(stderr, "%s: must specify -o out.mbtiles or -e directory\n", argv[0]);
exit(EXIT_ARGS); exit(EXIT_ARGS);
@@ -3821,7 +3752,7 @@ int main(int argc, char **argv) {
auto input_ret = read_input(sources, name ? name : out_mbtiles ? out_mbtiles auto input_ret = read_input(sources, name ? name : out_mbtiles ? out_mbtiles
: out_dir, : out_dir,
maxzoom, minzoom, basezoom, basezoom_marker_width, outdb, out_dir, &exclude, &include, exclude_all, filter, droprate, buffer, tmpdir, gamma, read_parallel, forcetable, attribution, gamma != 0, file_bbox, file_bbox1, file_bbox2, prefilter, postfilter, description, guess_maxzoom, guess_cluster_maxzoom, &attribute_types, argv[0], &attribute_accum, attribute_descriptions, commandline, minimum_maxzoom); maxzoom, minzoom, basezoom, basezoom_marker_width, outdb, out_dir, &exclude, &include, exclude_all, filter, droprate, buffer, tmpdir, read_parallel, forcetable, attribution, file_bbox, file_bbox1, file_bbox2, prefilter, postfilter, description, guess_maxzoom, guess_cluster_maxzoom, &attribute_types, argv[0], &attribute_accum, attribute_descriptions, commandline, minimum_maxzoom);
ret = std::get<0>(input_ret); ret = std::get<0>(input_ret);
man/tippecanoe.1
+1 -12
View File
@@ -612,13 +612,6 @@ preference to retaining points in sparse areas and dropping points in dense area
.IP \(bu 2 .IP \(bu 2
\fB\fC\-aC\fR or \fB\fC\-\-cluster\-densest\-as\-needed\fR: If a tile is too large, try to reduce its size by increasing the minimum spacing between features, and leaving one placeholder feature from each group. The remaining feature will be given a \fB\fC"clustered": true\fR attribute to indicate that it represents a cluster, a \fB\fC"point_count"\fR attribute to indicate the number of features that were clustered into it, and a \fB\fC"sqrt_point_count"\fR attribute to indicate the relative width of a feature to represent the cluster. If the features being clustered are points, the representative feature will be located at the average of the original points' locations; otherwise, one of the original features will be left as the representative. \fB\fC\-aC\fR or \fB\fC\-\-cluster\-densest\-as\-needed\fR: If a tile is too large, try to reduce its size by increasing the minimum spacing between features, and leaving one placeholder feature from each group. The remaining feature will be given a \fB\fC"clustered": true\fR attribute to indicate that it represents a cluster, a \fB\fC"point_count"\fR attribute to indicate the number of features that were clustered into it, and a \fB\fC"sqrt_point_count"\fR attribute to indicate the relative width of a feature to represent the cluster. If the features being clustered are points, the representative feature will be located at the average of the original points' locations; otherwise, one of the original features will be left as the representative.
.RE .RE
.SS Dropping tightly overlapping features
.RS
.IP \(bu 2
\fB\fC\-g\fR \fIgamma\fP or \fB\fC\-\-gamma=_gamma\fR_: Rate at which especially dense dots are dropped (default 0, for no effect). A gamma of 2 reduces the number of dots less than a pixel apart to the square root of their original number.
.IP \(bu 2
\fB\fC\-aG\fR or \fB\fC\-\-increase\-gamma\-as\-needed\fR: If a tile is too large, try to reduce it to under 500K by increasing the \fB\fC\-g\fR gamma. The discovered gamma applies to the entire zoom level. You probably want to use \fB\fC\-\-drop\-densest\-as\-needed\fR instead.
.RE
.SS Line and polygon simplification .SS Line and polygon simplification
.RS .RS
.IP \(bu 2 .IP \(bu 2
@@ -759,7 +752,7 @@ as a series of newline\-delimited GeoJSON objects on the standard input, and \fB
set of GeoJSON features from the filter's standard output. set of GeoJSON features from the filter's standard output.
.PP .PP
The prefilter receives the features at the highest available resolution, before line simplification, The prefilter receives the features at the highest available resolution, before line simplification,
polygon topology repair, gamma calculation, dynamic feature dropping, or other internal processing. polygon topology repair, dynamic feature dropping, or other internal processing.
The postfilter receives the features at tile resolution, after simplification, cleaning, and dropping. The postfilter receives the features at tile resolution, after simplification, cleaning, and dropping.
.PP .PP
The layer name is provided as part of the \fB\fCtippecanoe\fR element of the feature and must be passed through The layer name is provided as part of the \fB\fCtippecanoe\fR element of the feature and must be passed through
@@ -865,10 +858,6 @@ a different zoom specified with \fB\fC\-B\fR if you have precise but sparse data
I don't know why 2.5 is the appropriate number, but the densities of many different I don't know why 2.5 is the appropriate number, but the densities of many different
data sets fall off at about this same rate. You can use \-r to specify a different rate. data sets fall off at about this same rate. You can use \-r to specify a different rate.
.PP .PP
You can use the gamma option to thin out especially dense clusters of points.
For any area where dots are closer than one pixel together (at whatever zoom level),
a gamma of 3, for example, will reduce these clusters to the cube root of their original density.
.PP
For line features, it drops any features that are too small to draw at all. For line features, it drops any features that are too small to draw at all.
This still leaves the lower zooms too dark (and too dense for the 500K tile limit, This still leaves the lower zooms too dark (and too dense for the 500K tile limit,
in some places), so I need to figure out an equitable way to throw features away. in some places), so I need to figure out an equitable way to throw features away.
options.hpp
-1
View File
@@ -10,7 +10,6 @@
#define A_DETECT_SHARED_BORDERS ((int) 'b') #define A_DETECT_SHARED_BORDERS ((int) 'b')
#define A_PREFER_RADIX_SORT ((int) 'R') #define A_PREFER_RADIX_SORT ((int) 'R')
#define A_CALCULATE_FEATURE_DENSITY ((int) 'g') #define A_CALCULATE_FEATURE_DENSITY ((int) 'g')
#define A_INCREASE_GAMMA_AS_NEEDED ((int) 'G')
#define A_DROP_DENSEST_AS_NEEDED ((int) 's') #define A_DROP_DENSEST_AS_NEEDED ((int) 's')
#define A_DROP_FRACTION_AS_NEEDED ((int) 'd') #define A_DROP_FRACTION_AS_NEEDED ((int) 'd')
#define A_DROP_SMALLEST_AS_NEEDED ((int) 'n') #define A_DROP_SMALLEST_AS_NEEDED ((int) 'n')
serial.cpp
+1 -1
View File
@@ -719,7 +719,7 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf) {
} }
} }
if (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED] || additional[A_CALCULATE_FEATURE_DENSITY] || additional[A_DROP_SMALLEST_AS_NEEDED] || additional[A_COALESCE_SMALLEST_AS_NEEDED] || additional[A_INCREASE_GAMMA_AS_NEEDED] || additional[A_GENERATE_POLYGON_LABEL_POINTS] || sst->uses_gamma || cluster_distance != 0) { if (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED] || additional[A_CALCULATE_FEATURE_DENSITY] || additional[A_DROP_SMALLEST_AS_NEEDED] || additional[A_COALESCE_SMALLEST_AS_NEEDED] || additional[A_GENERATE_POLYGON_LABEL_POINTS] || cluster_distance != 0) {
sf.index = bbox_index; sf.index = bbox_index;
} else { } else {
sf.index = 0; sf.index = 0;
serial.hpp
-1
View File
@@ -171,7 +171,6 @@ struct serialization_state {
int basezoom = 0; int basezoom = 0;
bool filters = false; bool filters = false;
bool uses_gamma = false;
std::map<std::string, layermap_entry> *layermap = NULL; std::map<std::string, layermap_entry> *layermap = NULL;
tests/muni/out/-Z11_-z11_-g2.json
-6399
View File
File diff suppressed because it is too large Load Diff
tests/muni/out/-Z11_-z13_-M10000_-aG.json
-15565
View File
File diff suppressed because it is too large Load Diff
tests/muni/out/-Z11_-z13_-rf2000_-g2.json
-21228
View File
File diff suppressed because it is too large Load Diff
tile.cpp
+7 -57
View File
@@ -1373,8 +1373,6 @@ struct write_tile_args {
compressor **geomfile = NULL; compressor **geomfile = NULL;
double todo = 0; double todo = 0;
std::atomic<long long> *along = NULL; std::atomic<long long> *along = NULL;
double gamma = 0;
double gamma_out = 0;
int child_shards = 0; int child_shards = 0;
int *geomfd = NULL; int *geomfd = NULL;
off_t *geom_size = NULL; off_t *geom_size = NULL;
@@ -1954,7 +1952,7 @@ void coalesce_geometry(partial &p, serial_feature &sf) {
p.geoms.push_back(sf.geometry); p.geoms.push_back(sf.geometry);
} }
long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, char *stringpool, int z, const unsigned tx, const unsigned ty, const int detail, int min_detail, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, compressor **geomfile, int minzoom, int maxzoom, double todo, std::atomic<long long> *along, long long alongminus, double gamma, int child_shards, long long *pool_off, unsigned *initial_x, unsigned *initial_y, std::atomic<int> *running, double simplification, std::vector<std::map<std::string, layermap_entry>> *layermaps, std::vector<std::vector<std::string>> *layer_unmaps, size_t tiling_seg, size_t pass, unsigned long long mingap, long long minextent, double fraction, const char *prefilter, const char *postfilter, struct json_object *filter, write_tile_args *arg, atomic_strategy *strategy, bool compressed_input, struct node *shared_nodes_map, size_t nodepos) { long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, char *stringpool, int z, const unsigned tx, const unsigned ty, const int detail, int min_detail, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, compressor **geomfile, int minzoom, int maxzoom, double todo, std::atomic<long long> *along, long long alongminus, int child_shards, long long *pool_off, unsigned *initial_x, unsigned *initial_y, std::atomic<int> *running, double simplification, std::vector<std::map<std::string, layermap_entry>> *layermaps, std::vector<std::vector<std::string>> *layer_unmaps, size_t tiling_seg, size_t pass, unsigned long long mingap, long long minextent, double fraction, const char *prefilter, const char *postfilter, struct json_object *filter, write_tile_args *arg, atomic_strategy *strategy, bool compressed_input, struct node *shared_nodes_map, size_t nodepos) {
double merge_fraction = 1; double merge_fraction = 1;
double mingap_fraction = 1; double mingap_fraction = 1;
double minextent_fraction = 1; double minextent_fraction = 1;
@@ -1992,10 +1990,10 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
double fraction_accum = 0; double fraction_accum = 0;
unsigned long long previndex = 0, density_previndex = 0, merge_previndex = 0; unsigned long long density_previndex = 0, merge_previndex = 0;
unsigned long long extent_previndex = 0; unsigned long long extent_previndex = 0;
double scale = (double) (1LL << (64 - 2 * (z + 8))); double scale = (double) (1LL << (64 - 2 * (z + 8)));
double gap = 0, density_gap = 0; double density_gap = 0;
double spacing = 0; double spacing = 0;
long long original_features = 0; long long original_features = 0;
@@ -2154,14 +2152,6 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
multiplier_seq = retain_points_multiplier - 1; multiplier_seq = retain_points_multiplier - 1;
} }
if (gamma > 0) {
if (manage_gap(sf.index, &previndex, scale, gamma, &gap) && find_partial(partials, sf, which_partial, layer_unmaps, LLONG_MAX, multiplier_seq)) {
preserve_attributes(arg->attribute_accum, sf, stringpool, pool_off, partials[which_partial]);
strategy->dropped_by_gamma++;
continue;
}
}
// Cap the indices, rather than sampling them like extents (areas), // Cap the indices, rather than sampling them like extents (areas),
// because choose_mingap cares about the distance between *surviving* // because choose_mingap cares about the distance between *surviving*
// features, not between *original* features, so we can't just store // features, not between *original* features, so we can't just store
@@ -2777,24 +2767,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
fprintf(stderr, "tile %d/%u/%u has %zu features, >%zu \n", z, tx, ty, totalsize, max_tile_features); fprintf(stderr, "tile %d/%u/%u has %zu features, >%zu \n", z, tx, ty, totalsize, max_tile_features);
} }
if (additional[A_INCREASE_GAMMA_AS_NEEDED] && gamma < 10) { if (mingap < ULONG_MAX && (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED])) {
if (gamma < 1) {
gamma = 1;
} else {
gamma = gamma * 1.25;
}
if (gamma > arg->gamma_out) {
arg->gamma_out = gamma;
arg->still_dropping = true;
}
if (!quiet) {
fprintf(stderr, "Going to try gamma of %0.3f to make it fit\n", gamma);
}
line_detail++; // to keep it the same when the loop decrements it
continue;
} else if (mingap < ULONG_MAX && (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED])) {
mingap_fraction = mingap_fraction * max_tile_features / totalsize * 0.90; mingap_fraction = mingap_fraction * max_tile_features / totalsize * 0.90;
unsigned long long mg = choose_mingap(indices, mingap_fraction); unsigned long long mg = choose_mingap(indices, mingap_fraction);
if (mg <= mingap) { if (mg <= mingap) {
@@ -2881,23 +2854,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
} }
} }
if (additional[A_INCREASE_GAMMA_AS_NEEDED] && gamma < 10) { if (mingap < ULONG_MAX && (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED])) {
if (gamma < 1) {
gamma = 1;
} else {
gamma = gamma * 1.25;
}
if (gamma > arg->gamma_out) {
arg->gamma_out = gamma;
arg->still_dropping = true;
}
if (!quiet) {
fprintf(stderr, "Going to try gamma of %0.3f to make it fit\n", gamma);
}
line_detail++; // to keep it the same when the loop decrements it
} else if (mingap < ULONG_MAX && (additional[A_DROP_DENSEST_AS_NEEDED] || additional[A_COALESCE_DENSEST_AS_NEEDED] || additional[A_CLUSTER_DENSEST_AS_NEEDED])) {
mingap_fraction = mingap_fraction * max_tile_size / (kept_adjust * compressed.size()) * 0.90; mingap_fraction = mingap_fraction * max_tile_size / (kept_adjust * compressed.size()) * 0.90;
unsigned long long mg = choose_mingap(indices, mingap_fraction); unsigned long long mg = choose_mingap(indices, mingap_fraction);
if (mg <= mingap) { if (mg <= mingap) {
@@ -3052,7 +3009,7 @@ void *run_thread(void *vargs) {
// fprintf(stderr, "%d/%u/%u\n", z, x, y); // fprintf(stderr, "%d/%u/%u\n", z, x, y);
long long len = write_tile(&dc, &geompos, arg->stringpool, z, x, y, z == arg->maxzoom ? arg->full_detail : arg->low_detail, arg->min_detail, arg->outdb, arg->outdir, arg->buffer, arg->fname, arg->geomfile, arg->minzoom, arg->maxzoom, arg->todo, arg->along, geompos, arg->gamma, arg->child_shards, arg->pool_off, arg->initial_x, arg->initial_y, arg->running, arg->simplification, arg->layermaps, arg->layer_unmaps, arg->tiling_seg, arg->pass, arg->mingap, arg->minextent, arg->fraction, arg->prefilter, arg->postfilter, arg->filter, arg, arg->strategy, arg->compressed, arg->shared_nodes_map, arg->nodepos); long long len = write_tile(&dc, &geompos, arg->stringpool, z, x, y, z == arg->maxzoom ? arg->full_detail : arg->low_detail, arg->min_detail, arg->outdb, arg->outdir, arg->buffer, arg->fname, arg->geomfile, arg->minzoom, arg->maxzoom, arg->todo, arg->along, geompos, arg->child_shards, arg->pool_off, arg->initial_x, arg->initial_y, arg->running, arg->simplification, arg->layermaps, arg->layer_unmaps, arg->tiling_seg, arg->pass, arg->mingap, arg->minextent, arg->fraction, arg->prefilter, arg->postfilter, arg->filter, arg, arg->strategy, arg->compressed, arg->shared_nodes_map, arg->nodepos);
if (len < 0) { if (len < 0) {
int *err = &arg->err; int *err = &arg->err;
@@ -3117,7 +3074,7 @@ void *run_thread(void *vargs) {
return NULL; return NULL;
} }
int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<unsigned> *midx, std::atomic<unsigned> *midy, int &maxzoom, int minzoom, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, const char *tmpdir, double gamma, int full_detail, int low_detail, int min_detail, long long *pool_off, unsigned *initial_x, unsigned *initial_y, double simplification, double maxzoom_simplification, std::vector<std::map<std::string, layermap_entry>> &layermaps, const char *prefilter, const char *postfilter, std::map<std::string, attribute_op> const *attribute_accum, struct json_object *filter, std::vector<strategy> &strategies, int iz, struct node *shared_nodes_map, size_t nodepos) { int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<unsigned> *midx, std::atomic<unsigned> *midy, int &maxzoom, int minzoom, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, const char *tmpdir, int full_detail, int low_detail, int min_detail, long long *pool_off, unsigned *initial_x, unsigned *initial_y, double simplification, double maxzoom_simplification, std::vector<std::map<std::string, layermap_entry>> &layermaps, const char *prefilter, const char *postfilter, std::map<std::string, attribute_op> const *attribute_accum, struct json_object *filter, std::vector<strategy> &strategies, int iz, struct node *shared_nodes_map, size_t nodepos) {
last_progress = 0; last_progress = 0;
// The existing layermaps are one table per input thread. // The existing layermaps are one table per input thread.
@@ -3251,7 +3208,6 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<
int err = INT_MAX; int err = INT_MAX;
double zoom_gamma = gamma;
unsigned long long zoom_mingap = ((1LL << (32 - z)) / 256 * cluster_distance) * ((1LL << (32 - z)) / 256 * cluster_distance); unsigned long long zoom_mingap = ((1LL << (32 - z)) / 256 * cluster_distance) * ((1LL << (32 - z)) / 256 * cluster_distance);
long long zoom_minextent = 0; long long zoom_minextent = 0;
double zoom_fraction = 1; double zoom_fraction = 1;
@@ -3276,8 +3232,6 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<
args[thread].geomfile = sub + thread * (TEMP_FILES / threads); args[thread].geomfile = sub + thread * (TEMP_FILES / threads);
args[thread].todo = todo; args[thread].todo = todo;
args[thread].along = &along; // locked with var_lock args[thread].along = &along; // locked with var_lock
args[thread].gamma = zoom_gamma;
args[thread].gamma_out = zoom_gamma;
args[thread].mingap = zoom_mingap; args[thread].mingap = zoom_mingap;
args[thread].mingap_out = zoom_mingap; args[thread].mingap_out = zoom_mingap;
args[thread].minextent = zoom_minextent; args[thread].minextent = zoom_minextent;
@@ -3344,10 +3298,6 @@ int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<
err = *((int *) retval); err = *((int *) retval);
} }
if (args[thread].gamma_out > zoom_gamma) {
zoom_gamma = args[thread].gamma_out;
again = true;
}
if (args[thread].mingap_out > zoom_mingap) { if (args[thread].mingap_out > zoom_mingap) {
zoom_mingap = args[thread].mingap_out; zoom_mingap = args[thread].mingap_out;
again = true; again = true;
tile.hpp
+3 -3
View File
@@ -53,10 +53,10 @@ struct strategy {
strategy() = default; strategy() = default;
}; };
// long long write_tile(char **geom, char *stringpool, unsigned *file_bbox, int z, unsigned x, unsigned y, int detail, int min_detail, int basezoom, sqlite3 *outdb, const char *outdir, double droprate, int buffer, const char *fname, FILE **geomfile, int file_minzoom, int file_maxzoom, double todo, char *geomstart, long long along, double gamma, int nlayers, std::atomic<strategy> *strategy); // long long write_tile(char **geom, char *stringpool, unsigned *file_bbox, int z, unsigned x, unsigned y, int detail, int min_detail, int basezoom, sqlite3 *outdb, const char *outdir, double droprate, int buffer, const char *fname, FILE **geomfile, int file_minzoom, int file_maxzoom, double todo, char *geomstart, long long along, int nlayers, std::atomic<strategy> *strategy);
int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<unsigned> *midx, std::atomic<unsigned> *midy, int &maxzoom, int minzoom, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, const char *tmpdir, double gamma, int full_detail, int low_detail, int min_detail, long long *pool_off, unsigned *initial_x, unsigned *initial_y, double simplification, double maxzoom_simplification, std::vector<std::map<std::string, layermap_entry> > &layermap, const char *prefilter, const char *postfilter, std::map<std::string, attribute_op> const *attribute_accum, struct json_object *filter, std::vector<strategy> &strategies, int iz, struct node *shared_nodes_map, size_t nodepos); int traverse_zooms(int *geomfd, off_t *geom_size, char *stringpool, std::atomic<unsigned> *midx, std::atomic<unsigned> *midy, int &maxzoom, int minzoom, sqlite3 *outdb, const char *outdir, int buffer, const char *fname, const char *tmpdir, int full_detail, int low_detail, int min_detail, long long *pool_off, unsigned *initial_x, unsigned *initial_y, double simplification, double maxzoom_simplification, std::vector<std::map<std::string, layermap_entry> > &layermap, const char *prefilter, const char *postfilter, std::map<std::string, attribute_op> const *attribute_accum, struct json_object *filter, std::vector<strategy> &strategies, int iz, struct node *shared_nodes_map, size_t nodepos);
int manage_gap(unsigned long long index, unsigned long long *previndex, double scale, double gamma, double *gap); int manage_gap(unsigned long long index, unsigned long long *previndex, double scale, double *gap);
#endif #endif