mirror of
https://github.com/felt/tippecanoe.git
synced 2026-10-02 08:25:40 +02:00
Implement tile-join's --use-attribute-for-id
The option had been listed in tile-join's option table since 533e000 (#361)
removed the code behind it, so it parsed and then exited with "Unrecognized
option." #409 dropped the leftover entry rather than ship a usage message
advertising an option that didn't work. Implement it instead.
The ID handling that #361 removed only ever applied to attributes that came
back from a SQLite join, and that join went away in the same commit, so there
is no longer anywhere in this tool to hang it on. Instead the option takes the
ID from an attribute of either kind that tile-join does have: one already
present in the source tiles, or one joined from a CSV with -c, with a joined
value superseding a tile value of the same name just as it does for ordinary
attributes. The attribute is consumed rather than copied through, and it is
checked before -x, -y, -X, and --exclude-all-tile-attributes, so the ID can
come from an attribute that isn't kept -- which is the useful case, as in
`-x GEOID10 --use-attribute-for-id=GEOID10`.
The conversion from an attribute value to an ID, along with the warnings for
the values that can't be represented as one, moves out of serialize_feature()
into attribute_to_feature_id() in mvt.cpp, which both tools link, so that
tile-join reports the same problems the same way instead of silently
truncating. Its one difference is that tile-join converts stringified numbers
without any additional option, since it has no equivalent of tippecanoe's -aI.
Along the way, an attribute whose value is 0 is now usable as a feature ID.
The check that an ID survives a round trip through its string form compared
std::to_string() against the value with its leading zeros stripped, and
stripping the only digit of "0" left "" to compare against "0", so a zero was
rejected as too large to represent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DRXWaDXVEZV4t1ddR6ZRcw
This commit is contained in:
Claude
@@ -6,6 +6,12 @@
|
||||
Features exactly at the threshold are kept rather than dropped. (#384, #385)
|
||||
* Add `--exclude-all-tile-geometries` to tile-join, to produce tiles that
|
||||
carry only attributes. (#382)
|
||||
* Add `--use-attribute-for-id=`*name* to tile-join, which had been advertised
|
||||
as an option for years without an implementation. The attribute can be one
|
||||
from the source tiles or one joined from a CSV, and supplies the ID even if
|
||||
the attribute filters would otherwise remove it. An attribute value of `0`
|
||||
is now usable as a feature ID in tippecanoe as well, instead of being
|
||||
reported as too large to represent.
|
||||
* Generate each tool's usage message from the same option table that
|
||||
`getopt_long()` reads, so the hand-written lists in tile-join,
|
||||
tippecanoe-overzoom, tippecanoe-json-tool, tippecanoe-decode, and
|
||||
|
||||
@@ -460,6 +460,10 @@ join-test: tippecanoe tippecanoe-decode tile-join
|
||||
./tile-join --quiet --force --tile-stats-sample-values-limit=1 -o tests/join-population/joined-tile-stats-sample-values-limit.mbtiles -x GEOID10 -c tests/join-population/population.csv tests/join-population/tabblock_06001420.mbtiles
|
||||
./tile-join --quiet --force --tile-stats-values-limit=1 -o tests/join-population/joined-tile-stats-values-limit.mbtiles -x GEOID10 -c tests/join-population/population.csv tests/join-population/tabblock_06001420.mbtiles
|
||||
./tile-join -q -f -i -o tests/join-population/joined-i.mbtiles -x GEOID10 -c tests/join-population/population.csv tests/join-population/tabblock_06001420.mbtiles
|
||||
# --use-attribute-for-id can take the ID from an attribute that -x removes,
|
||||
# and from one that only exists because it was joined from the CSV
|
||||
./tile-join -q -f --use-attribute-for-id=GEOID10 -o tests/join-population/joined-id.mbtiles -x GEOID10 -c tests/join-population/population.csv tests/join-population/tabblock_06001420.mbtiles
|
||||
./tile-join -q -f --use-attribute-for-id=population --exclude-all-tile-attributes -o tests/join-population/joined-id-from-csv.mbtiles -c tests/join-population/population.csv tests/join-population/tabblock_06001420.mbtiles
|
||||
./tile-join -q -f -o tests/join-population/merged.mbtiles tests/join-population/tabblock_06001420.mbtiles tests/join-population/macarthur.mbtiles tests/join-population/macarthur2.mbtiles
|
||||
./tile-join -q -f -c tests/join-population/windows.csv -o tests/join-population/windows.mbtiles tests/join-population/macarthur.mbtiles
|
||||
./tippecanoe-decode -x generator --maximum-zoom=11 --minimum-zoom=4 tests/join-population/joined.mbtiles > tests/join-population/joined.mbtiles.json.check
|
||||
@@ -469,6 +473,8 @@ join-test: tippecanoe tippecanoe-decode tile-join
|
||||
./tippecanoe-decode -x generator --maximum-zoom=11 --minimum-zoom=4 tests/join-population/joined-tile-stats-values-limit.mbtiles > tests/join-population/joined-tile-stats-values-limit.mbtiles.json.check
|
||||
./tippecanoe-decode -x generator --maximum-zoom=11 --minimum-zoom=4 tests/join-population/joined-tile-stats-sample-values-limit.mbtiles > tests/join-population/joined-tile-stats-sample-values-limit.mbtiles.json.check
|
||||
./tippecanoe-decode -x generator tests/join-population/joined-i.mbtiles > tests/join-population/joined-i.mbtiles.json.check
|
||||
./tippecanoe-decode -x generator --maximum-zoom=11 --minimum-zoom=4 tests/join-population/joined-id.mbtiles > tests/join-population/joined-id.mbtiles.json.check
|
||||
./tippecanoe-decode -x generator --maximum-zoom=11 --minimum-zoom=4 tests/join-population/joined-id-from-csv.mbtiles > tests/join-population/joined-id-from-csv.mbtiles.json.check
|
||||
./tippecanoe-decode -x generator tests/join-population/merged.mbtiles > tests/join-population/merged.mbtiles.json.check
|
||||
./tippecanoe-decode -x generator tests/join-population/windows.mbtiles > tests/join-population/windows.mbtiles.json.check
|
||||
cmp tests/join-population/joined.mbtiles.json.check tests/join-population/joined.mbtiles.json
|
||||
@@ -478,9 +484,12 @@ join-test: tippecanoe tippecanoe-decode tile-join
|
||||
cmp tests/join-population/joined-tile-stats-sample-values-limit.mbtiles.json.check tests/join-population/joined-tile-stats-sample-values-limit.mbtiles.json
|
||||
cmp tests/join-population/joined-tile-stats-values-limit.mbtiles.json.check tests/join-population/joined-tile-stats-values-limit.mbtiles.json
|
||||
cmp tests/join-population/joined-i.mbtiles.json.check tests/join-population/joined-i.mbtiles.json
|
||||
cmp tests/join-population/joined-id.mbtiles.json.check tests/join-population/joined-id.mbtiles.json
|
||||
cmp tests/join-population/joined-id-from-csv.mbtiles.json.check tests/join-population/joined-id-from-csv.mbtiles.json
|
||||
cmp tests/join-population/merged.mbtiles.json.check tests/join-population/merged.mbtiles.json
|
||||
cmp tests/join-population/windows.mbtiles.json.check tests/join-population/windows.mbtiles.json
|
||||
rm -f tests/join-population/joined-null.mbtiles tests/join-population/joined-null.mbtiles.json.check
|
||||
rm -f tests/join-population/joined-id.mbtiles tests/join-population/joined-id.mbtiles.json.check tests/join-population/joined-id-from-csv.mbtiles tests/join-population/joined-id-from-csv.mbtiles.json.check
|
||||
./tile-join -q -f -l macarthur -n "macarthur name" -N "macarthur description" -A "macarthur's attribution" -o tests/join-population/just-macarthur.mbtiles tests/join-population/merged.mbtiles
|
||||
./tile-join -q -f -L macarthur -o tests/join-population/no-macarthur.mbtiles tests/join-population/merged.mbtiles
|
||||
./tippecanoe-decode -x generator tests/join-population/just-macarthur.mbtiles > tests/join-population/just-macarthur.mbtiles.json.check
|
||||
|
||||
@@ -844,6 +844,10 @@ The options are:
|
||||
* `-J` *filter-file* or `--feature-filter-file`=*filter-file*: Like `-j`, but read the filter from a file.
|
||||
* `-pe` or `--empty-csv-columns-are-null`: Treat empty CSV columns as nulls rather than as empty strings.
|
||||
|
||||
### Setting feature IDs
|
||||
|
||||
* `--use-attribute-for-id=`*name*: Use the attribute with the specified *name*, whether it was already present in the source tiles or was joined from a CSV with `-c`, as if it were specified as the feature ID. The attribute is not also copied to the output. It is used for the ID even if `-x`, `-y`, `-X`, or `--exclude-all-tile-attributes` would otherwise remove it, so you can take the ID from an attribute that you don't want to keep. Unlike in tippecanoe, an attribute that is a stringified number is converted without any additional option. If its value can't be represented as a feature ID, a warning is printed and it is kept as an ordinary attribute instead.
|
||||
|
||||
### Setting or disabling tile size limits
|
||||
|
||||
* `-pk` or `--no-tile-size-limit`: Don't skip tiles larger than 500K.
|
||||
|
||||
@@ -888,6 +888,9 @@ The options are:
|
||||
\fB-J\fR \fIfilter-file\fP or \fB--feature-filter-file\fR=\fIfilter-file\fP: Like \fB-j\fR, but read the filter from a file.
|
||||
.IP \(bu 2
|
||||
\fB-pe\fR or \fB--empty-csv-columns-are-null\fR: Treat empty CSV columns as nulls rather than as empty strings.
|
||||
.SS Setting feature IDs
|
||||
.IP \(bu 2
|
||||
\fB--use-attribute-for-id=\fR\fIname\fP: Use the attribute with the specified \fIname\fP, whether it was already present in the source tiles or was joined from a CSV with \fB-c\fR, as if it were specified as the feature ID. The attribute is not also copied to the output. It is used for the ID even if \fB-x\fR, \fB-y\fR, \fB-X\fR, or \fB--exclude-all-tile-attributes\fR would otherwise remove it, so you can take the ID from an attribute that you don't want to keep. Unlike in tippecanoe, an attribute that is a stringified number is converted without any additional option. If its value can't be represented as a feature ID, a warning is printed and it is kept as an ordinary attribute instead.
|
||||
.SS Setting or disabling tile size limits
|
||||
.IP \(bu 2
|
||||
\fB-pk\fR or \fB--no-tile-size-limit\fR: Don't skip tiles larger than 500K.
|
||||
|
||||
@@ -820,6 +820,68 @@ serial_val mvt_value_to_serial_val(mvt_value const &v) {
|
||||
return sv;
|
||||
}
|
||||
|
||||
static std::string strip_zeroes(std::string s) {
|
||||
// Doesn't do anything special with '-' followed by leading zeros
|
||||
// since integer IDs must be positive.
|
||||
//
|
||||
// The last digit is kept even if it is a zero, so that a value of "0"
|
||||
// is the ID 0 instead of comparing unequal to it and being rejected.
|
||||
|
||||
while (s.size() > 1 && s[0] == '0') {
|
||||
s.erase(s.begin());
|
||||
}
|
||||
|
||||
return s;
|
||||
}
|
||||
|
||||
// --use-attribute-for-id: convert the value of the attribute that was named as the
|
||||
// source of the feature ID into an ID. Returns false, after warning once about each
|
||||
// of the ways the conversion can fail, if the value can't be represented as an ID,
|
||||
// in which case the caller should treat it as an ordinary attribute instead.
|
||||
//
|
||||
// Values that aren't numbers are only converted if the caller says to, since in
|
||||
// tippecanoe that requires -aI. tile-join, which has no such option, always does.
|
||||
bool attribute_to_feature_id(std::string const &key, serial_val const &val, bool convert_numeric, unsigned long long *id) {
|
||||
if (val.s.size() == 0 || (val.type != mvt_double && !convert_numeric)) {
|
||||
static bool warned = false;
|
||||
|
||||
if (!warned) {
|
||||
fprintf(stderr, "Warning: Attribute \"%s\"=\"%s\" as feature ID is not a number\n", key.c_str(), val.s.c_str());
|
||||
warned = true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
char *err;
|
||||
long long id_value = strtoull(val.s.c_str(), &err, 10);
|
||||
|
||||
if (err != NULL && *err != '\0') {
|
||||
static bool warned_frac = false;
|
||||
|
||||
if (!warned_frac) {
|
||||
fprintf(stderr, "Warning: Can't represent non-integer feature ID %s\n", val.s.c_str());
|
||||
warned_frac = true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
if (std::to_string(id_value) != strip_zeroes(val.s)) {
|
||||
static bool warned = false;
|
||||
|
||||
if (!warned) {
|
||||
fprintf(stderr, "Warning: Can't represent too-large feature ID %s\n", val.s.c_str());
|
||||
warned = true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
*id = id_value;
|
||||
return true;
|
||||
}
|
||||
|
||||
// This extracts an integer value from an mvt_value
|
||||
long long mvt_value_to_long_long(mvt_value const &v) {
|
||||
switch (v.type) {
|
||||
|
||||
@@ -250,6 +250,7 @@ bool is_unsigned_integer(const char *s, unsigned long long *v);
|
||||
|
||||
struct serial_val;
|
||||
serial_val mvt_value_to_serial_val(mvt_value const &v);
|
||||
bool attribute_to_feature_id(std::string const &key, serial_val const &val, bool convert_numeric, unsigned long long *id);
|
||||
|
||||
void get_bbox(std::vector<mvt_geometry> const &geom,
|
||||
long long *xmin, long long *ymin, long long *xmax, long long *ymax,
|
||||
|
||||
+7
-42
@@ -378,17 +378,6 @@ static long long scale_geometry(struct serialization_state *sst, long long *bbox
|
||||
return geom.size();
|
||||
}
|
||||
|
||||
static std::string strip_zeroes(std::string s) {
|
||||
// Doesn't do anything special with '-' followed by leading zeros
|
||||
// since integer IDs must be positive
|
||||
|
||||
while (s.size() > 0 && s[0] == '0') {
|
||||
s.erase(s.begin());
|
||||
}
|
||||
|
||||
return s;
|
||||
}
|
||||
|
||||
int nodecmp(const void *void1, const void *void2) {
|
||||
node *n1 = (node *) void1;
|
||||
node *n2 = (node *) void2;
|
||||
@@ -811,39 +800,15 @@ int serialize_feature(struct serialization_state *sst, serial_feature &sf, std::
|
||||
}
|
||||
|
||||
if (*sf.full_keys[i] == attribute_for_id) {
|
||||
if (sf.full_values[i].type != mvt_double && !additional[A_CONVERT_NUMERIC_IDS]) {
|
||||
static bool warned = false;
|
||||
unsigned long long id;
|
||||
|
||||
if (!warned) {
|
||||
fprintf(stderr, "Warning: Attribute \"%s\"=\"%s\" as feature ID is not a number\n", sf.full_keys[i]->c_str(), sf.full_values[i].s.c_str());
|
||||
warned = true;
|
||||
}
|
||||
} else {
|
||||
char *err;
|
||||
long long id_value = strtoull(sf.full_values[i].s.c_str(), &err, 10);
|
||||
if (attribute_to_feature_id(*sf.full_keys[i], sf.full_values[i], additional[A_CONVERT_NUMERIC_IDS], &id)) {
|
||||
sf.id = id;
|
||||
sf.has_id = true;
|
||||
|
||||
if (err != NULL && *err != '\0') {
|
||||
static bool warned_frac = false;
|
||||
|
||||
if (!warned_frac) {
|
||||
fprintf(stderr, "Warning: Can't represent non-integer feature ID %s\n", sf.full_values[i].s.c_str());
|
||||
warned_frac = true;
|
||||
}
|
||||
} else if (std::to_string(id_value) != strip_zeroes(sf.full_values[i].s)) {
|
||||
static bool warned = false;
|
||||
|
||||
if (!warned) {
|
||||
fprintf(stderr, "Warning: Can't represent too-large feature ID %s\n", sf.full_values[i].s.c_str());
|
||||
warned = true;
|
||||
}
|
||||
} else {
|
||||
sf.id = id_value;
|
||||
sf.has_id = true;
|
||||
|
||||
sf.full_keys.erase(sf.full_keys.begin() + i);
|
||||
sf.full_values.erase(sf.full_values.begin() + i);
|
||||
continue;
|
||||
}
|
||||
sf.full_keys.erase(sf.full_keys.begin() + i);
|
||||
sf.full_values.erase(sf.full_values.begin() + i);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
+67
-10
@@ -59,6 +59,7 @@ bool exclude_all = false;
|
||||
bool exclude_all_tile_attributes = false;
|
||||
bool exclude_all_tile_geometries = false;
|
||||
std::vector<std::string> unidecode_data;
|
||||
std::string attribute_for_id;
|
||||
|
||||
bool want_overzoom = false;
|
||||
int buffer = 5;
|
||||
@@ -98,6 +99,36 @@ struct arg {
|
||||
double minlon2, maxlon2;
|
||||
};
|
||||
|
||||
// one output feature: the attributes it will be written with, which are the ones
|
||||
// copied from the source tile plus any that were joined to it from the CSV
|
||||
struct match {
|
||||
bool has_id = false;
|
||||
unsigned long long id;
|
||||
std::map<std::string, std::pair<mvt_value, serial_val>> attributes;
|
||||
std::vector<std::string> key_order;
|
||||
};
|
||||
|
||||
// --use-attribute-for-id: if this is the attribute that was named as the source of the
|
||||
// feature ID, use its value as the ID and report that it should not also be copied
|
||||
// through as an attribute.
|
||||
//
|
||||
// This is checked before the -x, -y, -X, and --exclude-all-tile-attributes filters,
|
||||
// so that the attribute that supplies the ID doesn't have to be one that is kept.
|
||||
static bool used_for_id(match &m, std::string const &key, serial_val const &sv) {
|
||||
if (attribute_for_id.size() == 0 || key != attribute_for_id || sv.type == mvt_null) {
|
||||
return false;
|
||||
}
|
||||
|
||||
unsigned long long id;
|
||||
if (!attribute_to_feature_id(key, sv, true, &id)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
m.has_id = true;
|
||||
m.id = id;
|
||||
return true;
|
||||
}
|
||||
|
||||
void append_tile(std::string message, int z, unsigned x, unsigned y, std::map<std::string, layermap_entry> &layermap, std::vector<std::string> &header, std::map<std::string, std::vector<std::string>> &mapping, sqlite3 * /* db */, std::set<std::string> &exclude, std::set<std::string> &include, std::set<std::string> &keep_layers, std::set<std::string> &remove_layers, int ifmatched, mvt_tile &outtile, json_object *filter, struct arg *a) {
|
||||
mvt_tile tile;
|
||||
int features_added = 0;
|
||||
@@ -183,13 +214,6 @@ void append_tile(std::string message, int z, unsigned x, unsigned y, std::map<st
|
||||
continue;
|
||||
}
|
||||
|
||||
struct match {
|
||||
bool has_id = false;
|
||||
unsigned long long id;
|
||||
std::map<std::string, std::pair<mvt_value, serial_val>> attributes;
|
||||
std::vector<std::string> key_order;
|
||||
};
|
||||
|
||||
std::vector<match> matches;
|
||||
bool matched = false;
|
||||
|
||||
@@ -213,7 +237,12 @@ void append_tile(std::string message, int z, unsigned x, unsigned y, std::map<st
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!exclude_all_tile_attributes) {
|
||||
// the attribute that supplies the feature ID is still checked
|
||||
// against the CSV below, but is not copied through
|
||||
|
||||
bool is_id = used_for_id(m, key, sv);
|
||||
|
||||
if (!exclude_all_tile_attributes && !is_id) {
|
||||
if (include.count(std::string(key)) || (!exclude_all && exclude.count(std::string(key)) == 0 && exclude_attributes.count(std::string(key)) == 0)) {
|
||||
m.attributes.insert(std::pair<std::string, std::pair<mvt_value, serial_val>>(key, std::pair<mvt_value, serial_val>(val, sv)));
|
||||
m.key_order.push_back(key);
|
||||
@@ -244,6 +273,22 @@ void append_tile(std::string message, int z, unsigned x, unsigned y, std::map<st
|
||||
|
||||
const char *sjoinkey = joinkey.c_str();
|
||||
|
||||
serial_val joinsv;
|
||||
joinsv.type = attr_type;
|
||||
joinsv.s = joinval;
|
||||
|
||||
if (used_for_id(m, joinkey, joinsv)) {
|
||||
// a joined ID supersedes one from the tile, as
|
||||
// joined attributes supersede tile attributes
|
||||
|
||||
auto fa = m.attributes.find(sjoinkey);
|
||||
if (fa != m.attributes.end()) {
|
||||
m.attributes.erase(fa);
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
if (include.count(joinkey) || (!exclude_all && exclude.count(joinkey) == 0 && exclude_attributes.count(joinkey) == 0 && attr_type != mvt_null)) {
|
||||
mvt_value outval;
|
||||
if (attr_type == mvt_string) {
|
||||
@@ -286,13 +331,20 @@ void append_tile(std::string message, int z, unsigned x, unsigned y, std::map<st
|
||||
m.id = feat.id;
|
||||
m.has_id = feat.has_id;
|
||||
|
||||
if (!exclude_all_tile_attributes) {
|
||||
// the tags are still walked if the attributes are all being excluded,
|
||||
// in case one of them is the one that supplies the feature ID
|
||||
|
||||
if (!exclude_all_tile_attributes || attribute_for_id.size() > 0) {
|
||||
for (size_t t = 0; t + 1 < feat.tags.size(); t += 2) {
|
||||
const std::string &key = layer.keys[feat.tags[t]];
|
||||
mvt_value &val = layer.values[feat.tags[t + 1]];
|
||||
serial_val sv = mvt_value_to_serial_val(val);
|
||||
|
||||
if (include.count(key) || (!exclude_all && exclude.count(key) == 0 && exclude_attributes.count(key) == 0)) {
|
||||
if (used_for_id(m, key, sv)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!exclude_all_tile_attributes && (include.count(key) || (!exclude_all && exclude.count(key) == 0 && exclude_attributes.count(key) == 0))) {
|
||||
m.attributes.insert(std::pair<std::string, std::pair<mvt_value, serial_val>>(key, std::pair<mvt_value, serial_val>(val, sv)));
|
||||
m.key_order.push_back(key);
|
||||
}
|
||||
@@ -1295,6 +1347,9 @@ static const struct option long_options[] = {
|
||||
{"feature-filter-file", required_argument, 0, 'J'},
|
||||
{"feature-filter", required_argument, 0, 'j'},
|
||||
|
||||
{"Setting feature IDs", 0, 0, 0},
|
||||
{"use-attribute-for-id", required_argument, 0, '~'},
|
||||
|
||||
{"Setting or disabling tile size limits", 0, 0, 0},
|
||||
{"no-tile-size-limit", no_argument, &pk, 1},
|
||||
{"no-tile-compression", no_argument, &pC, 1},
|
||||
@@ -1540,6 +1595,8 @@ int main(int argc, char **argv) {
|
||||
exclude_all_tile_attributes = true;
|
||||
} else if (strcmp(opt, "exclude-all-tile-geometries") == 0) {
|
||||
exclude_all_tile_geometries = true;
|
||||
} else if (strcmp(opt, "use-attribute-for-id") == 0) {
|
||||
attribute_for_id = optarg;
|
||||
} else {
|
||||
fprintf(stderr, "%s: Unrecognized option --%s\n", argv[0], opt);
|
||||
exit(EXIT_ARGS);
|
||||
|
||||
Reference in New Issue
Block a user