Improve tiling speed (#195)

* Add a way to run tippecanoe single-threaded for profiling

* Do less work when the tilestats sample values list is already full

* Save a copy when retrieving the attribute key

* Fewer atomic operations

* Move string hashing from mbtiles to text

* Only do approximate attribute deduplication when writing tiles

* Feature dropping tests are sensitive to exact tile size

* All tile creators now create a string pool for the tile

* Features clipped away to nothing should not participate in that tile

* Revert "Only do approximate attribute deduplication when writing tiles"

This reverts commit c42b34b498.

* Also revert the related test changes

* Revert "Revert "Only do approximate attribute deduplication when writing tiles""

This reverts commit 18509876c3.

* Be more specific about the string hash function

* Use fnv1a instead of std::hash for everything

* Reduce the chance of hash collisions

* Stick a hash search on the front of the tree search in addpool

* Eliminate repeated hashing of the same string

* Switch instead of ifs in json parsing

* A few more cases to populate the hash in addpool

* Store the hash in the tree instead of recalculating

* Add explanatory comment for mysterious argument

* Fewer copies in attribute stringification

* Clean up ancient weirdness in JSON attribute stringification

* More serial_val cleanup

* Pass a serial_feature to rewrite instead of many broken-down arguments

* Get rid of the multiple geometries within `partial`

* Revert "Pass a serial_feature to rewrite instead of many broken-down arguments"

This reverts commit 6f4ab9b725.

* Goodbye, struct coalesce

* Revert "Features clipped away to nothing should not participate in that tile"

This reverts commit 124462fbdc.

* Migrating fields from partial to serial_feature

* Name reconciliation between serial_feature and partial

* Replace struct partial with an augmented serial_feature

* Fix some overzealous search-and-replace renaming

* Don't say struct so often

* Remove more of the former partial construction

* Commenting and cleaning up

* Trying again to avoid all these arguments to rewrite

* I swear I did this same thing before and it didn't work.

* More rewrite cleanup

* Exile --detect-shared-borders to its own file

* Add missing headers

* More commenting and cleanup

* More comments

* Sprinkle consts around

* Emplacing and std::moving

* More cleanup

* That shouldn't have worked after a std::move

* Don't need to allocate memory to compare keys

* Reduce use of the global string pool in tiling

* Another avoidable mvt_value construction

* Further reduction to explicit string pool passing

* These reverses are no longer optimizations

* These layernames can all be references

* Don't drag an unused layername string around with every feature

* Heed a compiler warning about potential buffer overflow

* Fix my confusion about which feature's string pool is relevant

* Avoid some unnecessary allocations in attribute accumulation

* Maybe faster serialization?

* Eliminate a comparison

* Do the same here

* Save a couple of allocations when parsing numbers in JSON

* Immediately assign features to layers instead of subdividing later

* Maintain tilestats for tippecanoe:retain_points_multiplier_sequence

* Crunch out more duplicate attribute values when writing out the tile

* Do tilestats for tippecanoe:retain_points_multiplier_first too

* Shell filters need to be real threads, even if nothing else does

* Simplify tippecanoe_minzoom/maxzoom representation

* Update version and changelog

plugin.cpp

@@ -25,6 +25,7 @@
 #include "geometry.hpp"
 #include "serial.hpp"
 #include "errors.hpp"
+#include "thread.hpp"
 
 extern "C" {
 #include "jsonpull/jsonpull.h"
@@ -87,6 +88,7 @@ static std::vector<mvt_geometry> to_feature(drawvec &geom) {
 // Reads from the postfilter
 std::vector<mvt_layer> parse_layers(int fd, int z, unsigned x, unsigned y, std::vector<std::map<std::string, layermap_entry>> *layermaps, size_t tiling_seg, std::vector<std::vector<std::string>> *layer_unmaps, int extent) {
 	std::map<std::string, mvt_layer> ret;
+	std::shared_ptr<std::string> tile_stringpool = std::make_shared<std::string>();
 
 	FILE *f = fdopen(fd, "r");
 	if (f == NULL) {
@@ -262,23 +264,16 @@ std::vector<mvt_layer> parse_layers(int fd, int z, unsigned x, unsigned y, std::
 			}
 
 			for (size_t i = 0; i < properties->value.object.length; i++) {
-				int tp = -1;
-				std::string s;
-
-				stringify_value(properties->value.object.values[i], tp, s, "Filter output", jp->line, j);
+				serial_val sv = stringify_value(properties->value.object.values[i], "Filter output", jp->line, j);
 
 				// Nulls can be excluded here because this is the postfilter
 				// and it is nearly time to create the vector representation
 
-				if (tp >= 0 && tp != mvt_null) {
-					mvt_value v = stringified_to_mvt_value(tp, s.c_str());
+				if (sv.type != mvt_null) {
+					mvt_value v = stringified_to_mvt_value(sv.type, sv.s.c_str(), tile_stringpool);
 					l->second.tag(feature, std::string(properties->value.object.keys[i]->value.string.string), v);
 
-					serial_val attrib;
-					attrib.type = tp;
-					attrib.s = s;
-
-					add_to_tilestats(ts->second.tilestats, std::string(properties->value.object.keys[i]->value.string.string), attrib);
+					add_to_tilestats(ts->second.tilestats, std::string(properties->value.object.keys[i]->value.string.string), sv);
 				}
 			}
 
@@ -503,24 +498,17 @@ serial_feature parse_feature(json_pull *jp, int z, unsigned x, unsigned y, std::
 			}
 
 			for (size_t i = 0; i < properties->value.object.length; i++) {
-				serial_val v;
-				v.type = -1;
-
-				stringify_value(properties->value.object.values[i], v.type, v.s, "Filter output", jp->line, j);
+				serial_val v = stringify_value(properties->value.object.values[i], "Filter output", jp->line, j);
 
 				// Nulls can be excluded here because the expression evaluation filter
 				// would have already run before prefiltering
 
-				if (v.type >= 0 && v.type != mvt_null) {
+				if (v.type != mvt_null) {
 					sf.full_keys.push_back(std::string(properties->value.object.keys[i]->value.string.string));
 					sf.full_values.push_back(v);
 
-					serial_val attrib;
-					attrib.s = v.s;
-					attrib.type = v.type;
-
 					if (!postfilter) {
-						add_to_tilestats(ts->second.tilestats, std::string(properties->value.object.keys[i]->value.string.string), attrib);
+						add_to_tilestats(ts->second.tilestats, std::string(properties->value.object.keys[i]->value.string.string), v);
 					}
 				}
 			}
@@ -643,6 +631,7 @@ std::vector<mvt_layer> filter_layers(const char *filter, std::vector<mvt_layer>
 	wa.extent = extent;
 
 	pthread_t writer;
+	// this does need to be a real thread, so we can pipe both to and from it
 	if (pthread_create(&writer, NULL, run_writer, &wa) != 0) {
 		perror("pthread_create (filter writer)");
 		exit(EXIT_PTHREAD);