diegoripley/tippecanoe
1
0
Fork 0
mirror of https://github.com/felt/tippecanoe.git synced 2026-10-02 08:25:40 +02:00
Code Issues Packages Projects Releases Wiki Activity

Reduce thrashing during feature ingestion and tiling (#56)

Browse Source 
* Remove the concept of "separate metadata"

This was an extra level of attribute indirection (features point
to metadata records which point to key and value strings) which was
intended to reduce the size of temporary storage for features with
large numbers of attributes that were also spread across large numbers
of tiles at maxzoom.

For other kinds of features, the extra indirection slowed things down
instead, and, especially when maxzoom guessing was being used, many more
features were having their metadata externalized than could actually
benefit from it.

* Shave a few bytes off temporary files by using more unsigned integers

* Flush stderr after logging progress

* Revert "Shave a few bytes off temporary files by using more unsigned integers"

This reverts commit eef29084ec.

* Limit the size of the string pools and trees to fit in memory

* Add missing #include

* Move the string pool and search tree from mmap to allocated memory

* Sort in allocated rather than mapped memory too

* Also use pread instead of mapping to read in the data to sort

* When the pool gets too big, switch to just the file, not memory

* Switch string pool from memory to disk when memory is 10% full

* Add to-memory versions of the serialization functions

* Crashy work in progress toward compression

* Fix the pointer bug that was causing the crash

* Serialize features into memory rather than straight to disk

* Compress individual features in the temporary files

* Don't need to store the length of the geometry

* Remove per-feature compression; move minzoom back into the object

* Start adding a stream compressor object

* Track file position within fwrite_check()

* Add compressed stream writer functions

* Pull the writing of the serialized feature out to the callers

* Starting toward compression again from a different point

* Hook up more compression functions

* Remove unused code from the other day

* Make enough deflate calls to flush out all the buffered data

* Start on decompression

* Tile number is uncompressed, tile content is compressed

* Work on alternating compressed and uncompressed in decompression

* Closer, but still doesn't work

* Sort of works

* Works until we get to concatenated tiles

* More attempts that don't work

* One bug down

* It made a tileset!

* Handle nonzero initial zooms

* Fix seeking within compressed feature streams

* Tests pass!

* Remove debug spew

* Oops: remember to delete the temporary files so they don't hang around

* Test that fails with the current compression code

* Properly account for bytes read while closing the compressed stream

* Limit the number of warnings about bad label points

* A little more armor when closing decompression

* This time for sure

* A different, less fragile, test that failed previously with compression

* Move feature stream compression to its own file

* Remove now-unused code to deserialize from a file

* Forgot to add the new files

* Remove a little debugging logging

* Add a couple of comments on what it means to be within decompression

* Fix indentation

* Update changelog. Remove stray debugging comment.
This commit is contained in:
Erica Fischer
2023-02-14 12:47:40 -08:00
committed by GitHub
parent b155b4671b
commit f2127fec97
23 changed files with 85678 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serial.hpp
+11 -24
View File
@@ -11,14 +11,19 @@
#include "mbtiles.hpp"
#include "jsonpull/jsonpull.h"
size_t fwrite_check(const void *ptr, size_t size, size_t nitems, FILE *stream, const char *fname);
size_t fwrite_check(const void *ptr, size_t size, size_t nitems, FILE *stream, std::atomic<long long> *fpos, const char *fname);
void serialize_int(FILE *out, int n, std::atomic<long long> *fpos, const char *fname);
void serialize_long_long(FILE *out, long long n, std::atomic<long long> *fpos, const char *fname);
void serialize_ulong_long(FILE *out, unsigned long long n, std::atomic<long long> *fpos, const char *fname);
void serialize_byte(FILE *out, signed char n, std::atomic<long long> *fpos, const char *fname);
void serialize_uint(FILE *out, unsigned n, std::atomic<long long> *fpos, const char *fname);
void serialize_string(FILE *out, const char *s, std::atomic<long long> *fpos, const char *fname);
void serialize_int(std::string &out, int n);
void serialize_long_long(std::string &out, long long n);
void serialize_ulong_long(std::string &out, unsigned long long n);
void serialize_byte(std::string &out, signed char n);
void serialize_uint(std::string &out, unsigned n);
void deserialize_int(char **f, int *n);
void deserialize_long_long(char **f, long long *n);
@@ -26,12 +31,6 @@ void deserialize_ulong_long(char **f, unsigned long long *n);
void deserialize_uint(char **f, unsigned *n);
void deserialize_byte(char **f, signed char *n);
int deserialize_int_io(FILE *f, int *n, std::atomic<long long> *geompos);
int deserialize_long_long_io(FILE *f, long long *n, std::atomic<long long> *geompos);
int deserialize_ulong_long_io(FILE *f, unsigned long long *n, std::atomic<long long> *geompos);
int deserialize_uint_io(FILE *f, unsigned *n, std::atomic<long long> *geompos);
int deserialize_byte_io(FILE *f, signed char *n, std::atomic<long long> *geompos);
struct serial_val {
int type = 0;
std::string s = "";
@@ -61,8 +60,6 @@ struct serial_feature {
std::vector<long long> keys{};
std::vector<long long> values{};
// If >= 0, metadata is external
long long metapos = 0;
// XXX This isn't serialized. Should it be here?
long long bbox[4] = {0, 0, 0, 0};
@@ -72,54 +69,45 @@ struct serial_feature {
bool dropped = false;
};
void serialize_feature(FILE *geomfile, serial_feature *sf, std::atomic<long long> *geompos, const char *fname, long long wx, long long wy, bool include_minzoom);
serial_feature deserialize_feature(FILE *geoms, std::atomic<long long> *geompos_in, char *metabase, long long *meta_off, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y);
std::string serialize_feature(serial_feature *sf, long long wx, long long wy);
serial_feature deserialize_feature(std::string &geoms, unsigned z, unsigned tx, unsigned ty, unsigned *initial_x, unsigned *initial_y);
struct reader {
int metafd = -1;
int poolfd = -1;
int treefd = -1;
int geomfd = -1;
int indexfd = -1;
FILE *metafile = NULL;
struct memfile *poolfile = NULL;
struct memfile *treefile = NULL;
FILE *geomfile = NULL;
FILE *indexfile = NULL;
std::atomic<long long> metapos;
std::atomic<long long> geompos;
std::atomic<long long> indexpos;
long long file_bbox[4] = {0, 0, 0, 0};
struct stat geomst {};
struct stat metast {};
char *geom_map = NULL;
reader()
: metapos(0), geompos(0), indexpos(0) {
: geompos(0), indexpos(0) {
}
reader(reader const &r) {
metafd = r.metafd;
poolfd = r.poolfd;
treefd = r.treefd;
geomfd = r.geomfd;
indexfd = r.indexfd;
metafile = r.metafile;
poolfile = r.poolfile;
treefile = r.treefile;
geomfile = r.geomfile;
indexfile = r.indexfile;
long long p = r.metapos;
metapos = p;
p = r.geompos;
long long p = r.geompos;
geompos = p;
p = r.indexpos;
@@ -128,7 +116,6 @@ struct reader {
memcpy(file_bbox, r.file_bbox, sizeof(file_bbox));
geomst = r.geomst;
metast = r.metast;
geom_map = r.geom_map;
}