diegoripley/tippecanoe
1
0
Fork 0
mirror of https://github.com/felt/tippecanoe.git synced 2026-10-02 16:35:40 +02:00
Code Issues Packages Projects Releases Wiki Activity

Run prefilter/postfilter shells via single-threaded worker processes

Browse Source 
Forking and waitpid()-ing for the -C/-c shell filters used to happen
directly in the multi-threaded tiling threads, which made fork() expensive
(full COW of the parent's address space) and unsafe (other threads could
hold libc/malloc locks at fork time). Now we pre-fork a small pool of
single-threaded worker processes once, before any tiling or reader threads
are spawned, and the tiling threads ask a worker to fork+exec the shell
child on their behalf, receiving the input/output pipe fds back via
SCM_RIGHTS. The fork() and waitpid() happen entirely inside the
single-threaded worker, where they are cheap and only block that worker.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Erica Fischer
2026-05-30 14:50:37 -07:00
co-authored by Cursor
parent cb6cacef15
commit 4f4214ee23
4 changed files with 443 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tile.cpp
+3 -12
View File
@@ -1770,7 +1770,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
}
int prefilter_write = -1, prefilter_read = -1;
pid_t prefilter_pid = 0;
filter_handle prefilter_handle;
FILE *prefilter_fp = NULL;
pthread_t prefilter_writer;
run_prefilter_args rpa; // here so it stays in scope until joined
@@ -1783,7 +1783,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
}
if (prefilter != NULL) {
setup_filter(prefilter, &prefilter_write, &prefilter_read, &prefilter_pid, z, tx, ty);
setup_filter(prefilter, &prefilter_write, &prefilter_read, &prefilter_handle, z, tx, ty);
prefilter_fp = fdopen(prefilter_write, "w");
if (prefilter_fp == NULL) {
perror("freopen prefilter");
@@ -2294,16 +2294,7 @@ long long write_tile(decompressor *geoms, std::atomic<long long> *geompos_in, ch
perror("close output from prefilter");
exit(EXIT_CLOSE);
}
while (1) {
int stat_loc;
if (waitpid(prefilter_pid, &stat_loc, 0) < 0) {
perror("waitpid for prefilter\n");
exit(EXIT_PTHREAD);
}
if (WIFEXITED(stat_loc) || WIFSIGNALED(stat_loc)) {
break;
}
}
wait_filter(prefilter_handle);
void *ret;
if (pthread_join(prefilter_writer, &ret) != 0) {
perror("pthread_join prefilter writer");