dataforcanada/d4c-infra-distribution
1
0
Fork 0
mirror of https://github.com/dataforcanada/d4c-infra-distribution.git synced 2026-06-13 22:20:55 +02:00
Code Issues Packages Projects Releases Wiki Activity

Got it working. Will start running it tomorrow early in the morning, going to chill

Browse Source
This commit is contained in:
Diego Ripley
2026-03-08 15:46:25 -04:00
parent b2dfd76165
commit c9d051bc59
3 changed files with 153 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/05_cloudflare_http_ingestor/README.md
+6 -3
View File
@@ -16,10 +16,13 @@ Client POST ──▶ Worker ──stream──▶ S3 PutObject / Multipart
**Two upload paths are used automatically:**
| Source provides `Content-Length`? | Upload method | Memory overhead |
| Condition | Upload method | Memory overhead |
|---|---|---|
| Yes | Single streaming `PUT` | ~0 (pipe-through) |
| No | Multipart upload in 5 MiB chunks | ≤ 5 MiB |
| Known size ≤ 100 MiB | Single streaming `PUT` | ~0 (pipe-through) |
| Unknown size **or** > 100 MiB | Multipart upload in 5 MiB chunks | ≤ 5 MiB |
> Files larger than 100 MiB always use multipart upload because Cloudflare
> Workers enforce a body-size limit on single outbound `fetch()` requests.
## Setup