Replace httpc backend with gun for HTTP/2 multiplexing support#68
Open
dcrockwell wants to merge 6 commits intodevelopfrom
Open
Replace httpc backend with gun for HTTP/2 multiplexing support#68dcrockwell wants to merge 6 commits intodevelopfrom
dcrockwell wants to merge 6 commits intodevelopfrom
Conversation
## Why This Change Was Made - The HTTP client had 6 hardcoded httpc configuration values baked into the Erlang shim (connect_timeout, autoredirect, max_sessions, max_pipeline_length, keep_alive_timeout, max_keep_alive_length). Users had no way to tune TCP connection timeouts, disable redirect following, or adjust connection pool settings for their workload. This is a published library — users shouldn't have to fork it to change a timeout. ## What Was Changed - Added `connect_timeout(ms)` and `auto_redirect(enabled)` builder functions on ClientRequest for per-request settings, with corresponding getters and private resolve functions that apply the previous hardcoded defaults when not set - Added `TransportConfig` opaque type with `transport_config()` factory, 4 builder functions (max_sessions, max_pipeline_length, keep_alive_timeout, max_keep_alive_length), 4 getters, and `configure_transport()` to apply settings globally via ETS + httpc:set_options - Added `dream_http_client_transport_config` ETS table creation to dream_http_client_app:start/2 - Updated configure_httpc/0 to read from ETS (falling back to hardcoded defaults if configure_transport was never called) - Updated all 3 Erlang shim request functions (request_sync/7, request_stream/8, request_stream_messages/8) and stream_owner_loop/6 to accept and use the new parameters - Updated internal.gleam FFI signatures and start_httpc_stream to pass connect_timeout_ms and autoredirect - Updated YielderState, RecordingYielderState, and all 7 caller functions in client.gleam to resolve and propagate the new parameters - 14 new tests, 3 test snippets, README Configuration section, CHANGELOG 5.2.0 entry, release notes, version bump 5.1.3 → 5.2.0 ## Note to Future Engineer - The Erlang FFI arity chain is: Gleam external declaration → Erlang export list → Erlang function head. If you change one, you must change all three or you get a runtime badarity crash that won't show up until an actual HTTP request is made. The compiler won't catch FFI arity mismatches. - configure_transport writes to ETS AND calls httpc:set_options immediately. configure_httpc reads from ETS on every request. Yes, this means the settings are applied twice on the first request after configure_transport is called. No, this is not a bug — it's the price of making runtime reconfiguration work without a restart. You're welcome. - The resolve functions (resolve_connect_timeout, resolve_auto_redirect) are where the defaults live. If you want to change a default, change it there AND in the ETS fallback branch of configure_httpc AND in the TransportConfig factory. Three places. We know. It's Erlang's fault.
## Why This Change Was Made - The application needs to make tens of thousands of concurrent HTTP requests to HTTP/2 servers (OpenAI, Anthropic, etc.). Erlang's built-in httpc is limited to HTTP/1.1 with one-connection-per-request, which means 10k requests = 10k TCP connections. Gun supports HTTP/2 multiplexing, allowing thousands of concurrent streams over a handful of connections. - TransportConfig was redesigned with 13 gun-native fields (connection pooling, timeouts, HTTP/2 flow control, keepalive) to expose full configurability rather than the 4 httpc-specific fields that would have been immediately obsolete. ## What Was Changed - Replaced dream_httpc_shim.erl (1177 lines) with dream_http_shim.erl using gun:open/await_up/get/post/await/await_body for sync, pull-stream, and message-stream request modes - Added dream_http_conn_manager.erl: a gen_server managing gun connections via an ETS bag table with per-host round-robin selection, idle reaping, dead connection cleanup, and crash recovery in init/1 - Redesigned TransportConfig from 4 httpc fields to 13 gun-native fields with full builder/getter API - Implemented manual auto-redirect (301/302/303/307/308, max 5 hops) with relative URL resolution since gun does not follow redirects natively - Added stale connection retry: requests that hit a server-closed connection are retried once on a fresh connection - Added mock server redirect endpoints and 11 integration tests covering all redirect status codes across send(), stream_yielder(), and start_stream() - Updated all FFI references from dream_httpc_shim to dream_http_shim - Updated CHANGELOG, README, release notes, test snippets ## Note to Future Engineer - Gun sends connection-level messages (gun_up/gun_down) to the owner process (the gen_server) and stream-level messages (gun_response/gun_data) to the reply_to process (the request caller). This split is load-bearing — do not try to consolidate message handling into the gen_server or you'll create a bottleneck that serializes all HTTP traffic through one process. - The ETS table is a `bag` (not `set`) because we store multiple connections per host. This means ets:select_replace doesn't work — touch/1 does match_delete + insert instead. Yes, we learned this the hard way. - drain_stream uses a hardcoded 5000ms timeout, not the request timeout. This is intentional — it's draining a redirect response body, not waiting for user data. If you change it to the request timeout, you'll wonder why redirects occasionally take 30 seconds. You're welcome. - gun:await defaults to 5 seconds if you don't pass a timeout. Every call site explicitly passes TimeoutMs. If you add a new gun:await call and forget the timeout, you'll get mysterious 5-second failures in production. Ask me how I know.
dcrockwell
changed the title
The httpc-to-gun swap added gun as a transitive dependency of dream_http_client, but downstream manifest.toml files were stale and missing it, causing "no such file or directory: gun.app" at runtime in CI integration tests.
…ency
## Why This Change Was Made
- After fixing mist's HTTP/2 frame handling for RFC 9113 compliance, dream needed its own server-side tests proving the full h2c pipeline works end-to-end
- The mist dependency needed to move from a local path to the GitHub branch with the HTTP/2 fixes, so CI and collaborators can resolve it
- dream_http_client switched to a local path dev-dependency to access the Http2Only protocol API (not yet published on Hex)
## What Was Changed
- Added 6 BDD h2c integration tests in test/dream/servers/mist/h2c_test.gleam: GET, path params, JSON with content-type, 404 error routing, POST with body echo, concurrent multiplexed requests
- Each test starts its own server on a distinct port (19980-19985) to avoid TCP TIME_WAIT conflicts
- Switched mist dependency from `{ path = "../mist" }` to `{ git = "https://github.com/TrustBound/mist.git", ref = "fix/http2-support" }`
- Switched dream_http_client dev-dep from Hex version to `{ path = "modules/http_client" }` for access to protocols(Http2Only) API
- Fixed hooks.gleam: client.new -> client.new() and readiness check now treats ResponseError (HTTP 404) as "server is responding"
- Fixed opensearch module and 4 examples to use new structured error types (RequestError.error: TransportError, StreamFailure)
- Removed direct mist dependency from all 11 examples and regenerated manifests (they get mist transitively from dream)
- Includes all dream_http_client v5.2.0 changes: structured error types, h2c protocol support, log level configuration
## Note to Future Engineer
- Examples should NOT have mist as a direct dependency — they get it transitively through dream. Adding it back will cause dependency resolution failures when dream uses a git or path dep for mist
- If "starts on expected port" test starts failing, check that is_server_responding handles ResponseError (404 from empty router) as a success signal
- All code that pattern-matches on RequestError must use `error: TransportError` not `message: String`; use `client.transport_error_to_string()` for readable strings
- stream_yielder now returns Result(BytesTree, StreamFailure) instead of Result(BytesTree, String); use `client.stream_failure_to_string()` to convert
## Why This Change Was Made
- `send_request` in `dream_http_shim.erl` used dynamic dispatch (`gun:Method/4`)
which is inconsistent across HTTP methods. For PUT/POST/PATCH, `gun:put/3` etc.
call `headers()` with `nofin`, expecting a follow-up `gun:data/4` that never
arrives. For GET, `gun:get/4` treats the 4th arg as `ReqOpts` (not `Body`),
crashing with `{badmap, <<>>}`. Both sides wait forever on PUT/POST/PATCH.
## What Was Changed
- Replaced `gun:Method/4` dynamic dispatch with `gun:request/5` which always
sends `{request, ...}` with body and `fin` flag, regardless of HTTP method
- Added `method_atom_to_binary/1` to convert method atoms to uppercase binaries
- Added PATCH /patch endpoint to mock server (view, controller, router)
- Added 5 regression tests for empty-body PUT, POST, PATCH, GET, DELETE
- Updated CHANGELOG with Fixed entry
## Note to Future Engineer
- gun's method-specific functions have wildly inconsistent arity semantics:
`gun:get/4` takes ReqOpts, `gun:put/4` takes Body. `gun:put/3` sends nofin,
`gun:get/3` sends fin. Just use `gun:request/5` and save yourself the therapy.
## Why This Change Was Made - During debugging of the HTTP/2 stream actor lifecycle bug, mist was pointed at a local path (../mist) for rapid iteration. Now that the fix is committed and pushed to TrustBound/mist fix/http2-support, the dependency is restored to the git reference. ## What Was Changed - gleam.toml: mist dependency from `path = "../mist"` back to `git = "https://github.com/TrustBound/mist.git", ref = "fix/http2-support"` - manifest.toml: updated automatically by gleam build ## Note to Future Engineer - If you need to iterate on mist locally again, swap the dep back to path. Just don't commit it. We've been through this before. Twice.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
httpcto gun 2.2.0gen_serverwith per-host round-robin, idle reaping, and crash recoveryTransportConfigredesigned with 13 gun-native fields for full control over connection management, timeouts, HTTP/2 flow control, and keepaliveconnect_timeoutandauto_redirectcontrols added toClientRequestsend(),stream_yielder(), orstart_stream()Why
Erlang's
httpcis limited to HTTP/1.1 with one-connection-per-request. For applications that need high concurrency — e.g. making tens of thousands of requests to HTTP/2 endpoints like OpenAI or Anthropic — that means tens of thousands of TCP connections. Gun supports native HTTP/2 multiplexing via ALPN negotiation, allowing thousands of concurrent streams over a few connections. By swapping the framework's underlying HTTP backend, any application that needs this capability gets it for free.What
New files
dream_http_shim.erl— Gun-backed Erlang FFI shim replacingdream_httpc_shim.erl. Implementsrequest_sync/7,request_stream/8,request_stream_messages/8using gun primitives. Includes manual auto-redirect (301/302/303/307/308, max 5 hops with relative URL resolution), stale connection retry, decompression, and header normalization.dream_http_conn_manager.erl— OTPgen_servermanaging gun connections via an ETSbagtable. Per-host multi-connection pooling with round-robin selection, idle connection reaping, dead connection cleanup, and crash recovery ininit/1.redirect_test.gleam— 11 integration tests covering all redirect status codes acrosssend(),stream_yielder(), andstart_stream().Deleted files
dream_httpc_shim.erl— Replaced bydream_http_shim.erlModified files
client.gleam—TransportConfigredesigned from 4 httpc fields to 13 gun-native fields with full builder/getter API. Per-requestconnect_timeoutandauto_redirectadded. Module doc updated to reference gun.internal.gleam— All@externalreferences updated fromdream_httpc_shimtodream_http_shimdream_http_client_app.erl— Createsdream_http_client_connectionsETS bag tabledream_http_client_sup.erl— Addeddream_http_conn_manageras supervised child workergleam.toml— Addedgun >= 2.2.0 and < 3.0.0dependencyTransportConfig (13 fields)
max_connectionsidle_timeoutdefault_connect_timeoutdomain_lookup_timeouttls_handshake_timeoutretryretry_timeoutkeepalivekeepalive_tolerancemax_concurrent_streamsinitial_connection_window_sizeinitial_stream_window_sizeclosing_timeoutTest plan
gleam format --checkpassesauto_redirect(False)send(),stream_yielder(),start_stream()ets_table_ownership_test