GSoC 2026 Proposal Draft: HTTP Streaming

[Piyush-Goenka]

Project Proposal

1. Project Title

HTTP Streaming Support for Rage Framework — Mentored by @rsamoilov @tonekk

---

2. Project Size

Medium (~175 hours)

This scope is appropriate because the work involves:

• Designing and implementing a new streaming path in Iodine's C layer
• Implementing http_stream from scratch for chunked transfer encoding
• Building the RackStream writer with full backpressure and lifecycle management
• Adding fiber-safe resume bridges between Iodine's event loop and Rage's fiber scheduler
• Implementing the Rage-side developer API with enumerator support
• Writing integration tests covering all edge cases across both repositories

This is not a small patch — it touches core networking internals in C and the Ruby-facing API layer. It is scoped to HTTP/1.1 only in this phase, keeping it within the 175-hour medium boundary.

---

3. Project Length

Medium project (~175 hours) over 12 weeks.

Proposed coding period: May 27 – August 25, 2026

No major scheduling conflicts during this period. I will be available full-time for the duration of the program. Any minor constraints will be communicated to mentors in advance.

---

4. Abstract

Modern streaming applications like ChatGPT, Claude, and Grok deliver content incrementally using HTTP Chunked Transfer Encoding. Rage, a fiber-based Ruby web framework built on the Iodine server, currently buffers all responses before sending — making real-time streaming impossible. This project implements core HTTP streaming support across two repositories: the Iodine C layer (transport, backpressure, chunked encoding) and the Rage Ruby layer (developer-friendly API, fiber integration). The result will be a declarative, protocol-agnostic streaming API — render stream: enumerator — that unlocks real-time capabilities in Rage while keeping the interface abstract enough to support HTTP/2 in the future.

---

5. Problem Statement

Scope

This project is explicitly scoped to:

✔️ HTTP Streaming via Chunked Transfer Encoding
✔️ HTTP/1.1 transport (HTTP/2 design-ready)
✔️ Developer-friendly protocol-agnostic API

✖️ NO Server-Sent Events (SSE) — out of scope
✖️ NO WebSocket changes — existing path untouched
✖️ NO HTTP/2 implementation — future work only

SSE support is a natural future extension of this work — the architecture is explicitly designed to support it with minimal additional effort.

---

What Exists Today

Iodine currently handles Rack responses in two ways:

String body — entire response sent in one shot:

[200, headers, "hello world"] → http_send_body → finish/close

each body — all chunks collected into a C buffer, then sent as one shot:

[200, headers, body.each] → collect ALL chunks → http_send_body → finish/close

Codebase Evidence

• iodine_http.c — Iodine accepts Enumerator bodies since Enumerator responds to #each, but the full #each loop is collected into a C buffer before sending — not streamed progressively.
• iodine/README.md — explicitly states "Rack streaming is not supported".
• api.rb — Rage controller builds @__body as an array. No streaming path exists in Rage today.

The Core Problem

Even when an app produces chunks incrementally via .each, Iodine buffers everything first. The user receives nothing until the entire response is collected:

What apps want:           What Iodine does today:
chunk 1 -> send now       chunk 1 ─┐
chunk 2 -> send now       chunk 2 ─┤-> collect all -> send once
chunk 3 -> send now       chunk 3 ─┘

This makes real-time streaming — word-by-word LLM output, live data feeds, large file downloads — impossible without a protocol upgrade.

Why Rage is the Right Framework to Fix This

Rage is built on fibers — every request runs in its own fiber. Pausing a fiber on backpressure costs nothing and releases the thread for Iodine to drain. This makes Rage architecturally perfect for streaming. The missing piece is the transport layer support in Iodine.

---

6. Proposed Solution

Two Repositories, Clear Boundary

This project spans two repositories with a clean separation of responsibilities:

rage-rb/iodine (https://github.com/rage-rb/iodine)  -> C layer
  transport, chunked encoding, backpressure,
  socket lifecycle, RackStream writer

rage-rb/rage   (https://github.com/rage-rb/rage)    -> Ruby layer
  developer API, enumerator wrapping,
  fiber scheduling, protocol abstraction

---

Approach Selection — Option A vs Option B

Two approaches were considered for implementing HTTP streaming in Iodine:

Option A adds a new streaming path directly in the normal Rack HTTP response flow, detecting a body.call(stream) body type and handling it incrementally. Option B reuses the existing rack.upgrade mechanism — the same path used today for WebSocket upgrades.

Option B was rejected because rack.upgrade changes the semantics of the connection entirely — the app takes full ownership of the raw socket. Using it for HTTP streaming would require apps to use upgrade semantics for normal responses, break the standard Rack response contract, and make HTTP/2 support impossible without rewriting app-facing code. It mixes two fundamentally different concerns into one abstraction.

Option A is the correct path because it preserves the standard Rack HTTP contract, leaves existing String and each body paths completely unchanged, and keeps the transport layer extensible for HTTP/2 without any app-facing changes. Streaming becomes a first-class Rack body type — not a protocol upgrade.

---

Key Design Decisions

HTTP streaming in normal Rack HTTP path, not rack.upgrade — streaming is added as a new body type in the standard response flow. This preserves the Rack contract, leaves existing paths untouched, and keeps the design extensible for HTTP/2.

body.call(stream) as the streaming contract — the new body type follows the Rack streaming spec directly. Iodine detects this body type and hands a RackStream writer to the app. This is the boundary between Iodine's transport layer and the Ruby application layer.

Enumerator normalization in Rage, not Iodine — when the developer writes render stream: enumerator, Rage wraps the enumerator into a body.call(stream) proc internally. Iodine only ever sees call(stream). This keeps C scope minimal and follows Rage's cooperative concurrency model — if Iodine iterated the enumerator directly in C, any blocking operation inside it would freeze the entire event loop.

Explicit :would_block over implicit blocking — stream.write returns :would_block when the buffer is full instead of blocking internally. The producer calls Fiber.yield, releasing the GVL so Iodine can drain. This is the only design compatible with an event-loop server.

Dual callbacks on backpressure — when a fiber pauses on :would_block, both on_drain and on_close are registered. on_drain handles normal resume, on_close ensures the fiber always wakes on disconnect. Registering only on_drain would cause fibers to sleep forever if the client disconnects while blocked.

---

Backpressure Strategy

Two backpressure strategies were explored for handling fast producers.

The first approach has stream.write return :would_block explicitly when the buffer exceeds the high watermark. The producer calls Fiber.yield, releasing the GVL so Iodine can drain the buffer freely. Once pending bytes drop below the low watermark, on_drain fires, IodineCaller acquires the GVL, and the fiber resumes. The event loop is never blocked.

The second approach has write block internally until the buffer drains — simpler for the caller, but it holds the GVL inside the wait loop. Since Iodine needs the GVL to run on_drain, the buffer never drains, the loop never exits, and the server deadlocks. A single streaming request would freeze all connections.

The first approach is the only design compatible with an event-loop server and is what this project implements.

---

Developer-Facing API — Rage Side

The public API is declarative and enumerator-based:

# Simple case — stream any enumerable:
render stream: "Hello, world".each_char

# Custom enumerator — full control:
stream = Enumerator.new do |y|
  data_source.each do |chunk|
    y << chunk
    sleep 1        # Rage fiber scheduler handles this
  end
end
render stream: stream

Developer never sees :would_block, Fiber.yield, or stream.close. All backpressure and lifecycle management is hidden inside Rage.

---

Enumerator Normalization — Where the Conversion Happens

Decision: Rage normalizes Enumerator → body.call(stream) proc.

This was the most carefully considered architectural decision of this proposal. Two options were explored:

Option 1 — Rage converts (chosen)
Rage wraps the Enumerator into a body.call(stream) proc internally. Iodine only ever sees call(stream).

Option 2 — Iodine C converts (rejected)
Iodine detects Enumerator in C and iterates natively via rb_funcall.

Why Option 2 is fundamentally wrong:

For an event-driven server, the highest priority is never blocking the event loop. If Iodine C calls enumerator.next and the Ruby code inside the enumerator performs a blocking operation — reading a file, querying a database, making an HTTP request — the C layer waits for Ruby to return, freezing the entire event loop and starving all other connections.

Why Option 1 is the only correct approach:

Option 1 follows Rage's cooperative concurrency model correctly:

• Rage handles the high-level logic of iterating the enumerator inside the Fiber
• Iodine provides the low-level stream object and signals backpressure
• Fiber.yield is the glue that allows Ruby code to pause and hand control back to Iodine

Performance concern is negligible: Ruby handles millions of method calls per second. The overhead of the Ruby loop is minimal compared to the cost of network I/O.

Internal conversion in Rage:

# Developer writes: render stream: my_enumerator
# Rage converts internally to:

body = proc do |stream|
  enumerator.each do |chunk|
    loop do
      result = stream.write(chunk)
      case result
      when :ok          then break
      when :would_block then Fiber.yield   # glue — releases GVL
      when :closed, :disconnected, :error
        raise IOError, "stream terminated"
      end
    end
  end
ensure
  stream.close
end

[200, headers, body]   # handed to Iodine → triggers call(stream) path

---

New Rack Body Detection Branch — Iodine Side

A new call(stream) branch is added to Iodine's body detection in iodine_http.c:

Rack body type?
├── String        → existing one-shot send (unchanged)
├── each          → existing buffered compatibility path (unchanged)
└── call(stream)  → NEW: create RackStream writer → body.call(stream)

Existing paths are completely untouched — backwards compatibility guaranteed.

---

RackStream Writer Internals

Every streaming connection is managed by a stream_ctx_t struct in C:

typedef struct {
  intptr_t uuid;         // socket connection (facil.io handle)
  stream_state_t state;  // lifecycle state
  VALUE fiber;           // Fiber.current — producer fiber reference
  size_t high_watermark; // 64KB — pause threshold
  size_t low_watermark;  // 16KB — resume threshold
  int blocked;           // backpressure flag
} stream_ctx_t;

Initialized at call(stream) detection — the earliest point where all fields are available:

• uuid — client already connected
• state — set to IDLE
• fiber — Fiber.current available
• high_watermark / low_watermark — pre-configured constants
• blocked — set to false

stream_ctx_t Memory Lifecycle

stream_ctx_t cannot rely on Ruby GC alone because it owns C-side connection state and also holds Ruby references. The proposed ownership model is:

• Allocation point — allocate stream_ctx_t when Iodine enters the new body.call(stream) branch in iodine_http.c, right before creating the RackStream writer object.
• Primary owner — the active HTTP streaming response path owns the context, not the generic Iodine::Connection attachment system. That system currently manages upgraded WebSocket/SSE/raw connection objects, but this feature stays in normal HTTP response flow, so streaming needs its own teardown path.
• Ruby reference safety — the RackStream Ruby object must mark the fiber reference during GC and release it only after terminal teardown so the producer fiber cannot be collected while blocked.
• Normal terminal cleanup — when the stream reaches CLOSED after stream.close / final chunk / response finish, teardown detaches drain/close callbacks, clears Ruby references, and frees stream_ctx_t exactly once.
• Disconnect cleanup — if the client disconnects before normal completion, the disconnect callback marks the context terminal, wakes any blocked producer, unregisters callbacks, and runs the same one-shot free path.
• Double-free protection — the context needs a terminal/freed guard so write-path errors, disconnect callbacks, and normal close cannot all free it independently. Cleanup must be centralized in one final teardown function.

This means custom teardown logic will be required for HTTP streaming; Iodine's existing upgraded-connection attachment lifecycle is useful as a reference, but it does not automatically own or free stream_ctx_t in the normal HTTP response path.

---

stream.write Decision Flow

Every stream.write(data) call executes these checks in strict O(1) order:

1. closed/error?           → return :closed immediately    (flag check)
2. socket disconnected?    → return :disconnected           (fio_is_valid)
3. data is String?         → TypeError if not               (Check_Type)
4. pending >= HARD_MAX     → fail stream, return :error     (safety cutoff)
5. pending >= HIGH (64KB)  → return :would_block            (normal backpressure)
6. http_stream send chunk  → return :ok or :error           (actual send)

Cheapest checks first. Most expensive operation (actual send) only happens after all cheap checks pass.

---

Three-Threshold Backpressure Policy

LOW_WATERMARK  = 16KB  → resume producer after drain
HIGH_WATERMARK = 64KB  → pause producer, return :would_block
HARD_MAX       = 256KB → last line of defense, fail stream

Hysteresis between HIGH and LOW prevents stuttering. Producer never resumes at HIGH — waits until LOW, giving breathing room before the next pause.

HARD_MAX policy — Fail Stream:

• Drop chunk → data silently lost, stream corrupted — rejected
• Fail Stream → close stream cleanly, client gets error, can retry — chosen
• Disconnect → TCP teardown, too aggressive — rejected

Nine backpressure rules:

1. Hysteresis — two thresholds not one
2. HIGH/LOW defined as explicit constants
3. pending >= HIGH → :would_block immediately
4. Resume only when pending <= LOW
5. write is O(1) — returns immediately always
6. Never busy-wait — Fiber.yield only
7. close is idempotent in all states
8. Disconnect → all further writes fail fast
9. No unbounded buffering inside writer

---

Stream State Machine

IDLE → HEADERS_SENT → STREAMING
STREAMING ↔ BLOCKED (backpressure)
STREAMING/BLOCKED → CLOSING → CLOSED
STREAMING/BLOCKED/CLOSING → ERROR → CLOSED

Full transition table:

From	Trigger	To
IDLE	first write	HEADERS_SENT
HEADERS_SENT	headers flushed	STREAMING
STREAMING	pending >= HIGH	BLOCKED
BLOCKED	pending <= LOW	STREAMING
STREAMING	close called	CLOSING
BLOCKED	close called while sleeping	CLOSING
CLOSING	final chunk sent + cleanup	CLOSED
STREAMING	write failure / disconnect	ERROR
BLOCKED	disconnect while sleeping	ERROR
CLOSING	write failure during flush	ERROR
ERROR	cleanup done	CLOSED

---

Code Snippets

1. rack_stream_write in C

static VALUE rack_stream_write(VALUE self, VALUE data) {
  stream_ctx_t *ctx = get_ctx(self);

  // fail fast checks — O(1)
  if (ctx->state == CLOSED || ctx->state == ERROR)
    return SYM_CLOSED;

  if (!fio_is_valid(ctx->uuid))
    return SYM_DISCONNECTED;

  Check_Type(data, T_STRING);

  // HARD_MAX check — last line of defense, Fail Stream
  if (fio_pending(ctx->uuid) >= HARD_MAX) {
    ctx->state = ERROR;
    return SYM_ERROR;  // terminal error; teardown handled by stream/connection lifecycle
  }

  // HIGH_WATERMARK check — normal backpressure
  if (fio_pending(ctx->uuid) >= ctx->high_watermark) {
    ctx->blocked = 1;
    ctx->state = BLOCKED;
    ensure_drain_subscription(ctx);  // register on_drain + on_close
    return SYM_WOULD_BLOCK;
  }

  // actual send — http_stream implemented from scratch
  if (http_stream(ctx->h, RSTRING_PTR(data), RSTRING_LEN(data)) < 0) {
    ctx->state = ERROR;
    return SYM_ERROR; // terminal error; teardown handled by stream/connection lifecycle
  }

  return SYM_OK;
}

2. on_drain + on_close resume bridge in C

// Normal resume — buffer drained below LOW_WATERMARK
static void on_stream_drain(intptr_t uuid, void *ctx_ptr) {
  stream_ctx_t *ctx = (stream_ctx_t *)ctx_ptr;

  if (fio_pending(uuid) > ctx->low_watermark)
    return;  // not drained enough yet — keep waiting

  if (ctx->blocked && ctx->fiber != Qnil) {
    ctx->blocked = 0;
    ctx->state = STREAMING;
    // IodineCaller acquires GVL safely before resuming Ruby fiber
    IodineCaller.call2(rb_fiber_resume, ctx->fiber, 0, NULL);
  }
}

// Disconnect resume — client gone while fiber sleeping
static void on_stream_close(intptr_t uuid, void *ctx_ptr) {
  stream_ctx_t *ctx = (stream_ctx_t *)ctx_ptr;

  if (ctx->blocked && ctx->fiber != Qnil) {
    ctx->state = ERROR;
    // disconnect is already a terminal lifecycle event
    // wake fiber with error signal so it never sleeps forever
    IodineCaller.call2(rb_fiber_resume, ctx->fiber, 0, NULL);
  }
}

Note: rb_fiber_resume is called via IodineCaller to safely acquire the GVL before executing Ruby code from within Iodine's C event loop. Direct rb_fiber_resume call without GVL would crash.

3. Rage-side enumerator wrapping

# lib/rage/streaming.rb
module Rage
  class Streaming
    def initialize(enumerator)
      @enumerator = enumerator
    end

    def call(stream)
      @enumerator.each do |chunk|
        loop do
          result = stream.write(chunk)
          case result
          when :ok          then break
          when :would_block
            Fiber.yield    # releases GVL — Iodine drains — on_drain resumes
          when :closed, :disconnected, :error
            raise IOError, "stream terminated"
          end
        end
      end
    ensure
      stream.close
    end
  end
end

4. http_stream — to be implemented from scratch

// This function does not exist in Iodine today.
// Core GSoC deliverable — implements HTTP/1.1 chunked encoding.
static int http_stream(http_s *h, const char *data, size_t length) {
  // format chunk as HTTP/1.1 chunked transfer encoding:
  // <size in hex>\r\n
  // <data>\r\n
  char size_buf[32];
  int size_len = snprintf(size_buf, sizeof(size_buf), "%zx\r\n", length);

  // write size line
  fio_write(h->connection, size_buf, size_len);
  // write data
  fio_write(h->connection, data, length);
  // write terminator
  fio_write(h->connection, "\r\n", 2);

  return 0;  // return negative on failure
}

---

Existing Rage Primitives to Hook Into

Rage already has fiber infrastructure. This project hooks into existing primitives — not building from scratch:

lib/rage/fiber.rb              → Fiber.await — use for drain/resume
lib/rage/fiber_scheduler.rb    → pause/block/unblock mechanics
lib/rage/middleware/
  fiber_wrapper.rb             → request pause/resume integration
iodine/lib/iodine/
  connection.rb                → on_drained callback — existing hook

---

Complete Data Flow

Developer writes:
render stream: my_enumerator

Rage:
  detects render stream: key
  wraps enumerator in Rage::Streaming
  returns [200, headers, streaming_body]

Iodine:
  detects body.call(stream) type
  creates RackStream writer (stream_ctx_t)
  calls body.call(stream)

Rage::Streaming#call executes in Fiber:
  iterates enumerator
  calls stream.write(chunk) per chunk
  :ok          → continue
  :would_block → Fiber.yield (releases GVL)
                 Iodine drains buffer
                 on_drain fires
                 IodineCaller → rb_fiber_resume
                 fiber wakes, continues
  :closed/:error → IOError raised, stream stops

stream.close:
  final chunk sent
  HTTP response finalized
  connection closed cleanly

---

Target Files

Iodine — Modify:

• iodine/ext/iodine/http_internal.h — stream state types, ctx struct declaration
• iodine/ext/iodine/http.h — transport lifecycle API declarations
• iodine/ext/iodine/http.c — generic HTTP stream primitives
• iodine/ext/iodine/http1.c — implement http_stream chunked encoding
• iodine/ext/iodine/iodine_http.c — add call(stream) branch to body detection

Iodine — Create:

• iodine/ext/iodine/iodine_rack_stream.h — RackStream writer header
• iodine/ext/iodine/iodine_rack_stream.c — full RackStream writer implementation
• iodine/spec/integration/response_streaming_spec.rb — integration tests
• iodine/spec/support/apps/response_streaming.ru — test Rack app fixture

Iodine — Update:

• iodine/README.md — streaming API documentation

Rage — Modify:

• lib/rage/controller/api.rb — add render stream: support
• lib/rage/fiber.rb — hook drain/resume into existing Fiber.await
• lib/rage/fiber_scheduler.rb — integrate stream pause/unblock
• lib/rage/rack_adapter.rb — detect streaming response type

Rage — Create:

• lib/rage/streaming.rb — Rage::Streaming enumerator wrapper
• spec/stream_spec.rb — streaming integration tests

Rage — Update:

• README.md — developer guide for streaming API

---

Edge Cases Covered (Behavior Matrix)

ID	Scenario	Behavior
01	status=200, streaming body	normal incremental streaming
02	status=204 with streaming	ignore body, finalize headers only
03	status=304 with streaming	ignore body, finalize headers only
04	HEAD request + streaming	no body bytes sent
05	no explicit content-length	chunked encoding framing used
06	conflicting body-length header	strip/reject in stream mode
07	header mutation after first write	reject, log error
08	non-string chunk	TypeError
09	first write is empty string	header-flush-only, no-op
10	large chunk write	split safely, no blocking loop
11	write before flush	send headers lazily then chunk
12	write after close	return :closed immediately
13	double close	idempotent, no crash
14	stream never closed by app	auto-close at end of call path
15	exception before first write	normal 500 error response
16	exception after first write	log + close stream safely
17	body.close raises	log, continue cleanup
18	pending below HIGH	write → :ok
19	pending crosses HIGH	write → :would_block, mark blocked
20	writes while blocked	immediate :would_block, no growth
21	drain, pending still high	remain blocked
22	drain, pending <= LOW	resume fiber, continue writing
23	blocked then close called	CLOSING state, finish when safe
24	disconnect before first write	fail fast, closed/error state
25	disconnect mid-stream	fail fast, cleanup done
26	disconnect while blocked	on_close fires, IOError, cleanup
27	write syscall failure	ERROR state + cleanup
28	many concurrent streams	no starvation, no loop blocking
29	hot producer + normal requests	normal requests stay responsive
30	existing String body	unchanged
31	existing each body	unchanged, buffered compatibility
32	upgrade flows (rack.upgrade)	unchanged
33	fiber pause/resume interaction	no deadlocks, state preserved
34	TLS + plain HTTP	same stream contract
35	GC/finalizer edge	no leaked handles or subscriptions

---

7. Project Deliverables

Iodine (C Layer)

1. http_stream function — core chunked transfer encoding, implemented from scratch
2. stream_ctx_t struct — RackStream writer context with all 6 fields
3. rack_stream_write — O(1) write with all checks and return semantics
4. rack_stream_close — idempotent, safe in all states
5. on_drain callback — buffer drain resume with LOW_WATERMARK check
6. on_close callback — disconnect safety, fiber never sleeps forever
7. Three-threshold backpressure — LOW/HIGH/HARD_MAX with Fail Stream policy
8. State machine enforcement — IDLE → CLOSED/ERROR lifecycle
9. Existing path compatibility — String and each paths completely unchanged

Rage (Ruby Layer)

10. render stream: enumerator — declarative developer API
11. Rage::Streaming — enumerator → body.call(stream) conversion
12. Fiber integration — hooks into existing Fiber.await and fiber_scheduler.rb
13. Error handling — log + close on exception after first write

Both Repos

14. Integration tests — producer flood, concurrent requests, disconnect mid-stream, backpressure resume
15. Documentation — updated READMEs with streaming API guide

---

8. Timeline / Milestones

Community Bonding (May 1 – May 26)

• Deep dive into iodine_http.c, http1.c, http.c source
• Study facil.io APIs: fio_pending, fio_write, fio_is_valid, socket lifecycle
• Study IodineCaller and iodine_connection_fire_event mechanisms
• Study existing Rage fiber primitives: fiber.rb, fiber_scheduler.rb, fiber_wrapper.rb
• Study iodine/lib/iodine/connection.rb on_drained callback
• Finalize http_stream function signature with mentors
• Set up local development and test environment for both repos
• Discuss any remaining design questions with @tonekk and @rsamoilov

Phase 1 — Core C Layer (Week 1–3)

• Implement http_stream in http1.c — chunked transfer encoding
• Add transport lifecycle API (begin, write, finish) to http.c
• Add http_internal.h stream state types and structs
• Unit test C layer in isolation with simple Rack app

Phase 2 — RackStream Writer (Week 4–5)

• Implement stream_ctx_t struct in iodine_rack_stream.h
• Implement rack_stream_write — all 6 O(1) checks + return semantics
• Implement rack_stream_close — idempotent, safe in blocked state
• Implement rack_stream_closed?
• Implement on_stream_drain — LOW_WATERMARK check + IodineCaller resume
• Implement on_stream_close — disconnect safety, always wake fiber
• Register dual callbacks on backpressure trigger

Phase 3 — Iodine Integration (Week 6–7)

• Extend iodine_http.c to detect call(stream) body type
• Add new streaming branch to Rack body detection
• Ensure existing String and each paths completely unchanged
• Verify upgrade flows (rack.upgrade, WebSocket) unaffected
• Core integration test: prove body.call(stream) crosses the C-to-Ruby bridge and sends the first chunk before full response
  completion
• Manual end-to-end streaming validation at Iodine level

Phase 4 — Rage API (Week 8–9)

• Implement Rage::Streaming enumerator wrapper in lib/rage/streaming.rb
• Add render stream: support to lib/rage/controller/api.rb
• Hook into existing Fiber.await in lib/rage/fiber.rb
• Integrate with fiber_scheduler.rb pause/unblock mechanics
• Update lib/rage/rack_adapter.rb to detect streaming response
• Backpressure integration test: fast producer triggers :would_block, fiber yields, on_drain resumes, stream continues
• Manual end-to-end test across both repos

Phase 5 — Backpressure & Edge Cases (Week 10–11)

• Implement HIGH/LOW/HARD_MAX three-threshold system
• Implement HARD_MAX Fail Stream overload action
• Handle all 35 behavior matrix edge cases
• HEAD request body suppression
• 204/304 status body suppression
• Header mutation rejection after first write
• Large chunk safe splitting
• GC/finalizer safety — release socket, cancel callbacks
• Double close idempotency

Phase 6 — Testing & Documentation (Week 12)

• Integration test: producer flood
• Integration test: concurrent requests remain responsive
• Integration test: disconnect mid-stream cleanup
• Integration test: backpressure resume correctness
• Complete remaining integration coverage across both repositories
• Update iodine/README.md with streaming API guide
• Update rage/README.md with render stream: developer guide
• Final review with mentors
• Submit final evaluation

---

9. Related Work

Existing Streaming in Iodine

Iodine supports persistent connections via rack.upgrade for WebSocket. This project adds streaming to the normal HTTP response path without requiring protocol upgrades — a fundamentally different and complementary capability.

Rails ActionController::Live

Rails implements streaming via ActionController::Live with response.stream.write. This project takes inspiration from that pattern but adapts it to Rage's fiber model — using a declarative render stream: enumerator API that is more idiomatic and requires no explicit ensure stream.close.

Node.js Streams

Node.js pioneered the highWaterMark / backpressure vocabulary. This project uses the same producer-consumer pattern with hysteresis (HIGH/LOW watermarks), adapted for Ruby's fiber model and Iodine's event loop.

Rack Streaming Spec

The Rack spec defines body.call(stream) as the interface for streaming bodies. This project implements full Rack streaming compliance in Iodine for the first time.

HTTP/2 Compatibility

The API is designed to be protocol-agnostic. The Rage::Streaming interface abstracts the transport layer entirely — future HTTP/2 support requires only a new Iodine backend, not any app-facing API changes.

SSE Extensibility

Although SSE is explicitly out of scope for this project, the architecture is designed so SSE becomes a natural and minimal extension.

SSE is HTTP streaming with a different Content-Type (text/event-stream) and a specific chunk format (data: <payload>\n\n). Everything this project builds — stream_ctx_t, backpressure system, state machine, on_drain/on_close callbacks, fiber scheduler integration — is completely reusable for SSE without any changes.

Adding SSE in a future project would require only:

Rage:   lib/rage/sse.rb      → SSE chunk formatter  (~30 lines)
        api.rb               → render sse: key       (~5 lines)

Iodine: http_sse_stream()    → SSE headers instead
                               of chunked encoding   (~20 lines)

The transport contract, backpressure logic, state machine, and fiber integration stay completely untouched. This project builds the foundation — SSE is the natural next step.

---

10. Previous Contributions

• Actively studied the Rage and Iodine codebases — read iodine_http.c, http1.c, api.rb, fiber.rb, and UPGRADE_FLOW.md to understand the existing architecture before proposing any changes
• Opened rage-rb/rage#221 as a first contribution to the codebase
• Engaged with @rsamoilov on Discord across multiple technical discussions — working through the architecture, iterating on the design, and understanding the implementation constraints of the Iodine event loop before writing this proposal

---

11. About You

My name is Piyush Goenka. I am a second-year B.Sc. Computer Science student at BITS Pilani.

I was drawn to this project because it sits at the intersection of systems programming and real-world application — the same producer-consumer backpressure patterns that power LLM streaming, live data feeds, and incremental content delivery. Implementing this at the server level, in C and Ruby, is exactly the kind of deep systems work I want to do.

Open Source

I am an active open source contributor. I contributed extensively to the Palisadoes Foundation — the organization behind the Talawa project, a modular open source platform used by community-based organizations worldwide. My contributions were consistent and substantial enough that I was made a maintainer of the organization. This means I have real experience navigating large unfamiliar codebases, following contributor conventions, writing reviewable code, and taking responsibility for the quality of a shared project.

Achievements

I won Smart India Hackathon (SIH), India's largest national-level hackathon, which required building and shipping a working technical solution under a tight deadline — a skill directly relevant to a GSoC project with fixed weekly milestones.

I am comfortable contributing to codebases I did not write, communicating clearly with maintainers, and learning quickly from feedback. I am genuinely curious about low-level systems and enjoy going deep on problems rather than staying at the surface.

---

12. Availability

• Weekly availability: 35–40 hours per week
• Full-time availability during the coding period
• Timezone: IST (UTC+5:30) — I will adjust my working hours to overlap with mentor availability and ensure regular synchronous communication during the coding period
• No major exams or internship conflicts during May–August 2026
• Will communicate any minor scheduling issues proactively

---

13. Communication Plan

• Weekly progress updates posted to GitHub Discussions or Discord
• Daily async updates on Discord during active implementation weeks
• GitHub PRs for all code — small, reviewable, well-described commits
• Proactive communication — will flag blockers immediately rather than waiting for weekly check-ins
• Available on Discord during maintainer-friendly hours

---

14. Additional Notes

Why This Project Is Hard (And Why I Am Ready For It)

This project is rated Hard — not because the API shape is complex, but because of deep implementation challenges:

1. http_stream does not exist — must be implemented from scratch in C
2. GVL safety — rb_fiber_resume requires GVL acquisition via IodineCaller, not a direct call
3. Disconnect safety — dual on_drain + on_close callbacks required so fiber never sleeps forever
4. Backpressure correctness — O(1) write, no busy-wait, hysteresis between HIGH and LOW
5. Cooperative concurrency — Iodine C must never call enumerator.next directly — Ruby blocking operations inside enumerators would freeze the entire event loop

All five of these challenges were identified and have concrete solutions documented in this proposal before submission.

Protocol Agnostic Design

Although the initial implementation targets HTTP/1.1, every design decision keeps HTTP/2 in mind:

• Rage::Streaming abstracts the transport layer
• Developer API (render stream: enumerator) is transport-agnostic
• Future HTTP/2 support requires only a new Iodine backend
• No app-facing API changes needed for HTTP/2

Cooperative Concurrency Model

The three-layer model that makes this work:

Rage    → high level: iterates enumerator inside Fiber
Iodine  → low level: provides stream object, signals backpressure
Fiber.yield → the glue: pauses Ruby, hands control to event loop

This is why Rage's fiber architecture makes it uniquely suited for this feature — and why this project is the right time to implement it.

Beyond GSoC

GSoC is not the end goal for me — it is a starting point. I am genuinely interested in Rage as a project and in the broader problem of making Ruby web infrastructure faster and more capable. The mentorship and the deep dive into Iodine's internals during this project will leave me in a strong position to keep contributing meaningfully.

After the summer I intend to continue — whether that means working on follow-up features like SSE or HTTP/2 streaming, reviewing PRs, helping new contributors navigate the codebase, or tackling other open issues. I want to give back to the community that I will have learned from, not just complete a project and move on.

[rsamoilov]

Hi @Piyush-Goenka

This is an exceptionally strong and mature proposal as is.

Several minor comments:

• I would recommend adding a brief note on the memory management lifecycle for stream_ctx_t. Does Iodine's connection attachment system handle freeing this automatically on disconnect, or will you need custom teardown logic?
• Ensure your final implementation draft explicitly shows how the HARD_MAX check fits into the rack_stream_write flow, as the current snippet only demonstrates the high_watermark check.
• Consider moving at least one core integration test up to Phase 3 or 4. Proving the C-to-Ruby bridge works early will severely de-risk the final weeks of your project.

[Piyush-Goenka]

Hi @rsamoilov
Thank you for the detailed feedback. I have updated the proposal with all three changes:

1. Added a stream_ctx_t memory lifecycle section covering allocation, ownership, GC safety, disconnect cleanup, and double-free protection. Should I also add the required code snippets about how is this managed ?

2. Added the HARD_MAX check explicitly to both the rack_stream_write snippet and the stream.write decision flow
3. Moved integration tests earlier — C-to-Ruby bridge test in Phase 3, backpressure resume test in Phase 4

Is there anything else that is required to be changed in the proposal apart from this ?

[rsamoilov]

Hi @Piyush-Goenka , this looks great!

[Piyush-Goenka]

@rsamoilov
Thanks a lot for your help and guidance.