io_uring Support

[qbacpey]

TODO

API

• io_uring_prep_xxx:
• io_uring_submit:
• io_uring_submit_and_wait:
• io_uring_wait_cqe:
• io_uring_peek_cqe:

Data Structure

• unordered_map<struct io_uring *, struct io_uring_sqe *> (requests_to_submit): Track the requests to be submitted. Key is the io_uring obj this request belongs to, value is the struct io_uring_sqe * of the requests that used as arguments of io_uring_sqe_set_data.
• unordered_map<struct io_uring *, unordered_map<__u64, Enum::RequestQueue>> (ongoing_requests): Track the submitted requests.
  • When liburing waiting function (e.g. io_uring_wait_cqe) is called, TMIO would mark all the prending requests for this struct io_uring as "required finsihed". Also would mark the request pointed by struct io_uring_cqe **cqe_ptr as finsihed (might use the io_uring_cqe_get_data for doing this in a clever ways).
  • When liburing checking function (e.g. io_uring_cqe_get_data) is called, TMIO would mark the related request as "actually finished".

Helper Thread

Because the reap API all inplement with static inline, not able to intercept them with LD_PRELOAD, consider use a helper thread to do this.

Can have a std::async helper thread to maintain the data strucutre of TMIO, the best interveal can be approach by check how many io_uring have their complemntion request not checking.

Testing

• Using all liburing/test for testing.

[qbacpey]

Design Doucement

By using io_uring, many unnecessary syscalls would be elimiated, so it would be hard to intercept some IO request (e.g. SQPOLL allow we directly put the SQE in the kernel-user shared MEM). But if we choose to intercept io_uring, all the submitting and reaping of IO requests would need to go through APIs provided by liburing, so we can still intercept IO requests by intercept those APIs.

When using liburing, you never directly call the io_uring_enter() system call. That is usually taken care of by liburing’s io_uring_submit() function. It automatically determines if you are using polling mode or not and deals with when your program needs to call io_uring_enter() without you having to bother about it.

Normal Submit Process

Accoring to the Lord of the io_uring normal process of using liburing would be:

/* Get an SQE */
struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
/* Setup a readv operation */
io_uring_prep_readv(sqe, file_fd, fi->iovecs, blocks, 0);
/* Set user data */
io_uring_sqe_set_data(sqe, fi);
/* Finally, submit the request */
io_uring_submit(ring);

Given that one program might have multiple io_uring instance, we would use a std::vector<RingTraceState> tracked_rings_; to manage the relationship of requests to differnet io_uring instance, in case we mistakenly marks some requests as requested or finished.

When some requests are submited, we need to intercept io_uring_submit, iterate all the sqe* of the ring before the io_uring internel tail, check their opcode and related arguments, initlize related request in the TMIO data structure.

For the sake of performance, the main thread is the only thread that would check the io_uring submittion queue. After it finished checking, it would try to acquire the ring_processing_lock_ for handing requests:

1. Success: The main thread would become the "worker"

• Scans the io_uring submission queue for new SQE, parse it and add the data to appropriate set and call the Write/Read_Async_Start.
• Then reap all the requests in the completion queue.
• At last it would trigger the condition variable for waking up the polling thread.

2. Fail: The main thread would only put the requests in the lock-free staged_requests_queue_ and waiting the polling threads to process them.

While for the polling thread, it would regularily check the lock-free staged_requests_queue_, find if there are new requests arrived and put them into the std::vector<RingTraceState> tracked_rings_;. The other things is just like the "worker". But the difference is just that since there might be mutliple io_uring instances, the polling thread need to iterate all the current RingTraceState. And if there aren't any activate pending request, the polling thread would sleep for saving resources.

Also we would use the __u64 user_data; as request marker.

Normal Complete Process

Accoring to Lord of the io_uring there exist two kinds completion API:

• io_uring_wait_cqe:
  • Blocks until at least one completion is available.
  • Cannot specifiy to wait for a particular request with user_data
• io_uring_peek_cqe:
  • Returns an I/O completion, if one is readily available. Doesn’t wait.

Remember:

If the application merely wishes to peek at the completion and not wait for an event to become available, io_uring_peek_cqe(3)does that. For both use cases, the application must call io_uring_cqe_seen(3)once it is done with this completion event. Repeated calls to io_uring_peek_cqe(3)or io_uring_wait_cqe(3)will otherwise keep returning the same event.

Given above informaiton, we should consider intecept the API as follow:

• io_uring_wait_cqe: Mark all the requests realteds to the ring as "Requested Complete", after the function return, also mark the related request as "Actually Completed".
• io_uring_cqe_get_data: Might also intercept this function for mark the "Actual Finsihed".
• io_uring_peek_cqe: Mark all the requests realteds to the ring as "Requested Complete".

Fixed Buffer

The code after integrated Fixed buffers is basicly the same as normal process, the only differnet would be:

• First use io_uring_register_buffers() to register the reusable buffer that would be shared between user space and kernel space. So that it can elimiate the copy process from kernel to user space (Zero Copy of Linux).
• When need to preapre IO, need to use the special fixed API io_uring_prep_write_fixed() and io_uring_prep_read_fixed().

So, we only need to intercept two more API would be fine, doesn't need anythins special.

Polling

io_uring support three kinds of polling:

• IORING_SETUP_IOPOLL
  • Only support file with O_DIRECT
  • Need hardward support
  • User need to call io_uring_enter to explicit notice io_uring to put the CQE into completion queue.
• IORING_SETUP_HYBRID_IOPOLL
  • Basically same as IORING_SETUP_IOPOLL
  • Would be delay a bit before doing completion side polling.
• IORING_SETUP_SQPOLL
  • A kernel thread would be created to reap SQEs from the submit queue automatically.
  • With this flag, the application can even submit and reap I/Os without doing a single system call.

However liburing only explicitly mention the support about IORING_SETUP_SQPOLL, so we would only considering it in TMIO.

Anyway, the process also same as Normal Process, liburing make it transparent.