Can I co_await an operation executed by one io_context in a coroutine executed by another in Asio? - c++

Question

I have an HTTP server library writen by Asio with one io_context per thread model. It has an io_context_pool which retrieves one io_context orderly. So when the server starts, io_context #1 will be used to execute the acceptor and io_context #2 will be used for the first connection and io_context #3 will be with the second and so on.
It also wait for signal to call the io_context.stop() orderly to stop the server and it works well.
I would like to refactor it with C++20 coroutine. I've almost finished but I found that if signal reveived to call io_context.stop(), heap used after free will be reported by sanitizer. I've no idea how to solve this.
Here is a simplified example:
constexpr int wait_second = 2;
constexpr int run_second = 3;
asio::awaitable<void> co_main(asio::io_context& other_ctx){
asio::steady_timer timer(other_ctx);
timer.expires_after(std::chrono::seconds(wait_second));
co_await timer.async_wait(asio::use_awaitable);
std::cout<<"timer"<<std::endl;
}
int main() {
asio::io_context ctx;
asio::io_context other_ctx;
asio::executor_work_guard<asio::io_context::executor_type> g1(ctx.get_executor());
asio::executor_work_guard<asio::io_context::executor_type> g2(other_ctx.get_executor());
asio::co_spawn(ctx, co_main(other_ctx), asio::detached);
std::thread([&]{
std::this_thread::sleep_for(std::chrono::seconds(run_second));
ctx.stop();
other_ctx.stop();
}).detach();
std::thread t1([&]{ctx.run();});
std::thread t2([&]{other_ctx.run();});
t1.join();
t2.join();
return 0;
}
The coroutine co_main is executed by ctx but the timer in it is executed by other_ctx. The wait_second represents the waiting duration of the timer while run_second represent after how many seconds the ctx and io_context will be stopped.
If the wait_second is less than run_second, everything works fine but if wait_second is longer, heap used after free will be reported by sanitizer. So is this the right way to use asio::awaitable? Can I mix use the io_context with coroutine?

Answer 1

Yes this usage is fine. Not optimal, but fine.
The thing to note about binding IO objects to an executor is that it mostly serves two purposes:
firstly it indicates which execution context holds the service instance for the IO object's implementation
secondly it serves as the default(!) executor for completion handlers
Note that 1. implies that it may incur more overhead than sharing the context, and 2. means that there is little behavioral difference in this case. It's the default only, so the awaitable's coro executor overrides it here. This may not be as you expected. See e.g. this similar question Boost asio steady_timer work on different strand than intended
What's the sanitizer issue
It's a lifetime issue. Let's instrument a bit and use BOOST_ASIO_ENABLE_HANDLER_TRACKING as well:
#include <boost/asio.hpp>
#include <iostream>
using namespace std::chrono_literals;
namespace asio = boost::asio;
constexpr auto wait_for = 200ms, run_for = 300ms;
asio::awaitable<void> co_main(asio::io_context& other_ctx){
asio::steady_timer timer{other_ctx.get_executor(), wait_for};
co_await timer.async_wait(asio::use_awaitable);
std::cout << "timer" << std::endl;
}
int main() {
{
asio::io_context other_ctx;
{
asio::io_context ctx;
//auto g1 = ctx.get_executor();
//auto g2 = other_ctx.get_executor();
asio::co_spawn(ctx, co_main(other_ctx), asio::detached);
std::thread stopper([&] {
std::this_thread::sleep_for(run_for);
ctx.stop();
other_ctx.stop();
});
std::thread t1([&] { ctx.run(); });
std::thread t2([&] { other_ctx.run(); });
stopper.join();
t1.join();
t2.join();
std::cout << "All joined" << std::endl;
std::cout << "ctx: " << &ctx << std::endl;
std::cout << "other_ctx: " << &other_ctx << std::endl;
}
std::cout << "ctx destructed" << std::endl;
}
std::cout << "other_ctx destructed" << std::endl;
}
After all threads are done (and joined), the destructor of other_ctx abandons all operations in the operation queue. Some of the completions are bound to the executor of ctx (because of the coro, see above) which is already gone. This leads to the use-after-scope (truncating for SO):
#asio|1661689229.839262|0^1|in 'co_spawn_entry_point' (/home/sehe/custom/superboost/boost/asio/impl/co_spawn.hpp:154)
#asio|1661689229.839262|0*1|io_context#0x7ffe27cf56a0.execute
#asio|1661689229.839715|>1|
#asio|1661689229.839961|1*2|deadline_timer#0x6110000000a0.async_wait
#asio|1661689229.839986|<1|
All joined
ctx: 0x7ffe27cf56a0
other_ctx: 0x7ffe27cf5680
ctx destructed
#asio|1661689230.139986|2*3|io_context#0x7ffe27cf56a0.execute
=================================================================
==16129==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffe27cf56a8 at pc 0x55bace9962f6 bp 0x7ffe27cf4270 sp 0x7ffe27cf4260
READ of size 8 at 0x7ffe27cf56a8 thread T0
#0 0x55bace9962f5 in void boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>::....hpp:319
#1 0x55bace968067 in void boost::asio::execution::detail::any_executor_base::execute<boost::asio::d....hpp:611
#2 0x55bace968067 in std::enable_if<boost_asio_execution_execute_fn::call_traits<boost_asio_executi....hpp:208
#3 0x55bace968067 in void boost::asio::detail::initiate_post_with_executor<boost::asio::any_io_exec....hpp:122
#4 0x55bace968067 in void boost::asio::detail::completion_handler_async_result<boost::asio::detail:....hpp:482
#5 0x55bace968067 in boost::asio::constraint<boost::asio::detail::async_result_has_initiate_memfn<b....hpp:896
#6 0x55bace968067 in auto boost::asio::post<boost::asio::any_io_executor, boost::asio::detail::awai....hpp:242
#7 0x55bace969443 in boost::asio::detail::awaitable_thread<boost::asio::any_io_executor>::~awaitabl....hpp:673
#8 0x55bace99ecb0 in boost::asio::detail::awaitable_handler_base<boost::asio::any_io_executor, void....hpp:29
#9 0x55bace99ecb0 in boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::sy....hpp:78
#10 0x55bace99ecb0 in boost::asio::detail::binder1<boost::asio::detail::awaitable_handler<boost::as....hpp:139
#11 0x55bace99ecb0 in boost::asio::detail::wait_handler<boost::asio::detail::awaitable_handler<boos....hpp:67
#12 0x55bace91be12 in boost::asio::detail::scheduler_operation::destroy() /home/sehe/custom/superbo....hpp:45
#13 0x55bace91be12 in void boost::asio::detail::op_queue_access::destroy<boost::asio::detail::sched....hpp:47
#14 0x55bace91be12 in boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>::~op_....hpp:81
#15 0x55bace91be12 in boost::asio::detail::scheduler::abandon_operations(boost::asio::detail::op_qu....ipp:444
#16 0x55bace91be12 in boost::asio::detail::epoll_reactor::shutdown() /home/sehe/custom/superboost/b....ipp:92
#17 0x55bace8ebc8c in boost::asio::detail::service_registry::shutdown_services() /home/sehe/custom/....ipp:44
#18 0x55bace8ebc8c in boost::asio::execution_context::shutdown() /home/sehe/custom/superboost/boost....ipp:41
#19 0x55bace8ebc8c in boost::asio::execution_context::~execution_context() /home/sehe/custom/superb....ipp:34
#20 0x55bace88d684 in boost::asio::io_context::~io_context() /home/sehe/custom/superboost/boost/asi....ipp:56
#21 0x55bace88d684 in main /home/sehe/Projects/stackoverflow/test.cpp:16
#22 0x7efe676d0c86 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21c86)
#23 0x55bace88e859 in _start (/home/sehe/Projects/stackoverflow/build/sotest+0x1a5859)
Address 0x7ffe27cf56a8 is located in stack of thread T0 at offset 904 in frame
#0 0x55bace88bd2f in main /home/sehe/Projects/stackoverflow/test.cpp:14
This frame has 34 object(s):
// ... skipped
[128, 136) '<unknown>'
[160, 168) 'stopper' (line 24)
[192, 200) 't1' (line 30)
[224, 232) '<unknown>'
[256, 264) 't2' (line 31)
[288, 296) '<unknown>'
// ... skipped
[832, 840) '<unknown>'
[864, 880) 'other_ctx' (line 16)
[896, 912) 'ctx' (line 18) <== Memory access at offset 904 is inside this variable
[928, 944) '<unknown>'
[960, 1016) '<unknown>'
[1056, 1112) '<unknown>'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-scope /home/sehe/custom/superboost/boost/asio/impl/io_context.hpp:319 in void boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>::execute<boost::asio::detail::executor_function>(boost::asio::detail::executor_function&&) const
Shadow bytes around the buggy address:
0x100044f96a80: f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2
0x100044f96a90: f8 f2 f2 f2 f8 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
0x100044f96aa0: 00 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2
0x100044f96ab0: f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2
0x100044f96ac0: f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f2 f2 f2
=>0x100044f96ad0: 00 00 f2 f2 f8[f8]f2 f2 f8 f8 f2 f2 f8 f8 f8 f8
0x100044f96ae0: f8 f8 f8 f2 f2 f2 f2 f2 f8 f8 f8 f8 f8 f8 f8 f3
0x100044f96af0: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
0x100044f96b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100044f96b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100044f96b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==16129==ABORTING
Ironically, flipping the order of executor destruction doesn't help:
{
asio::io_context ctx;
{
asio::io_context other_ctx;
// ... skiped unchanged
}
std::cout << "other_ctx destructed" << std::endl;
}
std::cout << "ctx destructed" << std::endl;
Now we get the inverse dependency: after all threads have joined, the destructor of other_ctx correctly queues completion for the async_wait on ctx. But once ctx destructor shuts down its services, it will destroy the coro stack frame, which includes the timer object which ... has a stale reference to the other_ctx destructor.
#asio|1661689960.225134|0^1|in 'co_spawn_entry_point' (/home/sehe/custom/superboost/boost/asio/impl/co_spawn.hpp:154)
#asio|1661689960.225134|0*1|io_context#0x7ffd9e839d80.execute
#asio|1661689960.225576|>1|
#asio|1661689960.225876|1*2|deadline_timer#0x6110000000a0.async_wait
#asio|1661689960.225899|<1|
All joined
ctx: 0x7ffd9e839d80
other_ctx: 0x7ffd9e839da0
#asio|1661689960.525759|2*3|io_context#0x7ffd9e839d80.execute
#asio|1661689960.525773|~2|
other_ctx destructed
/usr/include/c++/10/coroutine:128: runtime error: member call on null pointer of type 'struct steady_timer'
/home/sehe/custom/superboost/boost/asio/basic_waitable_timer.hpp:382:3: runtime error: member access within null pointer of type 'struct basic_waitable_timer'
/home/sehe/custom/superboost/boost/asio/basic_waitable_timer.hpp:382:3: runtime error: member call on null pointer of type 'struct io_object_impl'
/home/sehe/custom/superboost/boost/asio/detail/io_object_impl.hpp:97:5: runtime error: member access within null pointer of type 'struct io_object_impl'
AddressSanitizer:DEADLYSIGNAL
=================================================================
==22649==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x558135f6b18f bp 0x7ffd9e837d10 sp 0x7ffd9e837a00 T0)
==22649==The signal is caused by a READ memory access.
==22649==Hint: address points to the zero page.
#0 0x558135f6b18f in boost::asio::detail::io_object_impl<boost::asio::detail::deadline_timer_service<bo....hpp:97
#1 0x558135e6d07a in boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wai....hpp:382
#2 0x558135e6d07a in co_main(boost::asio::io_context&) [clone .actor] /home/sehe/custom/superboost/boos....hpp:380
#3 0x558135f0b5e9 in std::__n4861::coroutine_handle<void>::destroy() const /usr/include/c++/10/coroutine:128
#4 0x558135f0b5e9 in boost::asio::detail::awaitable_frame_base<boost::asio::any_io_executor>::destroy()....hpp:496
#5 0x558135f0b5e9 in boost::asio::awaitable<void, boost::asio::any_io_executor>::~awaitable() /home/seh....hpp:77
#6 0x558135f0b5e9 in boost::asio::awaitable<boost::asio::detail::awaitable_thread_entry_point, boost::a....actor] /home/sehe/custom/superboost/boost/asio/impl/co_spawn.hpp:183
#7 0x558135ea6d1d in std::__n4861::coroutine_handle<void>::destroy() const /usr/include/c++/10/coroutine:128
#8 0x558135ea6d1d in boost::asio::detail::awaitable_frame_base<boost::asio::any_io_executor>::destroy()....hpp:496
#9 0x558135ea6d1d in boost::asio::awaitable<boost::asio::detail::awaitable_thread_entry_point, boost::a....hpp:77
#10 0x558135ea6d1d in boost::asio::detail::awaitable_thread<boost::asio::any_io_executor>::~awaitable_t....hpp:674
#11 0x558135ea6d1d in boost::asio::detail::binder0<boost::asio::detail::awaitable_thread<boost::asio::a....hpp:32
#12 0x558135ea6d1d in void boost::asio::detail::executor_function::complete<boost::asio::detail::binder....hpp:110
#13 0x558135f6ecfb in boost::asio::detail::executor_function::~executor_function() /home/sehe/custom/su....hpp:55
#14 0x558135f6ecfb in boost::asio::detail::executor_op<boost::asio::detail::executor_function, std::all....hpp:62
#15 0x558135ec4893 in boost::asio::detail::scheduler_operation::destroy() /home/sehe/custom/superboost/....hpp:45
#16 0x558135ec4893 in boost::asio::detail::scheduler::shutdown() /home/sehe/custom/superboost/boost/asi....ipp:176
#17 0x558135ebfc8c in boost::asio::detail::service_registry::shutdown_services() /home/sehe/custom/supe....ipp:44
#18 0x558135ebfc8c in boost::asio::execution_context::shutdown() /home/sehe/custom/superboost/boost/asi....ipp:41
#19 0x558135ebfc8c in boost::asio::execution_context::~execution_context() /home/sehe/custom/superboost....ipp:34
#20 0x558135e61684 in boost::asio::io_context::~io_context() /home/sehe/custom/superboost/boost/asio/im....ipp:56
#21 0x558135e61684 in main /home/sehe/Projects/stackoverflow/test.cpp:16
#22 0x7fbd0bd07c86 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21c86)
#23 0x558135e62859 in _start (/home/sehe/Projects/stackoverflow/build/sotest+0x1a5859)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/sehe/custom/superboost/boost/asio/detail/io_object_impl.hpp:97 in boost::asio::detail::io_object_impl<boost::asio::detail::deadline_timer_service<boost::asio::detail::chrono_time_traits<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > >, boost::asio::any_io_executor>::~io_object_impl()
==22649==ABORTING
The problem appears to be that the coro frame outlives the lifetime of other_ctx here.
Conclusion
You have to avoid destruction of both contexts from inter-depending cyclically. This practically means avoiding uncompleted operations in their operation queues.
It's good to note that the problems mostly start with using io_context::stop() to stop context mid-operation. In addition, there doesn't currently seem to be a reason run() the other_ctx in the first place.
Consider using async completion / awaitable operators with cancellation.