tokio-runtime-worker panic at startup with ptmgr code

When I run with !903 (merged) and !902 (merged) merged, I get this panic at startup:

thread 'tokio-runtime-worker' panicked at 'assertion failed: !ret.inner.is_empty()', /home/nickm/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/select_all.rs:40:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:142:14
   2: core::panicking::panic
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:48:5
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   5: tokio::runtime::task::raw::poll
   6: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
   7: tokio::runtime::task::raw::poll
   8: tokio::runtime::task::UnownedTask<S>::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

cc @eta