Make Rc<T>::deref and Arc<T>::deref zero-cost

[EFanZh]

Currently, Rc<T> and Arc<T> store pointers to RcInner<T> and ArcInner<T>. This PR changes the pointers so that they point to T directly instead.

This is based on the assumption that we access the T value more frequently than accessing reference counts. With this change, accessing the data can be done without offsetting pointers from RcInner<T> and ArcInner<T> to their contained data. This change might also enables some possibly useful future optimizations, such as:

• Convert &[Rc<T>] into &[&T] within O(1) time.
• Convert &[Rc<T>] into Vec<&T> utilizing memcpy.
• Convert &Option<Rc<T>> into Option<&T> without branching.
• Make Rc<T> and Arc<T> FFI compatible types where T: Sized.

[rustbot]

r? @jhpratt

rustbot has assigned @jhpratt.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[marmeladema]

Would it potentially enable those types to have an ffi compatible ABI? So that they could be returned and passed directly from /to ffi function, like Box?

[EFanZh]

Would it potentially enable those types to have an ffi compatible ABI? So that they could be returned and passed directly from /to ffi function, like Box?

I think in theory it is possible, at least for sized types, but I am not familiar with how to formally make it so.

[jhpratt]

r? libs

[joboet]

@EFanZh Is this ready for review? If so, please un-draft the PR.

[EFanZh]

@joboet: The source code part is mostly done, but I haven’t finished updating LLDB and CDB pretty printers. The CI doesn’t seem to run those tests.

[joboet]

No worries! I just didn't want to keep you waiting in case you had forgotten to change the state.
@rustbot author

[EFanZh]

I have rewritten this PR as a proof of concept for the idea I mentioned in #132553 (comment). Specifically, I added an alloc::raw_rc module for storing common logic between Rc and Arc. In this way, it could be easier to add new features, optimizations and bugfixes for both Rc and Arc. I also adopted @scottmcm’s method of storing reference counts adjacent to value field. Here is a brief description of some important types I added:

/// Shared reference counts type for both `Rc` and `Arc`.
/// The `UnsafeCell<usize>` type is used as `Cell<usize>` for `Rc`
/// and `AtomicUsize` for `Arc`.
pub struct RefCounts {
    pub weak: UnsafeCell<usize>,
    pub strong: UnsafeCell<usize>,
}

/// Base implementation for `rc::Weak` and `sync::Weak`.
pub struct RawWeak<T, A>
where
    T: ?Sized,
{
    /// Dangling or points to the value field of an rc allocation.
    ptr: NonNull<T>,
    alloc: A,
}

/// Base implementation for `rc::Rc` and `sync::Arc`.
#[repr(transparent)]
pub struct RawRc<T, A>
where
    T: ?Sized,
{
    /// A `RawRc` is just a `RawWeak` with a strong reference owned by itself.
    weak: RawWeak<T, A>,
    _phantom_data: PhantomData<T>,
}

/// Base implementation for `rc::UniqueRc` and possible future `sync::UniqueArc`.
#[repr(transparent)]
pub struct RawUniqueRc<T, A>
where
    T: ?Sized,
{
    /// A `RawUniqueRc` is just a `RawWeak` with an initialized value but has a
    /// strong reference count of zero.
    weak: RawWeak<T, A>,
}

With these types above, the standard library types are implemented as:

mod rc {
    #[repr(transparent)]
    pub struct Weak<T: ?Sized, A: Allocator = Global> {
        raw_weak: RawWeak<T, A>,
    }

    #[repr(transparent)]
    pub struct Rc<T: ?Sized, A: Allocator = Global> {
        raw_rc: RawRc<T, A>,
    }

    #[repr(transparent)]
    pub struct UniqueRc<T: ?Sized, A: Allocator = Global> {
        raw_unique_rc: RawUniqueRc<T, A>,
    }
}

mod sync {
    #[repr(transparent)]
    pub struct Weak<T: ?Sized, A: Allocator = Global> {
        raw_weak: RawWeak<T, A>,
    }

    #[repr(transparent)]
    pub struct Arc<T: ?Sized, A: Allocator = Global> {
        raw_rc: RawRc<T, A>,
    }
}

The different behaviors of rc and sync types are represented with an RcOps trait:

pub trait RcOps {
    /// Used by `RawRc::clone` and `RawWeak::clone`.
    unsafe fn increment_ref_count(count: &UnsafeCell<usize>);

    /// Used by `RawRc::drop` and `RawWeak::drop`.
    unsafe fn decrement_ref_count(count: &UnsafeCell<usize>) -> bool;

    /// Used by `RawWeak::upgrade`.
    unsafe fn upgrade(strong_count: &UnsafeCell<usize>) -> bool;

    /// Used by `RawRc::downgrade`.
    unsafe fn downgrade(weak_count: &UnsafeCell<usize>);

    /// Used by `RawRc::try_unwrap`.
    unsafe fn lock_strong_count(strong_count: &UnsafeCell<usize>) -> bool;

    /// Used by `RawUniqueRc::into_rc`.
    unsafe fn unlock_strong_count(strong_count: &UnsafeCell<usize>);

    /// Used by `RawRc::get_mut`.
    unsafe fn is_unique(strong_count: &UnsafeCell<usize>, weak_count: &UnsafeCell<usize>) -> bool;

    /// Used by `RawRc::make_mut`.
    #[cfg(not(no_global_oom_handling))]
    unsafe fn make_unique<T, A, F, G>(rc: &mut RawRc<T, A>, by_clone: F, by_move: G)
    where
        T: ?Sized,
        F: FnOnce(&mut RawRc<T, A>),
        G: FnOnce(&mut RawRc<T, A>);
}

Both rc and sync modules have their own RcOps implementation types. Their implementation types are passed to raw_rc module as generic arguments. For example:

mod raw_rc {
  impl<T, A> RawRc<T, A> where T: ?Sized {
      pub unsafe fn clone<R>(&self) -> Self where A: Clone, R: RcOps {
          unsafe {
              R::increment_ref_count(self.strong_count());
  
              Self::from_weak(self.weak.clone_without_increment_weak_count())
          }
      }
  }
}

mod rc {
    enum RcOps {}

    /// Specifies behaviors of `rc::Rc`, `rc::Weak` and `rc::UniqueRc`.
    impl raw_rc::RcOps for RcOps { ... }

    impl<T: ?Sized, A: Allocator + Clone> Clone for Rc<T, A> {
        fn clone(&self) -> Self {
            Self { raw_rc: unsafe { self.raw_rc.clone::<RcOps>() } }
        }
    }
}

mod sync {
    enum RcOps {}

    /// Specifies behaviors of `sync::Arc` and `sync::Weak`.
    impl raw_rc::RcOps for RcOps { ... }

    impl<T: ?Sized, A: Allocator + Clone> Clone for Arc<T, A> {
        fn clone(&self) -> Self {
            Self { raw_rc: unsafe { self.raw_rc.clone::<RcOps>() } }
        }
    }
}

This PR is not a completed work:

• Documentations are missing.
• Binary size and compilation time haven’t been optimized.
• #[inline] and #[track_caller] attributes haven’t been considered.
• Names of types, traits and functions are somewhat arbitrarily decided, better names may needed to be used.
• Hidden bugs may exist.
• Commit history needs to be reorganized for merging.

Now, I think this PR is in a good shape to decide whether it is something the standard library wants. @joboet, @scottmcm: what do you think? Is it worth completing this PR?

[safinaskar]

I think in theory it is possible, at least for sized types, but I am not familiar with how to formally make it so.

If you really want to do this, then you should place allocator in allocated block, not in Rc itself. And then you can add #[repr(transparent)] to RawWeak. And then Rc will became totally ABI-compatible with raw pointer (see here for more details https://doc.rust-lang.org/nightly/std/primitive.fn.html#abi-compatibility ). Also, this will mean that you don't need to clone allocator when you do Rc::clone()

[safinaskar]

Ping @scottmcm (see my prev. comment). Also, here allocators for Rc was originally introduced: #89132 and here discussed: https://rust-lang.zulipchat.com/#narrow/channel/197181-t-libs.2Fwg-allocators/topic/Perf.20for.20.2389132.20.28Arc.2FRc.20allocator_api.29

[EFanZh]

If you really want to do this, then you should place allocator in allocated block, not in Rc itself.

@safinaskar: Personally, I am fine with the idea, but I think it’s better to implement it in a separate PR, since it will introduce API changes (For example, Arc::<T,A>::into_raw_with_allocator may additionally require A being Clone).

As for #[repr(transparent)] support, it can also be done later with some unknown method (maybe compiler magic, some new attribute like #[repr(transparent_if_possible)], or moving the allocator into the allocated block like you suggested).