Skip to content

RWKV/RWKV-block

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

[WIP] RWKV: Receptive Key Weight Value

WIP: This is meant to be reference block implmentaiton for various RWKV modules. It is not considered complete

Pytorch design decisions

  • RWKV states are passed around with native tuples, and list intentionally, benchmarking show this has a measurable speed bump compared to data state classes when needed.
  • init state tune weights should be made avaliable via init_state.x.wkv

About

PyTorch implementation of RWKV blocks

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published