Skip to content

byte pair encoder (BPE)

BPE = Byte Pair Encoding an encoder that takes text -> tokenIds

takes common byte pairs and merges them into 1 tokenId, adds it to the overall token vocabulary