Gpt Tokenizer Versions Save

JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.

2.1.2

7 months ago

2.1.2 (2023-10-07)

Bug Fixes

  • bind encodeChat and encodeChatGenerator (86c270c), closes #15

2.1.1

11 months ago

2.1.1 (2023-06-01)

Bug Fixes

  • missing default exports due to limitations of ESM (2a55474), closes #11

2.1.0

11 months ago

2.1.0 (2023-06-01)

Bug Fixes

  • correct excluded path (71af9d3)
  • workaround for webpack not exposing the default export in UMD correctly (84887b4), closes #12

Features

2.0.0

11 months ago

2.0.0 (2023-05-24)

Features

  • complete rewrite to support different models (eedd944), closes #5 #6
  • more tests and better README (e660a25)

BREAKING CHANGES

  • default encoder is now GPT3.5 / GPT4

2.0.0-beta.2

11 months ago

2.0.0-beta.2 (2023-05-24)

Features

  • more tests and better README (e660a25)

2.0.0-beta.1

11 months ago

2.0.0-beta.1 (2023-05-23)

Features

  • complete rewrite to support different models (eedd944), closes #5 #6

BREAKING CHANGES

  • default encoder is now GPT3.5 / GPT4

1.0.5

11 months ago

1.0.5 (2023-05-16)

Bug Fixes

1.0.4

1 year ago

1.0.4 (2023-05-01)

Bug Fixes

  • eslint and jest complaints (8b1cb01)
  • importing/requiring under certain setups was not working (8b71131), closes #4

1.0.3

1 year ago

1.0.3 (2023-04-18)

Bug Fixes

1.0.2

1 year ago

1.0.2 (2023-04-18)

Bug Fixes