kani (カニ) is a highly hackable microframework for chat-based language models with tool use/function calling. (NLP-OSS @ EMNLP 2023)
As of Nov 6, 2023, OpenAI added the ability for a single assistant message to request calling multiple functions in
parallel, and wrapped all function calls in a ToolCall
wrapper. In order to add support for this in kani while
maintaining backwards compatibility with OSS function calling models, a ChatMessage
now actually maintains the
following internal representation:
ChatMessage.function_call
is actually an alias for ChatMessage.tool_calls[0].function
. If there is more
than one tool call in the message, when trying to access this property, kani will raise an exception.
To translate kani's FUNCTION message types to OpenAI's TOOL message types, the OpenAIEngine now performs a translation based on binding free tool call IDs to following FUNCTION messages deterministically.
To the kani end user, there should be no change to how functions are defined and called. One breaking change was necessary:
Kani.do_function_call
and Kani.handle_function_call_exception
now take an additional tool_call_id
parameter, which may break overriding functions. The documentation has been updated to encourage overriders to handle *args, **kwargs
to prevent this happening again.kani can now handle making multiple function calls in parallel if the model requests it. Rather than returning an ASSISTANT message with a single function_call
, an engine can now return a list of tool_calls
. kani will resolve these tool calls in parallel using asyncio, and add their results to the chat history in the order of the list provided.
Returning a single function_call
will continue to work for backwards compatibility.
OpenAIChatMessage
s as input rather than kani.ChatMessage
in order to better type-validate API requestsThe Message Parts API is intended to provide a foundation for future multimodal LLMs and other engines that require engine-specific input without compromising kani's model-agnostic design. This is accomplished by allowing ChatMessage.content
to be a list of MessagePart
objects, in addition to a string.
This change is fully backwards-compatible and will not affect existing code.
When writing code with compatibility in mind, the ChatMessage
class exposes ChatMessage.text
(always a string or None) and ChatMessage.parts
(always a list of message parts), which we recommend using instead of ChatMessage.content
. These properties are dynamically generated based on the underlying content, and it is safe to mix messages with different content types in a single Kani.
Generally, message part classes are defined by an engine, and consumed by the developer. Message parts can be used in any role’s message - for example, you might use a message part in an assistant message to separate out a chain of thought from a user reply, or in a user message to supply an image to a multimodal model.
For more information, see the Message Parts documentation.
Up next: we're adding support for multimodal vision-language models like LLaVA and GPT-Vision through a kani extension!
[INST]
wrapper. See the tests for how kani translates consecutive message types into the LLaMA prompt.Kani.full_round
now emits every message generated during the round, not just assistant messages
FUNCTION
messages, and potentially SYSTEM
messages from a function exception handler.Kani.full_round_str
's default behaviour is unchanged.Kani.full_round_str
now takes in a message_formatter
rather than a function_call_formatter
ASSISTANT
messages.Kani.do_function_call
now returns a FunctionCallResult
rather than a bool
Kani.add_to_history
in the override, save the ChatMessage to a variableFunctionCallResult(is_model_turn=<old return value>, message=<message from above>)
Kani.handle_function_call_exception
now returns a ExceptionHandleResult
rather than a bool
Kani.add_to_history
in the override, save the ChatMessage to a variableExceptionHandleResult(should_retry=<old return value>, message=<message from above>)
kani.utils.message_formatters
kani.ExceptionHandleResult
and kani.FunctionCallResult
ChatMessage.copy_with
could cause unset values to appear in JSON serializationsKani.chat_round
to use Kani.full_round
when AI functions are definedKani.add_to_history
, a method that is called whenever kani adds a new message to the chat contexthttpclient.BaseClient.request
now returns a Response
to aid low-level implementation
.get()
and .post()
are unchanged