LiteLLM

dbally.llms.litellm.LiteLLM

LiteLLM(model_name: str = 'gpt-3.5-turbo', default_options: Optional[LiteLLMOptions] = None, *, base_url: Optional[str] = None, api_key: Optional[str] = None, api_version: Optional[str] = None)

Bases: LLM[LiteLLMOptions]

Class for interaction with any LLM supported by LiteLLM API.

Constructs a new LiteLLM instance.

PARAMETER	DESCRIPTION
model_name	Name of the LiteLLM supported model to be used. Default is "gpt-3.5-turbo". TYPE: str DEFAULT: 'gpt-3.5-turbo'
default_options	Default options to be used. TYPE: Optional[LiteLLMOptions] DEFAULT: None
base_url	Base URL of the LLM API. TYPE: Optional[str] DEFAULT: None
api_key	API key to be used. API key to be used. If not specified, an environment variable will be used, for more information, follow the instructions for your specific vendor in the LiteLLM documentation. TYPE: Optional[str] DEFAULT: None
api_version	API version to be used. If not specified, the default version will be used. TYPE: Optional[str] DEFAULT: None

RAISES	DESCRIPTION
ImportError	If the litellm package is not installed.

Source code in src/dbally/llms/litellm.py

def __init__(
    self,
    model_name: str = "gpt-3.5-turbo",
    default_options: Optional[LiteLLMOptions] = None,
    *,
    base_url: Optional[str] = None,
    api_key: Optional[str] = None,
    api_version: Optional[str] = None,
) -> None:
    """
    Constructs a new LiteLLM instance.

    Args:
        model_name: Name of the [LiteLLM supported model](https://docs.litellm.ai/docs/providers) to be used.\
            Default is "gpt-3.5-turbo".
        default_options: Default options to be used.
        base_url: Base URL of the LLM API.
        api_key: API key to be used. API key to be used. If not specified, an environment variable will be used,
            for more information, follow the instructions for your specific vendor in the\
            [LiteLLM documentation](https://docs.litellm.ai/docs/providers).
        api_version: API version to be used. If not specified, the default version will be used.

    Raises:
        ImportError: If the litellm package is not installed.
    """
    if not HAVE_LITELLM:
        raise ImportError("You need to install litellm package to use LiteLLM models")

    super().__init__(model_name, default_options)
    self.base_url = base_url
    self.api_key = api_key
    self.api_version = api_version

model_name instance-attribute

model_name = model_name

default_options instance-attribute

default_options = default_options or _options_cls()

base_url instance-attribute

base_url = base_url

api_key instance-attribute

api_key = api_key

api_version instance-attribute

api_version = api_version

client property

client: LiteLLMClient

Client for the LLM.

generate_text async

generate_text(prompt: PromptTemplate, *, event_tracker: Optional[EventTracker] = None, options: Optional[LLMOptions] = None) -> str

Prepares and sends a prompt to the LLM and returns the response.

PARAMETER	DESCRIPTION
prompt	Formatted prompt template with conversation and response parsing configuration. TYPE: PromptTemplate
event_tracker	Event store used to audit the generation process. TYPE: Optional[EventTracker] DEFAULT: None
options	Options to use for the LLM client. TYPE: Optional[LLMOptions] DEFAULT: None

RETURNS	DESCRIPTION
str	Text response from LLM.

RAISES	DESCRIPTION
LLMError	If LLM text generation fails.

Source code in src/dbally/llms/base.py

async def generate_text(
    self,
    prompt: PromptTemplate,
    *,
    event_tracker: Optional[EventTracker] = None,
    options: Optional[LLMOptions] = None,
) -> str:
    """
    Prepares and sends a prompt to the LLM and returns the response.

    Args:
        prompt: Formatted prompt template with conversation and response parsing configuration.
        event_tracker: Event store used to audit the generation process.
        options: Options to use for the LLM client.

    Returns:
        Text response from LLM.

    Raises:
        LLMError: If LLM text generation fails.
    """
    options = (self.default_options | options) if options else self.default_options
    event = LLMEvent(prompt=prompt.chat, type=type(prompt).__name__)
    event_tracker = event_tracker or EventTracker()

    async with event_tracker.track_event(event) as span:
        event.response = await self.client.call(
            conversation=prompt.chat,
            options=options,
            event=event,
            json_mode=prompt.json_mode,
        )
        span(event)

    return event.response

count_tokens

count_tokens(prompt: PromptTemplate) -> int

Counts tokens in the prompt.

PARAMETER	DESCRIPTION
prompt	Formatted prompt template with conversation and response parsing configuration. TYPE: PromptTemplate

RETURNS	DESCRIPTION
int	Number of tokens in the prompt.

Source code in src/dbally/llms/litellm.py

def count_tokens(self, prompt: PromptTemplate) -> int:
    """
    Counts tokens in the prompt.

    Args:
        prompt: Formatted prompt template with conversation and response parsing configuration.

    Returns:
        Number of tokens in the prompt.
    """
    return sum(litellm.token_counter(model=self.model_name, text=message["content"]) for message in prompt.chat)

dbally.llms.clients.litellm.LiteLLMClient

LiteLLMClient(model_name: str, *, base_url: Optional[str] = None, api_key: Optional[str] = None, api_version: Optional[str] = None)

Bases: LLMClient[LiteLLMOptions]

Client for the LiteLLM that supports calls to 100+ LLMs APIs, including OpenAI, Anthropic, VertexAI, Hugging Face and others.

Constructs a new LiteLLMClient instance.

PARAMETER	DESCRIPTION
model_name	Name of the model to use. TYPE: str
base_url	Base URL of the LLM API. TYPE: Optional[str] DEFAULT: None
api_key	API key used to authenticate with the LLM API. TYPE: Optional[str] DEFAULT: None
api_version	API version of the LLM API. TYPE: Optional[str] DEFAULT: None

RAISES	DESCRIPTION
ImportError	If the litellm package is not installed.

Source code in src/dbally/llms/clients/litellm.py

def __init__(
    self,
    model_name: str,
    *,
    base_url: Optional[str] = None,
    api_key: Optional[str] = None,
    api_version: Optional[str] = None,
) -> None:
    """
    Constructs a new LiteLLMClient instance.

    Args:
        model_name: Name of the model to use.
        base_url: Base URL of the LLM API.
        api_key: API key used to authenticate with the LLM API.
        api_version: API version of the LLM API.

    Raises:
        ImportError: If the litellm package is not installed.
    """
    if not HAVE_LITELLM:
        raise ImportError("You need to install litellm package to use LiteLLM models")

    super().__init__(model_name)
    self.base_url = base_url
    self.api_key = api_key
    self.api_version = api_version

model_name instance-attribute

model_name = model_name

base_url instance-attribute

base_url = base_url

api_key instance-attribute

api_key = api_key

api_version instance-attribute

api_version = api_version

call async

call(conversation: ChatFormat, options: LiteLLMOptions, event: LLMEvent, json_mode: bool = False) -> str

Calls the appropriate LLM endpoint with the given prompt and options.

PARAMETER	DESCRIPTION
conversation	List of dicts with "role" and "content" keys, representing the chat history so far. TYPE: ChatFormat
options	Additional settings used by the LLM. TYPE: LiteLLMOptions
event	Container with the prompt, LLM response and call metrics. TYPE: LLMEvent
json_mode	Force the response to be in JSON format. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
str	Response string from LLM.

RAISES	DESCRIPTION
LLMConnectionError	If there is a connection error with the LLM API.
LLMStatusError	If the LLM API returns an error status code.
LLMResponseError	If the LLM API response is invalid.

Source code in src/dbally/llms/clients/litellm.py

async def call(
    self,
    conversation: ChatFormat,
    options: LiteLLMOptions,
    event: LLMEvent,
    json_mode: bool = False,
) -> str:
    """
    Calls the appropriate LLM endpoint with the given prompt and options.

    Args:
        conversation: List of dicts with "role" and "content" keys, representing the chat history so far.
        options: Additional settings used by the LLM.
        event: Container with the prompt, LLM response and call metrics.
        json_mode: Force the response to be in JSON format.

    Returns:
        Response string from LLM.

    Raises:
        LLMConnectionError: If there is a connection error with the LLM API.
        LLMStatusError: If the LLM API returns an error status code.
        LLMResponseError: If the LLM API response is invalid.
    """
    response_format = {"type": "json_object"} if json_mode else None

    try:
        response = await litellm.acompletion(
            messages=conversation,
            model=self.model_name,
            base_url=self.base_url,
            api_key=self.api_key,
            api_version=self.api_version,
            response_format=response_format,
            **options.dict(),
        )
    except litellm.openai.APIConnectionError as exc:
        raise LLMConnectionError() from exc
    except litellm.openai.APIStatusError as exc:
        raise LLMStatusError(exc.message, exc.status_code) from exc
    except litellm.openai.APIResponseValidationError as exc:
        raise LLMResponseError() from exc

    event.completion_tokens = response.usage.completion_tokens
    event.prompt_tokens = response.usage.prompt_tokens
    event.total_tokens = response.usage.total_tokens

    return response.choices[0].message.content

dbally.llms.clients.litellm.LiteLLMOptions dataclass

LiteLLMOptions(frequency_penalty: Union[Optional[float], NotGiven] = NOT_GIVEN, max_tokens: Union[Optional[int], NotGiven] = NOT_GIVEN, n: Union[Optional[int], NotGiven] = NOT_GIVEN, presence_penalty: Union[Optional[float], NotGiven] = NOT_GIVEN, seed: Union[Optional[int], NotGiven] = NOT_GIVEN, stop: Union[Optional[Union[str, List[str]]], NotGiven] = NOT_GIVEN, temperature: Union[Optional[float], NotGiven] = NOT_GIVEN, top_p: Union[Optional[float], NotGiven] = NOT_GIVEN)

Bases: LLMOptions

Dataclass that represents all available LLM call options for the LiteLLM client. Each of them is described in the LiteLLM documentation.

frequency_penalty class-attribute instance-attribute

frequency_penalty: Union[Optional[float], NotGiven] = NOT_GIVEN

max_tokens class-attribute instance-attribute

max_tokens: Union[Optional[int], NotGiven] = NOT_GIVEN

n class-attribute instance-attribute

n: Union[Optional[int], NotGiven] = NOT_GIVEN

presence_penalty class-attribute instance-attribute

presence_penalty: Union[Optional[float], NotGiven] = NOT_GIVEN

seed class-attribute instance-attribute

seed: Union[Optional[int], NotGiven] = NOT_GIVEN

stop class-attribute instance-attribute

stop: Union[Optional[Union[str, List[str]]], NotGiven] = NOT_GIVEN

temperature class-attribute instance-attribute

temperature: Union[Optional[float], NotGiven] = NOT_GIVEN

top_p class-attribute instance-attribute

top_p: Union[Optional[float], NotGiven] = NOT_GIVEN

dict

dict() -> Dict[str, Any]

Creates a dictionary representation of the LLMOptions instance. If a value is None, it will be replaced with a provider-specific not-given sentinel.

RETURNS	DESCRIPTION
Dict[str, Any]	A dictionary representation of the LLMOptions instance.

Source code in src/dbally/llms/clients/base.py

def dict(self) -> Dict[str, Any]:
    """
    Creates a dictionary representation of the LLMOptions instance.
    If a value is None, it will be replaced with a provider-specific not-given sentinel.

    Returns:
        A dictionary representation of the LLMOptions instance.
    """
    options = asdict(self)
    return {
        key: self._not_given if value is None or isinstance(value, NotGiven) else value
        for key, value in options.items()
    }