EventHandler Base Class

db-ally provides an EventHandler abstract class that can be used to log the runs of db-ally to any external systems.

Tip

To learn how to create a cutom EventHandler, visit How-To: Create your own event handler.

Lifecycle

Each run of [dbally.Collection.ask][] will trigger all instances of EventHandler that were passed to the Collection's constructor (or the dbally.create_collection function).

1. EventHandler.request_start is called with RequestStart, it can return a context object that will be passed to next calls.
2. For each event that occurs during the run, EventHandler.event_start is called with the context object returned by EventHandler.request_start and an Event object. It can return context for the EventHandler.event_end method.
3. When the event ends EventHandler.event_end is called with the context object returned by EventHandler.event_start and an Event object.
4. On the end of the run EventHandler.request_end is called with the context object returned by EventHandler.request_start and the RequestEnd.

sequenceDiagram
  participant C as Collection.ask()
  participant E as EventHandler
  C->>E: request_start(RequestStart(question, self.name))
  activate E
  E->>C: optional RequestCtx
  deactivate E
  loop every event
        C->>E: event_start(LLMEvent, RequestCtx)
        activate E
        E->>C: optional EventCtx
        deactivate E
        activate C
        C->>E: event_end(LLMEvent, RequestCtx, EventCtx)
        deactivate C
  end
  C->>E: request_end(RequestEnd, RequestCtx)

Currently handled events:

• Every call to the LLM

dbally.audit.EventHandler

Bases: Generic[RequestCtx, EventCtx], ABC

A base class that every custom handler should inherit from

request_start abstractmethod async

request_start(user_request: RequestStart) -> RequestCtx

Function that is called at the beginning of every Collection.ask execution.

PARAMETER	DESCRIPTION
user_request	Object containing name of collection and asked query TYPE: RequestStart

RETURNS	DESCRIPTION
RequestCtx	Implementation-specific request context object, which is passed to the future callbacks

Source code in src/dbally/audit/event_handlers/base.py

@abc.abstractmethod
async def request_start(self, user_request: RequestStart) -> RequestCtx:
    """
    Function that is called at the beginning of every `Collection.ask` execution.

    Args:
        user_request: Object containing name of collection and asked query

    Returns:
        Implementation-specific request context object, which is passed to the future callbacks
    """

event_start abstractmethod async

event_start(event: Event, request_context: RequestCtx) -> EventCtx

Function that is called during every event execution.

PARAMETER	DESCRIPTION
event	db-ally event to be logged with all the details. TYPE: Event
request_context	Optional context passed from request_start method TYPE: RequestCtx

RETURNS	DESCRIPTION
EventCtx	Implementation-specific request context object, which is passed to the event_end callback

Source code in src/dbally/audit/event_handlers/base.py

@abc.abstractmethod
async def event_start(self, event: Event, request_context: RequestCtx) -> EventCtx:
    """
    Function that is called during every event execution.


    Args:
        event: db-ally event to be logged with all the details.
        request_context: Optional context passed from request_start method

    Returns:
        Implementation-specific request context object, which is passed to the `event_end` callback
    """

event_end abstractmethod async

event_end(event: Optional[Event], request_context: RequestCtx, event_context: EventCtx) -> None

Function that is called during every event execution.

PARAMETER	DESCRIPTION
event	db-ally event to be logged with all the details. TYPE: Optional[Event]
request_context	Optional context passed from request_start method TYPE: RequestCtx
event_context	Optional context passed from event_start method TYPE: EventCtx

Source code in src/dbally/audit/event_handlers/base.py

@abc.abstractmethod
async def event_end(self, event: Optional[Event], request_context: RequestCtx, event_context: EventCtx) -> None:
    """
    Function that is called during every event execution.

    Args:
        event: db-ally event to be logged with all the details.
        request_context: Optional context passed from request_start method
        event_context: Optional context passed from event_start method
    """

request_end abstractmethod async

request_end(output: RequestEnd, request_context: RequestCtx) -> None

Log the end of the request.

PARAMETER	DESCRIPTION
output	The output of the request. TYPE: RequestEnd
request_context	Optional context passed from request_start method TYPE: RequestCtx

Source code in src/dbally/audit/event_handlers/base.py

@abc.abstractmethod
async def request_end(self, output: RequestEnd, request_context: RequestCtx) -> None:
    """
    Log the end of the request.

    Args:
        output: The output of the request.
        request_context: Optional context passed from request_start method
    """

dbally.audit.events.RequestStart dataclass

RequestStart(collection_name: str, question: str)

Class representing request start data.

collection_name instance-attribute

collection_name: str

question instance-attribute

question: str

dbally.audit.events.RequestEnd dataclass

RequestEnd(result: ExecutionResult)

Class representing request end data.

result instance-attribute

result: ExecutionResult

dbally.audit.events.Event dataclass

Event()

Bases: ABC

Base class for all events.

dbally.audit.events.LLMEvent dataclass

LLMEvent(prompt: Union[str, ChatFormat], type: str, response: Optional[str] = None, completion_tokens: Optional[int] = None, prompt_tokens: Optional[int] = None, total_tokens: Optional[int] = None)

Bases: Event

Class for LLM event.

prompt instance-attribute

prompt: Union[str, ChatFormat]

type instance-attribute

type: str

response class-attribute instance-attribute

response: Optional[str] = None

completion_tokens class-attribute instance-attribute

completion_tokens: Optional[int] = None

prompt_tokens class-attribute instance-attribute

prompt_tokens: Optional[int] = None

total_tokens class-attribute instance-attribute

total_tokens: Optional[int] = None

dbally.audit.events.SimilarityEvent dataclass

SimilarityEvent(store: str, fetcher: str, input_value: str, output_value: Optional[str] = None)

Bases: Event

SimilarityEvent is fired when a SimilarityIndex lookup is performed.

store instance-attribute

store: str

fetcher instance-attribute

fetcher: str

input_value instance-attribute

input_value: str

output_value class-attribute instance-attribute

output_value: Optional[str] = None

dbally.audit.spans.EventSpan

EventSpan()

Helper class for logging events.

Source code in src/dbally/audit/spans.py

def __init__(self) -> None:
    self.data: Optional[Event] = None

data instance-attribute

data: Optional[Event] = None