RequestGate

The requests that the app sends at once to one self-hosted server (llm-openai-compatible §OCP-W6).

Declared in src/libs/libllm/llm/clients/OnlineTransport.hpp.

Detailed description

A llama.cpp server with one slot queues a second request and sends it no byte until the first one ends. The app keeps its own queue instead, so a waiting turn can say that it waits and how many turns are ahead of it. One gate exists for each provider, shared by every client of its models, on any thread: of() gives it, and a mutex guards it. A start function runs in the thread of its context object.

The gate also keeps whether the provider refused `stream_options`, for the life of the process (§OCP-W7).

Member function documentation

ahead

int RequestGate::ahead(quint64 ticket) const

How many turns wait before ticket, or -1 when it runs or is not known.

claimSlotsProbe

bool RequestGate::claimSlotsProbe()

Returns true once: to the first client of a self-hosted provider whose slots no test or fetch read, so that it reads them from the server's properties (§OCP-M2). Until the answer, the gate has one slot.

enter

quint64 RequestGate::enter(QObject *context, std::function<void ()> start)

Run start in the thread of context when a slot is free: at once, or when a request before it leaves. The ticket names the turn for leave() and ahead().

leave

void RequestGate::leave(quint64 ticket)

The turn of ticket ends, or it no longer waits. Its slot goes to the first turn that waits.

of

[static] QSharedPointer<RequestGate> RequestGate::of(const QString &providerId)

The gate of the provider providerId. Every caller of one provider gets the same gate while one of them holds it.

refuseStreamOptions

void RequestGate::refuseStreamOptions()

The provider refused `stream_options`: no request to it sends the field again while the app runs (§OCP-W7).

setCapacity

void RequestGate::setCapacity(int capacity)

Allow capacity requests at once, at least one: the server's slots. More capacity starts the waiting turns at once.