Job resources

Reference for declaring what a batch job consumes. For why the model is shaped this way, see How jobs are scheduled.

Declaring a resource

A job carries requests and limits on its record:

BatchJob job;
job.type = QStringLiteral("cfd/solve");

QVariantMap requests{
    {QStringLiteral("octomy.io/cpu-milli"),   4000},
    {QStringLiteral("example.com/su2-processes"), 4},
};
QVariantMap limits{
    {QStringLiteral("octomy.io/cpu-milli"),   8000},
    {QStringLiteral("example.com/su2-processes"), 4},
};
job.resources.insert(QStringLiteral("requests"), requests);
job.resources.insert(QStringLiteral("limits"),   limits);

Both keys are optional. A job that declares neither is a BestEffort job, which is correct and is what every existing OctoMY batch job is.

Declare what you will use, then use only that

A worker that starts threads of its own must cap the fan-out first and declare the cap. Declare the machine and the job almost never admits. Declare less than you take and the budget is wrong, and nothing reports it — the scheduler believes the declaration while the machine obeys the code.

Bounds detection is the worked example in the tree. It sampled across idealThreadCount() - 1 threads, so on a 64-core box one job took 63 of them from a slot the scheduler counted as one. It now caps at four and declares 4000:

// The cap and the request are one function each, and the submitters call
// the second. Two numbers meant to be one number drift.
int BoundsDetectionWorker::samplingThreadCount()
{
    constexpr int kMaxSamplingThreads = 4;
    return qMax(1, qMin(kMaxSamplingThreads, QThread::idealThreadCount()));
}

This made bounds detection slower on any machine with more than five cores, on purpose. A declared number the scheduler can hold is worth more than an undeclared worker that takes the box.

Naming rules

Rule Valid Invalid
A domain prefix is required example.com/cfd-cores cfd-cores
The domain must contain a dot octomy.io/gpu octomy/gpu
Quantities are whole numbers 500 0.5

octomy.io/ is reserved for this project. Use your own domain for anything else; it does not need to resolve, it needs to be yours.

A name that breaks these rules is rejected when the provider registers, not when a job is placed. That is deliberate — the error reaches the person who chose the name while they are still working on it.

Shipped conventions

Two resources ship, and neither is built in. No code path in libbatch treats them specially; they exist because most jobs want them.

Name Unit Measured from
octomy.io/cpu-milli Thousandths of a core. One core is 1000 QThread::idealThreadCount()
octomy.io/memory-bytes Bytes /proc/meminfo MemTotal

Express fractions by choosing a smaller unit. Half a core is 500 of cpu-milli, never 0.5 of a cpu.

Advertising your own resource

A node advertises a resource by registering a provider that can measure it:

class Su2ProcessProvider : public batch::IResourceProvider
{
public:
    QString resourceName() const override
    {
        return QStringLiteral("example.com/su2-processes");
    }

    qint64 capacity() const override
    {
        return mDetectedProcessSlots;   // measured, not configured
    }

    bool isAvailable() const override
    {
        return mSolverFound;            // false hides the resource entirely
    }
};

Three things are worth getting right.

Measure, do not accept a claim. The rule the fleet already applies is that a node reports what it can verify and never what an operator asserts. An SU2 provider should report how many solver processes this box can actually run, having started one and read its version — not a number from a config file.

isAvailable() == false is not zero. Returning false hides the resource completely, and that is the honest answer when the measurement cannot be made. Returning 0 says "I have none of this", which is a different statement. A provider that guesses routes work confidently to a machine that cannot run it, and the failure surfaces an hour later somewhere else.

Register once, at startup:

QString error;
if (!manager.registerProvider(QSharedPointer<batch::IResourceProvider>(new Su2ProcessProvider), &error)) {
    qWarning() << "resource provider rejected:" << error;
}

What a refusal tells you

Admission never partially succeeds. A job that cannot have everything it requested gets nothing, and the result names both the step that refused and the amount that was short:

needs example.com/cfd-cores 16, short by example.com/cfd-cores 8

A resource the node has never heard of counts as zero, not as unlimited. A job asking for something nowhere advertised is refused rather than silently admitted.

Checking a declaration against reality

A declaration is a number a person wrote, and it goes stale. batch.drift reports, per job type and resource, what was declared against what was measured:

batch.drift {"jobType": "cfd/solve", "minSamples": 20}

Two shapes are worth acting on, and they are opposite defects:

  • Measured far below declared. The job reserves capacity it never uses, so every node it lands on has less to give than it thinks. A job type using a quarter of its request wastes three quarters of a reservation on every placement.
  • Measured far above declared. The job takes more than it reserved, so the node is oversubscribed without the scheduler knowing. Every node it lands on becomes slower than its budget promised.

A resource that was measured but never declared is reported too, with a declaration of zero. That is the worse of the two: the scheduler cannot see the consumer at all.

Nothing corrects drift automatically, and that is deliberate. An automatic correction would make the declaration meaningless and the defect permanent, because the number a person would have fixed is the one the machine is quietly working around. The report exists so somebody edits the number.

Nothing is reported below minSamples runs. Drift computed from two runs is not drift.

In this section