Batch artifacts
A job that runs on another machine cannot read a path on yours. It reads artifacts: files named by the hash of their bytes. The same bytes have the same name on every node, so a job sees the same inputs wherever it runs, and its outputs come back the same way.
The job contract
A job that reads or writes files states them in its data and its result.
In job.data |
Meaning |
|---|---|
inputs |
The files the job reads: a list of {name, hash, size}. name is a relative path in the job's folder |
params |
The job type's own small parameters |
inputHashes |
The input hashes, so placement prefers a worker that already holds them |
| In the result | Meaning |
|---|---|
outputs |
The files the job wrote: a list of {name, hash, size} |
values |
The job type's own small results, such as a triangle count |
A hash is 64 lower-case hex digits. A name never starts at a root, never holds : and never climbs
out of the job's folder with ...
What happens to a job's files
- Before the run. The node makes a folder for the job and writes each input into it by name. An input the node does not hold fails the job, naming the input.
- The run. The job reads its inputs and writes its outputs in that folder. It names its outputs in its result, by name only.
- After the run. The node puts each named output into its store and fills in the hash and size. An output the job named and did not write fails the job. Then the folder is deleted.
The same steps run for a job on your own node and for a job on a worker.
Between two nodes
- Inputs travel before the start. The coordinator asks the worker which inputs it already holds, and sends only the others. An input two jobs share crosses once.
- Outputs travel before the result. The worker sends its outputs, then the result that names them.
- Every artifact is checked on arrival. The receiver checks the size and the hash of the bytes. A mismatch is refused, naming what arrived, and the job fails. It is not retried, because the same bytes would hash the same again.
- A link that fails is retried, and the transfer resumes. If inputs cannot be sent, the job waits
to be placed again. If a worker cannot send its outputs, it holds the result and sends both later.
An artifact sent again to the same node continues where the last send stopped. The node keeps the
part that arrived in
blob-spoolin its base folder, for up to a day. - An artifact can be any size. Each end reads and writes it as a file, a block at a time, so a large artifact needs disk space, not memory.
Tasks
| Task | What it does |
|---|---|
batch.artifact.put |
Put a local file into this node's store. Returns its hash and size |
batch.artifact.get |
Write the artifact with a hash to a local path |
batch.artifact.transfers |
The artifacts on their way to other nodes now: each one's size, and bytesDone, the bytes that have arrived so far |
batch.job.submit with inputFiles |
Each local file becomes an input artifact, by the name you give it |
batch.job.get |
Shows a job's inputs, and its outputs in the result |
batch.job.collect |
Write every output of a finished job into a folder, each by its name |
No task returns artifact bytes in its answer. The bytes go to a path on the node that runs the task.
Example
Run a job on a file, wherever placement puts it, and collect what it wrote:
{"task": "batch.job.submit",
"args": {"type": "construct.mesh", "inputFiles": {"scene": "/home/leo/scenes/gyroid.construct"}}}
{"task": "batch.job.collect", "args": {"jobId": "42", "folder": "/home/leo/meshes"}}