Live editing API

This extends Agent API v1 with reservations in open documents, including unsaved documents. The agent stays outside Unity. Image generation is performed by the agent’s own tools; these commands neither call a model nor download images. All calls run on Unity’s main thread in Edit Mode. They never trigger C# recompilation, refresh the AssetDatabase, or save the document. FX delivery can compile transient GPU shaders; this is separate from script compilation.

Immediate editing batches

For parameter-only FX edits, preset insertion, reordering/copying/baking FX, layer structure changes and blur/healing strokes, use whimtex_assistant_execute. It takes the same operations as the batch/headless backend, an explicit open sessionId and the document expectedRevision from op:inspect. It forms one Undo step, never saves, and requires all pending jobs to finish first. See shared editing operations for the complete contract. This is an immediate edit, not a replacement for reserving an output while doing lengthy generation. Use whimtex_fx_catalog to discover installed presets and whimtex_render_probe to inspect FX input/output or isolated channels without changing the document.

Make Seamless is a targeted layer (type:"makeSeamless"), not an FX preset. Use shared add/set operations with settings.makeSeamless and a target operation for its source. All four methods, units, new-layer defaults and independent edge selectors are documented in Make Seamless settings. Use whimtex_describe for defaults/enums and inspect the session before updating existing settings; changing the method retains its other parameters. For packed maps, quiltingChannels:"Independent" affects patch matching, while processRed / processGreen / processBlue / processAlpha control which channels receive the result in every method.

Connect and discover

For an authorized content-generation request, reserve and capture context in one call:

unity command whimtex_assistant_begin --requestId 'NEW-UUID' --name 'Balcony' --source merged --area selection --project-path 'D:/Projects/MyGame' --format json

requestId is required; generate a unique value before invoking the command and retain the exact arguments for retries. Optional arguments: name (Generating…), source (none), area (canvas), sessionId, sourceLayerId, selectionMode (strict), padding (-1: mode default). This shortcut creates a new layer at root index 0; use JSON begin for other placement or replacePixels. Direct equivalent: WhimTexApi.LiveBegin(requestId, name, source, area, sessionId, sourceLayerId, selectionMode, padding).

With no sessionId, begin uses the only open document or the currently focused WhimTex window. Only when multiple windows are open and none currently has focus, it falls back to the open window with the highest focusOrder. Each OnFocus advances a shared counter; window orders survive script reload, and the counter resumes above restored orders. API reads do not change focus history. Missing or tied history returns session_ambiguous without writing. Focus a window or specify its session explicitly in that case. A retry resolves its original receipt before considering the current focus, so switching windows cannot redirect a duplicate request. Use globally unique request IDs. For source:layer, omitting sourceLayerId means the active layer at begin.

This is an explicit mutation, not a read with side effects. Reading sessions or inspecting a document never creates a placeholder. No full snapshot is required before an ordinary begin. The response includes a compact context with document name/path, activeLayerId, selectedLayerIds, selectionActive and selectionBounds, all captured at begin. It does not enumerate layer settings. Rendering and PNG encoding are still synchronous when a source image is requested; context and reservation are captured in one main-thread call, not at the instant a chat message was sent.

The existing optional Pipeline adapter exposes:

unity command whimtex_assistant_sessions --project-path 'D:/Projects/MyGame' --format json
unity command whimtex_assistant_live --requestPath 'D:/Projects/MyGame/Temp/WhimTex/request.json' --project-path 'D:/Projects/MyGame' --format json

Direct equivalents, without Pipeline:

WhimTexApi.LiveSessions();
WhimTexApi.LiveFile(absoluteRequestPath);
WhimTexApi.LiveJson(requestJson);

Use the DCFApixels.WhimTex namespace. Check the returned JSON success as well as transport success. Read-only discovery returns sessions, each with sessionId, name, assetPath, dimensions and focused status, plus focusOrder (0 means no recorded focus). assetPath identifies the bound TIFF or JSON document, not its temporary in-memory compositor. A blank path means no saved document binding; a source image opened for editing is not necessarily a saved layered document. If multiple documents are open, use the user’s requested document; ask when ambiguous. Do not guess from a layer name.

{"apiVersion":1,"op":"inspect","sessionId":"SESSION"}

Inspection returns the regular document snapshot, activeLayerId, selectedLayerIds and the current canvas selection. Pending entries expose jobId, contentLocked and status text. The session is pinned to that window/document pair: switching documents, closing the window or reloading scripts invalidates delivery. Discovery does not open windows or change selection.

Reserve first, generate later 

{
  "apiVersion":1,"op":"begin","sessionId":"SESSION",
  "requestId":"a-new-caller-generated-uuid","name":"Moonlit mist"
}

The result contains jobId, reserved layerId, state, optional capture and saved:false. Use a fresh requestId for each task. Retrying the identical begin request returns the existing job; reusing it for a different request fails. Keep the original JSON for transport retries.

Begin field Meaning
sessionId, requestId requestId required; sessionId optional for unambiguous automatic document selection
name Optional; defaults to Generating…. Normally give a useful name immediately
parent, index Group ID and insertion index; default root, top (0)
source none (default), merged, or layer
sourceLayerId For source:layer; omitted means the active layer. An explicit ID may identify any valid source
area canvas (default) or selection; selection must be active and nonempty
selectionMode Selection only: strict (default) enforces the mask; guide treats it as a placement reference
padding Context around selection bounds, 0..4096 pixels; defaults to 32 for strict, 128 for guide. Selection only. JSON does not accept the CLI’s -1 sentinel
destination newLayer (default) or replacePixels
targetLayerId Explicit Drawing ID required for replacePixels

The source and destination are independent. Choose them from the user’s request. For example, removing an object from the visible composition usually needs source:merged; fixing the content of a particular layer may need source:layer. Do not require all tasks to use Copy Merged.

A reservation is an inert layer, ignored by compositing, effect inputs and clipping chains. Users may rename, hide, move or group it immediately. Content settings and painting are unavailable. Completion preserves its current ID, name, visibility and position. Duplicating, merging or converting a reservation (including a group containing one) is unavailable until generation finishes.

Several results from the same context 

{"apiVersion":1,"op":"fork","jobId":"PENDING-JOB","requestId":"NEW-UUID","name":"Second result"}

Fork inserts a separate named reservation directly below the source reservation in its current container. It reuses the immutable capture files, frozen selection and selectionMode, without a new render or sampling current pixels. Each fork has independent job/layer IDs, cancellation and completion. Only pending newLayer jobs can be forked; fork before completing the source. Deleting the source afterward does not cancel its forks. requestId is idempotent with identical JSON. Name defaults to the source reservation’s current name. Canvas resize and existing resource limits are checked.

Capture a selected region 

{
  "apiVersion":1,"op":"begin","sessionId":"SESSION",
  "requestId":"another-uuid","name":"Background repair",
  "source":"merged","area":"selection","padding":48,
  "destination":"newLayer"
}

capture.imagePath is a cropped PNG with surrounding context. capture.maskPath, when present, is a matching grayscale mask of the frozen selection. Its meaning depends on selectionMode:

  • strict (default): white means editable, black means preserve. The API enforces this mask on completion as well as candidate previews; soft selection values become partial coverage.
  • guide: white marks the preferred subject area; black is surrounding context, not protected pixels. The selection is a reference for generation, not an enforced editing mask. Result pixels and shadows can extend outside it, within the padded capture crop. The API does not automatically remove the returned background; request transparent surroundings when generating a cutout on a new layer.

Choose strict for precise masked edits or explicit outside-area protection. Choose guide for object placement with shadows/soft edges that need extra room. A request to fill the selection with the main object need not constrain its shadow. Keep unrelated surroundings unchanged in the generation prompt. capture.selectionMode, capture.maskEnforced and capture.maskMeaning report the frozen policy; canvas captures have null selectionMode and false maskEnforced. Unknown modes are errors.

For example, reserve a new object with room for its shadow:

{
  "apiVersion":1,"op":"begin","requestId":"new-object-uuid","name":"Balcony",
  "source":"merged","area":"selection","selectionMode":"guide","padding":160
}

Choose padding before capture; both modes remain bounded by the canvas. In guide mode the generator must not use the selection as a hard editing mask. With replacePixels, the entire crop is replaced, including black-mask context and any transparent pixels in the result. Prefer newLayer for additions unless replacing this wider area is part of the user’s request. Preview before delivery when useful.

capture.region is [x,y,width,height] in bottom-left canvas pixels. PNGs have ordinary image orientation; do not manually flip them. Generate the entire context crop, not just its white mask area. Capture pixels are sRGB 8-bit PNG, excluding display channels, EV and Post FX. HDR is clamped for the generation input. The original HDR document is not changed by capture. Single-layer capture includes that layer’s transform, Swizzle and FX before its outer opacity/blend, even if it is hidden. Groups are isolated color sources; child visibility is respected.

Selection, its interpretation, crop bounds and source pixels are frozen at begin. Later selection changes do not change the task. Context files live under Temp/WhimTex/Agent/<jobId>/, not Assets. Returned paths are absolute.

Complete with an image 

{
  "apiVersion":1,"op":"complete","jobId":"JOB",
  "imagePath":"C:/Temp/generated.png","fit":"stretch"
}

PNG input is decoded directly into owned Drawing pixels at its original resolution; there is no intermediate File layer or imported source asset. New images become Standard Drawing layers. The result is not automatically saved. The user can inspect it, paint on it and save normally.

fit is contain or stretch. For new layers it sets Transform scale and position, never resizes the stored image. Contain preserves aspect ratio and centers within the captured region; stretch fits the complete generated image to that region. Defaults: contain for a full canvas, stretch for a selection crop. For regional edits return the full captured crop with the same framing. In strict mode the frozen selection is mapped into source pixels and multiplied into their alpha; RGB and source dimensions are preserved. Selection edges have the precision of the generated source’s pixel grid. Guide mode preserves the returned RGBA without mask multiplication and uses the same crop transform. For replacePixels, the existing target’s resolution is retained and nonmatching input is resampled in linear, premultiplied color. PNGs are interpreted as sRGB color, not raw linear data textures. PNG input is limited to 64 MiB and 16,777,216 pixels.

For replacePixels, only the original Drawing’s pixel content is updated. Its identity, transform, blend, FX, name and visibility remain. The selection is mapped into Drawing coordinates; rotation and scale are supported. Repeating/mirrored transforms are rejected because one stored pixel can appear both inside and outside the selected region. Use a new layer for such edits. A transformed source can still be sampled for a new-layer result. Existing FX and swizzles still apply after replacement; prefer a new layer when editing their already-processed appearance.

Pixel replacement checks the target’s content/settings fingerprint. Independent edits to other layers, renaming, visibility and reordering do not invalidate it. If the target changed, revision_conflict leaves the reservation intact. Do not refresh the fingerprint and overwrite the user’s newer work: start a new-layer job, or recapture after agreeing on a new edit.

Complete with layer parameters 

{
  "apiVersion":1,"op":"complete","jobId":"JOB",
  "layer":{
    "type":"noise",
    "settings":{"noise":{"scale":8,"seed":472}}
  }
}

layer accepts type, settings, transform, fx, and effect-only input/target. input accepts Previous, Specific (default), or AllBelow; only Specific accepts a target. AllBelow reads visible lower siblings composited on transparency within the current group, not an external backdrop, and remains stack-relative after placement changes. Types and settings are the same as regular API add; discover defaults with whimtex_describe and consult the API reference. settings.name and settings.enabled are forbidden: those belong to the reservation and may already have been changed by the user. Groups start empty. Shader Processor uses the same inline fx contract as other layers, including groups.

Supply exactly one of layer or imagePath. Parameter completion requires area:canvas and destination:newLayer. For selection-masked or pixel-replacement jobs, complete with an image.

Inline Shader FX

Use a shaderProcessor for an effect on the composed lower stack; use fx on a normal layer to process just that layer. Reserve with begin first, then preview or complete with code embedded in the request:

{
  "apiVersion":1,"op":"complete","jobId":"RESERVATION",
  "layer":{"type":"shaderProcessor","fx":[{
    "code":"// @param float _Gain = 1.25\\nfloat4 ApplyFX(float2 uv, float4 color) { color.rgb *= _Gain; return color; }"
  }]}
}

For generated Drawing content, imagePath and fx can be supplied together at the request root. For parameter layers, put fx inside layer or at the root, never both. Pixel-replacement jobs do not accept FX changes; lock the existing layer for a separate FX edit instead.

Each fx entry has op (add by default, replace, remove) and an optional FX index. Add appends by default, or inserts at the specified index. Replace/remove require an explicit index from the layer’s fx snapshot. Indices refer to the entire FX list, including Material references, and each operation uses the list after the preceding operation. Unmentioned entries stay unchanged. Replace copies the new code into a fresh embedded FX; it never edits a shared external asset. Remove accepts only op/index. Add/replace accept code; define parameters with HLSL @param declarations.

  • Code: 1..65,536 characters of HLSL with float4 ApplyFX(float2 uv, float4 color).
  • color and SampleInput(uv) are straight RGBA in linear working space. Return straight RGBA; preserve color.a unless changing transparency is intentional. Layer opacity/blend run afterwards.
  • _MainTex, _MainTex_TexelSize, _InputSize, _CanvasSize, _PreviewScale are supplied by the editor. _InputSize/_CanvasSize hold width, height, reciprocal width, reciprocal height. _PreviewScale is full-size pixels per preview pixel. UnityCG.cginc is already included.
  • #include works with existing Assets/Packages paths and paths relative to the document (Assets before its first save). Prefer explicit project paths. An FX does not require its own shader file.
  • Parameters: at most 128, each with name, type and value. Bool takes JSON true/false (sent to HLSL as float 0/1); Float and Enum take finite numbers in -1,000,000..1,000,000 (Enum uses the declared numeric option value, not its label). Color takes four components, encoded RGB in -107..107 and alpha in 0..1, converted to linear RGB for the shader. Vector2, Vector3 and Vector take two, three and four raw components respectively, each in -1,000,000..1,000,000. Normal takes three components and normalizes them. The remaining types are documented below. Use valid unique HLSL identifiers; do not redeclare the generated uniforms in code.
  • Texture2D accepts an existing Assets/Packages texture path, "self" for the image before this FX, "none" for no source, or {"layer":"EXISTING-LAYER-ID"}. Layer references use stable IDs, not names or @aliases. Sampling the document’s own saved output is rejected; use self instead. Texture uniforms include <name>_TexelSize.
  • Point accepts two finite numbers in [-1000000,1000000], normalized bottom-left-origin UV coordinates. (0,0) and (1,1) are canvas corners, not value limits. Code declares it as // @param point _Center = (0.5, 0.5); its editor handle can also be dragged outside the canvas. Snapshot and update values use the two-number array form, the same JSON shape as Vector2.
  • Transform2D accepts value: {"position":[0.5,0.5],"size":[1,1],"rotation":0}; omitted fields use these defaults, not the previous parameter value. Alternatively use value: {"matrix":[1,0.2,0,0,1,0,0.15,0,1]} for skew/perspective: nine row-major doubles mapping local UV to input UV. Matrix and TRS fields cannot be combined. The matrix must be invertible with no horizon crossing the unit rectangle. Inspection returns either TRS fields or matrix. Position/size are normalized to the input image, rotation is in degrees. Size components must have magnitude at least 0.00001. Generates <name>_ToLocal(uv) and <name>_ToInput(uv) helpers.

  • Curve accepts a string containing keys((time, value, inTangent, outTangent, inWeight, outWeight, weightedMode), ...).

    "one" is also accepted: two keys (0,1), (1,1), constant output 1. The strings "linear", "easeIn" (quadratic slow start), "easeOut" (quadratic slow finish) and "easeInOut" are also accepted as normalized 0→1 curve factories. For example {"name":"_Profile","type":"Curve","value":"keys((0,0,1,1,0,0,0),(1,1,1,1,0,0,0))"}. Snapshots return the same string. Code uses // @param curve _Profile and _Profile_Sample(t); see the curve contract. Values update without recompilation.

  • Gradient accepts the same gradient value (color-stop array or object with colors, alphas, mode, wrapMode, smoothness, colorSpace) as layer gradients. The built-in Rounded algorithm rounds held boundaries at full smoothness and may approximate their interior stop values; see the gradient contract. Generates <name>_Sample(t); values outside 0..1 use the gradient’s Clamp, Repeat, or Mirror wrap mode. Code declarations may use // @param gradient _Ramp (opaque black-to-white default) or a two-color initializer, for example // @param gradient _Ramp = #7EF3FF -> #B270FF; RGBA tuples are also supported. Omitted API overrides retain the code-declared defaults; see the gradient syntax. A supplied gradient without mode uses Perceptual; explicit modes are retained. JSON RGB is limited to -107..107 and alpha to 0..1, not the wider standalone gradient clipboard range. Picker Preview EV, Channels and RGB/HSV preferences do not transform API values or populate History; see the API color contract and HDR behavior.
  • Inline code may instead declare parameters using HLSL metadata. If JSON values are supplied as well, every entry must match a code declaration by name and type; those values override defaults. The first-line catalog marker is required only for catalog files.
  • At most 16 FX operations per request and 32 resulting FX entries. Keep GPU work bounded: no unbounded loops or enormous per-pixel sampling loops. Successful compilation does not prove that a shader is fast or numerically stable; inspect a small preview and use HDR Debug as needed.

The API compiles a temporary GPU shader before delivery. This is not a project/C# recompilation, AssetDatabase refresh or an exported .shader file. Compile errors return shader_compile_failed with compiler diagnostics; the live content stays unchanged, and the pending job can be corrected. Previews do not publish candidate effects or add Undo entries. Completion is one Undo action. Embedded code, parameters and compiled shader follow the document’s usual save lifecycle. Inspection exposes each FX’s code, parameters, diagnostics and pending-change state. Parameters also expose stable id, nullable minimum/maximum, and structured Transform2D values. catalogPath identifies a linked HLSL source when present; inline replacement does not modify that source. Groups support FX directly. FX process the combined children before group opacity and blending and automatically isolate a Pass Through group using Normal blending.

Edit and lock an existing layer

For custom FX code or layer-settings candidate edits, do not create a replacement placeholder: lock the existing layer’s content instead. Linked preset insertion and parameter-only FX set use immediate editing batches without a lock. The lock’s changes.fx schema does not accept presetId or set, and an active lock blocks immediate batches. A sequence of immediate set and render/probe calls edits the live document with Undo; it is not a detached candidate preview.

unity command whimtex_assistant_lock --requestId 'NEW-UUID' --layerId 'LAYER-GUID' --project-path 'D:/Projects/MyGame' --format json

Direct equivalent: WhimTexApi.LiveLock(requestId, layerId, sessionId, expectedRevision). Or send op:lock with requestId/layerId, optional sessionId and expectedRevision through LiveJson/File. The optional expectedRevision is the layer’s contentRevision, not the document revision. Automatic window choice and idempotent request retries follow begin’s rules. The returned job owns the lock, and context.layer contains the captured layer settings and FX. Nothing new is inserted; the layer keeps rendering its previous result while the agent works.

{
  "apiVersion":1,"op":"complete","jobId":"EDIT-JOB",
  "changes":{"fx":[{
    "op":"add","code":"float4 ApplyFX(float2 uv, float4 color) { return float4(1 - color.rgb, color.a); }"
  }]}
}

Use op:preview with the same changes and optional view/maxSize/outputPath for a detached trial (default view is composite). changes accepts settings, transform, fx; name/visibility are not editable through this job. The layer’s ID, pixels, type, tree position and unrelated FX remain. This lock workflow does not replace Drawing pixels: use begin/replacePixels for that operation.

While locked, painting, fill, transform, opacity/blend and settings/FX editing are disabled for this layer. Users may still rename, hide, move or delete it. A group’s lock covers its own settings, not its children; children can continue changing. Destructive conversions/merges involving a locked layer are blocked. Path-based batches targeting its saved document are rejected while the edit is pending. Other live jobs may work on other layers, but cannot lock or replace pixels on this target concurrently.

Successful completion releases the lock. {"apiVersion":1,"op":"unlock","jobId":"EDIT-JOB"} releases it without changes and is safe to repeat. The user can choose Cancel Agent Edit in Layer Settings or the context menu; cancel with sessionId/layerId also works. A terminal fail releases an edit lock. If pausing or abandoning work, unlock/fail rather than leaving the user waiting. Locks are session-local, not saved: close/switch, deletion, script reload and Undo/Redo cancel them; Redo never reacquires one. A late completion cannot revive a cancelled job. Save As transfers the job to the new document copy. If content changed by another route, revision_conflict prevents overwriting it: unlock, inspect and agree on a new edit rather than bypassing the check.

Optional trial renders

Before completing, use the same candidate with op:preview:

{
  "apiVersion":1,"op":"preview","jobId":"JOB","view":"composite","maxSize":768,
  "layer":{"type":"noise","settings":{"noise":{"scale":8,"seed":472}}}
}

view is layer (default) or composite. The candidate replaces the reservation only on a detached copy. Its name/visibility/placement are current, and other layers reflect the document at preview time. The live document and its Undo history are unchanged. The trial returns an absolute outputPath. No trial is mandatory: use it when visual feedback helps choose parameters. If the user hid the reservation, a composite preview also respects that visibility. A layer-only preview still shows its content, so hiding a reservation does not prevent the agent from checking it.

To inspect a current document without a candidate:

{"apiVersion":1,"op":"render","sessionId":"SESSION","maxSize":1024}

Optional sourceLayerId renders one layer. Both render commands accept maxSize 1..4096 and optional project-relative outputPath under Temp/WhimTex/*.png. Files never overwrite; omit the path for an automatically unique name. These renders require a graphics device.

Status, cancellation and recovery 

{"apiVersion":1,"op":"status","jobId":"JOB"}

Status returns pending, completed or cancelled. On a generation failure, report it to the user and optionally attach a message without discarding the reservation:

{"apiVersion":1,"op":"fail","jobId":"JOB","message":"Image generation failed; waiting for retry."}

The user can cancel from Layer Settings or delete the row. Agents can cancel active or interrupted reservations by session and layer ID:

{"apiVersion":1,"op":"cancel","sessionId":"SESSION","layerId":"RESERVATION"}
  • Completion is one Undo action. Its repeated identical request returns the completed status without inserting again. A different completion request for an already-completed job is rejected.
  • Removing/undoing a reservation cancels its delivery. Redo does not restart generation. Undoing a completed result restores an inactive reservation; cancel it or start a new job, rather than replaying an old completion. A completed status acknowledges that the command ran, not that its result has never subsequently been undone or deleted.
  • Closing/switching the source window or reloading scripts interrupts delivery. Serialized reservations remain recognizable and removable, but jobs/masks are session-local and are not resumed automatically. Saving normally keeps jobs alive; Save As transfers active jobs to the newly saved copy in that window.
  • document_busy means wait for the current paint/transform/selection/drag gesture or delayed text edit to end, then retry the same request. No command ends the user’s gesture or discards typed text. Canvas resize produces a conflict; cancel and capture again.
  • After timeout, inspect status before retrying. Do not create a second reservation merely because the first response was lost. After an unknown job/session error, rediscover and inspect; never silently route a late result into another document.
  • rollback_failed reports applied:true conservatively because the document may be partially changed. Stop automatic retries and inspect the document before attempting recovery.
  • Limits: 16 pending jobs, 64 MiB total retained selection coverage, 512 job receipts per script session, 1024 layers, 16,777,216 canvas pixels and 67,108,864 owned Drawing pixels. Preview/capture pixels are transient and are not stored in Undo. Temporary PNG files remain available for inspection until Temp is cleaned; the API does not delete user-provided images.

Agent guidance

LayerToLocal(uv) converts canvas UV to the owning layer’s local UV, including parent group transforms and perspective. Use it for procedural shapes that must follow the layer transform. ApplyFX UV and SampleInput remain canvas-space; do not pass local UV to SampleInput. The helper does not wrap or clamp coordinates.

The portable skill is Skills~/whimtex-live/SKILL.md. It describes intent-based source selection, reservation ownership and recovery without requiring an embedded chat UI or a separate model integration inside Unity.