Every fresh chat starts from the How can I help? welcome state. A newly opened browser tab or window also starts clean, so the product does not silently attach you to an older conversation unless you explicitly continue one from History.

• New Chat clears the active conversation binding, resets the thread, drops transient run state, and brings back the welcome screen.
• View in History opens an old session read-only so you can inspect what happened without changing follow-up state.
• Continue in History rebinds the composer to that session so the next message is treated as a real follow-up rather than a brand-new task.
• Parallel runs can still exist in the background; use the Active pill to jump between them when more than one workflow is live.

Composer

• Press Enter to send and Shift + Enter for a new line.
• Use the + menu for files, folders, Drive, OneDrive, Co-Pilot quick starts, and context actions.
• Attached files are injected as live request context. Attached folders are indexed so the agent can selectively open specific files when needed instead of loading everything at once.
• The clock picker schedules the message instead of running it immediately, turning the same request into a future job.

Live Workflows

• Workflow plans are shown as a to-do list with To do, Doing, Done, and Issue states.
• If the agent asks a clarifying question, reply in the same chat and the run resumes with the new information.
• Final reports stay in the transcript so the chat itself becomes the audit trail of what was attempted, completed, skipped, or left unresolved.

Complex work uses a staged pipeline so planning, information gathering, execution, and quality checks do not collapse into a single opaque model response.

1. Planner turns the goal into a concrete task sequence, tool plan, or explicit todo structure.
2. Researcher gathers facts, files, emails, pages, prior session evidence, and live search results that may matter.
3. Curator compresses and filters the raw evidence so execution sees only the context that is actually useful.
4. Executor performs the work, including tool calls, deterministic actions, browser interaction, and content generation.
5. Validator checks whether the result is complete, coherent, and grounded enough to finish safely.
6. Report Out turns the run state into the final user-facing answer, summary, or completion report.

The product does not always use every stage. A simple request can bypass most of this. A harder task can loop through planning, execution, and validation more than once before it returns a final answer.

Fast Paths

• Simple email and calendar lookups often bypass the full workflow and go through specialized logic paths that are faster and more stable.
• Common actions like reply, send, create meeting, update meeting, folder moves, labels, attachments, and free-slot finding use dedicated logic instead of asking the model to improvise everything.
• Multi-goal requests are expanded into explicit todo steps so the system does not stop after only the first successful subtask.

Safety

• Approval mode pauses before sensitive skills when enabled in Settings.
• Run Policy controls step limits, tool-call limits, retries, runtime budget, blocked skills, and approval-required skills.
• Validation can trigger re-planning, fallback routing, clarification, or issue reporting when something is incomplete or risky.

Co-Pilot is the proactive briefing layer. It reads your connected inboxes and calendars and turns them into structured summaries.

• Morning Brief focuses on today: priorities, meetings, and messages needing action.
• Follow-up Brief focuses on open loops, pending replies, and stalled work.
• Each brief has its own toggle, daily time, instructions, and Run Now action.
• Folder and label descriptions from Outlook and Google can be added so Co-Pilot understands why a mailbox section matters.
• Supported channel command: copilot: morning brief or another configured brief name.

Skills are the tools the agent can call. The app separates them into Built-in Skills and Custom Skills.

• Built-in Skills cover search, files, shell, browser, email, calendar, cloud storage, messaging, and scheduling.
• Custom Skills live in the tools/ folder and can be turned on or off in Settings.
• MCP server skills are discovered automatically after you configure an MCP server.
• Disabled skills are never selected. Approval-required skills pause before execution.

Custom Skill Basics

• Each custom skill is one Python file.
• Define a unique name, a clear description, an argument schema, and a run(args) entry point.
• Use shared helpers from tools/_std.py for consistent success and error output.
• Restart the server after editing custom skill files.

Use the gear icon to open Settings. The most important sections are Models, Integrations, Channels, Server, and Updates.

Models

• Main Model is the default for planning, execution, and reporting.
• Fast Model is used for lighter or helper stages.
• Fallback Model is used when the preferred model is unavailable.
• Recommended defaults: glm-5.1:cloud, gpt-oss:20b-cloud, qwen3.5:9b.

Integrations

• Outlook uses Outlook Desktop on Windows and does not need API keys.
• Google Workspace needs OAuth credentials and a browser authorization step after setup.
• OneDrive can work from a local sync folder or through Microsoft Graph OAuth.
• Browser relies on Playwright plus Chromium.
• MCP Servers add external tool catalogs through Model Context Protocol.

Channels

• WhatsApp is now a QR-based setup. Enable it, then go to Settings > Channels > WhatsApp and scan the code.
• Slack uses Bot Token plus App Token with Socket Mode.
• Telegram uses a BotFather token.

Channel commands include status, pause, resume, stop, schedule: <interval> <goal>, schedules, unschedule: <id>, copilot: <name>, and help. In practice, normal natural-language requests still work best, and commands are mostly useful for messaging channels where you want quick operational control.

Server and Updates

• Run options control internet access, email access, approval mode, and fast mode.
• Show Thinking reveals reasoning blocks for supported models.
• Use Start, Reset, or Stop to manage the backend.
• Updates preserve settings, memory, and custom skills.

Enter	Send message
Shift + Enter	Insert a new line
Escape	Close the current overlay

• Use New Chat whenever you want a clean thread with the welcome screen.
• Use the Active pill to switch between running workflows.
• The left sidebar keeps History, Schedules, and Identity available without leaving the current page.
• The browser pane can be refreshed, expanded, or collapsed from its own controls.
• The schedule picker next to the composer turns the next message into a scheduled run.

The system uses several context, memory, routing, execution, and persistence layers at once. They are not interchangeable. Some exist to answer the current request, some exist to preserve continuity, and some exist only so the runtime can inspect itself and improve future runs.

Reading guide

• Start with the lifecycle if you want the shortest end-to-end explanation of what happens from message send to final answer.
• Jump to routing and orchestrator sections if you want to understand how the app decides between direct replies, fast paths, and full workflows.
• Use the memory, tooling, and frontend sections if you want to understand where continuity, side effects, and UI updates actually come from.

Architecture at a glance

Composer/UI
   -> request packaging + attachments + session binding
   -> backend routing + task understanding + context assembly
   -> direct reply OR deterministic fast path OR full orchestrator
   -> validation + fallback/replan/clarification when needed
   -> final report + session history + memory/log side effects

Cross-cutting layers:
   settings/keyring | managed integrations | tool registry | memory/wiki/user profile | websocket events

End-to-end request lifecycle

1. User input enters the composer. A request may contain plain text, follow-up wording, attached files, attached folders, selected cloud items, scheduled timing, and whatever session is currently bound in the UI.
2. The frontend packages the request. The browser sends the message to the backend together with the active session id, attachment references, scheduling data, and the relevant UI toggles such as model selection or internet access.
3. The backend restores live state. The server resolves settings, active model choices, auth state, connected providers, current session context, and any prior workflow state needed for continuation.
4. Request classification begins. The runtime decides whether the goal is direct chat, a fast deterministic action, a live follow-up on prior tool output, or a full orchestrated run.
5. Context is assembled. The system merges the current message, active session history, stored lookup records, attached context, managed context sources, relevant durable memory, and user identity where appropriate.
6. Routing chooses the cheapest safe path. If the task can be solved deterministically, the app stays out of the heavy workflow. If not, it escalates into the orchestrator.
7. Execution runs with live progress events. The runtime emits status, step, reasoning, todo, thought-summary, and completion events back to the UI over WebSocket.
8. Validation decides whether to stop or continue. The run can finish, ask a clarification, retry with a fallback, re-plan, or report an issue.
9. Results are persisted. The session transcript, tool trace, selected records, final report, and learning or memory side effects are stored in the appropriate layers.
10. The interface becomes the audit trail. The final answer stays in chat, the session lands in History, and any scheduled or long-lived state remains available for follow-up.

What happens before the model is even called

• Settings load first. The backend reads settings.json, resolves secret placeholders from keyring where needed, and determines which providers, channels, policies, and models are currently active.
• Session state is restored. If the message belongs to an existing session, the backend reloads prior messages, selected email or calendar records, stored lookup results, last tool traces, and run metadata used for follow-up continuity.
• Task understanding runs early. The system tries to infer the real task type, such as email lookup, email update, calendar create, calendar move, browser search, filesystem request, or general chat, before it spends tokens on a generic LLM pass.
• Fast-path guards run early. If the product already has a reliable path for a request class, it avoids unnecessary planning and goes directly into the relevant functional runtime.

Context Layers

Current request	Your latest message, live toggles, selected models, browser state, and anything attached to the current send action.
Current session	The active conversation history, recent tool results, run state, and short-range continuity used for follow-up questions.
Attached context	Files, folders, Google Drive items, and OneDrive items attached to the active request.
Managed context sources	Mailbox folders, labels, local folders, and cloud folders configured in Settings > Context so the agent can search them when relevant.
Relevant Memory	Durable user-facing wiki memory such as ongoing topics, important reference threads, and stable preferences that matter in future dialogue.
User Identity	The structured profile in user.md. This stores durable personal facts and communication preferences.
Semantic history recall	Search across earlier sessions when older work is relevant but not already present in the active session.
System Log	Operational traces for learning, reflection, corrections, memory maintenance, and server events. This is diagnostic, not conversational memory.

How routing chooses the execution path

• Direct respond path is used for normal conversational answers, lightweight interpretation, and cases where no external action is required.
• Deterministic action path is used when the product can reliably map a request to a purpose-built function such as reading emails, drafting replies, searching a mailbox, or mutating calendar items.
• Context follow-up path is used when the current message depends on earlier results, for example open the second one, summarize the PDF, or move that meeting.
• Full orchestrator path is used when the task needs planning, multi-tool coordination, browser interaction, file work, research, or iterative validation.

The practical goal of this routing layer is to keep simple work fast and stable while still allowing complex tasks to expand into a real agent run when necessary.

What the memory layers are for

• Session memory keeps the current conversation coherent while a topic is still active.
• Wiki memory stores durable user-relevant context rather than low-level engineering incidents or temporary runtime noise.
• User identity stores stable preferences such as tone, language, recurring personal facts, and long-lived user profile details.
• Managed context sources are not memory by themselves; they are curated places the agent may search live when a request needs external data.
• Private sessions are excluded from durable memory and learning.

Functional section map

• Lifecycle explains the chronological flow of one request.
• Routing explains how the app chooses the cheapest correct execution path.
• Orchestrator explains the staged agent runtime for harder tasks.
• Tooling and provider runtimes explain where side effects and structured business actions come from.
• Memory, learning, and frontend updates explain what persists after the answer and how the UI reflects a live run.

How the orchestrator actually runs

1. Planner stage. The planner receives the goal, the current task state, run policy, available tools, prior relevant memory, and any user-supplied context. Its job is not just to brainstorm, but to produce an actionable structure the rest of the runtime can execute.
2. Research stage. If a task needs live information, the researcher gathers it from the appropriate systems: email, calendar, browser, filesystem, managed context sources, search results, or prior workflow artifacts.
3. Curator stage. The curator trims the evidence down to the minimum context that still supports the next decision. This is important because raw research output is often too noisy or too large to pass directly into later stages.
4. Executor stage. The executor calls tools, performs browser steps, updates files, drafts messages, or produces structured outputs. This stage is where most side effects happen.
5. Validator stage. The validator checks whether the objective is actually complete, whether the evidence is strong enough, and whether a retry or re-plan is needed.
6. Report stage. The system converts the internal run state into a user-facing answer, preserving enough detail to be useful without dumping raw internal traces into the final response.

The orchestrator is policy-bound. It tracks maximum steps, maximum tool calls, validation retries, token budgets, runtime budgets, approval rules, disabled tools, and dynamic policy adjustments based on earlier outcomes. This means the runtime is not only trying to solve the task, but also trying to do so within operational constraints.

Tooling model under the hood

• Built-in tools cover browser work, shell commands, workspace search, file access, managed integrations, messaging channels, and other first-party capabilities.
• Custom tools live in the tools/ folder and are loaded dynamically so you can add project-specific functionality without rewriting the core app.
• MCP tools come from configured Model Context Protocol servers and extend the system with external tool catalogs.
• Approval-gated tools can be marked as sensitive so the runtime pauses before executing them when the approval policy requires it.
• Disabled tools are kept out of planning and execution entirely, which is different from approval gating.

How email and calendar stay reliable

• The product does not treat email and calendar as generic text tasks. It uses dedicated intent extraction, record matching, follow-up resolution, and provider-specific action runtimes.
• Earlier lookup results are stored in structured form so follow-ups can reference the first one, that draft, or the meeting tomorrow without re-inventing selection logic each time.
• Provider scope matters. The runtime can distinguish Outlook versus Google requests, preserve that scope across follow-ups, and still fall back intelligently when both ecosystems are enabled.
• This is why the product can behave more like a task system than a chatbot when you work with inbox and calendar actions.

How attachments and external context are processed

• Attached files are treated as immediate request context. They can be summarized, searched, or passed into the right downstream runtime.
• Attached folders are not blindly inlined. They are indexed so the agent can inspect the folder and open the right file when needed.
• Cloud attachments from Google Drive and OneDrive are normalized into the same execution model so the rest of the runtime can work with them consistently.
• Managed context sources are long-lived, curated search surfaces. They are available across future requests and are different from one-off attachments added to a single message.

How memory, learning, and reflection differ

• Conversation memory exists to answer the next user request better.
• User identity exists to preserve stable profile and preference information.
• Wiki memory exists to keep durable, user-relevant knowledge available across future sessions.
• System log exists to explain what the runtime did internally. It is operational evidence, not user memory.
• Learning rules exist to modify future prompting, routing, or planner behavior when repeated patterns show that the system should adapt.
• Reflection is the app evaluating how a run went so future runs can be more reliable, not simply summarizing the user conversation.

Learning and System Log

• The System Log records corrections, reflections, hypothesis decisions, KPI snapshots, learning updates, memory events, and server starts.
• Repeated correction patterns can be promoted into learning rules that influence future prompt or planner behavior.
• The log is for inspection and debugging; it is separate from the user-facing Relevant Memory shown in Context.
• Older operational log entries are pruned automatically so the log does not grow without bounds.

What the frontend is really doing during a run

• The frontend is not only rendering chat bubbles. It also tracks the active session binding, active run id, visible todo state, thinking visibility, browser preview state, attachment state, schedules, and settings overlays.
• Most long-running work is streamed back over WebSocket as typed events. The UI listens for these events and updates the transcript, progress list, thought bubbles, browser preview, and final report in near real time.
• The result is a split architecture: HTTP handles settings, uploads, and ordinary API calls, while WebSocket carries the live conversational run.

A concrete mental model

Think of the app as five cooperating systems running behind one chat box: a UI session manager, a routing layer, a set of deterministic business runtimes, an agent orchestrator, and a memory and learning layer. The product feels simple from the front, but under the hood it keeps deciding which of those systems should take the lead for the current message.

That is the core design principle of AgentOrchestrator: do not send everything to one giant model prompt. Instead, preserve structured state, choose the narrowest correct path, expose progress, and only expand into a full autonomous workflow when the task truly requires it.