tui-use（推使用）

TUI-Use: Operating TUI Programs via tui-agent

This skill teaches you how to use the tui-agent CLI tool to start, interact with, and control TUI (Terminal User Interface) programs. Think of tui-agent as your eyes and hands for terminal programs — it lets you see what's on screen, type text, press keys, click with the mouse, and wait for things to happen.

Prerequisites

tui-agent must be installed and available in PATH, you can clone from tui-use github. Install from the sibling tui-agent/ directory:

pip install -e tui-agent/

Architecture at a Glance

You (via Bash tool)  -->  tui-agent CLI  -->  Daemon (auto-started)  -->  PTY  -->  TUI Program
                     <--  stdout/JSON    <--  Unix Socket             <--  pyte virtual screen

The daemon starts automatically on first use — no manual setup needed.

Core Workflow

Every TUI interaction follows this pattern:

1. Start a session (launches the TUI program)
2. Wait for the program to be ready
3. Capture the screen to see what's displayed
4. Interact (type text, send keys, click, scroll)
5. Capture again to verify the result
6. Stop the session when done

This capture-act-capture loop is the fundamental rhythm. Always capture the screen before and after important actions so you can verify what happened.

Command Reference

Session Management

Start a TUI program:

tui-agent start --name <session-name> [--cols 120] [--rows 40] -- <command> [args...]

• --name: A unique identifier for this session (you'll use it in all subsequent commands)
• --cols/--rows: Terminal dimensions (default: 80x24). Use larger values for complex UIs
• Everything after -- is the command to run

List active sessions:

tui-agent list

Check session status:

tui-agent status --name <session-name>

Stop a session:

tui-agent stop --name <session-name>
# Or stop everything:
tui-agent stop --all

Seeing the Screen

Capture full screen:

tui-agent capture --name <session-name>

Capture with cursor position:

tui-agent capture --name <session-name> --cursor

This appends cursor coordinates to the output — helpful for knowing where you are in an editor.

Capture a specific region:

tui-agent capture --name <session-name> --top 0 --left 0 --height 5 --width 40

Save a snapshot for later comparison:

tui-agent capture --name <session-name> --save my_snapshot

Compare current screen to a saved snapshot:

tui-agent diff --name <session-name> --snapshot my_snapshot

Read scrollback history (content that scrolled off the top):

tui-agent scrollback --name <session-name> --lines 200

Typing and Keys

Type text:

tui-agent type --name <session-name> "hello world"

Type text and press Enter:

tui-agent type --name <session-name> --enter "ls -la"

Send special keys:

tui-agent key --name <session-name> <key1> [key2] [key3...]

Important: key vs type — The key command only accepts special key names listed below. For regular characters (letters, numbers, symbols) like vim's i, o, dd, :wq, always use type. Use key only for keys that aren't printable characters.

Supported key names (case-insensitive):

• Navigation: up, down, left, right, home, end, pageup, pagedown
• Editing: enter/return, tab, backspace, delete, insert, space, escape/esc
• Function keys: f1 through f12
• Ctrl combos: ctrl-a through ctrl-z

Paste text (uses bracketed paste mode, safe for editors):

tui-agent paste --name <session-name> "multi-line\
text content"

Use paste instead of type when inserting multi-line content into editors — bracketed paste mode prevents the editor from interpreting line breaks as commands.

Mouse Operations

Click at a position (0-indexed row and column):

tui-agent click --name <session-name> --row 5 --col 10

Scroll up/down:

tui-agent scroll-up --name <session-name> --row 12 --col 40 --lines 3
tui-agent scroll-down --name <session-name> --row 12 --col 40 --lines 3

Waiting for Conditions

Waiting is critical for reliable automation. TUI programs take time to render, and acting too fast leads to errors.

Wait for specific text to appear on screen:

tui-agent wait --name <session-name> --text "Ready" --timeout 10

Wait for text to disappear:

tui-agent wait --name <session-name> --text "Loading..." --absent --timeout 15

Wait for screen to stop changing (stable for N seconds):

tui-agent wait --name <session-name> --stable 2.0 --timeout 10

Use --stable when you don't know exactly what text to expect, but need the program to finish rendering.

Resizing the Terminal

tui-agent resize --name <session-name> --cols 160 --rows 50

Best Practices

Always wait before capturing

TUI programs need time to render. After starting a session or sending input, always wait before capturing:

tui-agent start --name ed -- vim file.py
tui-agent wait --name ed --stable 1.0         # Wait for vim to fully load
tui-agent capture --name ed --cursor           # Now safe to read the screen

Use meaningful session names

Name sessions after their purpose, not the program:

• Good: editor, monitor, file-browser
• Bad: s1, test, abc

Capture before and after actions

This is how you verify that your actions had the intended effect:

tui-agent capture --name ed --save before_edit
tui-agent type --name ed "i"                   # Enter insert mode (single char, use type)
tui-agent type --name ed "new line of code"
tui-agent key --name ed escape                 # Escape is a special key, use key
tui-agent capture --name ed                    # Check the result

Choose appropriate terminal size

• For simple programs (bash, basic TUIs): 80x24 (default) is fine
• For editors or complex UIs: use 120x40 or larger
• For programs with wide tables: increase cols to 160+

Clean up sessions when done

Always stop sessions you no longer need:

tui-agent stop --name ed

Or stop all sessions if you're done:

tui-agent stop --all

Handle errors gracefully

If a wait command times out (exit code 1), capture the screen to understand what happened instead of blindly retrying:

tui-agent wait --name ed --text "Success" --timeout 5
# If this fails, capture to see the actual state:
tui-agent capture --name ed

Common Workflows

Editing a File with nano

# Start nano
tui-agent start --name ed --cols 120 --rows 40 -- nano myfile.py
tui-agent wait --name ed --stable 1.0
tui-agent capture --name ed

# Type some code
tui-agent type --name ed "print('hello world')"
tui-agent key --name ed enter
tui-agent type --name ed "print('goodbye')"
tui-agent capture --name ed

# Save (Ctrl-O, then Enter to confirm filename)
tui-agent key --name ed ctrl-o
tui-agent wait --name ed --stable 0.5
tui-agent key --name ed enter
tui-agent wait --name ed --text "Wrote"

# Exit (Ctrl-X)
tui-agent key --name ed ctrl-x
tui-agent stop --name ed

Editing a File with vim

Note: vim may not respond to input on some platforms (see Platform Notes). If vim works in your environment:

# Start vim (use -u NONE to avoid config issues)
tui-agent start --name ed --cols 120 --rows 40 -- vim -u NONE -N myfile.py
tui-agent wait --name ed --stable 1.0
tui-agent capture --name ed

# Navigate to a specific line (e.g., line 10)
tui-agent type --name ed ":10"
tui-agent key --name ed enter
tui-agent capture --name ed

# Enter insert mode and add text
tui-agent type --name ed "o"                    # Open new line below (normal char, use type)
tui-agent type --name ed "    print('hello')"
tui-agent key --name ed escape                  # Back to normal mode

# Save and quit
tui-agent type --name ed ":wq"
tui-agent key --name ed enter
tui-agent wait --name ed --stable 1.0

Monitoring System with htop

# Start htop
tui-agent start --name mon --cols 160 --rows 50 -- htop
tui-agent wait --name mon --stable 2.0
tui-agent capture --name mon

# Search for a process
tui-agent key --name mon f3                     # F3 is a special key, use key
tui-agent type --name mon "python"
tui-agent key --name mon enter
tui-agent capture --name mon

# Scroll through the list
tui-agent scroll-down --name mon --row 25 --col 80 --lines 10
tui-agent capture --name mon

# Quit
tui-agent type --name mon "q"                   # Regular char, use type
tui-agent stop --name mon

Running a Shell Command and Reading Output

# Start a bash session
tui-agent start --name sh -- bash --norc --noprofile
tui-agent wait --name sh --stable 1.0

# Run a command
tui-agent type --name sh --enter "ls -la /tmp"
tui-agent wait --name sh --stable 1.0
tui-agent capture --name sh

# If output is long, check scrollback
tui-agent scrollback --name sh --lines 100

# Exit
tui-agent type --name sh --enter "exit"
tui-agent stop --name sh

Interacting with a ncurses Application

# Start any ncurses app (example: midnight commander)
tui-agent start --name fm --cols 120 --rows 40 -- mc
tui-agent wait --name fm --stable 2.0
tui-agent capture --name fm

# Navigate using arrow keys
tui-agent key --name fm down down down
tui-agent capture --name fm

# Press enter to open a directory
tui-agent key --name fm enter
tui-agent wait --name fm --stable 1.0
tui-agent capture --name fm

# Use function keys
tui-agent key --name fm f5                      # Copy dialog
tui-agent capture --name fm

# Quit
tui-agent key --name fm f10
tui-agent stop --name fm

Platform Notes

vim on macOS: The system vim (/usr/bin/vim) may not respond to type input when launched directly via tui-agent start. This appears to be related to how macOS vim detects terminal capabilities. Workarounds:

• Use nano or pico instead — they work reliably with tui-agent
• Use vim -u NONE -N to disable vimrc and run in nocompatible mode
• Install a different vim build (e.g., via Homebrew: brew install vim)
• For simple file edits, consider using shell commands (sed, awk) instead of a TUI editor

Other programs: Most terminal programs (bash, htop, mc, ncurses apps) work well out of the box with tui-agent. If a program doesn't respond to input, try increasing the wait time or using --stable with a longer duration.

Troubleshooting

"session not found" error: The session may have been stopped or the program exited. Check with tui-agent list.

Screen capture is empty or garbled: The program may not have finished rendering. Add a longer --stable wait or increase --timeout.

Keys don't seem to work: Some programs expect specific key sequences. Capture the screen to verify the program's current state (e.g., is vim in insert mode or normal mode?).

Program exited unexpectedly: Use tui-agent status --name <session> to check the exit code. The program may have crashed or received a signal.

Daemon not running: The daemon starts automatically. If issues persist, check if the socket file exists at /tmp/tui-agent-*.sock. You can manually start the daemon or simply retry the command.