AugmentClaude
TestingDevOps

Codex E2E Test

By liaohch3· liaohch3/claude-tap· 0

Run end-to-end validation tests with screenshot evidence for pull requests.

Installation

  1. Make sure Claude is on your device and in your terminal.

    Skills load from ~/.claude/skills/ when Claude Code starts up — so you need it on your machine first. If you don't have it yet, install it once with the command below, then run claude in any terminal to verify.

    One-time setup
    npm i -g @anthropic-ai/claude-code

    Already have it? Skip ahead.

  2. Paste into Claude Code or into your terminal.
    Install
    Sign up to copy
    Sign up free to reveal

    This copies the whole skill folder into ~/.claude/skills/codex-e2e-test-liaohch3/ — the SKILL.md plus any scripts, reference docs, or templates the skill ships with. Safe default: works for every skill.

    Faster alternative (instruction-only skills)

    Skips the clone and grabs only the SKILL.md file. Don't use this if the skill ships Python scripts, reference markdowns, or asset templates — they won't be downloaded and the skill will fail when it tries to load them.

    Quick install (SKILL.md only)
    Sign up to copy
    Sign up free to reveal
  3. Restart Claude Code.

    Quit and reopen Claude Code (or any other agent that loads from ~/.claude/skills/). New skills are picked up on startup.

  4. Just ask Claude.

    Skills auto-activate when your request matches the skill's description — no slash command needed. Trigger phrases live in the skill's own frontmatter; you can read them in the “What this skill does” section above.

Prefer to read the source first? Open on GitHub.

When Claude uses it

Run PR-grade real Codex E2E validation through claude-tap, including resume turns, multiple tool calls, optional image input, viewer verification, and screenshot evidence.

What this skill does

Codex E2E Test Skill

Run real end-to-end validation that starts claude-tap from local source, connects to the real Codex CLI via OAuth, captures OpenAI Responses API traces, and produces viewer screenshots suitable for PR evidence.

Use this skill for every PR that changes capture, proxying, viewer rendering, session/dashboard behavior, client launch logic, trace ordering, content blocks, tools, token usage, or screenshot/demo assets. If a PR cannot run this flow, state why in the PR and cover the same risk with another real client trace.

Prerequisites

  • codex CLI installed (npm install -g @openai/codex) and authenticated via OAuth
  • Python dev dependencies: uv sync --extra dev
  • Playwright installed: uv run playwright install chromium

Verify OAuth works:

codex exec "say hello" --dangerously-bypass-approvals-and-sandbox

If it fails with token errors, re-authenticate:

codex auth login

Key Difference from Claude E2E

Codex uses the OpenAI Responses API (/v1/responses) instead of Anthropic Messages API. With OAuth authentication, the upstream is https://chatgpt.com/backend-api/codex, not https://api.openai.com.

The proxy must be told the correct target with --tap-target.

Run a Real Codex E2E Trace

Prefer the resume + multimodal flow below for PR evidence. The simple commands are only smoke tests for checking local setup.

Simple (single tool call)

claude-tap --tap-client codex \
  --tap-target https://chatgpt.com/backend-api/codex \
  --tap-output-dir /tmp/codex-e2e \
  --tap-no-open \
  -- exec "say hello" \
  --dangerously-bypass-approvals-and-sandbox

Multi-call (triggers multiple API requests)

Use a task that requires shell tool use — this forces the agent to make multiple Responses API calls (models lookup + actual responses):

claude-tap --tap-client codex \
  --tap-target https://chatgpt.com/backend-api/codex \
  --tap-output-dir /tmp/codex-e2e \
  --tap-no-open \
  -- exec "Read pyproject.toml and tell me the project name and version" \
  --dangerously-bypass-approvals-and-sandbox

Expected: 4+ API calls (2x GET /v1/models + 2x POST /v1/responses).

Resume + Multimodal Content-Block Trace

Use this flow for viewer changes that affect message rendering, content block boundaries, tool call ordering, images, or copy/select behavior. It creates a real Codex session, resumes it at least once, forces multiple shell tool calls per user turn, and attaches an actual image so the trace includes multimodal content. This is the default PR evidence flow.

1. Prepare an isolated workspace

mkdir -p /tmp/claude-tap-real-codex-workspace
printf 'project = "claude-tap-real-codex-e2e"\n' \
  > /tmp/claude-tap-real-codex-workspace/project.toml

Create or copy a small valid PNG into the workspace. If you need a deterministic local image, generate it with Python's standard library:

python3 - <<'PY'
from pathlib import Path
import struct
import zlib

def chunk(kind: bytes, data: bytes) -> bytes:
    return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)

width = height = 8
rows = b"".join(b"\x00" + (b"\x2f\x80\xed\xff" * width) for _ in range(height))
png = (
    b"\x89PNG\r\n\x1a\n"
    + chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 6, 0, 0, 0))
    + chunk(b"IDAT", zlib.compress(rows))
    + chunk(b"IEND", b"")
)
Path("/tmp/claude-tap-real-codex-workspace/input.png").write_bytes(png)
PY

2. Start a real Codex session through claude-tap

Run from the repository, but point Codex at the isolated workspace with -C. The prompt should explicitly require several tool calls so the viewer has enough messages, tool calls, and response blocks to inspect.

uv run claude-tap --tap-client codex \
  --tap-target https://chatgpt.com/backend-api/codex \
  --tap-output-dir /tmp/claude-tap-real-codex-traces \
  --tap-no-open \
  -- exec -C /tmp/claude-tap-real-codex-workspace \
  --image /tmp/claude-tap-real-codex-workspace/input.png \
  --dangerously-bypass-approvals-and-sandbox \
  "Inspect this workspace and the attached image. Use shell tools to run pwd, list files, inspect project.toml, inspect input.png, then write codex_e2e_report.txt with your findings. Keep all writes inside this workspace."

3. Resume the same Codex session with another real turn

Use the session id printed by the first Codex run when possible. Avoid relying on --last on busy maintainer machines because it can resume an unrelated recent Codex session.

uv run claude-tap --tap-client codex \
  --tap-target https://chatgpt.com/backend-api/codex \
  --tap-output-dir /tmp/claude-tap-real-codex-traces \
  --tap-no-open \
  -- exec resume <SESSION_ID_FROM_FIRST_RUN> \
  --image /tmp/claude-tap-real-codex-workspace/input.png \
  --dangerously-bypass-approvals-and-sandbox \
  "Continue the same investigation in /tmp/claude-tap-real-codex-workspace. Use shell tools to read /tmp/claude-tap-real-codex-workspace/codex_e2e_report.txt, compute the byte size of /tmp/claude-tap-real-codex-workspace/input.png, and write /tmp/claude-tap-real-codex-workspace/codex_e2e_followup.txt. Then summarize what changed since the previous turn."

4. Capture multi-position viewer screenshots

Take screenshots at multiple scroll positions, including a deeper position in the same detail pane. Store them under .agents/evidence/pr/<topic>/ and use raw.githubusercontent.com links in the PR body.

mkdir -p .agents/evidence/pr/codex-real-e2e

uv run python - <<'PY'
from pathlib import Path
from playwright.sync_api import sync_playwright

html_files = sorted(Path("/tmp/claude-tap-real-codex-traces").rglob("trace_*.html"))
if not html_files:
    raise SystemExit("No viewer HTML found in /tmp/claude-tap-real-codex-traces")
html = html_files[-1]
out_dir = Path(".agents/evidence/pr/codex-real-e2e")

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1760, "height": 1100}, device_scale_factor=1)
    page.goto(f"file://{html}", wait_until="domcontentloaded", timeout=10000)
    page.wait_for_selector(".sidebar-item", timeout=5000)
    page.evaluate("document.documentElement.setAttribute('data-theme', 'light')")

    # Select the last Responses call so resume context is visible.
    page.evaluate(
        """() => {
          const items = Array.from(document.querySelectorAll('.sidebar-item'));
          const responses = items.filter(item => item.textContent.includes('/v1/responses'));
          (responses.at(-1) || items.at(-1))?.click();
        }"""
    )
    page.wait_for_timeout(300)

    page.evaluate(
        """() => {
          for (const section of document.querySelectorAll('#detail .section')) {
            const title = section.querySelector('.title')?.textContent || '';
            const body = section.querySelector('.section-body');
            const header = section.querySelector('.section-header');
            if (!body || !header) continue;
            const shouldOpen = ['System Prompt', 'Messages', 'Response'].includes(title);
            const isOpen = body.classList.contains('open');
            if (shouldOpen !== isOpen) header.click();
          }
        }"""
    )
    page.wait_for_timeout(200)

    def shot(name: str, scroll_top: int) -> None:
        page.evaluate("y => { const d = document.querySelector('#detail'); if (d) d.scrollTop = y; }", scroll_top)
        page.wait_for_timeout(200)
        page.screenshot(path=str(out_dir / name), full_page=False)

    shot("codex-real-top.png", 0)
    shot("codex-real-mid.png", 700)
    shot("codex-real-deep.png", 1400)

    image_count = page.evaluate(
        """() => {
          const img = document.querySelector('#detail img');
          if (!img) return 0;
          img.scrollIntoView({ block: 'center', inline: 'nearest' });
          return document.querySelectorAll('#detail img').length;
        }"""
    )
    if image_count:
        page.wait_for_timeout(200)
        page.screenshot(path=str(out_dir / "codex-real-image.png"), full_page=False)

    browser.close()
PY

Validate screenshots:

uv run python scripts/check_screenshots.py .agents/evidence/pr/codex-real-e2e

5. Verify the trace

  • The output directory contains at least two real .jsonl traces and generated .html viewers from claude-tap.
  • The trace includes multiple POST /v1/responses entries with status 200 and non-zero token usage.
  • At least one request contains image content from the --image attachment.
  • The viewer shows Messages and Response sections with multiple content blocks separated cleanly.
  • Tool calls and tool results stay interleaved in the order they happened.
  • Copy buttons still copy the complete logical message/section text, not only one visual content block.
  • Text selection across adjacent content blocks remains possible in the browser.
  • Screenshot evidence includes at least two scroll depths for the same resumed trace, not only the top of the page.
  • If the trace contains image input, screenshot evidence includes the rendered image block or records why the client did not send image content into the API request body.

Taking Viewer Screenshots with Playwright

from playwright.sync_api import sync_playwright
import time, glob

html = glob.glob("/tmp/codex-e2e/trace_*.html")[-1]

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page(viewport={"width": 1440, "height": 1000})
    page.goto(f"file://{html}")
    page.wait_for_load_state("networkidle")
    time.sleep(1)

    # Select a Responses call (data-idx matches trace line index)
    page.click('.sidebar-item[data-idx="1"]')
    time.sleep(0.5)

    # Collapse System Prompt, keep Messages open
    page.evaluate("""() => {
        const h = document.querySelectorAll('.section-header')[1];
        const next = h.nextElementSibling;
        if (next && getComputedStyle(next).display !== 'none') h.click();
    }""")

    # Scroll to Messages section
    page.evaluate("""() => {
        document.querySelectorAll('.section-header')[2]
          .scrollIntoView({behavior: 'instant', block: 'start'});
    }""")
    time.sleep(0.3)
    page.screenshot(path="/tmp/codex-e2e/messages.png")

    browser.close()

Verification Checklist

  • Trace .jsonl has ≥2 POST /v1/responses entries
  • Response status is 200 (not 401/502)
  • Token counts are non-zero in Responses calls
  • HTML viewer is generated (trace_*.html)
  • Sidebar shows multiple calls with model name and token counts
  • Messages section shows user message text (verifies #41 fix)
  • Response section shows assistant reply (verifies #40 fix)

Troubleshooting

SymptomCauseFix
WebSocket 502 then HTTP 401Default target api.openai.com rejects ChatGPT OAuth tokensUse --tap-target https://chatgpt.com/backend-api/codex
Missing scopes: api.responses.writeAPI key lacks Responses API accessUse OAuth (codex auth login) instead of OPENAI_API_KEY
Only 1 API callSimple prompt completed in one roundUse a task requiring tool use (file reads, shell commands)
OPENAI_BASE_URL is deprecated warningCodex v0.115+ prefers config.tomlHarmless — proxy still works via env var

Notes

  • Codex with OAuth uses WebSocket first, then falls back to HTTP/SSE when proxied. The fallback is transparent — traces capture the HTTP/SSE path correctly.
  • Each codex exec session also calls GET /v1/models for model discovery.
  • The --dangerously-bypass-approvals-and-sandbox flag is required for non-interactive exec.

Related skills

D

Documentation Co-Authoring

anthropics

Guide structured workflows for writing docs, proposals, and technical specs collaboratively.

Official
M

MCP Server Builder

anthropics

Build protocol servers that connect language models to external APIs and services.

OfficialComplete terms in LICENSE.txt
S

Skill Builder & Optimizer

anthropics

Create, edit, and optimize Claude skills with performance testing and benchmarking.

Official
M

Multi-Component Web Artifacts

anthropics

Build complex React artifacts with Tailwind CSS and shadcn/ui components.

OfficialComplete terms in LICENSE.txt